2024-03-17T23:30:48.008 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-17T23:30:48.008 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/pool-perm} --verbose --owner scheduled_pdonnell@teuthology --name pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi --archive /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608488 -- /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608488/orig.config.yaml 2024-03-17T23:30:48.052 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-17T23:30:48.145 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608488 branch: wip-batrick-testing-20240314.183243 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/pool-perm} email: pdonnell@redhat.com first_in_suite: false job_id: '7608488' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240314.183243 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - mons down, quorum - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 subvolume_version: 1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240314.183243 sha1: c382027d5c66ef4e0aa1131ea3e8080c658d3184 owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 4770 sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af sleep_before_teardown: 0 subset: 498/512 suite: fs suite_branch: wip-batrick-testing-20240314.183243 suite_path: /home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: c382027d5c66ef4e0aa1131ea3e8080c658d3184 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi150.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEuznq3ZAT3syTm6Yet4MN3i/Y3sAFNkEz27pRQMqOK0NnuEKnOkPmWZl97ElRgzeZNsnze7EUL85fmO4o5EZ6M= smithi157.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFQmXCS1lYraNILT38OgyDIOYfOlxAZXfrbtQYaTnpIiZzvkKmkBYkPjsJ1gnMMUKS7RHaMUrZ6fmo+5HkRvS2I= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_pool_perm teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-17_23:17:27 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-17T23:30:48.145 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa; will attempt to use it 2024-03-17T23:30:48.146 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks 2024-03-17T23:30:48.146 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-17T23:30:48.148 INFO:teuthology.task.internal:Checking packages... 2024-03-17T23:30:48.171 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af' 2024-03-17T23:30:48.171 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-17T23:30:48.171 INFO:teuthology.packaging:ref: None 2024-03-17T23:30:48.171 INFO:teuthology.packaging:tag: None 2024-03-17T23:30:48.171 INFO:teuthology.packaging:branch: wip-batrick-testing-20240314.183243 2024-03-17T23:30:48.171 INFO:teuthology.packaging:sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-17T23:30:48.172 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240314.183243 2024-03-17T23:30:48.495 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2162.g3d78f5ba 2024-03-17T23:30:48.496 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-17T23:30:48.503 INFO:teuthology.task.internal:no buildpackages task found 2024-03-17T23:30:48.503 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-17T23:30:48.508 INFO:teuthology.task.internal:Saving configuration 2024-03-17T23:30:48.524 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-17T23:30:48.531 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-17T23:30:48.552 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608488', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-17 23:24:02.780949', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEuznq3ZAT3syTm6Yet4MN3i/Y3sAFNkEz27pRQMqOK0NnuEKnOkPmWZl97ElRgzeZNsnze7EUL85fmO4o5EZ6M='} 2024-03-17T23:30:48.575 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608488', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-17 23:24:02.782200', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFQmXCS1lYraNILT38OgyDIOYfOlxAZXfrbtQYaTnpIiZzvkKmkBYkPjsJ1gnMMUKS7RHaMUrZ6fmo+5HkRvS2I='} 2024-03-17T23:30:48.575 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-17T23:30:48.599 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-17T23:30:48.599 INFO:teuthology.task.internal:roles: ubuntu@smithi157.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-17T23:30:48.599 INFO:teuthology.run_tasks:Running task console_log... 2024-03-17T23:30:48.713 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3c8c0d2160>, signals=[15]) 2024-03-17T23:30:48.713 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-17T23:30:48.720 INFO:teuthology.task.internal:Opening connections... 2024-03-17T23:30:48.720 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi150.front.sepia.ceph.com 2024-03-17T23:30:48.722 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-17T23:30:48.810 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi157.front.sepia.ceph.com 2024-03-17T23:30:48.811 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-17T23:30:48.900 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-17T23:30:48.908 DEBUG:teuthology.orchestra.run.smithi150:> uname -m 2024-03-17T23:30:48.925 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2024-03-17T23:30:48.926 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release 2024-03-17T23:30:48.985 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="CentOS Stream" 2024-03-17T23:30:48.985 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="9" 2024-03-17T23:30:48.985 INFO:teuthology.orchestra.run.smithi150.stdout:ID="centos" 2024-03-17T23:30:48.985 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="rhel fedora" 2024-03-17T23:30:48.985 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="9" 2024-03-17T23:30:48.985 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el9" 2024-03-17T23:30:48.986 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-17T23:30:48.986 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31" 2024-03-17T23:30:48.986 INFO:teuthology.orchestra.run.smithi150.stdout:LOGO="fedora-logo-icon" 2024-03-17T23:30:48.986 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-17T23:30:48.986 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://centos.org/" 2024-03-17T23:30:48.986 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-17T23:30:48.986 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-17T23:30:48.986 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-17T23:30:48.987 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2024-03-17T23:30:49.011 DEBUG:teuthology.orchestra.run.smithi157:> uname -m 2024-03-17T23:30:49.029 INFO:teuthology.orchestra.run.smithi157.stdout:x86_64 2024-03-17T23:30:49.030 DEBUG:teuthology.orchestra.run.smithi157:> cat /etc/os-release 2024-03-17T23:30:49.088 INFO:teuthology.orchestra.run.smithi157.stdout:NAME="CentOS Stream" 2024-03-17T23:30:49.088 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION="9" 2024-03-17T23:30:49.088 INFO:teuthology.orchestra.run.smithi157.stdout:ID="centos" 2024-03-17T23:30:49.088 INFO:teuthology.orchestra.run.smithi157.stdout:ID_LIKE="rhel fedora" 2024-03-17T23:30:49.088 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_ID="9" 2024-03-17T23:30:49.088 INFO:teuthology.orchestra.run.smithi157.stdout:PLATFORM_ID="platform:el9" 2024-03-17T23:30:49.088 INFO:teuthology.orchestra.run.smithi157.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-17T23:30:49.088 INFO:teuthology.orchestra.run.smithi157.stdout:ANSI_COLOR="0;31" 2024-03-17T23:30:49.088 INFO:teuthology.orchestra.run.smithi157.stdout:LOGO="fedora-logo-icon" 2024-03-17T23:30:49.088 INFO:teuthology.orchestra.run.smithi157.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-17T23:30:49.088 INFO:teuthology.orchestra.run.smithi157.stdout:HOME_URL="https://centos.org/" 2024-03-17T23:30:49.088 INFO:teuthology.orchestra.run.smithi157.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-17T23:30:49.089 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-17T23:30:49.089 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-17T23:30:49.089 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server 2024-03-17T23:30:49.110 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-17T23:30:49.120 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-17T23:30:49.126 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-17T23:30:49.126 DEBUG:teuthology.orchestra.run.smithi150:> test '!' -e /home/ubuntu/cephtest 2024-03-17T23:30:49.129 DEBUG:teuthology.orchestra.run.smithi157:> test '!' -e /home/ubuntu/cephtest 2024-03-17T23:30:49.145 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-17T23:30:49.151 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-17T23:30:49.151 DEBUG:teuthology.orchestra.run.smithi150:> test -z $(ls -A /var/lib/ceph) 2024-03-17T23:30:49.188 DEBUG:teuthology.orchestra.run.smithi157:> test -z $(ls -A /var/lib/ceph) 2024-03-17T23:30:49.215 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-17T23:30:49.270 INFO:teuthology.run_tasks:Running task kernel... 2024-03-17T23:30:49.288 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-17T23:30:49.288 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-17T23:30:49.289 DEBUG:teuthology.orchestra.run.smithi150:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-17T23:30:49.289 DEBUG:teuthology.orchestra.run.smithi157:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-17T23:30:49.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-17T23:30:49.305 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2024-03-17T23:30:49.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-17T23:30:49.306 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2024-03-17T23:30:49.364 INFO:teuthology.orchestra.run.smithi157.stdout:5.14.0-361.el9.x86_64 2024-03-17T23:30:49.364 INFO:teuthology.task.kernel:Running kernel on smithi157: 5.14.0-361.el9.x86_64 2024-03-17T23:30:49.364 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum install -y kernel 2024-03-17T23:30:49.365 INFO:teuthology.orchestra.run.smithi150.stdout:5.14.0-361.el9.x86_64 2024-03-17T23:30:49.366 INFO:teuthology.task.kernel:Running kernel on smithi150: 5.14.0-361.el9.x86_64 2024-03-17T23:30:49.366 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum install -y kernel 2024-03-17T23:30:51.439 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-17T23:30:51.531 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-17T23:30:52.848 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-17T23:30:53.165 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-17T23:30:54.039 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-17T23:30:56.365 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-03-17T23:31:00.282 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-17T23:31:01.842 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-03-17T23:31:08.145 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - BaseOS 556 kB/s | 8.0 MB 00:14 2024-03-17T23:31:10.896 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-03-17T23:31:15.031 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - CRB 8.9 MB/s | 6.1 MB 00:00 2024-03-17T23:31:16.574 INFO:teuthology.orchestra.run.smithi157.stdout:Extra Packages for Enterprise Linux 1.4 MB/s | 21 MB 00:14 2024-03-17T23:31:16.723 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 16 kB 00:00 2024-03-17T23:31:18.085 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-17T23:31:21.524 INFO:teuthology.orchestra.run.smithi157.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-03-17T23:31:23.174 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-17T23:31:23.655 INFO:teuthology.orchestra.run.smithi157.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-17T23:31:23.655 INFO:teuthology.orchestra.run.smithi157.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-17T23:31:23.765 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-17T23:31:23.771 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-17T23:31:23.771 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2024-03-17T23:31:23.771 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-17T23:31:23.771 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-17T23:31:23.771 INFO:teuthology.orchestra.run.smithi157.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-17T23:31:23.771 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-17T23:31:23.771 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-17T23:31:23.772 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-17T23:31:23.772 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-17T23:31:23.772 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:31:23.772 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-17T23:31:23.772 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2024-03-17T23:31:23.772 INFO:teuthology.orchestra.run.smithi157.stdout:Install 4 Packages 2024-03-17T23:31:23.772 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:31:23.772 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 100 M 2024-03-17T23:31:23.773 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 123 M 2024-03-17T23:31:23.773 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-17T23:31:24.216 INFO:teuthology.orchestra.run.smithi157.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 18 MB/s | 6.0 MB 00:00 2024-03-17T23:31:24.708 INFO:teuthology.orchestra.run.smithi157.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 25 MB/s | 21 MB 00:00 2024-03-17T23:31:25.342 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-17T23:31:25.342 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-17T23:31:25.377 INFO:teuthology.orchestra.run.smithi157.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 26 MB/s | 39 MB 00:01 2024-03-17T23:31:25.428 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-03-17T23:31:25.436 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-03-17T23:31:25.436 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2024-03-17T23:31:25.436 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-03-17T23:31:25.436 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-03-17T23:31:25.436 INFO:teuthology.orchestra.run.smithi150.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-17T23:31:25.436 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-03-17T23:31:25.436 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-17T23:31:25.437 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-17T23:31:25.437 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-17T23:31:25.437 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:31:25.437 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-03-17T23:31:25.437 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-03-17T23:31:25.437 INFO:teuthology.orchestra.run.smithi150.stdout:Install 4 Packages 2024-03-17T23:31:25.437 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:31:25.437 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 100 M 2024-03-17T23:31:25.437 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 123 M 2024-03-17T23:31:25.437 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-03-17T23:31:25.993 INFO:teuthology.orchestra.run.smithi157.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 19 MB/s | 33 MB 00:01 2024-03-17T23:31:25.993 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-17T23:31:25.993 INFO:teuthology.orchestra.run.smithi157.stdout:Total 45 MB/s | 100 MB 00:02 2024-03-17T23:31:26.351 INFO:teuthology.orchestra.run.smithi150.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 7.0 MB/s | 6.0 MB 00:00 2024-03-17T23:31:26.880 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-17T23:31:27.280 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-17T23:31:27.280 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-17T23:31:27.367 INFO:teuthology.orchestra.run.smithi150.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 11 MB/s | 21 MB 00:01 2024-03-17T23:31:27.835 INFO:teuthology.orchestra.run.smithi150.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 17 MB/s | 39 MB 00:02 2024-03-17T23:31:28.885 INFO:teuthology.orchestra.run.smithi150.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 13 MB/s | 33 MB 00:02 2024-03-17T23:31:28.885 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-03-17T23:31:28.885 INFO:teuthology.orchestra.run.smithi150.stdout:Total 29 MB/s | 100 MB 00:03 2024-03-17T23:31:28.951 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-17T23:31:28.952 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-17T23:31:29.765 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-03-17T23:31:30.154 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-03-17T23:31:30.154 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-03-17T23:31:31.486 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-03-17T23:31:31.487 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-03-17T23:31:31.942 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-17T23:31:32.634 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-17T23:31:32.768 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-17T23:31:34.216 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-03-17T23:31:34.893 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-17T23:31:35.043 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-17T23:31:35.636 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-17T23:31:35.822 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-17T23:31:37.859 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-17T23:31:38.046 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-17T23:31:40.587 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-17T23:31:40.782 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-17T23:31:42.791 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-17T23:31:42.922 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-17T23:31:45.600 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-17T23:31:47.672 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-17T23:32:14.850 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-17T23:32:14.880 INFO:teuthology.orchestra.run.smithi157.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-17T23:32:14.881 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:32:14.934 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-17T23:32:15.360 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-17T23:32:15.360 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-17T23:32:15.360 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-17T23:32:15.360 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-17T23:32:15.763 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-17T23:32:15.763 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:32:15.763 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-17T23:32:15.763 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-17T23:32:15.763 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-17T23:32:15.763 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-17T23:32:15.763 INFO:teuthology.orchestra.run.smithi157.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-17T23:32:15.763 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:32:15.763 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-17T23:32:16.281 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q kernel | sort -rV | head -n 1 2024-03-17T23:32:16.424 INFO:teuthology.orchestra.run.smithi157.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-17T23:32:16.424 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-17T23:32:16.424 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-17T23:32:16.424 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-17T23:32:16.424 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-17T23:32:16.425 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-17T23:32:16.425 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi157.front.sepia.ceph.com, path=None, version=distro) 2024-03-17T23:32:16.425 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q kernel | sort -rV | head -n 1 2024-03-17T23:32:16.548 INFO:teuthology.orchestra.run.smithi157.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-17T23:32:16.548 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-17T23:32:16.548 DEBUG:teuthology.orchestra.run.smithi157:> sudo rpm -qi grub2-tools 2024-03-17T23:32:16.590 INFO:teuthology.orchestra.run.smithi157.stdout:Name : grub2-tools 2024-03-17T23:32:16.590 INFO:teuthology.orchestra.run.smithi157.stdout:Epoch : 1 2024-03-17T23:32:16.590 INFO:teuthology.orchestra.run.smithi157.stdout:Version : 2.06 2024-03-17T23:32:16.590 INFO:teuthology.orchestra.run.smithi157.stdout:Release : 68.el9 2024-03-17T23:32:16.590 INFO:teuthology.orchestra.run.smithi157.stdout:Architecture: x86_64 2024-03-17T23:32:16.590 INFO:teuthology.orchestra.run.smithi157.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-17T23:32:16.590 INFO:teuthology.orchestra.run.smithi157.stdout:Group : Unspecified 2024-03-17T23:32:16.590 INFO:teuthology.orchestra.run.smithi157.stdout:Size : 8264254 2024-03-17T23:32:16.591 INFO:teuthology.orchestra.run.smithi157.stdout:License : GPLv3+ 2024-03-17T23:32:16.591 INFO:teuthology.orchestra.run.smithi157.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-17T23:32:16.591 INFO:teuthology.orchestra.run.smithi157.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-17T23:32:16.591 INFO:teuthology.orchestra.run.smithi157.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-17T23:32:16.591 INFO:teuthology.orchestra.run.smithi157.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-17T23:32:16.591 INFO:teuthology.orchestra.run.smithi157.stdout:Packager : builder@centos.org 2024-03-17T23:32:16.591 INFO:teuthology.orchestra.run.smithi157.stdout:Vendor : CentOS 2024-03-17T23:32:16.591 INFO:teuthology.orchestra.run.smithi157.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-17T23:32:16.591 INFO:teuthology.orchestra.run.smithi157.stdout:Summary : Support tools for GRUB. 2024-03-17T23:32:16.591 INFO:teuthology.orchestra.run.smithi157.stdout:Description : 2024-03-17T23:32:16.591 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:32:16.591 INFO:teuthology.orchestra.run.smithi157.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-17T23:32:16.591 INFO:teuthology.orchestra.run.smithi157.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-17T23:32:16.591 INFO:teuthology.orchestra.run.smithi157.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-17T23:32:16.591 INFO:teuthology.orchestra.run.smithi157.stdout:hardware devices. 2024-03-17T23:32:16.592 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:32:16.592 INFO:teuthology.orchestra.run.smithi157.stdout:This subpackage provides tools for support of all platforms. 2024-03-17T23:32:16.592 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-17T23:32:16.593 INFO:teuthology.task.kernel:Updating grub on smithi157 to boot 5.14.0-427.el9.x86_64 2024-03-17T23:32:16.593 DEBUG:teuthology.orchestra.run.smithi157:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-17T23:32:17.184 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-17T23:32:17.184 INFO:teuthology.orchestra.run.smithi150.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-17T23:32:17.184 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:32:17.217 INFO:teuthology.orchestra.run.smithi157.stderr:Generating grub configuration file ... 2024-03-17T23:32:17.269 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-17T23:32:17.685 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-17T23:32:17.685 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-17T23:32:17.685 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-17T23:32:17.685 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-17T23:32:18.123 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-17T23:32:18.123 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:32:18.123 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-03-17T23:32:18.123 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-17T23:32:18.123 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-17T23:32:18.123 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-17T23:32:18.123 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-17T23:32:18.123 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:32:18.123 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-03-17T23:32:18.524 INFO:teuthology.orchestra.run.smithi157.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-17T23:32:18.565 INFO:teuthology.orchestra.run.smithi157.stderr:done 2024-03-17T23:32:18.567 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2024-03-17T23:32:18.583 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.sGiwZ3UEMD 2024-03-17T23:32:18.583 DEBUG:teuthology.orchestra.run.smithi157:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.sGiwZ3UEMD 2024-03-17T23:32:18.638 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel | sort -rV | head -n 1 2024-03-17T23:32:18.653 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.sGiwZ3UEMD 2024-03-17T23:32:18.782 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-17T23:32:18.783 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-17T23:32:18.783 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-17T23:32:18.783 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-17T23:32:18.784 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-17T23:32:18.784 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-17T23:32:18.784 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi150.front.sepia.ceph.com, path=None, version=distro) 2024-03-17T23:32:18.784 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel | sort -rV | head -n 1 2024-03-17T23:32:18.802 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.sGiwZ3UEMD is 6KB 2024-03-17T23:32:18.850 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.sGiwZ3UEMD 2024-03-17T23:32:18.865 DEBUG:teuthology.orchestra.run.smithi157:> sudo /bin/ls /boot/loader/entries || true 2024-03-17T23:32:18.904 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-17T23:32:18.904 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-17T23:32:18.904 DEBUG:teuthology.orchestra.run.smithi150:> sudo rpm -qi grub2-tools 2024-03-17T23:32:18.929 INFO:teuthology.orchestra.run.smithi157.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-17T23:32:18.930 INFO:teuthology.orchestra.run.smithi157.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-17T23:32:18.930 INFO:teuthology.orchestra.run.smithi157.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-17T23:32:18.930 INFO:teuthology.orchestra.run.smithi157.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-17T23:32:18.931 DEBUG:teuthology.orchestra.run.smithi157:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-17T23:32:18.947 INFO:teuthology.orchestra.run.smithi150.stdout:Name : grub2-tools 2024-03-17T23:32:18.948 INFO:teuthology.orchestra.run.smithi150.stdout:Epoch : 1 2024-03-17T23:32:18.948 INFO:teuthology.orchestra.run.smithi150.stdout:Version : 2.06 2024-03-17T23:32:18.948 INFO:teuthology.orchestra.run.smithi150.stdout:Release : 68.el9 2024-03-17T23:32:18.948 INFO:teuthology.orchestra.run.smithi150.stdout:Architecture: x86_64 2024-03-17T23:32:18.948 INFO:teuthology.orchestra.run.smithi150.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-17T23:32:18.948 INFO:teuthology.orchestra.run.smithi150.stdout:Group : Unspecified 2024-03-17T23:32:18.948 INFO:teuthology.orchestra.run.smithi150.stdout:Size : 8264254 2024-03-17T23:32:18.948 INFO:teuthology.orchestra.run.smithi150.stdout:License : GPLv3+ 2024-03-17T23:32:18.948 INFO:teuthology.orchestra.run.smithi150.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-17T23:32:18.948 INFO:teuthology.orchestra.run.smithi150.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-17T23:32:18.948 INFO:teuthology.orchestra.run.smithi150.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-17T23:32:18.948 INFO:teuthology.orchestra.run.smithi150.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-17T23:32:18.948 INFO:teuthology.orchestra.run.smithi150.stdout:Packager : builder@centos.org 2024-03-17T23:32:18.949 INFO:teuthology.orchestra.run.smithi150.stdout:Vendor : CentOS 2024-03-17T23:32:18.949 INFO:teuthology.orchestra.run.smithi150.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-17T23:32:18.949 INFO:teuthology.orchestra.run.smithi150.stdout:Summary : Support tools for GRUB. 2024-03-17T23:32:18.949 INFO:teuthology.orchestra.run.smithi150.stdout:Description : 2024-03-17T23:32:18.949 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:32:18.949 INFO:teuthology.orchestra.run.smithi150.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-17T23:32:18.949 INFO:teuthology.orchestra.run.smithi150.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-17T23:32:18.949 INFO:teuthology.orchestra.run.smithi150.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-17T23:32:18.949 INFO:teuthology.orchestra.run.smithi150.stdout:hardware devices. 2024-03-17T23:32:18.949 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:32:18.949 INFO:teuthology.orchestra.run.smithi150.stdout:This subpackage provides tools for support of all platforms. 2024-03-17T23:32:18.950 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-17T23:32:18.951 INFO:teuthology.task.kernel:Updating grub on smithi150 to boot 5.14.0-427.el9.x86_64 2024-03-17T23:32:18.951 DEBUG:teuthology.orchestra.run.smithi150:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-17T23:32:19.094 DEBUG:teuthology.orchestra.run.smithi157:> sudo shutdown -r now 2024-03-17T23:32:19.574 INFO:teuthology.orchestra.run.smithi150.stderr:Generating grub configuration file ... 2024-03-17T23:32:20.961 INFO:teuthology.orchestra.run.smithi150.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-17T23:32:20.994 INFO:teuthology.orchestra.run.smithi150.stderr:done 2024-03-17T23:32:20.997 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2024-03-17T23:32:21.011 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.WtlGKVcc2f 2024-03-17T23:32:21.011 DEBUG:teuthology.orchestra.run.smithi150:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.WtlGKVcc2f 2024-03-17T23:32:21.082 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.WtlGKVcc2f 2024-03-17T23:32:21.221 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.WtlGKVcc2f is 6KB 2024-03-17T23:32:21.240 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.WtlGKVcc2f 2024-03-17T23:32:21.254 DEBUG:teuthology.orchestra.run.smithi150:> sudo /bin/ls /boot/loader/entries || true 2024-03-17T23:32:21.321 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-17T23:32:21.322 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-17T23:32:21.322 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-17T23:32:21.322 INFO:teuthology.orchestra.run.smithi150.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-17T23:32:21.324 DEBUG:teuthology.orchestra.run.smithi150:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-17T23:32:21.478 DEBUG:teuthology.orchestra.run.smithi150:> sudo shutdown -r now 2024-03-17T23:32:49.122 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-17T23:32:49.123 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-17T23:32:49.124 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-17T23:32:51.483 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-17T23:32:51.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-03-17T23:32:51.483 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-17T23:33:07.487 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-03-17T23:33:10.050 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-03-17T23:33:16.494 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-17T23:33:16.495 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-17T23:33:19.053 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-03-17T23:33:19.053 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-17T23:33:19.554 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-03-17T23:33:22.110 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.150 2024-03-17T23:33:31.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-17T23:33:31.565 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-17T23:33:34.113 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-03-17T23:33:34.114 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-17T23:33:34.622 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-03-17T23:33:49.638 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-17T23:33:49.639 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-17T23:34:08.163 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-03-17T23:34:26.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-17T23:34:26.180 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-17T23:34:26.584 DEBUG:teuthology.orchestra.run.smithi157:> true 2024-03-17T23:34:27.010 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-17T23:34:27.010 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-17T23:34:27.013 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2024-03-17T23:34:27.027 INFO:teuthology.orchestra.run.smithi157.stdout:5.14.0-427.el9.x86_64 2024-03-17T23:34:27.028 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-17T23:34:27.028 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-17T23:34:27.028 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-17T23:34:28.029 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-17T23:34:28.029 DEBUG:teuthology.orchestra.run.smithi157:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-17T23:34:28.092 INFO:teuthology.orchestra.run.smithi157.stdout:ttyS1 2024-03-17T23:34:28.139 DEBUG:teuthology.parallel:result is None 2024-03-17T23:34:34.118 DEBUG:teuthology.orchestra.remote:timed out 2024-03-17T23:34:49.123 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-03-17T23:34:49.124 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-17T23:34:49.662 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-03-17T23:34:50.044 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-03-17T23:34:50.044 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-17T23:34:50.045 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2024-03-17T23:34:50.059 INFO:teuthology.orchestra.run.smithi150.stdout:5.14.0-427.el9.x86_64 2024-03-17T23:34:50.060 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-17T23:34:50.060 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-17T23:34:50.060 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-17T23:34:51.061 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-17T23:34:51.061 DEBUG:teuthology.orchestra.run.smithi150:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-17T23:34:51.152 INFO:teuthology.orchestra.run.smithi150.stdout:ttyS1 2024-03-17T23:34:51.192 DEBUG:teuthology.parallel:result is None 2024-03-17T23:34:51.192 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-17T23:34:51.201 INFO:teuthology.task.internal:Creating test directory... 2024-03-17T23:34:51.203 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-17T23:34:51.205 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-17T23:34:51.225 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-17T23:34:51.233 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-17T23:34:51.295 INFO:teuthology.task.internal:Creating archive directory... 2024-03-17T23:34:51.295 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-17T23:34:51.299 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-17T23:34:51.331 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-17T23:34:51.339 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-17T23:34:51.339 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-17T23:34:51.360 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-17T23:34:51.398 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-17T23:34:51.399 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-17T23:34:51.412 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-17T23:34:51.414 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-17T23:34:51.415 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-17T23:34:51.422 INFO:teuthology.task.internal:Configuring sudo... 2024-03-17T23:34:51.423 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-17T23:34:51.457 DEBUG:teuthology.orchestra.run.smithi157:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-17T23:34:51.486 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-17T23:34:51.495 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-17T23:34:51.496 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-17T23:34:51.526 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-17T23:34:51.542 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-17T23:34:51.636 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-17T23:34:51.675 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-17T23:34:51.756 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-17T23:34:51.800 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:34:51.800 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-17T23:34:51.867 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-17T23:34:51.912 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-17T23:34:51.968 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-17T23:34:52.052 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-17T23:34:52.095 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:34:52.095 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-17T23:34:52.163 DEBUG:teuthology.orchestra.run.smithi150:> sudo service rsyslog restart 2024-03-17T23:34:52.165 DEBUG:teuthology.orchestra.run.smithi157:> sudo service rsyslog restart 2024-03-17T23:34:52.206 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-17T23:34:52.271 INFO:teuthology.orchestra.run.smithi157.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-17T23:34:52.659 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-17T23:34:52.667 INFO:teuthology.task.internal:Starting timer... 2024-03-17T23:34:52.667 INFO:teuthology.run_tasks:Running task pcp... 2024-03-17T23:34:52.678 INFO:teuthology.run_tasks:Running task selinux... 2024-03-17T23:34:52.687 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-03-17T23:34:52.687 DEBUG:teuthology.orchestra.run.smithi150:> sudo service auditd rotate 2024-03-17T23:34:52.743 INFO:teuthology.orchestra.run.smithi150.stdout:Rotating logs: 2024-03-17T23:34:52.745 DEBUG:teuthology.orchestra.run.smithi157:> sudo service auditd rotate 2024-03-17T23:34:52.800 INFO:teuthology.orchestra.run.smithi157.stdout:Rotating logs: 2024-03-17T23:34:52.802 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-17T23:34:52.802 DEBUG:teuthology.orchestra.run.smithi150:> /usr/sbin/getenforce 2024-03-17T23:34:52.826 INFO:teuthology.orchestra.run.smithi150.stdout:Permissive 2024-03-17T23:34:52.827 DEBUG:teuthology.orchestra.run.smithi157:> /usr/sbin/getenforce 2024-03-17T23:34:52.844 INFO:teuthology.orchestra.run.smithi157.stdout:Permissive 2024-03-17T23:34:52.844 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi150.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi157.front.sepia.ceph.com': 'permissive'} 2024-03-17T23:34:52.845 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-17T23:34:52.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-17T23:34:52.894 DEBUG:teuthology.orchestra.run.smithi157:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-17T23:34:52.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-17T23:34:52.922 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-17T23:34:52.922 DEBUG:teuthology.orchestra.run.smithi150:> sudo /usr/sbin/setenforce permissive 2024-03-17T23:34:52.969 DEBUG:teuthology.orchestra.run.smithi157:> sudo /usr/sbin/setenforce permissive 2024-03-17T23:34:53.013 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-17T23:34:53.023 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-17T23:34:53.041 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-17T23:34:53.042 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi150.front.sepia.ceph.com,smithi157.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-17T23:44:07.922 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi150.front.sepia.ceph.com'), Remote(name='ubuntu@smithi157.front.sepia.ceph.com')] 2024-03-17T23:44:07.923 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-03-17T23:44:07.924 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-17T23:44:08.012 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-03-17T23:44:08.084 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-03-17T23:44:08.084 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-17T23:44:08.085 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-17T23:44:08.173 DEBUG:teuthology.orchestra.run.smithi157:> true 2024-03-17T23:44:08.249 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-03-17T23:44:08.249 INFO:teuthology.run_tasks:Running task clock... 2024-03-17T23:44:08.258 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-17T23:44:08.259 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-17T23:44:08.259 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-17T23:44:08.261 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-17T23:44:08.261 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-17T23:44:08.289 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-17T23:44:08.306 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-17T23:44:08.320 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-17T23:44:08.336 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: ntpd: command not found 2024-03-17T23:44:08.336 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-17T23:44:08.349 INFO:teuthology.orchestra.run.smithi150.stdout:506 Cannot talk to daemon 2024-03-17T23:44:08.366 INFO:teuthology.orchestra.run.smithi157.stderr:sudo: ntpd: command not found 2024-03-17T23:44:08.366 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-17T23:44:08.379 INFO:teuthology.orchestra.run.smithi157.stdout:506 Cannot talk to daemon 2024-03-17T23:44:08.381 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-17T23:44:08.396 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-17T23:44:08.412 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-17T23:44:08.434 INFO:teuthology.orchestra.run.smithi150.stderr:bash: line 1: ntpq: command not found 2024-03-17T23:44:08.437 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-17T23:44:08.437 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2024-03-17T23:44:08.437 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-17T23:44:08.438 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-17T23:44:08.438 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-17T23:44:08.438 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-17T23:44:08.461 INFO:teuthology.orchestra.run.smithi157.stderr:bash: line 1: ntpq: command not found 2024-03-17T23:44:08.465 INFO:teuthology.orchestra.run.smithi157.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-17T23:44:08.466 INFO:teuthology.orchestra.run.smithi157.stdout:=============================================================================== 2024-03-17T23:44:08.466 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-17T23:44:08.466 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-17T23:44:08.466 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-17T23:44:08.466 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-17T23:44:08.466 INFO:teuthology.run_tasks:Running task install... 2024-03-17T23:44:08.474 DEBUG:teuthology.task.install:project ceph 2024-03-17T23:44:08.474 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'}} 2024-03-17T23:44:08.474 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'} 2024-03-17T23:44:08.474 INFO:teuthology.task.install:Using flavor: default 2024-03-17T23:44:08.482 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-17T23:44:08.482 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-17T23:44:08.482 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-17T23:44:08.483 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-17T23:44:08.485 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-17T23:44:08.485 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-17T23:44:08.626 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-17T23:44:08.627 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-17T23:44:08.699 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-17T23:44:08.699 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-17T23:44:08.740 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-17T23:44:08.740 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:44:08.740 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-17T23:44:08.769 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-17T23:44:08.769 DEBUG:teuthology.orchestra.run.smithi157:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-17T23:44:08.810 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-17T23:44:08.811 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:44:08.811 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-17T23:44:08.838 DEBUG:teuthology.orchestra.run.smithi157:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-17T23:44:08.839 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-17T23:44:08.839 DEBUG:teuthology.orchestra.run.smithi150:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-17T23:44:08.906 DEBUG:teuthology.orchestra.run.smithi150:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-17T23:44:08.917 DEBUG:teuthology.orchestra.run.smithi157:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-17T23:44:08.984 INFO:teuthology.orchestra.run.smithi157.stdout:check_obsoletes = 1 2024-03-17T23:44:08.986 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum clean all 2024-03-17T23:44:08.987 DEBUG:teuthology.orchestra.run.smithi150:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-17T23:44:09.054 INFO:teuthology.orchestra.run.smithi150.stdout:check_obsoletes = 1 2024-03-17T23:44:09.055 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum clean all 2024-03-17T23:44:09.220 INFO:teuthology.orchestra.run.smithi157.stdout:74 files removed 2024-03-17T23:44:09.248 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-17T23:44:09.295 INFO:teuthology.orchestra.run.smithi150.stdout:74 files removed 2024-03-17T23:44:09.322 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-17T23:44:09.908 INFO:teuthology.orchestra.run.smithi157.stdout:ceph packages for x86_64 241 kB/s | 83 kB 00:00 2024-03-17T23:44:10.042 INFO:teuthology.orchestra.run.smithi150.stdout:ceph packages for x86_64 206 kB/s | 83 kB 00:00 2024-03-17T23:44:10.350 INFO:teuthology.orchestra.run.smithi157.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-17T23:44:10.426 INFO:teuthology.orchestra.run.smithi150.stdout:ceph noarch packages 30 kB/s | 10 kB 00:00 2024-03-17T23:44:10.776 INFO:teuthology.orchestra.run.smithi157.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-03-17T23:44:10.960 INFO:teuthology.orchestra.run.smithi150.stdout:ceph source packages 3.8 kB/s | 1.9 kB 00:00 2024-03-17T23:44:11.219 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-17T23:44:11.629 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-17T23:44:12.612 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-17T23:44:13.096 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2024-03-17T23:44:13.636 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-17T23:44:15.664 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-17T23:44:22.327 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - BaseOS 923 kB/s | 8.0 MB 00:08 2024-03-17T23:44:24.253 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-17T23:44:27.512 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - CRB 746 kB/s | 6.1 MB 00:08 2024-03-17T23:44:28.323 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - CRB 9.8 MB/s | 6.1 MB 00:00 2024-03-17T23:44:28.929 INFO:teuthology.orchestra.run.smithi157.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 16 kB 00:00 2024-03-17T23:44:29.797 INFO:teuthology.orchestra.run.smithi157.stdout:Extra Packages for Enterprise Linux 28 MB/s | 21 MB 00:00 2024-03-17T23:44:29.974 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-03-17T23:44:31.609 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-17T23:44:34.681 INFO:teuthology.orchestra.run.smithi157.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-03-17T23:44:36.559 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-17T23:44:37.203 INFO:teuthology.orchestra.run.smithi157.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-17T23:44:37.204 INFO:teuthology.orchestra.run.smithi157.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-17T23:44:37.218 INFO:teuthology.orchestra.run.smithi157.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-17T23:44:37.219 INFO:teuthology.orchestra.run.smithi157.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-17T23:44:37.220 INFO:teuthology.orchestra.run.smithi157.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-17T23:44:37.221 INFO:teuthology.orchestra.run.smithi157.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-17T23:44:37.224 INFO:teuthology.orchestra.run.smithi157.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-17T23:44:37.227 INFO:teuthology.orchestra.run.smithi157.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-17T23:44:37.228 INFO:teuthology.orchestra.run.smithi157.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-17T23:44:37.309 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2024-03-17T23:44:37.324 INFO:teuthology.orchestra.run.smithi157.stdout:============================================================================================ 2024-03-17T23:44:37.324 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2024-03-17T23:44:37.324 INFO:teuthology.orchestra.run.smithi157.stdout:============================================================================================ 2024-03-17T23:44:37.324 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2024-03-17T23:44:37.324 INFO:teuthology.orchestra.run.smithi157.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-17T23:44:37.324 INFO:teuthology.orchestra.run.smithi157.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-17T23:44:37.324 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-17T23:44:37.325 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-17T23:44:37.325 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-17T23:44:37.325 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-17T23:44:37.325 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-17T23:44:37.325 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-17T23:44:37.325 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-17T23:44:37.325 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-17T23:44:37.325 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-17T23:44:37.325 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-17T23:44:37.325 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-17T23:44:37.325 INFO:teuthology.orchestra.run.smithi157.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-17T23:44:37.325 INFO:teuthology.orchestra.run.smithi157.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-17T23:44:37.326 INFO:teuthology.orchestra.run.smithi157.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-17T23:44:37.326 INFO:teuthology.orchestra.run.smithi157.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-17T23:44:37.326 INFO:teuthology.orchestra.run.smithi157.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-17T23:44:37.326 INFO:teuthology.orchestra.run.smithi157.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-17T23:44:37.326 INFO:teuthology.orchestra.run.smithi157.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-17T23:44:37.326 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-17T23:44:37.326 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-17T23:44:37.326 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-17T23:44:37.326 INFO:teuthology.orchestra.run.smithi157.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-17T23:44:37.326 INFO:teuthology.orchestra.run.smithi157.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-17T23:44:37.326 INFO:teuthology.orchestra.run.smithi157.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-17T23:44:37.326 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-17T23:44:37.327 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-17T23:44:37.327 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-17T23:44:37.327 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-17T23:44:37.327 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-17T23:44:37.327 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-17T23:44:37.327 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-17T23:44:37.327 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-17T23:44:37.327 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-17T23:44:37.327 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-17T23:44:37.327 INFO:teuthology.orchestra.run.smithi157.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-17T23:44:37.327 INFO:teuthology.orchestra.run.smithi157.stdout:Upgrading: 2024-03-17T23:44:37.327 INFO:teuthology.orchestra.run.smithi157.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-17T23:44:37.327 INFO:teuthology.orchestra.run.smithi157.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-17T23:44:37.327 INFO:teuthology.orchestra.run.smithi157.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-17T23:44:37.328 INFO:teuthology.orchestra.run.smithi157.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-17T23:44:37.328 INFO:teuthology.orchestra.run.smithi157.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-17T23:44:37.328 INFO:teuthology.orchestra.run.smithi157.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-17T23:44:37.328 INFO:teuthology.orchestra.run.smithi157.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-17T23:44:37.328 INFO:teuthology.orchestra.run.smithi157.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-17T23:44:37.328 INFO:teuthology.orchestra.run.smithi157.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-17T23:44:37.328 INFO:teuthology.orchestra.run.smithi157.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-17T23:44:37.328 INFO:teuthology.orchestra.run.smithi157.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-17T23:44:37.328 INFO:teuthology.orchestra.run.smithi157.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-17T23:44:37.328 INFO:teuthology.orchestra.run.smithi157.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-17T23:44:37.328 INFO:teuthology.orchestra.run.smithi157.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-17T23:44:37.328 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-17T23:44:37.329 INFO:teuthology.orchestra.run.smithi157.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-17T23:44:37.329 INFO:teuthology.orchestra.run.smithi157.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-17T23:44:37.329 INFO:teuthology.orchestra.run.smithi157.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-17T23:44:37.329 INFO:teuthology.orchestra.run.smithi157.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-17T23:44:37.329 INFO:teuthology.orchestra.run.smithi157.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-17T23:44:37.329 INFO:teuthology.orchestra.run.smithi157.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-17T23:44:37.329 INFO:teuthology.orchestra.run.smithi157.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-17T23:44:37.329 INFO:teuthology.orchestra.run.smithi157.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-17T23:44:37.329 INFO:teuthology.orchestra.run.smithi157.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-17T23:44:37.329 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2024-03-17T23:44:37.329 INFO:teuthology.orchestra.run.smithi157.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-17T23:44:37.329 INFO:teuthology.orchestra.run.smithi157.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-17T23:44:37.329 INFO:teuthology.orchestra.run.smithi157.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-17T23:44:37.330 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-17T23:44:37.330 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-17T23:44:37.330 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-17T23:44:37.330 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-17T23:44:37.330 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-17T23:44:37.330 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-17T23:44:37.330 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-17T23:44:37.330 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-17T23:44:37.330 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-17T23:44:37.330 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-17T23:44:37.330 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-17T23:44:37.330 INFO:teuthology.orchestra.run.smithi157.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-17T23:44:37.331 INFO:teuthology.orchestra.run.smithi157.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-17T23:44:37.331 INFO:teuthology.orchestra.run.smithi157.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-17T23:44:37.331 INFO:teuthology.orchestra.run.smithi157.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-17T23:44:37.331 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-17T23:44:37.331 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-17T23:44:37.331 INFO:teuthology.orchestra.run.smithi157.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-17T23:44:37.331 INFO:teuthology.orchestra.run.smithi157.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-17T23:44:37.331 INFO:teuthology.orchestra.run.smithi157.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-17T23:44:37.331 INFO:teuthology.orchestra.run.smithi157.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-17T23:44:37.331 INFO:teuthology.orchestra.run.smithi157.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-17T23:44:37.331 INFO:teuthology.orchestra.run.smithi157.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-17T23:44:37.331 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-17T23:44:37.331 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-17T23:44:37.332 INFO:teuthology.orchestra.run.smithi157.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-17T23:44:37.332 INFO:teuthology.orchestra.run.smithi157.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-17T23:44:37.332 INFO:teuthology.orchestra.run.smithi157.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-17T23:44:37.332 INFO:teuthology.orchestra.run.smithi157.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-17T23:44:37.332 INFO:teuthology.orchestra.run.smithi157.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-17T23:44:37.332 INFO:teuthology.orchestra.run.smithi157.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-17T23:44:37.332 INFO:teuthology.orchestra.run.smithi157.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-17T23:44:37.332 INFO:teuthology.orchestra.run.smithi157.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-17T23:44:37.332 INFO:teuthology.orchestra.run.smithi157.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-17T23:44:37.332 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-17T23:44:37.332 INFO:teuthology.orchestra.run.smithi157.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-17T23:44:37.332 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-17T23:44:37.332 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-17T23:44:37.332 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-17T23:44:37.333 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-17T23:44:37.333 INFO:teuthology.orchestra.run.smithi157.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-17T23:44:37.333 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-17T23:44:37.333 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-17T23:44:37.333 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-17T23:44:37.333 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-17T23:44:37.333 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-17T23:44:37.333 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-17T23:44:37.333 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-17T23:44:37.333 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-17T23:44:37.333 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-17T23:44:37.333 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-17T23:44:37.333 INFO:teuthology.orchestra.run.smithi157.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-17T23:44:37.334 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-17T23:44:37.334 INFO:teuthology.orchestra.run.smithi157.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-17T23:44:37.334 INFO:teuthology.orchestra.run.smithi157.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-17T23:44:37.334 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-17T23:44:37.334 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-17T23:44:37.334 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-17T23:44:37.334 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-17T23:44:37.334 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-17T23:44:37.334 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-17T23:44:37.334 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-17T23:44:37.334 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-17T23:44:37.335 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-17T23:44:37.335 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-17T23:44:37.335 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-17T23:44:37.335 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-17T23:44:37.335 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-17T23:44:37.335 INFO:teuthology.orchestra.run.smithi157.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-17T23:44:37.335 INFO:teuthology.orchestra.run.smithi157.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-17T23:44:37.335 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-17T23:44:37.335 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-17T23:44:37.335 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-17T23:44:37.335 INFO:teuthology.orchestra.run.smithi157.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-17T23:44:37.335 INFO:teuthology.orchestra.run.smithi157.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-17T23:44:37.335 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-17T23:44:37.336 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-17T23:44:37.336 INFO:teuthology.orchestra.run.smithi157.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-17T23:44:37.336 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-17T23:44:37.336 INFO:teuthology.orchestra.run.smithi157.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-17T23:44:37.336 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-17T23:44:37.336 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-17T23:44:37.336 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-17T23:44:37.337 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-17T23:44:37.337 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-17T23:44:37.337 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-17T23:44:37.337 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-17T23:44:37.337 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-17T23:44:37.337 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-17T23:44:37.337 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-17T23:44:37.337 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-17T23:44:37.338 INFO:teuthology.orchestra.run.smithi157.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-17T23:44:37.338 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-17T23:44:37.338 INFO:teuthology.orchestra.run.smithi157.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-17T23:44:37.338 INFO:teuthology.orchestra.run.smithi157.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-17T23:44:37.338 INFO:teuthology.orchestra.run.smithi157.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-17T23:44:37.338 INFO:teuthology.orchestra.run.smithi157.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-17T23:44:37.338 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-17T23:44:37.338 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-17T23:44:37.338 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-17T23:44:37.338 INFO:teuthology.orchestra.run.smithi157.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-17T23:44:37.338 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-17T23:44:37.338 INFO:teuthology.orchestra.run.smithi157.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-17T23:44:37.338 INFO:teuthology.orchestra.run.smithi157.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-17T23:44:37.339 INFO:teuthology.orchestra.run.smithi157.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-17T23:44:37.339 INFO:teuthology.orchestra.run.smithi157.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-17T23:44:37.339 INFO:teuthology.orchestra.run.smithi157.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-17T23:44:37.339 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-17T23:44:37.339 INFO:teuthology.orchestra.run.smithi157.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-17T23:44:37.339 INFO:teuthology.orchestra.run.smithi157.stdout:Installing weak dependencies: 2024-03-17T23:44:37.339 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-17T23:44:37.339 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:44:37.339 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2024-03-17T23:44:37.339 INFO:teuthology.orchestra.run.smithi157.stdout:============================================================================================ 2024-03-17T23:44:37.339 INFO:teuthology.orchestra.run.smithi157.stdout:Install 150 Packages 2024-03-17T23:44:37.339 INFO:teuthology.orchestra.run.smithi157.stdout:Upgrade 24 Packages 2024-03-17T23:44:37.339 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:44:37.340 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 224 M 2024-03-17T23:44:37.340 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2024-03-17T23:44:38.290 INFO:teuthology.orchestra.run.smithi157.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 27 kB/s | 6.6 kB 00:00 2024-03-17T23:44:38.657 INFO:teuthology.orchestra.run.smithi157.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 2.4 MB/s | 890 kB 00:00 2024-03-17T23:44:38.782 INFO:teuthology.orchestra.run.smithi157.stdout:(3/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 7.3 MB/s | 5.4 MB 00:00 2024-03-17T23:44:38.808 INFO:teuthology.orchestra.run.smithi157.stdout:(4/174): ceph-immutable-object-cache-19.0.0-216 964 kB/s | 145 kB 00:00 2024-03-17T23:44:39.024 INFO:teuthology.orchestra.run.smithi157.stdout:(5/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 9.8 MB/s | 2.4 MB 00:00 2024-03-17T23:44:39.072 INFO:teuthology.orchestra.run.smithi150.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-17T23:44:39.073 INFO:teuthology.orchestra.run.smithi150.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-17T23:44:39.087 INFO:teuthology.orchestra.run.smithi150.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-17T23:44:39.088 INFO:teuthology.orchestra.run.smithi150.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-17T23:44:39.089 INFO:teuthology.orchestra.run.smithi150.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-17T23:44:39.091 INFO:teuthology.orchestra.run.smithi150.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-17T23:44:39.093 INFO:teuthology.orchestra.run.smithi150.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-17T23:44:39.095 INFO:teuthology.orchestra.run.smithi150.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-17T23:44:39.097 INFO:teuthology.orchestra.run.smithi150.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-17T23:44:39.177 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-03-17T23:44:39.203 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================ 2024-03-17T23:44:39.203 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-03-17T23:44:39.203 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================ 2024-03-17T23:44:39.203 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-03-17T23:44:39.203 INFO:teuthology.orchestra.run.smithi150.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-17T23:44:39.203 INFO:teuthology.orchestra.run.smithi150.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-17T23:44:39.203 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-17T23:44:39.203 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-17T23:44:39.204 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-17T23:44:39.204 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-17T23:44:39.204 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-17T23:44:39.204 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-17T23:44:39.204 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-17T23:44:39.204 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-17T23:44:39.204 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-17T23:44:39.204 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-17T23:44:39.204 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-17T23:44:39.204 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-17T23:44:39.204 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-17T23:44:39.204 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-17T23:44:39.204 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-17T23:44:39.205 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-17T23:44:39.205 INFO:teuthology.orchestra.run.smithi150.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-17T23:44:39.205 INFO:teuthology.orchestra.run.smithi150.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-17T23:44:39.205 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-17T23:44:39.205 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-17T23:44:39.205 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-17T23:44:39.205 INFO:teuthology.orchestra.run.smithi150.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-17T23:44:39.205 INFO:teuthology.orchestra.run.smithi150.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-17T23:44:39.205 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-17T23:44:39.205 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-17T23:44:39.205 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-17T23:44:39.205 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-17T23:44:39.205 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-17T23:44:39.205 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-17T23:44:39.206 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-17T23:44:39.206 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-17T23:44:39.206 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-17T23:44:39.206 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-17T23:44:39.206 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-17T23:44:39.206 INFO:teuthology.orchestra.run.smithi150.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-17T23:44:39.206 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrading: 2024-03-17T23:44:39.206 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-17T23:44:39.206 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-17T23:44:39.206 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-17T23:44:39.206 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-17T23:44:39.206 INFO:teuthology.orchestra.run.smithi150.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-17T23:44:39.206 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-17T23:44:39.206 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-17T23:44:39.207 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-17T23:44:39.207 INFO:teuthology.orchestra.run.smithi150.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-17T23:44:39.207 INFO:teuthology.orchestra.run.smithi150.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-17T23:44:39.207 INFO:teuthology.orchestra.run.smithi150.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-17T23:44:39.207 INFO:teuthology.orchestra.run.smithi150.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-17T23:44:39.207 INFO:teuthology.orchestra.run.smithi150.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-17T23:44:39.207 INFO:teuthology.orchestra.run.smithi150.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-17T23:44:39.207 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-17T23:44:39.207 INFO:teuthology.orchestra.run.smithi150.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-17T23:44:39.207 INFO:teuthology.orchestra.run.smithi150.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-17T23:44:39.207 INFO:teuthology.orchestra.run.smithi150.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-17T23:44:39.207 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-17T23:44:39.207 INFO:teuthology.orchestra.run.smithi150.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-17T23:44:39.207 INFO:teuthology.orchestra.run.smithi150.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-17T23:44:39.208 INFO:teuthology.orchestra.run.smithi150.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-17T23:44:39.208 INFO:teuthology.orchestra.run.smithi150.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-17T23:44:39.208 INFO:teuthology.orchestra.run.smithi150.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-17T23:44:39.208 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-03-17T23:44:39.208 INFO:teuthology.orchestra.run.smithi150.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-17T23:44:39.208 INFO:teuthology.orchestra.run.smithi150.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-17T23:44:39.208 INFO:teuthology.orchestra.run.smithi150.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-17T23:44:39.208 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-17T23:44:39.208 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-17T23:44:39.208 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-17T23:44:39.208 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-17T23:44:39.208 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-17T23:44:39.208 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-17T23:44:39.209 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-17T23:44:39.209 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-17T23:44:39.209 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-17T23:44:39.209 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-17T23:44:39.209 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-17T23:44:39.209 INFO:teuthology.orchestra.run.smithi150.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-17T23:44:39.209 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-17T23:44:39.209 INFO:teuthology.orchestra.run.smithi150.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-17T23:44:39.209 INFO:teuthology.orchestra.run.smithi150.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-17T23:44:39.209 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-17T23:44:39.209 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-17T23:44:39.209 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-17T23:44:39.209 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-17T23:44:39.209 INFO:teuthology.orchestra.run.smithi150.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-17T23:44:39.209 INFO:teuthology.orchestra.run.smithi150.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-17T23:44:39.210 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-17T23:44:39.210 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-17T23:44:39.210 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-17T23:44:39.210 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-17T23:44:39.210 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-17T23:44:39.210 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-17T23:44:39.210 INFO:teuthology.orchestra.run.smithi150.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-17T23:44:39.210 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-17T23:44:39.210 INFO:teuthology.orchestra.run.smithi150.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-17T23:44:39.210 INFO:teuthology.orchestra.run.smithi150.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-17T23:44:39.210 INFO:teuthology.orchestra.run.smithi150.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-17T23:44:39.210 INFO:teuthology.orchestra.run.smithi150.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-17T23:44:39.210 INFO:teuthology.orchestra.run.smithi150.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-17T23:44:39.211 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-17T23:44:39.211 INFO:teuthology.orchestra.run.smithi150.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-17T23:44:39.211 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-17T23:44:39.211 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-17T23:44:39.211 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-17T23:44:39.211 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-17T23:44:39.211 INFO:teuthology.orchestra.run.smithi150.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-17T23:44:39.211 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-17T23:44:39.211 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-17T23:44:39.211 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-17T23:44:39.211 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-17T23:44:39.211 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-17T23:44:39.211 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-17T23:44:39.211 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-17T23:44:39.212 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-17T23:44:39.212 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-17T23:44:39.212 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-17T23:44:39.212 INFO:teuthology.orchestra.run.smithi150.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-17T23:44:39.212 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-17T23:44:39.212 INFO:teuthology.orchestra.run.smithi150.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-17T23:44:39.212 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-17T23:44:39.212 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-17T23:44:39.212 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-17T23:44:39.212 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-17T23:44:39.212 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-17T23:44:39.212 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-17T23:44:39.212 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-17T23:44:39.212 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-17T23:44:39.213 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-17T23:44:39.213 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-17T23:44:39.213 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-17T23:44:39.213 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-17T23:44:39.213 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-17T23:44:39.213 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-17T23:44:39.213 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-17T23:44:39.213 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-17T23:44:39.213 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-17T23:44:39.213 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-17T23:44:39.213 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-17T23:44:39.213 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-17T23:44:39.213 INFO:teuthology.orchestra.run.smithi150.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-17T23:44:39.213 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-17T23:44:39.213 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-17T23:44:39.214 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-17T23:44:39.214 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-17T23:44:39.214 INFO:teuthology.orchestra.run.smithi150.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-17T23:44:39.214 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-17T23:44:39.214 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-17T23:44:39.214 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-17T23:44:39.214 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-17T23:44:39.214 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-17T23:44:39.214 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-17T23:44:39.214 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-17T23:44:39.214 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-17T23:44:39.214 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-17T23:44:39.214 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-17T23:44:39.215 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-17T23:44:39.215 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-17T23:44:39.215 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-17T23:44:39.215 INFO:teuthology.orchestra.run.smithi150.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-17T23:44:39.215 INFO:teuthology.orchestra.run.smithi150.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-17T23:44:39.215 INFO:teuthology.orchestra.run.smithi150.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-17T23:44:39.215 INFO:teuthology.orchestra.run.smithi150.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-17T23:44:39.215 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-17T23:44:39.215 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-17T23:44:39.215 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-17T23:44:39.215 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-17T23:44:39.215 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-17T23:44:39.215 INFO:teuthology.orchestra.run.smithi150.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-17T23:44:39.215 INFO:teuthology.orchestra.run.smithi150.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-17T23:44:39.215 INFO:teuthology.orchestra.run.smithi150.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-17T23:44:39.216 INFO:teuthology.orchestra.run.smithi150.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-17T23:44:39.216 INFO:teuthology.orchestra.run.smithi150.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-17T23:44:39.216 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-17T23:44:39.216 INFO:teuthology.orchestra.run.smithi150.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-17T23:44:39.216 INFO:teuthology.orchestra.run.smithi150.stdout:Installing weak dependencies: 2024-03-17T23:44:39.216 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-17T23:44:39.216 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:44:39.216 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-03-17T23:44:39.216 INFO:teuthology.orchestra.run.smithi150.stdout:============================================================================================ 2024-03-17T23:44:39.216 INFO:teuthology.orchestra.run.smithi150.stdout:Install 150 Packages 2024-03-17T23:44:39.216 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrade 24 Packages 2024-03-17T23:44:39.216 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:44:39.217 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 224 M 2024-03-17T23:44:39.217 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-03-17T23:44:39.241 INFO:teuthology.orchestra.run.smithi157.stdout:(6/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 3.4 MB/s | 1.5 MB 00:00 2024-03-17T23:44:39.500 INFO:teuthology.orchestra.run.smithi157.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 9.9 MB/s | 4.7 MB 00:00 2024-03-17T23:44:40.059 INFO:teuthology.orchestra.run.smithi157.stdout:(8/174): ceph-common-19.0.0-2162.g3d78f5ba.el9. 10 MB/s | 21 MB 00:02 2024-03-17T23:44:40.125 INFO:teuthology.orchestra.run.smithi157.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 379 kB/s | 25 kB 00:00 2024-03-17T23:44:40.177 INFO:teuthology.orchestra.run.smithi150.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 32 kB/s | 6.6 kB 00:00 2024-03-17T23:44:40.351 INFO:teuthology.orchestra.run.smithi157.stdout:(10/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 12 MB/s | 9.8 MB 00:00 2024-03-17T23:44:40.409 INFO:teuthology.orchestra.run.smithi157.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 3.7 MB/s | 221 kB 00:00 2024-03-17T23:44:40.459 INFO:teuthology.orchestra.run.smithi157.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 544 kB/s | 27 kB 00:00 2024-03-17T23:44:40.510 INFO:teuthology.orchestra.run.smithi157.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 633 kB/s | 32 kB 00:00 2024-03-17T23:44:40.619 INFO:teuthology.orchestra.run.smithi157.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 6.7 MB/s | 747 kB 00:00 2024-03-17T23:44:40.685 INFO:teuthology.orchestra.run.smithi157.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 2.4 MB/s | 163 kB 00:00 2024-03-17T23:44:40.744 INFO:teuthology.orchestra.run.smithi157.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 2.1 MB/s | 127 kB 00:00 2024-03-17T23:44:40.810 INFO:teuthology.orchestra.run.smithi150.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.4 MB/s | 890 kB 00:00 2024-03-17T23:44:40.828 INFO:teuthology.orchestra.run.smithi157.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 5.9 MB/s | 501 kB 00:00 2024-03-17T23:44:40.944 INFO:teuthology.orchestra.run.smithi150.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 1.1 MB/s | 145 kB 00:00 2024-03-17T23:44:41.361 INFO:teuthology.orchestra.run.smithi157.stdout:(18/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 9.6 MB/s | 5.1 MB 00:00 2024-03-17T23:44:41.461 INFO:teuthology.orchestra.run.smithi157.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 452 kB/s | 45 kB 00:00 2024-03-17T23:44:41.537 INFO:teuthology.orchestra.run.smithi157.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.7 MB/s | 131 kB 00:00 2024-03-17T23:44:41.612 INFO:teuthology.orchestra.run.smithi157.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 2.1 MB/s | 162 kB 00:00 2024-03-17T23:44:41.696 INFO:teuthology.orchestra.run.smithi157.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 3.8 MB/s | 320 kB 00:00 2024-03-17T23:44:41.788 INFO:teuthology.orchestra.run.smithi157.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 3.2 MB/s | 298 kB 00:00 2024-03-17T23:44:41.871 INFO:teuthology.orchestra.run.smithi157.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.2 MB/s | 99 kB 00:00 2024-03-17T23:44:41.947 INFO:teuthology.orchestra.run.smithi157.stdout:(25/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-17T23:44:42.327 INFO:teuthology.orchestra.run.smithi150.stdout:(4/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 1.7 MB/s | 2.4 MB 00:01 2024-03-17T23:44:42.380 INFO:teuthology.orchestra.run.smithi157.stdout:(26/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 7.1 MB/s | 3.1 MB 00:00 2024-03-17T23:44:42.464 INFO:teuthology.orchestra.run.smithi157.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 2.0 MB/s | 170 kB 00:00 2024-03-17T23:44:42.539 INFO:teuthology.orchestra.run.smithi157.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 452 kB/s | 34 kB 00:00 2024-03-17T23:44:42.631 INFO:teuthology.orchestra.run.smithi157.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 1.5 MB/s | 139 kB 00:00 2024-03-17T23:44:43.036 INFO:teuthology.orchestra.run.smithi150.stdout:(5/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 2.1 MB/s | 1.5 MB 00:00 2024-03-17T23:44:43.115 INFO:teuthology.orchestra.run.smithi157.stdout:(30/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 4.3 MB/s | 17 MB 00:03 2024-03-17T23:44:43.157 INFO:teuthology.orchestra.run.smithi157.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 3.4 MB/s | 1.8 MB 00:00 2024-03-17T23:44:43.216 INFO:teuthology.orchestra.run.smithi157.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 4.2 MB/s | 248 kB 00:00 2024-03-17T23:44:43.244 INFO:teuthology.orchestra.run.smithi150.stdout:(6/174): ceph-common-19.0.0-2162.g3d78f5ba.el9. 6.3 MB/s | 21 MB 00:03 2024-03-17T23:44:43.257 INFO:teuthology.orchestra.run.smithi157.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 1.2 MB/s | 49 kB 00:00 2024-03-17T23:44:43.316 INFO:teuthology.orchestra.run.smithi157.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 283 kB/s | 16 kB 00:00 2024-03-17T23:44:43.400 INFO:teuthology.orchestra.run.smithi157.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 3.1 MB/s | 264 kB 00:00 2024-03-17T23:44:43.583 INFO:teuthology.orchestra.run.smithi157.stdout:(36/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 4.0 MB/s | 748 kB 00:00 2024-03-17T23:44:43.595 INFO:teuthology.orchestra.run.smithi150.stdout:(7/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 1.5 MB/s | 5.4 MB 00:03 2024-03-17T23:44:43.643 INFO:teuthology.orchestra.run.smithi157.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 313 kB/s | 18 kB 00:00 2024-03-17T23:44:43.775 INFO:teuthology.orchestra.run.smithi157.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.9 MB/s | 662 kB 00:00 2024-03-17T23:44:43.826 INFO:teuthology.orchestra.run.smithi157.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 18 MB/s | 920 kB 00:00 2024-03-17T23:44:43.859 INFO:teuthology.orchestra.run.smithi157.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-17T23:44:43.901 INFO:teuthology.orchestra.run.smithi157.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-17T23:44:43.927 INFO:teuthology.orchestra.run.smithi157.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 1.2 MB/s | 32 kB 00:00 2024-03-17T23:44:44.027 INFO:teuthology.orchestra.run.smithi157.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-03-17T23:44:44.052 INFO:teuthology.orchestra.run.smithi157.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 689 kB/s | 17 kB 00:00 2024-03-17T23:44:44.077 INFO:teuthology.orchestra.run.smithi157.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 404 kB/s | 10 kB 00:00 2024-03-17T23:44:44.119 INFO:teuthology.orchestra.run.smithi157.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-17T23:44:44.153 INFO:teuthology.orchestra.run.smithi157.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-17T23:44:44.212 INFO:teuthology.orchestra.run.smithi157.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-17T23:44:44.254 INFO:teuthology.orchestra.run.smithi157.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 14 MB/s | 578 kB 00:00 2024-03-17T23:44:44.296 INFO:teuthology.orchestra.run.smithi157.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-17T23:44:44.337 INFO:teuthology.orchestra.run.smithi157.stdout:(51/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 282 kB 00:00 2024-03-17T23:44:44.371 INFO:teuthology.orchestra.run.smithi157.stdout:(52/174): lua-5.4.4-4.el9.x86_64.rpm 5.4 MB/s | 187 kB 00:00 2024-03-17T23:44:44.413 INFO:teuthology.orchestra.run.smithi157.stdout:(53/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-03-17T23:44:44.447 INFO:teuthology.orchestra.run.smithi157.stdout:(54/174): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-17T23:44:44.554 INFO:teuthology.orchestra.run.smithi150.stdout:(8/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x86 13 MB/s | 17 MB 00:01 2024-03-17T23:44:44.572 INFO:teuthology.orchestra.run.smithi157.stdout:(55/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-17T23:44:44.604 INFO:teuthology.orchestra.run.smithi150.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 508 kB/s | 25 kB 00:00 2024-03-17T23:44:44.631 INFO:teuthology.orchestra.run.smithi157.stdout:(56/174): postgresql-13.11-1.el9.x86_64.rpm 26 MB/s | 1.5 MB 00:00 2024-03-17T23:44:44.681 INFO:teuthology.orchestra.run.smithi157.stdout:(57/174): postgresql-contrib-13.11-1.el9.x86_64 16 MB/s | 811 kB 00:00 2024-03-17T23:44:44.715 INFO:teuthology.orchestra.run.smithi157.stdout:(58/174): postgresql-private-libs-13.11-1.el9.x 3.9 MB/s | 134 kB 00:00 2024-03-17T23:44:44.729 INFO:teuthology.orchestra.run.smithi150.stdout:(10/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x8 2.8 MB/s | 4.7 MB 00:01 2024-03-17T23:44:44.879 INFO:teuthology.orchestra.run.smithi150.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 1.4 MB/s | 221 kB 00:00 2024-03-17T23:44:44.924 INFO:teuthology.orchestra.run.smithi157.stdout:(59/174): postgresql-server-13.11-1.el9.x86_64. 27 MB/s | 5.7 MB 00:00 2024-03-17T23:44:44.955 INFO:teuthology.orchestra.run.smithi150.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 361 kB/s | 27 kB 00:00 2024-03-17T23:44:44.974 INFO:teuthology.orchestra.run.smithi157.stdout:(60/174): protobuf-3.14.0-13.el9.x86_64.rpm 20 MB/s | 1.0 MB 00:00 2024-03-17T23:44:45.008 INFO:teuthology.orchestra.run.smithi157.stdout:(61/174): python3-attrs-20.3.0-7.el9.noarch.rpm 2.3 MB/s | 78 kB 00:00 2024-03-17T23:44:45.030 INFO:teuthology.orchestra.run.smithi150.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 424 kB/s | 32 kB 00:00 2024-03-17T23:44:45.158 INFO:teuthology.orchestra.run.smithi157.stdout:(62/174): python3-babel-2.9.1-2.el9.noarch.rpm 39 MB/s | 5.8 MB 00:00 2024-03-17T23:44:45.183 INFO:teuthology.orchestra.run.smithi157.stdout:(63/174): python3-jsonpatch-1.21-16.el9.noarch. 979 kB/s | 24 kB 00:00 2024-03-17T23:44:45.208 INFO:teuthology.orchestra.run.smithi157.stdout:(64/174): python3-jsonpointer-2.0-4.el9.noarch. 723 kB/s | 18 kB 00:00 2024-03-17T23:44:45.242 INFO:teuthology.orchestra.run.smithi157.stdout:(65/174): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-03-17T23:44:45.267 INFO:teuthology.orchestra.run.smithi157.stdout:(66/174): python3-markupsafe-1.1.1-12.el9.x86_6 1.2 MB/s | 32 kB 00:00 2024-03-17T23:44:45.314 INFO:teuthology.orchestra.run.smithi150.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 2.6 MB/s | 747 kB 00:00 2024-03-17T23:44:45.393 INFO:teuthology.orchestra.run.smithi157.stdout:(67/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-03-17T23:44:45.405 INFO:teuthology.orchestra.run.smithi150.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 1.7 MB/s | 163 kB 00:00 2024-03-17T23:44:45.435 INFO:teuthology.orchestra.run.smithi157.stdout:(68/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.1 MB/s | 217 kB 00:00 2024-03-17T23:44:45.468 INFO:teuthology.orchestra.run.smithi157.stdout:(69/174): python3-oauthlib-3.1.1-5.el9.noarch.r 5.1 MB/s | 174 kB 00:00 2024-03-17T23:44:45.497 INFO:teuthology.orchestra.run.smithi150.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.4 MB/s | 127 kB 00:00 2024-03-17T23:44:45.502 INFO:teuthology.orchestra.run.smithi157.stdout:(70/174): python3-packaging-20.9-5.el9.noarch.r 2.0 MB/s | 69 kB 00:00 2024-03-17T23:44:45.527 INFO:teuthology.orchestra.run.smithi157.stdout:(71/174): python3-prettytable-0.7.2-27.el9.noar 1.6 MB/s | 41 kB 00:00 2024-03-17T23:44:45.569 INFO:teuthology.orchestra.run.smithi157.stdout:(72/174): python3-protobuf-3.14.0-13.el9.noarch 5.6 MB/s | 238 kB 00:00 2024-03-17T23:44:45.603 INFO:teuthology.orchestra.run.smithi157.stdout:(73/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.9 MB/s | 132 kB 00:00 2024-03-17T23:44:45.645 INFO:teuthology.orchestra.run.smithi157.stdout:(74/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.9 MB/s | 211 kB 00:00 2024-03-17T23:44:45.670 INFO:teuthology.orchestra.run.smithi157.stdout:(75/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.8 MB/s | 47 kB 00:00 2024-03-17T23:44:45.695 INFO:teuthology.orchestra.run.smithi157.stdout:(76/174): python3-requests-oauthlib-1.3.0-12.el 1.7 MB/s | 43 kB 00:00 2024-03-17T23:44:45.731 INFO:teuthology.orchestra.run.smithi150.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 2.1 MB/s | 501 kB 00:00 2024-03-17T23:44:46.046 INFO:teuthology.orchestra.run.smithi157.stdout:(77/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-03-17T23:44:46.080 INFO:teuthology.orchestra.run.smithi157.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-17T23:44:46.113 INFO:teuthology.orchestra.run.smithi157.stdout:(79/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.2 MB/s | 41 kB 00:00 2024-03-17T23:44:46.155 INFO:teuthology.orchestra.run.smithi157.stdout:(80/174): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2024-03-17T23:44:46.189 INFO:teuthology.orchestra.run.smithi157.stdout:(81/174): uuid-1.6.2-55.el9.x86_64.rpm 1.6 MB/s | 56 kB 00:00 2024-03-17T23:44:46.231 INFO:teuthology.orchestra.run.smithi157.stdout:(82/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-17T23:44:46.264 INFO:teuthology.orchestra.run.smithi157.stdout:(83/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-17T23:44:46.306 INFO:teuthology.orchestra.run.smithi157.stdout:(84/174): fuse-2.9.9-15.el9.x86_64.rpm 1.9 MB/s | 79 kB 00:00 2024-03-17T23:44:46.348 INFO:teuthology.orchestra.run.smithi157.stdout:(85/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-17T23:44:46.407 INFO:teuthology.orchestra.run.smithi157.stdout:(86/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-17T23:44:46.441 INFO:teuthology.orchestra.run.smithi157.stdout:(87/174): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-17T23:44:46.482 INFO:teuthology.orchestra.run.smithi157.stdout:(88/174): python3-pycparser-2.20-6.el9.noarch.r 2.9 MB/s | 124 kB 00:00 2024-03-17T23:44:46.941 INFO:teuthology.orchestra.run.smithi157.stdout:(89/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 7.0 MB/s | 48 MB 00:06 2024-03-17T23:44:46.967 INFO:teuthology.orchestra.run.smithi157.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 387 kB/s | 187 kB 00:00 2024-03-17T23:44:47.058 INFO:teuthology.orchestra.run.smithi157.stdout:(91/174): ceph-mgr-diskprediction-local-19.0.0- 1.9 MB/s | 7.4 MB 00:03 2024-03-17T23:44:47.117 INFO:teuthology.orchestra.run.smithi157.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-17T23:44:47.159 INFO:teuthology.orchestra.run.smithi157.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.9 MB/s | 195 kB 00:00 2024-03-17T23:44:47.184 INFO:teuthology.orchestra.run.smithi157.stdout:(94/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.6 MB/s | 803 kB 00:00 2024-03-17T23:44:47.209 INFO:teuthology.orchestra.run.smithi157.stdout:(95/174): python3-requests-2.25.1-8.el9.noarch. 1.3 MB/s | 125 kB 00:00 2024-03-17T23:44:47.251 INFO:teuthology.orchestra.run.smithi157.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 234 kB/s | 21 kB 00:00 2024-03-17T23:44:47.276 INFO:teuthology.orchestra.run.smithi157.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 277 kB/s | 25 kB 00:00 2024-03-17T23:44:47.314 INFO:teuthology.orchestra.run.smithi150.stdout:(18/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 3.2 MB/s | 5.1 MB 00:01 2024-03-17T23:44:47.369 INFO:teuthology.orchestra.run.smithi157.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.1 MB/s | 245 kB 00:00 2024-03-17T23:44:47.394 INFO:teuthology.orchestra.run.smithi157.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.1 MB/s | 248 kB 00:00 2024-03-17T23:44:47.419 INFO:teuthology.orchestra.run.smithi157.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 4.2 MB/s | 216 kB 00:00 2024-03-17T23:44:47.444 INFO:teuthology.orchestra.run.smithi157.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 696 kB/s | 35 kB 00:00 2024-03-17T23:44:47.470 INFO:teuthology.orchestra.run.smithi157.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 893 kB/s | 44 kB 00:00 2024-03-17T23:44:47.495 INFO:teuthology.orchestra.run.smithi157.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-17T23:44:47.553 INFO:teuthology.orchestra.run.smithi157.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-03-17T23:44:47.612 INFO:teuthology.orchestra.run.smithi157.stdout:(105/174): openssl-devel-3.0.7-27.el9.x86_64.rp 10 MB/s | 4.1 MB 00:00 2024-03-17T23:44:47.731 INFO:teuthology.orchestra.run.smithi150.stdout:(19/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 15 MB/s | 48 MB 00:03 2024-03-17T23:44:47.757 INFO:teuthology.orchestra.run.smithi150.stdout:(20/174): python3-ceph-argparse-19.0.0-2162.g3d 102 kB/s | 45 kB 00:00 2024-03-17T23:44:47.782 INFO:teuthology.orchestra.run.smithi150.stdout:(21/174): python3-ceph-common-19.0.0-2162.g3d78 2.6 MB/s | 131 kB 00:00 2024-03-17T23:44:47.812 INFO:teuthology.orchestra.run.smithi157.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.7 MB/s | 863 kB 00:00 2024-03-17T23:44:47.849 INFO:teuthology.orchestra.run.smithi150.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 4.7 MB/s | 320 kB 00:00 2024-03-17T23:44:47.874 INFO:teuthology.orchestra.run.smithi150.stdout:(23/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 1.4 MB/s | 162 kB 00:00 2024-03-17T23:44:47.899 INFO:teuthology.orchestra.run.smithi150.stdout:(24/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 5.8 MB/s | 298 kB 00:00 2024-03-17T23:44:47.933 INFO:teuthology.orchestra.run.smithi150.stdout:(25/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.7 MB/s | 99 kB 00:00 2024-03-17T23:44:47.959 INFO:teuthology.orchestra.run.smithi150.stdout:(26/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.4 MB/s | 85 kB 00:00 2024-03-17T23:44:48.000 INFO:teuthology.orchestra.run.smithi150.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 4.0 MB/s | 170 kB 00:00 2024-03-17T23:44:48.051 INFO:teuthology.orchestra.run.smithi150.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 678 kB/s | 34 kB 00:00 2024-03-17T23:44:48.054 INFO:teuthology.orchestra.run.smithi157.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 80 kB/s | 19 kB 00:00 2024-03-17T23:44:48.101 INFO:teuthology.orchestra.run.smithi150.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 2.7 MB/s | 139 kB 00:00 2024-03-17T23:44:48.138 INFO:teuthology.orchestra.run.smithi157.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 938 kB/s | 548 kB 00:00 2024-03-17T23:44:48.172 INFO:teuthology.orchestra.run.smithi157.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 552 kB/s | 309 kB 00:00 2024-03-17T23:44:48.230 INFO:teuthology.orchestra.run.smithi157.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 283 kB/s | 26 kB 00:00 2024-03-17T23:44:48.272 INFO:teuthology.orchestra.run.smithi157.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 484 kB/s | 49 kB 00:00 2024-03-17T23:44:48.285 INFO:teuthology.orchestra.run.smithi150.stdout:(30/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 2.1 MB/s | 9.8 MB 00:04 2024-03-17T23:44:48.326 INFO:teuthology.orchestra.run.smithi150.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 8.0 MB/s | 1.8 MB 00:00 2024-03-17T23:44:48.331 INFO:teuthology.orchestra.run.smithi157.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 672 kB/s | 67 kB 00:00 2024-03-17T23:44:48.385 INFO:teuthology.orchestra.run.smithi150.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 4.2 MB/s | 248 kB 00:00 2024-03-17T23:44:48.414 INFO:teuthology.orchestra.run.smithi157.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-17T23:44:48.427 INFO:teuthology.orchestra.run.smithi150.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 1.2 MB/s | 49 kB 00:00 2024-03-17T23:44:48.469 INFO:teuthology.orchestra.run.smithi150.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 397 kB/s | 16 kB 00:00 2024-03-17T23:44:48.528 INFO:teuthology.orchestra.run.smithi150.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 4.4 MB/s | 264 kB 00:00 2024-03-17T23:44:48.531 INFO:teuthology.orchestra.run.smithi157.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.1 MB/s | 838 kB 00:00 2024-03-17T23:44:48.611 INFO:teuthology.orchestra.run.smithi150.stdout:(36/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 8.8 MB/s | 748 kB 00:00 2024-03-17T23:44:48.623 INFO:teuthology.orchestra.run.smithi157.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 475 kB/s | 43 kB 00:00 2024-03-17T23:44:48.648 INFO:teuthology.orchestra.run.smithi157.stdout:(116/174): python3-asyncssh-2.13.2-4.el9.noarch 2.3 MB/s | 548 kB 00:00 2024-03-17T23:44:48.653 INFO:teuthology.orchestra.run.smithi150.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 438 kB/s | 18 kB 00:00 2024-03-17T23:44:48.703 INFO:teuthology.orchestra.run.smithi150.stdout:(38/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 4.0 MB/s | 3.1 MB 00:00 2024-03-17T23:44:48.715 INFO:teuthology.orchestra.run.smithi157.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 351 kB/s | 32 kB 00:00 2024-03-17T23:44:48.757 INFO:teuthology.orchestra.run.smithi157.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-03-17T23:44:48.842 INFO:teuthology.orchestra.run.smithi157.stdout:(119/174): libarrow-9.0.0-10.el9.x86_64.rpm 5.6 MB/s | 4.4 MB 00:00 2024-03-17T23:44:48.863 INFO:teuthology.orchestra.run.smithi150.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.1 MB/s | 662 kB 00:00 2024-03-17T23:44:48.866 INFO:teuthology.orchestra.run.smithi157.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.1 MB/s | 172 kB 00:00 2024-03-17T23:44:48.892 INFO:teuthology.orchestra.run.smithi157.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 2.6 MB/s | 358 kB 00:00 2024-03-17T23:44:48.896 INFO:teuthology.orchestra.run.smithi150.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 5.4 MB/s | 920 kB 00:00 2024-03-17T23:44:48.921 INFO:teuthology.orchestra.run.smithi150.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-17T23:44:48.942 INFO:teuthology.orchestra.run.smithi157.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.9 MB/s | 303 kB 00:00 2024-03-17T23:44:48.946 INFO:teuthology.orchestra.run.smithi150.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-17T23:44:48.963 INFO:teuthology.orchestra.run.smithi150.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 765 kB/s | 32 kB 00:00 2024-03-17T23:44:48.967 INFO:teuthology.orchestra.run.smithi157.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 352 kB/s | 35 kB 00:00 2024-03-17T23:44:49.001 INFO:teuthology.orchestra.run.smithi157.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 1.8 MB/s | 203 kB 00:00 2024-03-17T23:44:49.039 INFO:teuthology.orchestra.run.smithi150.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 32 MB/s | 3.0 MB 00:00 2024-03-17T23:44:49.056 INFO:teuthology.orchestra.run.smithi150.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 188 kB/s | 17 kB 00:00 2024-03-17T23:44:49.072 INFO:teuthology.orchestra.run.smithi150.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 303 kB/s | 10 kB 00:00 2024-03-17T23:44:49.084 INFO:teuthology.orchestra.run.smithi157.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-03-17T23:44:49.097 INFO:teuthology.orchestra.run.smithi150.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-17T23:44:49.101 INFO:teuthology.orchestra.run.smithi157.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 106 kB/s | 11 kB 00:00 2024-03-17T23:44:49.114 INFO:teuthology.orchestra.run.smithi150.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-17T23:44:49.127 INFO:teuthology.orchestra.run.smithi157.stdout:(127/174): python3-grpcio-tools-1.46.7-10.el9.x 908 kB/s | 144 kB 00:00 2024-03-17T23:44:49.148 INFO:teuthology.orchestra.run.smithi150.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-17T23:44:49.173 INFO:teuthology.orchestra.run.smithi150.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 578 kB 00:00 2024-03-17T23:44:49.176 INFO:teuthology.orchestra.run.smithi157.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 192 kB/s | 18 kB 00:00 2024-03-17T23:44:49.190 INFO:teuthology.orchestra.run.smithi150.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-17T23:44:49.202 INFO:teuthology.orchestra.run.smithi157.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 231 kB/s | 23 kB 00:00 2024-03-17T23:44:49.216 INFO:teuthology.orchestra.run.smithi150.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-17T23:44:49.227 INFO:teuthology.orchestra.run.smithi157.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 194 kB/s | 19 kB 00:00 2024-03-17T23:44:49.241 INFO:teuthology.orchestra.run.smithi150.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-17T23:44:49.266 INFO:teuthology.orchestra.run.smithi150.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-17T23:44:49.269 INFO:teuthology.orchestra.run.smithi157.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 213 kB/s | 20 kB 00:00 2024-03-17T23:44:49.292 INFO:teuthology.orchestra.run.smithi150.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 690 kB/s | 34 kB 00:00 2024-03-17T23:44:49.320 INFO:teuthology.orchestra.run.smithi157.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 8.7 MB/s | 1.0 MB 00:00 2024-03-17T23:44:49.345 INFO:teuthology.orchestra.run.smithi157.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 395 kB/s | 46 kB 00:00 2024-03-17T23:44:49.378 INFO:teuthology.orchestra.run.smithi157.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 724 kB/s | 79 kB 00:00 2024-03-17T23:44:49.392 INFO:teuthology.orchestra.run.smithi150.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-17T23:44:49.434 INFO:teuthology.orchestra.run.smithi150.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 11 MB/s | 1.5 MB 00:00 2024-03-17T23:44:49.468 INFO:teuthology.orchestra.run.smithi150.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-17T23:44:49.487 INFO:teuthology.orchestra.run.smithi157.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 345 kB/s | 58 kB 00:00 2024-03-17T23:44:49.493 INFO:teuthology.orchestra.run.smithi150.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-17T23:44:49.512 INFO:teuthology.orchestra.run.smithi157.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 123 kB/s | 16 kB 00:00 2024-03-17T23:44:49.546 INFO:teuthology.orchestra.run.smithi157.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-03-17T23:44:49.588 INFO:teuthology.orchestra.run.smithi157.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 893 kB/s | 90 kB 00:00 2024-03-17T23:44:49.601 INFO:teuthology.orchestra.run.smithi150.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 43 MB/s | 5.7 MB 00:00 2024-03-17T23:44:49.613 INFO:teuthology.orchestra.run.smithi157.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 280 kB/s | 28 kB 00:00 2024-03-17T23:44:49.635 INFO:teuthology.orchestra.run.smithi150.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 7.3 MB/s | 1.0 MB 00:00 2024-03-17T23:44:49.638 INFO:teuthology.orchestra.run.smithi157.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 333 kB/s | 31 kB 00:00 2024-03-17T23:44:49.652 INFO:teuthology.orchestra.run.smithi150.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.6 MB/s | 78 kB 00:00 2024-03-17T23:44:49.689 INFO:teuthology.orchestra.run.smithi157.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-03-17T23:44:49.714 INFO:teuthology.orchestra.run.smithi157.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 588 kB/s | 59 kB 00:00 2024-03-17T23:44:49.739 INFO:teuthology.orchestra.run.smithi157.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 356 kB/s | 36 kB 00:00 2024-03-17T23:44:49.777 INFO:teuthology.orchestra.run.smithi150.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 41 MB/s | 5.8 MB 00:00 2024-03-17T23:44:49.781 INFO:teuthology.orchestra.run.smithi157.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 554 kB/s | 51 kB 00:00 2024-03-17T23:44:49.794 INFO:teuthology.orchestra.run.smithi150.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 174 kB/s | 24 kB 00:00 2024-03-17T23:44:49.811 INFO:teuthology.orchestra.run.smithi150.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 540 kB/s | 18 kB 00:00 2024-03-17T23:44:49.823 INFO:teuthology.orchestra.run.smithi157.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-17T23:44:49.836 INFO:teuthology.orchestra.run.smithi150.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-17T23:44:49.848 INFO:teuthology.orchestra.run.smithi157.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 824 kB/s | 90 kB 00:00 2024-03-17T23:44:49.861 INFO:teuthology.orchestra.run.smithi150.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 631 kB/s | 32 kB 00:00 2024-03-17T23:44:49.996 INFO:teuthology.orchestra.run.smithi150.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 4.8 MB 00:00 2024-03-17T23:44:50.021 INFO:teuthology.orchestra.run.smithi150.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 217 kB 00:00 2024-03-17T23:44:50.046 INFO:teuthology.orchestra.run.smithi150.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-17T23:44:50.049 INFO:teuthology.orchestra.run.smithi157.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 1.6 MB/s | 427 kB 00:00 2024-03-17T23:44:50.071 INFO:teuthology.orchestra.run.smithi150.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-17T23:44:50.097 INFO:teuthology.orchestra.run.smithi150.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 820 kB/s | 41 kB 00:00 2024-03-17T23:44:50.122 INFO:teuthology.orchestra.run.smithi150.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-17T23:44:50.141 INFO:teuthology.orchestra.run.smithi157.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 63 kB/s | 20 kB 00:00 2024-03-17T23:44:50.147 INFO:teuthology.orchestra.run.smithi150.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-17T23:44:50.166 INFO:teuthology.orchestra.run.smithi157.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 604 kB/s | 192 kB 00:00 2024-03-17T23:44:50.173 INFO:teuthology.orchestra.run.smithi150.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-17T23:44:50.190 INFO:teuthology.orchestra.run.smithi150.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-17T23:44:50.206 INFO:teuthology.orchestra.run.smithi150.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 1.2 MB/s | 43 kB 00:00 2024-03-17T23:44:50.208 INFO:teuthology.orchestra.run.smithi157.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-03-17T23:44:50.267 INFO:teuthology.orchestra.run.smithi157.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 5.4 MB/s | 321 kB 00:00 2024-03-17T23:44:50.332 INFO:teuthology.orchestra.run.smithi150.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 312 kB/s | 39 kB 00:00 2024-03-17T23:44:50.409 INFO:teuthology.orchestra.run.smithi157.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 16 MB/s | 2.3 MB 00:00 2024-03-17T23:44:50.542 INFO:teuthology.orchestra.run.smithi157.stdout:(153/174): NetworkManager-libnm-1.46.0-1.el9.x8 14 MB/s | 1.8 MB 00:00 2024-03-17T23:44:50.549 INFO:teuthology.orchestra.run.smithi150.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-03-17T23:44:50.574 INFO:teuthology.orchestra.run.smithi150.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 171 kB/s | 41 kB 00:00 2024-03-17T23:44:50.608 INFO:teuthology.orchestra.run.smithi150.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-17T23:44:50.617 INFO:teuthology.orchestra.run.smithi157.stdout:(154/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 7.0 MB/s | 3.1 MB 00:00 2024-03-17T23:44:50.641 INFO:teuthology.orchestra.run.smithi150.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 851 kB/s | 56 kB 00:00 2024-03-17T23:44:50.643 INFO:teuthology.orchestra.run.smithi157.stdout:(155/174): NetworkManager-team-1.46.0-1.el9.x86 424 kB/s | 42 kB 00:00 2024-03-17T23:44:50.667 INFO:teuthology.orchestra.run.smithi150.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2024-03-17T23:44:50.692 INFO:teuthology.orchestra.run.smithi150.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 876 kB/s | 44 kB 00:00 2024-03-17T23:44:50.701 INFO:teuthology.orchestra.run.smithi157.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-03-17T23:44:50.717 INFO:teuthology.orchestra.run.smithi150.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-17T23:44:50.727 INFO:teuthology.orchestra.run.smithi157.stdout:(157/174): NetworkManager-tui-1.46.0-1.el9.x86_ 2.2 MB/s | 248 kB 00:00 2024-03-17T23:44:50.742 INFO:teuthology.orchestra.run.smithi150.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-17T23:44:50.769 INFO:teuthology.orchestra.run.smithi157.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 544 kB/s | 36 kB 00:00 2024-03-17T23:44:50.776 INFO:teuthology.orchestra.run.smithi150.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-17T23:44:50.802 INFO:teuthology.orchestra.run.smithi150.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-17T23:44:50.811 INFO:teuthology.orchestra.run.smithi157.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.2 MB/s | 192 kB 00:00 2024-03-17T23:44:50.827 INFO:teuthology.orchestra.run.smithi150.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-17T23:44:50.878 INFO:teuthology.orchestra.run.smithi157.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.3 MB/s | 256 kB 00:00 2024-03-17T23:44:50.903 INFO:teuthology.orchestra.run.smithi157.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 251 kB/s | 23 kB 00:00 2024-03-17T23:44:50.910 INFO:teuthology.orchestra.run.smithi150.stdout:(90/174): ceph-mgr-diskprediction-local-19.0.0- 2.8 MB/s | 7.4 MB 00:02 2024-03-17T23:44:50.935 INFO:teuthology.orchestra.run.smithi150.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 1.4 MB/s | 187 kB 00:00 2024-03-17T23:44:50.961 INFO:teuthology.orchestra.run.smithi157.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 108 kB 00:00 2024-03-17T23:44:50.987 INFO:teuthology.orchestra.run.smithi157.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 155 kB 00:00 2024-03-17T23:44:51.053 INFO:teuthology.orchestra.run.smithi157.stdout:(164/174): librados2-19.0.0-2162.g3d78f5ba.el9. 3.7 MB/s | 3.4 MB 00:00 2024-03-17T23:44:51.060 INFO:teuthology.orchestra.run.smithi150.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 195 kB 00:00 2024-03-17T23:44:51.070 INFO:teuthology.orchestra.run.smithi157.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 767 kB/s | 64 kB 00:00 2024-03-17T23:44:51.086 INFO:teuthology.orchestra.run.smithi150.stdout:(93/174): jq-1.6-16.el9.x86_64.rpm 723 kB/s | 187 kB 00:00 2024-03-17T23:44:51.096 INFO:teuthology.orchestra.run.smithi157.stdout:(166/174): libmount-2.37.4-18.el9.x86_64.rpm 1.0 MB/s | 136 kB 00:00 2024-03-17T23:44:51.111 INFO:teuthology.orchestra.run.smithi150.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-03-17T23:44:51.130 INFO:teuthology.orchestra.run.smithi157.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 379 kB/s | 29 kB 00:00 2024-03-17T23:44:51.145 INFO:teuthology.orchestra.run.smithi150.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 803 kB 00:00 2024-03-17T23:44:51.221 INFO:teuthology.orchestra.run.smithi157.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 8.0 MB/s | 1.2 MB 00:00 2024-03-17T23:44:51.237 INFO:teuthology.orchestra.run.smithi150.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 143 kB/s | 21 kB 00:00 2024-03-17T23:44:51.262 INFO:teuthology.orchestra.run.smithi150.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 169 kB/s | 25 kB 00:00 2024-03-17T23:44:51.289 INFO:teuthology.orchestra.run.smithi157.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 11 MB/s | 2.2 MB 00:00 2024-03-17T23:44:51.339 INFO:teuthology.orchestra.run.smithi157.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 11 MB/s | 2.3 MB 00:00 2024-03-17T23:44:51.364 INFO:teuthology.orchestra.run.smithi157.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 415 kB/s | 31 kB 00:00 2024-03-17T23:44:51.389 INFO:teuthology.orchestra.run.smithi157.stdout:(172/174): util-linux-core-2.37.4-18.el9.x86_64 2.7 MB/s | 465 kB 00:00 2024-03-17T23:44:51.412 INFO:teuthology.orchestra.run.smithi150.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.6 MB/s | 248 kB 00:00 2024-03-17T23:44:51.446 INFO:teuthology.orchestra.run.smithi150.stdout:(99/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-03-17T23:44:51.456 INFO:teuthology.orchestra.run.smithi157.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 812 kB/s | 74 kB 00:00 2024-03-17T23:44:51.471 INFO:teuthology.orchestra.run.smithi150.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 3.6 MB/s | 216 kB 00:00 2024-03-17T23:44:51.489 INFO:teuthology.orchestra.run.smithi157.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 7.1 MB/s | 1.1 MB 00:00 2024-03-17T23:44:51.493 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2024-03-17T23:44:51.493 INFO:teuthology.orchestra.run.smithi157.stdout:Total 16 MB/s | 224 MB 00:14 2024-03-17T23:44:51.505 INFO:teuthology.orchestra.run.smithi150.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 597 kB/s | 35 kB 00:00 2024-03-17T23:44:51.530 INFO:teuthology.orchestra.run.smithi150.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 758 kB/s | 44 kB 00:00 2024-03-17T23:44:51.564 INFO:teuthology.orchestra.run.smithi150.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-03-17T23:44:51.706 INFO:teuthology.orchestra.run.smithi150.stdout:(104/174): openssl-devel-3.0.7-27.el9.x86_64.rp 7.4 MB/s | 4.1 MB 00:00 2024-03-17T23:44:51.947 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2024-03-17T23:44:51.956 INFO:teuthology.orchestra.run.smithi150.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 2.1 MB/s | 548 kB 00:00 2024-03-17T23:44:52.023 INFO:teuthology.orchestra.run.smithi150.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 4.5 MB/s | 309 kB 00:00 2024-03-17T23:44:52.065 INFO:teuthology.orchestra.run.smithi150.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 466 kB/s | 19 kB 00:00 2024-03-17T23:44:52.132 INFO:teuthology.orchestra.run.smithi150.stdout:(108/174): lua-devel-5.4.4-4.el9.x86_64.rpm 37 kB/s | 22 kB 00:00 2024-03-17T23:44:52.249 INFO:teuthology.orchestra.run.smithi150.stdout:(109/174): libarrow-9.0.0-10.el9.x86_64.rpm 24 MB/s | 4.4 MB 00:00 2024-03-17T23:44:52.257 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2024-03-17T23:44:52.257 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2024-03-17T23:44:52.291 INFO:teuthology.orchestra.run.smithi150.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 1.2 MB/s | 49 kB 00:00 2024-03-17T23:44:52.316 INFO:teuthology.orchestra.run.smithi150.stdout:(111/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 141 kB/s | 26 kB 00:00 2024-03-17T23:44:52.341 INFO:teuthology.orchestra.run.smithi150.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-17T23:44:52.400 INFO:teuthology.orchestra.run.smithi150.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 14 MB/s | 838 kB 00:00 2024-03-17T23:44:52.425 INFO:teuthology.orchestra.run.smithi150.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-03-17T23:44:52.459 INFO:teuthology.orchestra.run.smithi150.stdout:(115/174): python3-asyncssh-2.13.2-4.el9.noarch 9.1 MB/s | 548 kB 00:00 2024-03-17T23:44:52.476 INFO:teuthology.orchestra.run.smithi150.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 866 kB/s | 43 kB 00:00 2024-03-17T23:44:52.493 INFO:teuthology.orchestra.run.smithi150.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 961 kB/s | 32 kB 00:00 2024-03-17T23:44:52.509 INFO:teuthology.orchestra.run.smithi150.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 446 kB/s | 15 kB 00:00 2024-03-17T23:44:52.535 INFO:teuthology.orchestra.run.smithi150.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 4.0 MB/s | 172 kB 00:00 2024-03-17T23:44:52.585 INFO:teuthology.orchestra.run.smithi150.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-03-17T23:44:52.619 INFO:teuthology.orchestra.run.smithi150.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.5 MB/s | 303 kB 00:00 2024-03-17T23:44:52.636 INFO:teuthology.orchestra.run.smithi150.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 702 kB/s | 35 kB 00:00 2024-03-17T23:44:52.661 INFO:teuthology.orchestra.run.smithi150.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 4.7 MB/s | 203 kB 00:00 2024-03-17T23:44:52.711 INFO:teuthology.orchestra.run.smithi150.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-03-17T23:44:52.762 INFO:teuthology.orchestra.run.smithi150.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 16 MB/s | 2.0 MB 00:00 2024-03-17T23:44:52.787 INFO:teuthology.orchestra.run.smithi150.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-03-17T23:44:52.812 INFO:teuthology.orchestra.run.smithi150.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 353 kB/s | 18 kB 00:00 2024-03-17T23:44:52.838 INFO:teuthology.orchestra.run.smithi150.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 460 kB/s | 23 kB 00:00 2024-03-17T23:44:52.863 INFO:teuthology.orchestra.run.smithi150.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 385 kB/s | 19 kB 00:00 2024-03-17T23:44:52.888 INFO:teuthology.orchestra.run.smithi150.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-03-17T23:44:52.930 INFO:teuthology.orchestra.run.smithi150.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 15 MB/s | 1.0 MB 00:00 2024-03-17T23:44:52.955 INFO:teuthology.orchestra.run.smithi150.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 693 kB/s | 46 kB 00:00 2024-03-17T23:44:52.981 INFO:teuthology.orchestra.run.smithi150.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-03-17T23:44:53.006 INFO:teuthology.orchestra.run.smithi150.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-03-17T23:44:53.031 INFO:teuthology.orchestra.run.smithi150.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-17T23:44:53.057 INFO:teuthology.orchestra.run.smithi150.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 328 kB/s | 16 kB 00:00 2024-03-17T23:44:53.082 INFO:teuthology.orchestra.run.smithi150.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-03-17T23:44:53.115 INFO:teuthology.orchestra.run.smithi150.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 483 kB/s | 28 kB 00:00 2024-03-17T23:44:53.141 INFO:teuthology.orchestra.run.smithi150.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 524 kB/s | 31 kB 00:00 2024-03-17T23:44:53.166 INFO:teuthology.orchestra.run.smithi150.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-03-17T23:44:53.191 INFO:teuthology.orchestra.run.smithi150.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-03-17T23:44:53.217 INFO:teuthology.orchestra.run.smithi150.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 713 kB/s | 36 kB 00:00 2024-03-17T23:44:53.242 INFO:teuthology.orchestra.run.smithi150.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-03-17T23:44:53.267 INFO:teuthology.orchestra.run.smithi150.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-17T23:44:53.309 INFO:teuthology.orchestra.run.smithi150.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-03-17T23:44:53.343 INFO:teuthology.orchestra.run.smithi150.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.6 MB/s | 427 kB 00:00 2024-03-17T23:44:53.368 INFO:teuthology.orchestra.run.smithi150.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 341 kB/s | 20 kB 00:00 2024-03-17T23:44:53.410 INFO:teuthology.orchestra.run.smithi150.stdout:(148/174): protobuf-compiler-3.14.0-13.el9.x86_ 468 kB/s | 863 kB 00:01 2024-03-17T23:44:53.435 INFO:teuthology.orchestra.run.smithi150.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-03-17T23:44:53.485 INFO:teuthology.orchestra.run.smithi150.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-03-17T23:44:53.561 INFO:teuthology.orchestra.run.smithi150.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.4 MB/s | 321 kB 00:00 2024-03-17T23:44:53.744 INFO:teuthology.orchestra.run.smithi150.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 13 MB/s | 2.3 MB 00:00 2024-03-17T23:44:53.820 INFO:teuthology.orchestra.run.smithi150.stdout:(153/174): NetworkManager-libnm-1.46.0-1.el9.x8 24 MB/s | 1.8 MB 00:00 2024-03-17T23:44:53.862 INFO:teuthology.orchestra.run.smithi150.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 1.0 MB/s | 42 kB 00:00 2024-03-17T23:44:53.920 INFO:teuthology.orchestra.run.smithi150.stdout:(155/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 6.5 MB/s | 3.1 MB 00:00 2024-03-17T23:44:53.945 INFO:teuthology.orchestra.run.smithi150.stdout:(156/174): NetworkManager-tui-1.46.0-1.el9.x86_ 2.9 MB/s | 248 kB 00:00 2024-03-17T23:44:53.979 INFO:teuthology.orchestra.run.smithi150.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-03-17T23:44:53.996 INFO:teuthology.orchestra.run.smithi150.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 722 kB/s | 36 kB 00:00 2024-03-17T23:44:54.038 INFO:teuthology.orchestra.run.smithi150.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.2 MB/s | 192 kB 00:00 2024-03-17T23:44:54.063 INFO:teuthology.orchestra.run.smithi150.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.7 MB/s | 256 kB 00:00 2024-03-17T23:44:54.089 INFO:teuthology.orchestra.run.smithi150.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 460 kB/s | 23 kB 00:00 2024-03-17T23:44:54.106 INFO:teuthology.orchestra.run.smithi150.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.5 MB/s | 108 kB 00:00 2024-03-17T23:44:54.131 INFO:teuthology.orchestra.run.smithi150.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.6 MB/s | 155 kB 00:00 2024-03-17T23:44:54.156 INFO:teuthology.orchestra.run.smithi150.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 136 kB 00:00 2024-03-17T23:44:54.182 INFO:teuthology.orchestra.run.smithi150.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-03-17T23:44:54.207 INFO:teuthology.orchestra.run.smithi150.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 574 kB/s | 29 kB 00:00 2024-03-17T23:44:54.295 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2024-03-17T23:44:54.296 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2024-03-17T23:44:54.316 INFO:teuthology.orchestra.run.smithi150.stdout:(167/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 20 MB/s | 2.2 MB 00:00 2024-03-17T23:44:54.349 INFO:teuthology.orchestra.run.smithi150.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 7.2 MB/s | 1.2 MB 00:00 2024-03-17T23:44:54.416 INFO:teuthology.orchestra.run.smithi150.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 23 MB/s | 2.3 MB 00:00 2024-03-17T23:44:54.441 INFO:teuthology.orchestra.run.smithi150.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 5.0 MB/s | 465 kB 00:00 2024-03-17T23:44:54.467 INFO:teuthology.orchestra.run.smithi150.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 622 kB/s | 31 kB 00:00 2024-03-17T23:44:54.525 INFO:teuthology.orchestra.run.smithi150.stdout:(172/174): librados2-19.0.0-2162.g3d78f5ba.el9. 3.0 MB/s | 3.4 MB 00:01 2024-03-17T23:44:54.592 INFO:teuthology.orchestra.run.smithi150.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 7.1 MB/s | 1.1 MB 00:00 2024-03-17T23:44:54.617 INFO:teuthology.orchestra.run.smithi150.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 495 kB/s | 74 kB 00:00 2024-03-17T23:44:54.624 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-03-17T23:44:54.624 INFO:teuthology.orchestra.run.smithi150.stdout:Total 15 MB/s | 224 MB 00:15 2024-03-17T23:44:55.113 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-03-17T23:44:55.425 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-03-17T23:44:55.425 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-03-17T23:44:57.431 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-03-17T23:44:57.431 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-03-17T23:45:01.216 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2024-03-17T23:45:01.294 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-17T23:45:01.329 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-17T23:45:01.358 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-17T23:45:01.367 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-17T23:45:01.408 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-17T23:45:01.470 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-17T23:45:01.510 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-17T23:45:01.536 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-17T23:45:01.579 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-17T23:45:01.907 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-17T23:45:01.970 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-17T23:45:02.092 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-17T23:45:02.130 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-17T23:45:02.220 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-17T23:45:02.257 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-17T23:45:02.313 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-17T23:45:02.495 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-17T23:45:02.548 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-17T23:45:02.585 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-17T23:45:02.688 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-17T23:45:02.710 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-17T23:45:02.760 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-17T23:45:02.798 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-17T23:45:02.820 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-17T23:45:02.836 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-17T23:45:03.018 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-17T23:45:03.052 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-17T23:45:03.094 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-17T23:45:03.287 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-17T23:45:03.436 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-17T23:45:03.506 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-17T23:45:03.609 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-17T23:45:03.666 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-17T23:45:03.894 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-17T23:45:04.010 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-17T23:45:04.103 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-17T23:45:04.131 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-17T23:45:04.158 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-17T23:45:04.194 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-17T23:45:04.267 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-17T23:45:04.321 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-17T23:45:04.368 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-17T23:45:04.440 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-17T23:45:04.475 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-17T23:45:04.515 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-03-17T23:45:04.518 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-17T23:45:04.551 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-17T23:45:04.587 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-17T23:45:04.622 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-17T23:45:04.656 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-17T23:45:04.657 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-17T23:45:04.692 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-17T23:45:04.701 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-17T23:45:04.712 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-17T23:45:04.737 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-17T23:45:04.764 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-17T23:45:04.765 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-17T23:45:04.823 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-17T23:45:04.834 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-17T23:45:04.855 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-17T23:45:04.879 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-17T23:45:04.880 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-17T23:45:04.944 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-17T23:45:04.952 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-17T23:45:05.087 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-17T23:45:05.139 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-17T23:45:05.270 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-17T23:45:05.332 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-17T23:45:05.453 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-17T23:45:05.491 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-17T23:45:05.566 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-17T23:45:05.598 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-17T23:45:05.646 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-17T23:45:05.664 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-17T23:45:05.743 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-17T23:45:05.825 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-17T23:45:05.826 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-17T23:45:05.871 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-17T23:45:05.876 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-17T23:45:05.879 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-17T23:45:05.904 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-17T23:45:05.921 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-17T23:45:05.968 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-17T23:45:06.006 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-17T23:45:06.008 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-17T23:45:06.030 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-17T23:45:06.036 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-17T23:45:06.096 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-17T23:45:06.134 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-17T23:45:06.157 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-17T23:45:06.173 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-17T23:45:06.297 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-17T23:45:06.374 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-17T23:45:06.407 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-17T23:45:06.439 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-17T23:45:06.492 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-17T23:45:06.571 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-17T23:45:06.615 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-17T23:45:06.618 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-17T23:45:06.747 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-17T23:45:06.766 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-17T23:45:06.826 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-17T23:45:06.866 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-17T23:45:06.944 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-17T23:45:06.986 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-17T23:45:07.145 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-17T23:45:07.215 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-17T23:45:07.217 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-17T23:45:07.253 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-17T23:45:07.289 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-17T23:45:07.329 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-17T23:45:07.336 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-17T23:45:07.372 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-17T23:45:07.415 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-17T23:45:07.421 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-17T23:45:07.458 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-17T23:45:07.465 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-17T23:45:07.484 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-17T23:45:07.529 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-17T23:45:07.600 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-17T23:45:07.620 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-17T23:45:07.654 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-17T23:45:07.673 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-17T23:45:07.683 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-17T23:45:07.711 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-17T23:45:07.725 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-17T23:45:07.776 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-17T23:45:07.810 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-17T23:45:07.818 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-17T23:45:07.858 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-17T23:45:07.858 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-17T23:45:07.895 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-17T23:45:07.907 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-17T23:45:07.931 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-17T23:45:07.933 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-17T23:45:07.957 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-17T23:45:08.001 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-17T23:45:08.035 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-17T23:45:08.036 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-17T23:45:08.083 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-17T23:45:08.116 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-17T23:45:08.158 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-17T23:45:08.185 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-17T23:45:08.230 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-17T23:45:08.256 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-17T23:45:08.299 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-17T23:45:08.305 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-17T23:45:08.364 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-17T23:45:08.420 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-17T23:45:08.440 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-17T23:45:08.492 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-17T23:45:08.601 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-17T23:45:08.645 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-17T23:45:08.774 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-17T23:45:08.818 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-17T23:45:08.850 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-17T23:45:08.880 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-17T23:45:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-17T23:45:08.943 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-17T23:45:08.982 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-17T23:45:09.014 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-17T23:45:09.032 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-17T23:45:09.044 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-17T23:45:09.113 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-17T23:45:09.120 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-17T23:45:09.163 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-17T23:45:09.191 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-17T23:45:09.196 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-17T23:45:09.237 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-17T23:45:09.240 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-17T23:45:09.250 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-17T23:45:09.272 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-17T23:45:09.282 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-17T23:45:09.329 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-17T23:45:09.361 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-17T23:45:09.367 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-17T23:45:09.399 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-17T23:45:09.443 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-17T23:45:09.486 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-17T23:45:09.518 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-17T23:45:09.556 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-17T23:45:09.648 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-17T23:45:09.653 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-17T23:45:09.703 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-17T23:45:09.836 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-17T23:45:09.918 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-17T23:45:09.962 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-17T23:45:10.312 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-17T23:45:10.399 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-17T23:45:10.433 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-17T23:45:10.457 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-17T23:45:10.483 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-17T23:45:10.525 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-17T23:45:10.529 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-17T23:45:10.565 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-17T23:45:10.573 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-17T23:45:10.602 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-17T23:45:10.650 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-17T23:45:10.652 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-17T23:45:10.684 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-17T23:45:10.701 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-17T23:45:10.728 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-17T23:45:10.770 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-17T23:45:10.877 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-17T23:45:10.904 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-17T23:45:10.907 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-17T23:45:10.950 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-17T23:45:10.980 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-17T23:45:10.984 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-17T23:45:11.021 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-17T23:45:11.117 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-17T23:45:11.163 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-17T23:45:11.203 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-17T23:45:11.229 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-17T23:45:11.254 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-17T23:45:11.340 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-17T23:45:11.464 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-17T23:45:11.570 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-17T23:45:11.620 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-17T23:45:11.627 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-17T23:45:11.669 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-17T23:45:11.725 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-17T23:45:11.736 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-17T23:45:11.911 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-17T23:45:11.958 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-17T23:45:12.097 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-17T23:45:12.139 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-17T23:45:12.171 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-17T23:45:12.202 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-17T23:45:12.230 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-17T23:45:12.264 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-17T23:45:12.303 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-17T23:45:12.335 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-17T23:45:12.365 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-17T23:45:12.442 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-17T23:45:12.484 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-17T23:45:12.517 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-17T23:45:12.558 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-17T23:45:12.593 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-17T23:45:12.683 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-17T23:45:12.773 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-17T23:45:12.816 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-17T23:45:12.847 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-17T23:45:12.895 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-17T23:45:12.951 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-17T23:45:12.992 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-17T23:45:13.040 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-17T23:45:13.041 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-17T23:45:13.072 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-17T23:45:13.103 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-17T23:45:13.130 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-17T23:45:13.441 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-17T23:45:13.468 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-17T23:45:13.559 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-17T23:45:13.590 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-17T23:45:13.626 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-17T23:45:13.777 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-17T23:45:13.829 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-17T23:45:13.863 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-17T23:45:13.911 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-17T23:45:14.008 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-17T23:45:14.039 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-17T23:45:14.106 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-17T23:45:14.219 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-17T23:45:14.330 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-17T23:45:14.428 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-17T23:45:14.510 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-17T23:45:14.540 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-17T23:45:14.588 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-17T23:45:15.312 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-17T23:45:15.435 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-17T23:45:16.677 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-17T23:45:16.745 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-17T23:45:16.753 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-17T23:45:16.786 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-17T23:45:16.809 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-17T23:45:16.841 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-17T23:45:16.868 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-17T23:45:17.183 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-17T23:45:17.206 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-17T23:45:17.281 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-17T23:45:17.311 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-17T23:45:17.330 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-17T23:45:17.347 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-17T23:45:17.373 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-17T23:45:17.458 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-17T23:45:17.458 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-17T23:45:17.458 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-17T23:45:17.458 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:17.499 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-17T23:45:17.817 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-17T23:45:17.843 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-17T23:45:17.906 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-17T23:45:17.936 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-17T23:45:20.148 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-17T23:45:20.208 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-17T23:45:20.744 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-17T23:45:20.786 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-17T23:45:20.860 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-17T23:45:20.860 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-17T23:45:20.860 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-17T23:45:20.860 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:20.904 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-17T23:45:29.114 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-17T23:45:29.114 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /sys 2024-03-17T23:45:29.114 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /proc 2024-03-17T23:45:29.114 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /mnt 2024-03-17T23:45:29.114 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /var/tmp 2024-03-17T23:45:29.114 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /home 2024-03-17T23:45:29.114 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /root 2024-03-17T23:45:29.114 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /tmp 2024-03-17T23:45:29.114 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:29.278 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-17T23:45:29.294 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-17T23:45:29.294 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:45:29.294 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-17T23:45:29.294 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:30.765 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-17T23:45:30.790 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-17T23:45:30.791 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:45:30.791 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-17T23:45:30.791 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-17T23:45:30.791 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-17T23:45:30.791 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:31.015 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-17T23:45:31.037 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-17T23:45:31.038 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:45:31.038 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-17T23:45:31.038 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-17T23:45:31.038 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-17T23:45:31.038 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:31.491 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-17T23:45:31.513 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-17T23:45:31.513 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:45:31.514 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-17T23:45:31.514 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-17T23:45:31.514 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-17T23:45:31.514 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:31.617 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-17T23:45:32.044 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-17T23:45:32.083 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-17T23:45:32.115 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-17T23:45:32.176 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-17T23:45:32.206 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-17T23:45:32.440 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-17T23:45:32.473 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-17T23:45:32.581 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-17T23:45:32.581 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /sys 2024-03-17T23:45:32.581 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /proc 2024-03-17T23:45:32.581 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /mnt 2024-03-17T23:45:32.581 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /var/tmp 2024-03-17T23:45:32.581 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /home 2024-03-17T23:45:32.582 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /root 2024-03-17T23:45:32.582 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /tmp 2024-03-17T23:45:32.582 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:32.750 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-17T23:45:32.766 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-17T23:45:32.766 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:45:32.766 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-17T23:45:32.766 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:33.441 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-17T23:45:33.474 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-17T23:45:33.574 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-17T23:45:33.745 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-17T23:45:33.776 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-17T23:45:33.798 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-17T23:45:33.798 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:45:33.799 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-17T23:45:33.799 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-17T23:45:33.799 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-17T23:45:33.799 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:33.843 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-17T23:45:33.854 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-17T23:45:33.910 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-17T23:45:34.261 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-17T23:45:34.287 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-17T23:45:34.287 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:45:34.287 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-17T23:45:34.287 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-17T23:45:34.287 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-17T23:45:34.288 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:34.512 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-17T23:45:34.535 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-17T23:45:34.535 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:45:34.535 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-17T23:45:34.535 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-17T23:45:34.535 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-17T23:45:34.535 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:34.788 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-17T23:45:34.828 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-17T23:45:34.851 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-17T23:45:34.851 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:45:34.851 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-17T23:45:34.852 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-17T23:45:34.852 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-17T23:45:34.852 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:34.896 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-17T23:45:34.917 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-17T23:45:34.917 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:45:34.917 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-17T23:45:34.917 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:34.972 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-17T23:45:34.987 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-17T23:45:34.994 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-17T23:45:34.994 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:45:34.994 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-17T23:45:34.995 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-17T23:45:34.995 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-17T23:45:34.995 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:35.010 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-17T23:45:35.010 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:45:35.010 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-17T23:45:35.010 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-17T23:45:35.010 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-17T23:45:35.010 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:35.106 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-17T23:45:35.306 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-17T23:45:35.328 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-17T23:45:35.328 INFO:teuthology.orchestra.run.smithi157.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:45:35.328 INFO:teuthology.orchestra.run.smithi157.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-17T23:45:35.328 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-17T23:45:35.329 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-17T23:45:35.329 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:35.457 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-17T23:45:35.488 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-17T23:45:35.520 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-17T23:45:35.580 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-17T23:45:35.602 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-17T23:45:35.842 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-17T23:45:35.870 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-17T23:45:36.839 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-17T23:45:36.870 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-17T23:45:36.970 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-17T23:45:37.143 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-17T23:45:37.172 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-17T23:45:37.197 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-17T23:45:37.197 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:45:37.197 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-17T23:45:37.197 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-17T23:45:37.197 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-17T23:45:37.197 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:37.266 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-17T23:45:37.277 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-17T23:45:37.329 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-17T23:45:38.198 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-17T23:45:38.234 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-17T23:45:38.257 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-17T23:45:38.257 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:45:38.257 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-17T23:45:38.257 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-17T23:45:38.257 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-17T23:45:38.257 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:38.301 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-17T23:45:38.324 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-17T23:45:38.324 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:45:38.324 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-17T23:45:38.324 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:38.368 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-17T23:45:38.390 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-17T23:45:38.390 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:45:38.391 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-17T23:45:38.391 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-17T23:45:38.391 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-17T23:45:38.391 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:38.694 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-17T23:45:38.716 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-17T23:45:38.716 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-17T23:45:38.717 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-17T23:45:38.717 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-17T23:45:38.717 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-17T23:45:38.717 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:39.776 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-17T23:45:39.835 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-17T23:45:39.864 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-17T23:45:39.892 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-17T23:45:40.077 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-17T23:45:40.199 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-17T23:45:40.240 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-17T23:45:40.314 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-17T23:45:40.373 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-17T23:45:40.408 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-17T23:45:40.441 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-17T23:45:40.481 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-17T23:45:40.511 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-17T23:45:40.575 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-17T23:45:40.647 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-17T23:45:40.684 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-17T23:45:41.273 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-17T23:45:41.315 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-17T23:45:41.335 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-17T23:45:42.171 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-17T23:45:42.245 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-17T23:45:42.287 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-17T23:45:42.386 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-17T23:45:42.415 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-17T23:45:42.440 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-17T23:45:42.466 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-17T23:45:42.492 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-17T23:45:42.493 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-17T23:45:42.580 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-17T23:45:42.610 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-17T23:45:42.610 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-17T23:45:42.703 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-17T23:45:42.729 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-17T23:45:42.762 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-17T23:45:42.788 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-17T23:45:42.812 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-17T23:45:42.856 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-17T23:45:42.902 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-17T23:45:42.934 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-17T23:45:42.959 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-17T23:45:42.984 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-17T23:45:43.010 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-17T23:45:43.035 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-17T23:45:43.041 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-17T23:45:43.042 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-17T23:45:43.165 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-17T23:45:43.214 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-17T23:45:43.229 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-17T23:45:43.229 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-17T23:45:43.245 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-17T23:45:43.264 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-17T23:45:43.320 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-17T23:45:43.345 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-17T23:45:43.370 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-17T23:45:43.404 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-17T23:45:43.430 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-17T23:45:43.448 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-17T23:45:43.495 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-17T23:45:43.564 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-17T23:45:43.604 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-17T23:45:43.653 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-17T23:45:43.712 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-17T23:45:43.747 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-17T23:45:43.780 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-17T23:45:43.820 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-17T23:45:43.858 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-17T23:45:43.919 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-17T23:45:44.018 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-17T23:45:44.048 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-17T23:45:44.640 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-17T23:45:44.679 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-17T23:45:44.699 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-17T23:45:45.519 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-17T23:45:45.616 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-17T23:45:45.657 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-17T23:45:45.740 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-17T23:45:45.778 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-17T23:45:45.803 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-17T23:45:45.828 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-17T23:45:45.855 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-17T23:45:45.855 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-17T23:45:45.943 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-17T23:45:45.972 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-17T23:45:45.972 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-17T23:45:46.057 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-17T23:45:46.083 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-17T23:45:46.125 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-17T23:45:46.150 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-17T23:45:46.175 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-17T23:45:46.210 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-17T23:45:46.256 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-17T23:45:46.296 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-17T23:45:46.322 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-17T23:45:46.347 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-17T23:45:46.372 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-17T23:45:46.398 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-17T23:45:46.405 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-17T23:45:46.405 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-17T23:45:46.594 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-17T23:45:46.594 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-17T23:45:46.677 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-17T23:45:46.702 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-17T23:45:46.744 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-17T23:45:46.777 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-17T23:45:46.811 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-17T23:45:46.877 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-17T23:45:48.577 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-17T23:45:48.578 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-17T23:45:48.578 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-17T23:45:48.578 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-17T23:45:48.578 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-17T23:45:48.578 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-17T23:45:48.578 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-17T23:45:48.578 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-17T23:45:48.579 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-17T23:45:48.579 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-17T23:45:48.579 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-17T23:45:48.579 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-17T23:45:48.579 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-17T23:45:48.579 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-17T23:45:48.579 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-17T23:45:48.579 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-17T23:45:48.579 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-17T23:45:48.579 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-17T23:45:48.579 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-17T23:45:48.579 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-17T23:45:48.579 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-17T23:45:48.580 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-17T23:45:48.580 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-17T23:45:48.580 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-17T23:45:48.580 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-17T23:45:48.580 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-17T23:45:48.580 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-17T23:45:48.580 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-17T23:45:48.580 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-17T23:45:48.580 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-17T23:45:48.580 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-17T23:45:48.580 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-17T23:45:48.580 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-17T23:45:48.580 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-17T23:45:48.581 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-17T23:45:48.581 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-17T23:45:48.581 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-17T23:45:48.581 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-17T23:45:48.581 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-17T23:45:48.581 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-17T23:45:48.581 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-17T23:45:48.581 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-17T23:45:48.581 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-17T23:45:48.581 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-17T23:45:48.581 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-17T23:45:48.581 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-17T23:45:48.582 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-17T23:45:48.582 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-17T23:45:48.582 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-17T23:45:48.582 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-17T23:45:48.582 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-17T23:45:48.582 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-17T23:45:48.582 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-17T23:45:48.582 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-17T23:45:48.582 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-17T23:45:48.582 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-17T23:45:48.582 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-17T23:45:48.582 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-17T23:45:48.583 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-17T23:45:48.583 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-17T23:45:48.583 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-17T23:45:48.583 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-17T23:45:48.583 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-17T23:45:48.583 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-17T23:45:48.583 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-17T23:45:48.583 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-17T23:45:48.583 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-17T23:45:48.583 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-17T23:45:48.583 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-17T23:45:48.583 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-17T23:45:48.584 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-17T23:45:48.584 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-17T23:45:48.584 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-17T23:45:48.584 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-17T23:45:48.584 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-17T23:45:48.584 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-17T23:45:48.584 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-17T23:45:48.584 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-17T23:45:48.584 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-17T23:45:48.584 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-17T23:45:48.584 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-17T23:45:48.584 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-17T23:45:48.584 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-17T23:45:48.585 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-17T23:45:48.585 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-17T23:45:48.585 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-17T23:45:48.585 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-17T23:45:48.585 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-17T23:45:48.585 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-17T23:45:48.585 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-17T23:45:48.585 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-17T23:45:48.585 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-17T23:45:48.585 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-17T23:45:48.585 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-17T23:45:48.585 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-17T23:45:48.585 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-17T23:45:48.585 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-17T23:45:48.586 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-17T23:45:48.586 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-17T23:45:48.586 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-17T23:45:48.586 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-17T23:45:48.586 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-17T23:45:48.586 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-17T23:45:48.586 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-17T23:45:48.586 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-17T23:45:48.586 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-17T23:45:48.586 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-17T23:45:48.586 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-17T23:45:48.586 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-17T23:45:48.586 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-17T23:45:48.586 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-17T23:45:48.587 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-17T23:45:48.587 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-17T23:45:48.587 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-17T23:45:48.587 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-17T23:45:48.587 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-17T23:45:48.587 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-17T23:45:48.587 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-17T23:45:48.587 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-17T23:45:48.587 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-17T23:45:48.587 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-17T23:45:48.587 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-17T23:45:48.587 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-17T23:45:48.587 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-17T23:45:48.587 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-17T23:45:48.588 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-17T23:45:48.588 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-17T23:45:48.588 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-17T23:45:48.588 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-17T23:45:48.588 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-17T23:45:48.588 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-17T23:45:48.588 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-17T23:45:48.588 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-17T23:45:48.588 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-17T23:45:48.588 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-17T23:45:48.588 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-17T23:45:48.588 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-17T23:45:48.588 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-17T23:45:48.588 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-17T23:45:48.589 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-17T23:45:48.589 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-17T23:45:48.589 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-17T23:45:48.589 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-17T23:45:48.589 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-17T23:45:48.589 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-17T23:45:48.589 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-17T23:45:48.589 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-17T23:45:48.589 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-17T23:45:48.589 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-17T23:45:48.589 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-17T23:45:48.589 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-17T23:45:48.589 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-17T23:45:48.590 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-17T23:45:48.590 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-17T23:45:48.590 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-17T23:45:48.590 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-17T23:45:48.590 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-17T23:45:48.590 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-17T23:45:48.590 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-17T23:45:48.590 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-17T23:45:48.590 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-17T23:45:48.590 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-17T23:45:48.590 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-17T23:45:48.590 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-17T23:45:48.590 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-17T23:45:48.590 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-17T23:45:48.591 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-17T23:45:48.591 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-17T23:45:48.591 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-17T23:45:48.591 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-17T23:45:48.591 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-17T23:45:48.591 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-17T23:45:48.591 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-17T23:45:48.591 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-17T23:45:48.591 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-17T23:45:48.591 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-17T23:45:48.591 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-17T23:45:48.591 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-17T23:45:48.591 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-17T23:45:48.591 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-17T23:45:48.591 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-17T23:45:48.592 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-17T23:45:48.592 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-17T23:45:48.592 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-17T23:45:48.592 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-17T23:45:48.592 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-17T23:45:48.592 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-17T23:45:48.592 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-17T23:45:48.592 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-17T23:45:48.592 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-17T23:45:48.592 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-17T23:45:48.592 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-17T23:45:48.592 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-17T23:45:48.592 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-17T23:45:48.592 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-17T23:45:48.592 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-17T23:45:48.592 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-17T23:45:48.593 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-17T23:45:51.696 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-17T23:45:51.696 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:51.696 INFO:teuthology.orchestra.run.smithi157.stdout:Upgraded: 2024-03-17T23:45:51.696 INFO:teuthology.orchestra.run.smithi157.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-17T23:45:51.697 INFO:teuthology.orchestra.run.smithi157.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-17T23:45:51.697 INFO:teuthology.orchestra.run.smithi157.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-17T23:45:51.697 INFO:teuthology.orchestra.run.smithi157.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-17T23:45:51.697 INFO:teuthology.orchestra.run.smithi157.stdout: acl-2.3.1-4.el9.x86_64 2024-03-17T23:45:51.697 INFO:teuthology.orchestra.run.smithi157.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-17T23:45:51.698 INFO:teuthology.orchestra.run.smithi157.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-17T23:45:51.698 INFO:teuthology.orchestra.run.smithi157.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-17T23:45:51.698 INFO:teuthology.orchestra.run.smithi157.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-17T23:45:51.698 INFO:teuthology.orchestra.run.smithi157.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-17T23:45:51.698 INFO:teuthology.orchestra.run.smithi157.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-17T23:45:51.698 INFO:teuthology.orchestra.run.smithi157.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-17T23:45:51.699 INFO:teuthology.orchestra.run.smithi157.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-17T23:45:51.699 INFO:teuthology.orchestra.run.smithi157.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.699 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.699 INFO:teuthology.orchestra.run.smithi157.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-17T23:45:51.699 INFO:teuthology.orchestra.run.smithi157.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-17T23:45:51.699 INFO:teuthology.orchestra.run.smithi157.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-17T23:45:51.700 INFO:teuthology.orchestra.run.smithi157.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-17T23:45:51.700 INFO:teuthology.orchestra.run.smithi157.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-17T23:45:51.700 INFO:teuthology.orchestra.run.smithi157.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-17T23:45:51.700 INFO:teuthology.orchestra.run.smithi157.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-17T23:45:51.700 INFO:teuthology.orchestra.run.smithi157.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-17T23:45:51.700 INFO:teuthology.orchestra.run.smithi157.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-17T23:45:51.701 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2024-03-17T23:45:51.701 INFO:teuthology.orchestra.run.smithi157.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-17T23:45:51.701 INFO:teuthology.orchestra.run.smithi157.stdout: automake-1.16.2-8.el9.noarch 2024-03-17T23:45:51.701 INFO:teuthology.orchestra.run.smithi157.stdout: bison-3.7.4-5.el9.x86_64 2024-03-17T23:45:51.701 INFO:teuthology.orchestra.run.smithi157.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-17T23:45:51.702 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.702 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.702 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.702 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.702 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:45:51.703 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.703 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.703 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.703 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:45:51.703 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:45:51.703 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:45:51.704 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:45:51.704 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:45:51.704 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.704 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.704 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:45:51.704 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.705 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.705 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.705 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:45:51.705 INFO:teuthology.orchestra.run.smithi157.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:45:51.705 INFO:teuthology.orchestra.run.smithi157.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.705 INFO:teuthology.orchestra.run.smithi157.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.706 INFO:teuthology.orchestra.run.smithi157.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:45:51.706 INFO:teuthology.orchestra.run.smithi157.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-17T23:45:51.706 INFO:teuthology.orchestra.run.smithi157.stdout: flex-2.6.4-9.el9.x86_64 2024-03-17T23:45:51.706 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-17T23:45:51.706 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-17T23:45:51.707 INFO:teuthology.orchestra.run.smithi157.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-17T23:45:51.707 INFO:teuthology.orchestra.run.smithi157.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-17T23:45:51.707 INFO:teuthology.orchestra.run.smithi157.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-17T23:45:51.707 INFO:teuthology.orchestra.run.smithi157.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-17T23:45:51.707 INFO:teuthology.orchestra.run.smithi157.stdout: jq-1.6-16.el9.x86_64 2024-03-17T23:45:51.707 INFO:teuthology.orchestra.run.smithi157.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-17T23:45:51.708 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-17T23:45:51.708 INFO:teuthology.orchestra.run.smithi157.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-17T23:45:51.708 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.708 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.708 INFO:teuthology.orchestra.run.smithi157.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.708 INFO:teuthology.orchestra.run.smithi157.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-17T23:45:51.709 INFO:teuthology.orchestra.run.smithi157.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-17T23:45:51.709 INFO:teuthology.orchestra.run.smithi157.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-17T23:45:51.709 INFO:teuthology.orchestra.run.smithi157.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-17T23:45:51.709 INFO:teuthology.orchestra.run.smithi157.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-17T23:45:51.709 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.709 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.710 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-17T23:45:51.710 INFO:teuthology.orchestra.run.smithi157.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.710 INFO:teuthology.orchestra.run.smithi157.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-17T23:45:51.710 INFO:teuthology.orchestra.run.smithi157.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-17T23:45:51.710 INFO:teuthology.orchestra.run.smithi157.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-17T23:45:51.710 INFO:teuthology.orchestra.run.smithi157.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-17T23:45:51.711 INFO:teuthology.orchestra.run.smithi157.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-17T23:45:51.711 INFO:teuthology.orchestra.run.smithi157.stdout: lua-5.4.4-4.el9.x86_64 2024-03-17T23:45:51.711 INFO:teuthology.orchestra.run.smithi157.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-17T23:45:51.711 INFO:teuthology.orchestra.run.smithi157.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-17T23:45:51.711 INFO:teuthology.orchestra.run.smithi157.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-17T23:45:51.712 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-17T23:45:51.712 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-17T23:45:51.712 INFO:teuthology.orchestra.run.smithi157.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-17T23:45:51.712 INFO:teuthology.orchestra.run.smithi157.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-17T23:45:51.712 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-13.11-1.el9.x86_64 2024-03-17T23:45:51.712 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-17T23:45:51.713 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-17T23:45:51.713 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-17T23:45:51.713 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-17T23:45:51.713 INFO:teuthology.orchestra.run.smithi157.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-17T23:45:51.713 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-17T23:45:51.713 INFO:teuthology.orchestra.run.smithi157.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-17T23:45:51.714 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-17T23:45:51.714 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-17T23:45:51.714 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-17T23:45:51.714 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.714 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.714 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.715 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-17T23:45:51.715 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-17T23:45:51.715 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-17T23:45:51.715 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-17T23:45:51.715 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-17T23:45:51.716 INFO:teuthology.orchestra.run.smithi157.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-17T23:45:51.716 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-17T23:45:51.716 INFO:teuthology.orchestra.run.smithi157.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-17T23:45:51.716 INFO:teuthology.orchestra.run.smithi157.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-17T23:45:51.716 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-17T23:45:51.716 INFO:teuthology.orchestra.run.smithi157.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-17T23:45:51.717 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-17T23:45:51.717 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-17T23:45:51.717 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-17T23:45:51.717 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-17T23:45:51.717 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-17T23:45:51.718 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-17T23:45:51.718 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-17T23:45:51.718 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-17T23:45:51.718 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-17T23:45:51.718 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-17T23:45:51.718 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-17T23:45:51.719 INFO:teuthology.orchestra.run.smithi157.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-17T23:45:51.719 INFO:teuthology.orchestra.run.smithi157.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-17T23:45:51.719 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-17T23:45:51.719 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-17T23:45:51.719 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-17T23:45:51.719 INFO:teuthology.orchestra.run.smithi157.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-17T23:45:51.720 INFO:teuthology.orchestra.run.smithi157.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-17T23:45:51.720 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-17T23:45:51.720 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ply-3.11-14.el9.noarch 2024-03-17T23:45:51.720 INFO:teuthology.orchestra.run.smithi157.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-17T23:45:51.720 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-17T23:45:51.721 INFO:teuthology.orchestra.run.smithi157.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-17T23:45:51.721 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-17T23:45:51.721 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-17T23:45:51.721 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-17T23:45:51.721 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-17T23:45:51.721 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-17T23:45:51.722 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-17T23:45:51.722 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.722 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.722 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-17T23:45:51.722 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-17T23:45:51.723 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-17T23:45:51.723 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.723 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-17T23:45:51.723 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-17T23:45:51.723 INFO:teuthology.orchestra.run.smithi157.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-17T23:45:51.723 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-17T23:45:51.724 INFO:teuthology.orchestra.run.smithi157.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-17T23:45:51.724 INFO:teuthology.orchestra.run.smithi157.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-17T23:45:51.724 INFO:teuthology.orchestra.run.smithi157.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-17T23:45:51.724 INFO:teuthology.orchestra.run.smithi157.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-17T23:45:51.724 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-17T23:45:51.724 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-17T23:45:51.725 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-17T23:45:51.725 INFO:teuthology.orchestra.run.smithi157.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-17T23:45:51.725 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-17T23:45:51.725 INFO:teuthology.orchestra.run.smithi157.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-17T23:45:51.725 INFO:teuthology.orchestra.run.smithi157.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-17T23:45:51.726 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.726 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.726 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:51.726 INFO:teuthology.orchestra.run.smithi157.stdout: re2-1:20211101-3.el9.x86_64 2024-03-17T23:45:51.726 INFO:teuthology.orchestra.run.smithi157.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-17T23:45:51.726 INFO:teuthology.orchestra.run.smithi157.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-17T23:45:51.727 INFO:teuthology.orchestra.run.smithi157.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-17T23:45:51.727 INFO:teuthology.orchestra.run.smithi157.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-17T23:45:51.727 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-17T23:45:51.727 INFO:teuthology.orchestra.run.smithi157.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-17T23:45:51.727 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:51.727 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2024-03-17T23:45:51.731 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-17T23:45:51.731 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-17T23:45:51.731 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-17T23:45:51.732 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-17T23:45:51.732 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-17T23:45:51.732 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-17T23:45:51.732 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-17T23:45:51.732 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-17T23:45:51.733 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-17T23:45:51.733 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-17T23:45:51.733 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-17T23:45:51.733 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-17T23:45:51.733 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-17T23:45:51.734 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-17T23:45:51.734 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-17T23:45:51.734 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-17T23:45:51.734 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-17T23:45:51.734 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-17T23:45:51.735 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-17T23:45:51.735 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-17T23:45:51.735 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-17T23:45:51.735 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-17T23:45:51.735 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-17T23:45:51.735 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-17T23:45:51.736 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-17T23:45:51.736 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-17T23:45:51.736 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-17T23:45:51.736 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-17T23:45:51.736 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-17T23:45:51.736 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-17T23:45:51.736 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-17T23:45:51.737 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-17T23:45:51.737 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-17T23:45:51.737 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-17T23:45:51.737 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-17T23:45:51.737 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-17T23:45:51.737 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-17T23:45:51.737 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-17T23:45:51.738 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-17T23:45:51.738 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-17T23:45:51.738 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-17T23:45:51.738 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-17T23:45:51.738 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-17T23:45:51.738 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-17T23:45:51.739 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-17T23:45:51.739 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-17T23:45:51.739 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-17T23:45:51.739 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-17T23:45:51.739 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-17T23:45:51.740 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-17T23:45:51.740 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-17T23:45:51.740 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-17T23:45:51.740 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-17T23:45:51.740 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-17T23:45:51.741 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-17T23:45:51.741 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-17T23:45:51.741 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-17T23:45:51.741 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-17T23:45:51.741 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-17T23:45:51.741 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-17T23:45:51.742 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-17T23:45:51.742 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-17T23:45:51.742 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-17T23:45:51.742 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-17T23:45:51.742 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-17T23:45:51.742 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-17T23:45:51.743 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-17T23:45:51.743 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-17T23:45:51.743 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-17T23:45:51.743 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-17T23:45:51.743 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-17T23:45:51.743 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-17T23:45:51.743 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-17T23:45:51.743 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-17T23:45:51.743 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-17T23:45:51.743 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-17T23:45:51.744 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-17T23:45:51.744 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-17T23:45:51.744 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-17T23:45:51.744 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-17T23:45:51.744 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-17T23:45:51.744 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-17T23:45:51.744 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-17T23:45:51.744 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-17T23:45:51.744 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-17T23:45:51.744 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-17T23:45:51.744 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-17T23:45:51.744 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-17T23:45:51.744 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-17T23:45:51.744 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-17T23:45:51.745 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-17T23:45:51.745 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-17T23:45:51.745 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-17T23:45:51.745 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-17T23:45:51.745 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-17T23:45:51.745 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-17T23:45:51.745 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-17T23:45:51.745 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-17T23:45:51.745 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-17T23:45:51.745 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-17T23:45:51.745 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-17T23:45:51.745 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-17T23:45:51.745 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-17T23:45:51.745 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-17T23:45:51.745 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-17T23:45:51.746 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-17T23:45:51.746 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-17T23:45:51.746 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-17T23:45:51.746 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-17T23:45:51.746 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-17T23:45:51.746 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-17T23:45:51.746 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-17T23:45:51.746 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-17T23:45:51.746 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-17T23:45:51.746 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-17T23:45:51.746 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-17T23:45:51.746 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-17T23:45:51.746 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-17T23:45:51.747 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-17T23:45:51.747 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-17T23:45:51.747 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-17T23:45:51.747 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-17T23:45:51.747 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-17T23:45:51.747 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-17T23:45:51.747 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-17T23:45:51.747 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-17T23:45:51.747 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-17T23:45:51.747 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-17T23:45:51.747 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-17T23:45:51.747 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-17T23:45:51.747 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-17T23:45:51.747 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-17T23:45:51.748 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-17T23:45:51.748 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-17T23:45:51.748 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-17T23:45:51.748 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-17T23:45:51.748 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-17T23:45:51.748 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-17T23:45:51.748 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-17T23:45:51.748 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-17T23:45:51.748 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-17T23:45:51.748 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-17T23:45:51.748 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-17T23:45:51.748 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-17T23:45:51.748 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-17T23:45:51.748 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-17T23:45:51.748 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-17T23:45:51.749 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-17T23:45:51.749 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-17T23:45:51.749 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-17T23:45:51.749 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-17T23:45:51.749 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-17T23:45:51.749 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-17T23:45:51.749 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-17T23:45:51.749 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-17T23:45:51.749 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-17T23:45:51.749 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-17T23:45:51.749 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-17T23:45:51.749 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-17T23:45:51.749 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-17T23:45:51.749 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-17T23:45:51.750 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-17T23:45:51.750 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-17T23:45:51.750 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-17T23:45:51.750 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-17T23:45:51.750 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-17T23:45:51.750 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-17T23:45:51.750 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-17T23:45:51.750 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-17T23:45:51.750 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-17T23:45:51.750 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-17T23:45:51.750 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-17T23:45:51.750 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-17T23:45:51.750 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-17T23:45:51.751 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-17T23:45:51.751 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-17T23:45:51.751 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-17T23:45:51.751 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-17T23:45:51.751 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-17T23:45:51.751 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-17T23:45:51.751 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-17T23:45:51.751 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-17T23:45:51.751 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-17T23:45:51.751 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-17T23:45:51.751 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-17T23:45:51.751 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-17T23:45:51.751 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-17T23:45:51.752 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-17T23:45:51.752 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-17T23:45:51.752 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-17T23:45:51.752 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-17T23:45:51.752 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-17T23:45:51.752 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-17T23:45:51.752 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-17T23:45:51.752 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-17T23:45:51.752 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-17T23:45:51.752 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-17T23:45:51.752 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-17T23:45:52.182 DEBUG:teuthology.parallel:result is None 2024-03-17T23:45:54.786 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-17T23:45:54.786 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:54.786 INFO:teuthology.orchestra.run.smithi150.stdout:Upgraded: 2024-03-17T23:45:54.786 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-17T23:45:54.786 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-17T23:45:54.787 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-17T23:45:54.787 INFO:teuthology.orchestra.run.smithi150.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-17T23:45:54.787 INFO:teuthology.orchestra.run.smithi150.stdout: acl-2.3.1-4.el9.x86_64 2024-03-17T23:45:54.787 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-17T23:45:54.787 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-17T23:45:54.787 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-17T23:45:54.787 INFO:teuthology.orchestra.run.smithi150.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-17T23:45:54.787 INFO:teuthology.orchestra.run.smithi150.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-17T23:45:54.787 INFO:teuthology.orchestra.run.smithi150.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-17T23:45:54.787 INFO:teuthology.orchestra.run.smithi150.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-17T23:45:54.787 INFO:teuthology.orchestra.run.smithi150.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-17T23:45:54.787 INFO:teuthology.orchestra.run.smithi150.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.787 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.787 INFO:teuthology.orchestra.run.smithi150.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-17T23:45:54.788 INFO:teuthology.orchestra.run.smithi150.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-17T23:45:54.788 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-17T23:45:54.788 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-17T23:45:54.788 INFO:teuthology.orchestra.run.smithi150.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-17T23:45:54.788 INFO:teuthology.orchestra.run.smithi150.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-17T23:45:54.788 INFO:teuthology.orchestra.run.smithi150.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-17T23:45:54.788 INFO:teuthology.orchestra.run.smithi150.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-17T23:45:54.788 INFO:teuthology.orchestra.run.smithi150.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-17T23:45:54.788 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-03-17T23:45:54.788 INFO:teuthology.orchestra.run.smithi150.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-17T23:45:54.788 INFO:teuthology.orchestra.run.smithi150.stdout: automake-1.16.2-8.el9.noarch 2024-03-17T23:45:54.788 INFO:teuthology.orchestra.run.smithi150.stdout: bison-3.7.4-5.el9.x86_64 2024-03-17T23:45:54.788 INFO:teuthology.orchestra.run.smithi150.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-17T23:45:54.788 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.788 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.789 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.789 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.789 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:45:54.789 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.789 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.789 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.789 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:45:54.789 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:45:54.789 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:45:54.789 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:45:54.789 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:45:54.789 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.789 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.790 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:45:54.790 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.790 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.790 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.790 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:45:54.790 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:45:54.790 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.790 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.790 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-17T23:45:54.790 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-17T23:45:54.790 INFO:teuthology.orchestra.run.smithi150.stdout: flex-2.6.4-9.el9.x86_64 2024-03-17T23:45:54.790 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-17T23:45:54.790 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-17T23:45:54.791 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-17T23:45:54.791 INFO:teuthology.orchestra.run.smithi150.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-17T23:45:54.791 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-17T23:45:54.791 INFO:teuthology.orchestra.run.smithi150.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-17T23:45:54.791 INFO:teuthology.orchestra.run.smithi150.stdout: jq-1.6-16.el9.x86_64 2024-03-17T23:45:54.791 INFO:teuthology.orchestra.run.smithi150.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-17T23:45:54.791 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-17T23:45:54.791 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-17T23:45:54.791 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.791 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.791 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.791 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-17T23:45:54.791 INFO:teuthology.orchestra.run.smithi150.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-17T23:45:54.792 INFO:teuthology.orchestra.run.smithi150.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-17T23:45:54.792 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-17T23:45:54.792 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-17T23:45:54.792 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.792 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.792 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-17T23:45:54.792 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.792 INFO:teuthology.orchestra.run.smithi150.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-17T23:45:54.792 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-17T23:45:54.792 INFO:teuthology.orchestra.run.smithi150.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-17T23:45:54.792 INFO:teuthology.orchestra.run.smithi150.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-17T23:45:54.792 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-17T23:45:54.792 INFO:teuthology.orchestra.run.smithi150.stdout: lua-5.4.4-4.el9.x86_64 2024-03-17T23:45:54.792 INFO:teuthology.orchestra.run.smithi150.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-17T23:45:54.793 INFO:teuthology.orchestra.run.smithi150.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-17T23:45:54.793 INFO:teuthology.orchestra.run.smithi150.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-17T23:45:54.793 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-17T23:45:54.793 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-17T23:45:54.793 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-17T23:45:54.793 INFO:teuthology.orchestra.run.smithi150.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-17T23:45:54.793 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-13.11-1.el9.x86_64 2024-03-17T23:45:54.793 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-17T23:45:54.793 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-17T23:45:54.793 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-17T23:45:54.793 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-17T23:45:54.793 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-17T23:45:54.793 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-17T23:45:54.794 INFO:teuthology.orchestra.run.smithi150.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-17T23:45:54.794 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-17T23:45:54.794 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-17T23:45:54.794 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-17T23:45:54.794 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.794 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.794 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.794 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-17T23:45:54.794 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-17T23:45:54.794 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-17T23:45:54.794 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-17T23:45:54.795 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-17T23:45:54.795 INFO:teuthology.orchestra.run.smithi150.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-17T23:45:54.795 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-17T23:45:54.795 INFO:teuthology.orchestra.run.smithi150.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-17T23:45:54.795 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-17T23:45:54.795 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-17T23:45:54.795 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-17T23:45:54.795 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-17T23:45:54.795 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-17T23:45:54.795 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-17T23:45:54.795 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-17T23:45:54.795 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-17T23:45:54.795 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-17T23:45:54.795 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-17T23:45:54.796 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-17T23:45:54.796 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-17T23:45:54.796 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-17T23:45:54.796 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-17T23:45:54.796 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-17T23:45:54.796 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-17T23:45:54.796 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-17T23:45:54.796 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-17T23:45:54.796 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-17T23:45:54.796 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-17T23:45:54.796 INFO:teuthology.orchestra.run.smithi150.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-17T23:45:54.796 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-17T23:45:54.796 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ply-3.11-14.el9.noarch 2024-03-17T23:45:54.796 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-17T23:45:54.797 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-17T23:45:54.797 INFO:teuthology.orchestra.run.smithi150.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-17T23:45:54.797 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-17T23:45:54.797 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-17T23:45:54.797 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-17T23:45:54.797 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-17T23:45:54.797 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-17T23:45:54.797 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-17T23:45:54.797 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.797 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.797 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-17T23:45:54.797 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-17T23:45:54.797 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-17T23:45:54.797 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.798 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-17T23:45:54.798 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-17T23:45:54.798 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-17T23:45:54.798 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-17T23:45:54.798 INFO:teuthology.orchestra.run.smithi150.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-17T23:45:54.798 INFO:teuthology.orchestra.run.smithi150.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-17T23:45:54.798 INFO:teuthology.orchestra.run.smithi150.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-17T23:45:54.798 INFO:teuthology.orchestra.run.smithi150.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-17T23:45:54.798 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-17T23:45:54.798 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-17T23:45:54.798 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-17T23:45:54.798 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-17T23:45:54.798 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-17T23:45:54.799 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-17T23:45:54.799 INFO:teuthology.orchestra.run.smithi150.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-17T23:45:54.799 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.799 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.799 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-17T23:45:54.799 INFO:teuthology.orchestra.run.smithi150.stdout: re2-1:20211101-3.el9.x86_64 2024-03-17T23:45:54.799 INFO:teuthology.orchestra.run.smithi150.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-17T23:45:54.799 INFO:teuthology.orchestra.run.smithi150.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-17T23:45:54.799 INFO:teuthology.orchestra.run.smithi150.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-17T23:45:54.799 INFO:teuthology.orchestra.run.smithi150.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-17T23:45:54.799 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-17T23:45:54.800 INFO:teuthology.orchestra.run.smithi150.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-17T23:45:54.800 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:54.800 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-03-17T23:45:55.271 DEBUG:teuthology.parallel:result is None 2024-03-17T23:45:55.272 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-17T23:45:55.415 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-17T23:45:55.437 INFO:teuthology.orchestra.run.smithi150.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-17T23:45:55.437 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-17T23:45:55.437 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-17T23:45:55.439 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-17T23:45:55.659 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-17T23:45:55.685 INFO:teuthology.orchestra.run.smithi157.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-17T23:45:55.685 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-17T23:45:55.685 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-17T23:45:55.687 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-17T23:45:55.687 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:45:55.687 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-17T23:45:55.719 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:45:55.719 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-17T23:45:55.759 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-17T23:45:55.759 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:45:55.759 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/daemon-helper 2024-03-17T23:45:55.787 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-17T23:45:55.854 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:45:55.854 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/daemon-helper 2024-03-17T23:45:55.884 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-17T23:45:55.952 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-17T23:45:55.953 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:45:55.953 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-17T23:45:55.980 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-17T23:45:56.047 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:45:56.047 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-17T23:45:56.076 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-17T23:45:56.143 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-17T23:45:56.143 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:45:56.143 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/stdin-killer 2024-03-17T23:45:56.170 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-17T23:45:56.236 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:45:56.237 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/stdin-killer 2024-03-17T23:45:56.263 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-17T23:45:56.330 INFO:teuthology.run_tasks:Running task ceph... 2024-03-17T23:45:56.414 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-17T23:45:56.414 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 777 /var/log/ceph 2024-03-17T23:45:56.416 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 777 /var/log/ceph 2024-03-17T23:45:56.443 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-17T23:45:56.443 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-17T23:45:56.483 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-17T23:45:56.509 INFO:tasks.ceph:Creating extra log directories... 2024-03-17T23:45:56.510 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-17T23:45:56.550 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-17T23:45:56.585 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-17T23:45:56.586 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-17T23:45:56.586 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-17T23:45:56.586 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:45:56.586 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-17T23:45:56.630 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-17T23:45:56.748 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-17T23:45:56.775 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:45:56.775 DEBUG:teuthology.orchestra.run.smithi157:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-17T23:45:56.790 DEBUG:teuthology.orchestra.run.smithi157:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-17T23:45:56.908 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-17T23:45:56.940 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-17T23:45:56.940 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'mons down, quorum', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-17T23:45:56.940 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608488', 'branch': 'wip-batrick-testing-20240314.183243', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/pool-perm}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7608488', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-batrick-testing-20240314.183243'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'mons down, quorum', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN'], 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 1, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-batrick-testing-20240314.183243', 'sha1': 'c382027d5c66ef4e0aa1131ea3e8080c658d3184'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 4770, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'sleep_before_teardown': 0, 'subset': '498/512', 'suite': 'fs', 'suite_branch': 'wip-batrick-testing-20240314.183243', 'suite_path': '/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/batrick/ceph', 'suite_sha1': 'c382027d5c66ef4e0aa1131ea3e8080c658d3184', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi150.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEuznq3ZAT3syTm6Yet4MN3i/Y3sAFNkEz27pRQMqOK0NnuEKnOkPmWZl97ElRgzeZNsnze7EUL85fmO4o5EZ6M=', 'smithi157.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFQmXCS1lYraNILT38OgyDIOYfOlxAZXfrbtQYaTnpIiZzvkKmkBYkPjsJ1gnMMUKS7RHaMUrZ6fmo+5HkRvS2I='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'mons down, quorum', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN'], 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_pool_perm']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-17_23:17:27', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-17T23:45:56.941 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-17T23:45:56.943 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-17T23:45:56.990 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m0777 -- /var/run/ceph 2024-03-17T23:45:56.998 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m0777 -- /var/run/ceph 2024-03-17T23:45:57.057 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:45:57.057 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/scratch_devs of=/dev/stdout 2024-03-17T23:45:57.078 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-17T23:45:57.078 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_1 2024-03-17T23:45:57.132 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-17T23:45:57.133 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-17T23:45:57.133 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-17T23:45:57.133 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-17T23:45:57.133 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-03-17T23:45:57.133 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-03-17 23:45:47.234328319 +0000 2024-03-17T23:45:57.133 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-03-17 23:38:49.025592039 +0000 2024-03-17T23:45:57.133 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-03-17 23:38:49.025592039 +0000 2024-03-17T23:45:57.133 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-03-17 23:38:49.025592039 +0000 2024-03-17T23:45:57.133 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-17T23:45:57.198 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-03-17T23:45:57.199 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-03-17T23:45:57.199 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000256863 s, 2.0 MB/s 2024-03-17T23:45:57.200 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-17T23:45:57.255 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_2 2024-03-17T23:45:57.310 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-17T23:45:57.311 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-17T23:45:57.311 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-17T23:45:57.311 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-17T23:45:57.311 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-03-17T23:45:57.311 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-03-17 23:45:47.235328300 +0000 2024-03-17T23:45:57.311 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-03-17 23:38:49.546581917 +0000 2024-03-17T23:45:57.311 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-03-17 23:38:49.546581917 +0000 2024-03-17T23:45:57.311 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-03-17 23:38:49.546581917 +0000 2024-03-17T23:45:57.311 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-17T23:45:57.377 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-03-17T23:45:57.377 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-03-17T23:45:57.377 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000203926 s, 2.5 MB/s 2024-03-17T23:45:57.378 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-17T23:45:57.433 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_3 2024-03-17T23:45:57.488 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-17T23:45:57.488 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-17T23:45:57.488 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-17T23:45:57.488 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-17T23:45:57.488 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-03-17T23:45:57.488 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-03-17 23:45:47.235328300 +0000 2024-03-17T23:45:57.488 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-03-17 23:38:49.889575252 +0000 2024-03-17T23:45:57.488 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-03-17 23:38:49.889575252 +0000 2024-03-17T23:45:57.488 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-03-17 23:38:49.889575252 +0000 2024-03-17T23:45:57.488 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-17T23:45:57.553 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-03-17T23:45:57.553 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-03-17T23:45:57.553 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000188389 s, 2.7 MB/s 2024-03-17T23:45:57.554 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-17T23:45:57.611 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_4 2024-03-17T23:45:57.665 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-17T23:45:57.665 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-17T23:45:57.665 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-17T23:45:57.665 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-17T23:45:57.665 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-03-17T23:45:57.665 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-03-17 23:45:47.235328300 +0000 2024-03-17T23:45:57.665 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-03-17 23:38:50.247568294 +0000 2024-03-17T23:45:57.665 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-03-17 23:38:50.247568294 +0000 2024-03-17T23:45:57.665 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-03-17 23:38:50.247568294 +0000 2024-03-17T23:45:57.666 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-17T23:45:57.731 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-03-17T23:45:57.731 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-03-17T23:45:57.731 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000229548 s, 2.2 MB/s 2024-03-17T23:45:57.732 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-17T23:45:57.788 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-17T23:45:57.788 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:45:57.788 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/scratch_devs of=/dev/stdout 2024-03-17T23:45:57.805 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-17T23:45:57.805 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_1 2024-03-17T23:45:57.861 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-17T23:45:57.861 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-17T23:45:57.861 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-17T23:45:57.862 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-17T23:45:57.862 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-03-17T23:45:57.862 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-03-17 23:45:43.842026386 +0000 2024-03-17T23:45:57.862 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-03-17 23:43:46.660947908 +0000 2024-03-17T23:45:57.862 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-03-17 23:43:46.660947908 +0000 2024-03-17T23:45:57.862 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-03-17 23:43:46.660947908 +0000 2024-03-17T23:45:57.862 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-17T23:45:57.928 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-03-17T23:45:57.928 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-03-17T23:45:57.928 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.00010627 s, 4.8 MB/s 2024-03-17T23:45:57.929 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-17T23:45:57.986 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_2 2024-03-17T23:45:58.042 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-17T23:45:58.042 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-17T23:45:58.042 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-17T23:45:58.042 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-17T23:45:58.042 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-03-17T23:45:58.042 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-03-17 23:45:43.843026368 +0000 2024-03-17T23:45:58.042 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-03-17 23:43:46.999942042 +0000 2024-03-17T23:45:58.042 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-03-17 23:43:46.999942042 +0000 2024-03-17T23:45:58.042 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-03-17 23:43:46.999942042 +0000 2024-03-17T23:45:58.043 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-17T23:45:58.108 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-03-17T23:45:58.108 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-03-17T23:45:58.108 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 9.9899e-05 s, 5.1 MB/s 2024-03-17T23:45:58.109 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-17T23:45:58.165 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_3 2024-03-17T23:45:58.219 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-17T23:45:58.219 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-17T23:45:58.219 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-17T23:45:58.219 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-17T23:45:58.219 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-03-17T23:45:58.219 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-03-17 23:45:43.843026368 +0000 2024-03-17T23:45:58.219 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-03-17 23:43:47.325936401 +0000 2024-03-17T23:45:58.219 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-03-17 23:43:47.325936401 +0000 2024-03-17T23:45:58.220 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-03-17 23:43:47.325936401 +0000 2024-03-17T23:45:58.220 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-17T23:45:58.284 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-03-17T23:45:58.284 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-03-17T23:45:58.284 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000104628 s, 4.9 MB/s 2024-03-17T23:45:58.285 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-17T23:45:58.340 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_4 2024-03-17T23:45:58.395 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-17T23:45:58.395 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-17T23:45:58.395 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-17T23:45:58.395 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-17T23:45:58.395 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2024-03-17T23:45:58.395 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2024-03-17 23:45:43.843026368 +0000 2024-03-17T23:45:58.395 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2024-03-17 23:43:47.647930828 +0000 2024-03-17T23:45:58.395 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2024-03-17 23:43:47.647930828 +0000 2024-03-17T23:45:58.395 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: 2024-03-17 23:43:47.647930828 +0000 2024-03-17T23:45:58.396 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-17T23:45:58.460 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2024-03-17T23:45:58.460 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2024-03-17T23:45:58.460 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 9.9694e-05 s, 5.1 MB/s 2024-03-17T23:45:58.461 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-17T23:45:58.516 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-17T23:45:58.516 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi150.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi157.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-17T23:45:58.516 INFO:tasks.ceph:Generating config... 2024-03-17T23:45:58.518 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-17T23:45:58.518 INFO:tasks.ceph:[client] debug client = 20 2024-03-17T23:45:58.518 INFO:tasks.ceph:[client] debug ms = 1 2024-03-17T23:45:58.518 INFO:tasks.ceph:[client] pid file = 2024-03-17T23:45:58.518 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-17T23:45:58.518 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-17T23:45:58.518 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-17T23:45:58.518 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-17T23:45:58.518 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-17T23:45:58.518 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-17T23:45:58.518 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-17T23:45:58.519 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-17T23:45:58.519 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-17T23:45:58.519 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-17T23:45:58.519 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-17T23:45:58.519 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-17T23:45:58.519 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-17T23:45:58.519 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-17T23:45:58.519 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-17T23:45:58.519 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-17T23:45:58.519 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-17T23:45:58.519 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-17T23:45:58.519 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-17T23:45:58.519 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-17T23:45:58.519 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-17T23:45:58.519 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-17T23:45:58.519 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-17T23:45:58.520 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-17T23:45:58.520 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-17T23:45:58.520 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-17T23:45:58.520 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-17T23:45:58.520 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-17T23:45:58.520 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-17T23:45:58.520 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-17T23:45:58.520 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-17T23:45:58.520 INFO:tasks.ceph:Setting up mon.a... 2024-03-17T23:45:58.520 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-17T23:45:58.570 INFO:teuthology.orchestra.run.smithi150.stdout:creating /etc/ceph/ceph.keyring 2024-03-17T23:45:58.573 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-17T23:45:58.617 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-17T23:45:58.643 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.150'), ('mon.b', '172.21.15.157'), ('mon.c', '[v2:172.21.15.157:3301,v1:172.21.15.157:6790]')] 2024-03-17T23:45:58.644 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.150,172.21.15.157,[v2:172.21.15.157:3301,v1:172.21.15.157:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-17T23:45:58.644 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:45:58.645 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-17T23:45:58.698 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.150 --add b 172.21.15.157 --addv c '[v2:172.21.15.157:3301,v1:172.21.15.157:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-17T23:45:58.784 INFO:teuthology.orchestra.run.smithi150.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-17T23:45:58.784 INFO:teuthology.orchestra.run.smithi150.stdout:monmaptool: generated fsid a1c5acd3-e97f-470b-8226-335736a7795b 2024-03-17T23:45:58.784 INFO:teuthology.orchestra.run.smithi150.stdout:setting min_mon_release = quincy 2024-03-17T23:45:58.785 INFO:teuthology.orchestra.run.smithi150.stdout:epoch 0 2024-03-17T23:45:58.785 INFO:teuthology.orchestra.run.smithi150.stdout:fsid a1c5acd3-e97f-470b-8226-335736a7795b 2024-03-17T23:45:58.785 INFO:teuthology.orchestra.run.smithi150.stdout:last_changed 2024-03-17T23:45:58.784913+0000 2024-03-17T23:45:58.785 INFO:teuthology.orchestra.run.smithi150.stdout:created 2024-03-17T23:45:58.784913+0000 2024-03-17T23:45:58.785 INFO:teuthology.orchestra.run.smithi150.stdout:min_mon_release 17 (quincy) 2024-03-17T23:45:58.785 INFO:teuthology.orchestra.run.smithi150.stdout:election_strategy: 1 2024-03-17T23:45:58.785 INFO:teuthology.orchestra.run.smithi150.stdout:0: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.a 2024-03-17T23:45:58.785 INFO:teuthology.orchestra.run.smithi150.stdout:1: [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon.b 2024-03-17T23:45:58.785 INFO:teuthology.orchestra.run.smithi150.stdout:2: [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] mon.c 2024-03-17T23:45:58.785 INFO:teuthology.orchestra.run.smithi150.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-17T23:45:58.786 DEBUG:teuthology.orchestra.run.smithi150:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-17T23:45:58.801 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a1c5acd3-e97f-470b-8226-335736a7795b... 2024-03-17T23:45:58.801 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-17T23:45:58.844 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-17T23:45:58.891 INFO:teuthology.orchestra.run.smithi150.stdout:[global] 2024-03-17T23:45:58.891 INFO:teuthology.orchestra.run.smithi150.stdout: chdir = "" 2024-03-17T23:45:58.892 INFO:teuthology.orchestra.run.smithi150.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-17T23:45:58.892 INFO:teuthology.orchestra.run.smithi150.stdout: auth supported = cephx 2024-03-17T23:45:58.892 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.892 INFO:teuthology.orchestra.run.smithi150.stdout: filestore xattr use omap = true 2024-03-17T23:45:58.892 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.892 INFO:teuthology.orchestra.run.smithi150.stdout: mon clock drift allowed = 1.000 2024-03-17T23:45:58.892 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.892 INFO:teuthology.orchestra.run.smithi150.stdout: osd crush chooseleaf type = 0 2024-03-17T23:45:58.892 INFO:teuthology.orchestra.run.smithi150.stdout: auth debug = true 2024-03-17T23:45:58.892 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.892 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on old message = true 2024-03-17T23:45:58.892 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on bug = true 2024-03-17T23:45:58.893 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.893 INFO:teuthology.orchestra.run.smithi150.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-17T23:45:58.893 INFO:teuthology.orchestra.run.smithi150.stdout: mon pg warn max object skew = 0 2024-03-17T23:45:58.893 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.893 INFO:teuthology.orchestra.run.smithi150.stdout: # disable pg_autoscaler by default for new pools 2024-03-17T23:45:58.893 INFO:teuthology.orchestra.run.smithi150.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-17T23:45:58.893 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.893 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default size = 2 2024-03-17T23:45:58.893 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.893 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow primary affinity = true 2024-03-17T23:45:58.893 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow pg remap = true 2024-03-17T23:45:58.893 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on legacy crush tunables = false 2024-03-17T23:45:58.893 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on crush straw calc version zero = false 2024-03-17T23:45:58.893 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on no sortbitwise = false 2024-03-17T23:45:58.894 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on osd down out interval zero = false 2024-03-17T23:45:58.894 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on too few osds = false 2024-03-17T23:45:58.894 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-17T23:45:58.894 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-17T23:45:58.894 INFO:teuthology.orchestra.run.smithi150.stdout: mon_allow_pool_size_one = true 2024-03-17T23:45:58.894 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.894 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-17T23:45:58.894 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.894 INFO:teuthology.orchestra.run.smithi150.stdout: osd default data pool replay window = 5 2024-03-17T23:45:58.894 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.894 INFO:teuthology.orchestra.run.smithi150.stdout: mon allow pool delete = true 2024-03-17T23:45:58.894 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.894 INFO:teuthology.orchestra.run.smithi150.stdout: mon cluster log file level = debug 2024-03-17T23:45:58.895 INFO:teuthology.orchestra.run.smithi150.stdout: debug asserts on shutdown = true 2024-03-17T23:45:58.895 INFO:teuthology.orchestra.run.smithi150.stdout: mon health detail to clog = false 2024-03-17T23:45:58.895 INFO:teuthology.orchestra.run.smithi150.stdout: mon host = "172.21.15.150,172.21.15.157,[v2:172.21.15.157:3301,v1:172.21.15.157:6790]" 2024-03-17T23:45:58.895 INFO:teuthology.orchestra.run.smithi150.stdout: fsid = a1c5acd3-e97f-470b-8226-335736a7795b 2024-03-17T23:45:58.895 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.895 INFO:teuthology.orchestra.run.smithi150.stdout:[osd] 2024-03-17T23:45:58.895 INFO:teuthology.orchestra.run.smithi150.stdout: osd journal size = 100 2024-03-17T23:45:58.895 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.895 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub load threshold = 5.0 2024-03-17T23:45:58.895 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub max interval = 600 2024-03-17T23:45:58.895 INFO:teuthology.orchestra.run.smithi150.stdout: osd mclock profile = high_recovery_ops 2024-03-17T23:45:58.895 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.895 INFO:teuthology.orchestra.run.smithi150.stdout: osd recover clone overlap = true 2024-03-17T23:45:58.895 INFO:teuthology.orchestra.run.smithi150.stdout: osd recovery max chunk = 1048576 2024-03-17T23:45:58.895 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.896 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug shutdown = true 2024-03-17T23:45:58.896 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug op order = true 2024-03-17T23:45:58.896 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug verify stray on activate = true 2024-03-17T23:45:58.896 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug trim objects = true 2024-03-17T23:45:58.896 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.896 INFO:teuthology.orchestra.run.smithi150.stdout: osd open classes on start = true 2024-03-17T23:45:58.896 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug pg log writeout = true 2024-03-17T23:45:58.896 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.896 INFO:teuthology.orchestra.run.smithi150.stdout: osd deep scrub update digest min age = 30 2024-03-17T23:45:58.896 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.896 INFO:teuthology.orchestra.run.smithi150.stdout: osd map max advance = 10 2024-03-17T23:45:58.896 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.896 INFO:teuthology.orchestra.run.smithi150.stdout: journal zero on create = true 2024-03-17T23:45:58.896 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.896 INFO:teuthology.orchestra.run.smithi150.stdout: filestore ondisk finisher threads = 3 2024-03-17T23:45:58.897 INFO:teuthology.orchestra.run.smithi150.stdout: filestore apply finisher threads = 3 2024-03-17T23:45:58.897 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.897 INFO:teuthology.orchestra.run.smithi150.stdout: bdev debug aio = true 2024-03-17T23:45:58.897 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug misdirected ops = true 2024-03-17T23:45:58.897 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore block size = 96636764160 2024-03-17T23:45:58.897 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore fsck on mount = True 2024-03-17T23:45:58.897 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluefs = 20 2024-03-17T23:45:58.897 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluestore = 20 2024-03-17T23:45:58.897 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-03-17T23:45:58.897 INFO:teuthology.orchestra.run.smithi150.stdout: debug osd = 20 2024-03-17T23:45:58.897 INFO:teuthology.orchestra.run.smithi150.stdout: debug rocksdb = 10 2024-03-17T23:45:58.897 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd backfillfull_ratio = 0.85 2024-03-17T23:45:58.897 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd full ratio = 0.9 2024-03-17T23:45:58.897 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd nearfull ratio = 0.8 2024-03-17T23:45:58.897 INFO:teuthology.orchestra.run.smithi150.stdout: osd failsafe full ratio = 0.95 2024-03-17T23:45:58.898 INFO:teuthology.orchestra.run.smithi150.stdout: osd objectstore = bluestore 2024-03-17T23:45:58.898 INFO:teuthology.orchestra.run.smithi150.stdout: osd op complaint time = 180 2024-03-17T23:45:58.898 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.898 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr] 2024-03-17T23:45:58.898 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-03-17T23:45:58.898 INFO:teuthology.orchestra.run.smithi150.stdout: debug mgr = 20 2024-03-17T23:45:58.898 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2024-03-17T23:45:58.898 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2024-03-17T23:45:58.898 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2024-03-17T23:45:58.898 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2024-03-17T23:45:58.898 INFO:teuthology.orchestra.run.smithi150.stdout: mgr/telemetry/nag = false 2024-03-17T23:45:58.898 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.898 INFO:teuthology.orchestra.run.smithi150.stdout:[mon] 2024-03-17T23:45:58.898 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-03-17T23:45:58.899 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2024-03-17T23:45:58.899 INFO:teuthology.orchestra.run.smithi150.stdout: debug paxos = 20 2024-03-17T23:45:58.899 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2024-03-17T23:45:58.899 INFO:teuthology.orchestra.run.smithi150.stdout: mon data avail warn = 5 2024-03-17T23:45:58.899 INFO:teuthology.orchestra.run.smithi150.stdout: mon mgr mkfs grace = 240 2024-03-17T23:45:58.899 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2024-03-17T23:45:58.899 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd reporter subtree level = osd 2024-03-17T23:45:58.899 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd prime pg temp = true 2024-03-17T23:45:58.899 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2024-03-17T23:45:58.899 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.899 INFO:teuthology.orchestra.run.smithi150.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-17T23:45:58.899 INFO:teuthology.orchestra.run.smithi150.stdout: auth mon ticket ttl = 660 # 11m 2024-03-17T23:45:58.899 INFO:teuthology.orchestra.run.smithi150.stdout: auth service ticket ttl = 240 # 4m 2024-03-17T23:45:58.899 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.900 INFO:teuthology.orchestra.run.smithi150.stdout: # don't complain about insecure global_id in the test suite 2024-03-17T23:45:58.900 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-17T23:45:58.900 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-17T23:45:58.900 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.900 INFO:teuthology.orchestra.run.smithi150.stdout: # 1m isn't quite enough 2024-03-17T23:45:58.900 INFO:teuthology.orchestra.run.smithi150.stdout: mon_down_mkfs_grace = 2m 2024-03-17T23:45:58.900 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.900 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_filestore_osds = false 2024-03-17T23:45:58.900 INFO:teuthology.orchestra.run.smithi150.stdout: mon op complaint time = 120 2024-03-17T23:45:58.900 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:45:58.900 INFO:teuthology.orchestra.run.smithi150.stdout:[client] 2024-03-17T23:45:58.900 INFO:teuthology.orchestra.run.smithi150.stdout: rgw cache enabled = true 2024-03-17T23:45:58.900 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable ops log = true 2024-03-17T23:45:58.900 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable usage log = true 2024-03-17T23:45:58.901 INFO:teuthology.orchestra.run.smithi150.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-17T23:45:58.901 INFO:teuthology.orchestra.run.smithi150.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-17T23:45:58.901 INFO:teuthology.orchestra.run.smithi150.stdout: client mount timeout = 600 2024-03-17T23:45:58.901 INFO:teuthology.orchestra.run.smithi150.stdout: debug client = 20 2024-03-17T23:45:58.901 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-03-17T23:45:58.901 INFO:teuthology.orchestra.run.smithi150.stdout: pid file = "" 2024-03-17T23:45:58.901 INFO:teuthology.orchestra.run.smithi150.stdout: rados mon op timeout = 900 2024-03-17T23:45:58.901 INFO:teuthology.orchestra.run.smithi150.stdout: rados osd op timeout = 900 2024-03-17T23:45:58.901 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.a] 2024-03-17T23:45:58.901 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.b] 2024-03-17T23:45:58.901 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.c] 2024-03-17T23:45:58.901 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.a] 2024-03-17T23:45:58.901 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.c] 2024-03-17T23:45:58.901 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.b] 2024-03-17T23:45:58.901 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.d] 2024-03-17T23:45:58.902 INFO:teuthology.orchestra.run.smithi150.stdout:[mds] 2024-03-17T23:45:58.902 INFO:teuthology.orchestra.run.smithi150.stdout: debug mds = 20 2024-03-17T23:45:58.902 INFO:teuthology.orchestra.run.smithi150.stdout: debug mds balancer = 20 2024-03-17T23:45:58.902 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-03-17T23:45:58.902 INFO:teuthology.orchestra.run.smithi150.stdout: mds debug frag = True 2024-03-17T23:45:58.902 INFO:teuthology.orchestra.run.smithi150.stdout: mds debug scatterstat = True 2024-03-17T23:45:58.902 INFO:teuthology.orchestra.run.smithi150.stdout: mds op complaint time = 180 2024-03-17T23:45:58.902 INFO:teuthology.orchestra.run.smithi150.stdout: mds verify scatter = True 2024-03-17T23:45:58.902 INFO:teuthology.orchestra.run.smithi150.stdout: osd op complaint time = 180 2024-03-17T23:45:58.902 INFO:teuthology.orchestra.run.smithi150.stdout: rados mon op timeout = 900 2024-03-17T23:45:58.902 INFO:teuthology.orchestra.run.smithi150.stdout: rados osd op timeout = 900 2024-03-17T23:45:58.903 INFO:teuthology.orchestra.run.smithi157.stdout:[global] 2024-03-17T23:45:58.903 INFO:teuthology.orchestra.run.smithi157.stdout: chdir = "" 2024-03-17T23:45:58.903 INFO:teuthology.orchestra.run.smithi157.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-17T23:45:58.903 INFO:teuthology.orchestra.run.smithi157.stdout: auth supported = cephx 2024-03-17T23:45:58.903 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.903 INFO:teuthology.orchestra.run.smithi157.stdout: filestore xattr use omap = true 2024-03-17T23:45:58.903 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.903 INFO:teuthology.orchestra.run.smithi157.stdout: mon clock drift allowed = 1.000 2024-03-17T23:45:58.903 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.903 INFO:teuthology.orchestra.run.smithi157.stdout: osd crush chooseleaf type = 0 2024-03-17T23:45:58.903 INFO:teuthology.orchestra.run.smithi157.stdout: auth debug = true 2024-03-17T23:45:58.903 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.904 INFO:teuthology.orchestra.run.smithi157.stdout: ms die on old message = true 2024-03-17T23:45:58.904 INFO:teuthology.orchestra.run.smithi157.stdout: ms die on bug = true 2024-03-17T23:45:58.904 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.904 INFO:teuthology.orchestra.run.smithi157.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-17T23:45:58.904 INFO:teuthology.orchestra.run.smithi157.stdout: mon pg warn max object skew = 0 2024-03-17T23:45:58.904 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.904 INFO:teuthology.orchestra.run.smithi157.stdout: # disable pg_autoscaler by default for new pools 2024-03-17T23:45:58.904 INFO:teuthology.orchestra.run.smithi157.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-17T23:45:58.904 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.904 INFO:teuthology.orchestra.run.smithi157.stdout: osd pool default size = 2 2024-03-17T23:45:58.904 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.904 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd allow primary affinity = true 2024-03-17T23:45:58.904 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd allow pg remap = true 2024-03-17T23:45:58.904 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on legacy crush tunables = false 2024-03-17T23:45:58.905 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on crush straw calc version zero = false 2024-03-17T23:45:58.905 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on no sortbitwise = false 2024-03-17T23:45:58.905 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on osd down out interval zero = false 2024-03-17T23:45:58.905 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on too few osds = false 2024-03-17T23:45:58.905 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-17T23:45:58.905 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-17T23:45:58.905 INFO:teuthology.orchestra.run.smithi157.stdout: mon_allow_pool_size_one = true 2024-03-17T23:45:58.905 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.905 INFO:teuthology.orchestra.run.smithi157.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-17T23:45:58.905 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.905 INFO:teuthology.orchestra.run.smithi157.stdout: osd default data pool replay window = 5 2024-03-17T23:45:58.905 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.905 INFO:teuthology.orchestra.run.smithi157.stdout: mon allow pool delete = true 2024-03-17T23:45:58.905 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.905 INFO:teuthology.orchestra.run.smithi157.stdout: mon cluster log file level = debug 2024-03-17T23:45:58.906 INFO:teuthology.orchestra.run.smithi157.stdout: debug asserts on shutdown = true 2024-03-17T23:45:58.906 INFO:teuthology.orchestra.run.smithi157.stdout: mon health detail to clog = false 2024-03-17T23:45:58.906 INFO:teuthology.orchestra.run.smithi157.stdout: mon host = "172.21.15.150,172.21.15.157,[v2:172.21.15.157:3301,v1:172.21.15.157:6790]" 2024-03-17T23:45:58.906 INFO:teuthology.orchestra.run.smithi157.stdout: fsid = a1c5acd3-e97f-470b-8226-335736a7795b 2024-03-17T23:45:58.906 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.906 INFO:teuthology.orchestra.run.smithi157.stdout:[osd] 2024-03-17T23:45:58.906 INFO:teuthology.orchestra.run.smithi157.stdout: osd journal size = 100 2024-03-17T23:45:58.906 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.906 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub load threshold = 5.0 2024-03-17T23:45:58.906 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub max interval = 600 2024-03-17T23:45:58.906 INFO:teuthology.orchestra.run.smithi157.stdout: osd mclock profile = high_recovery_ops 2024-03-17T23:45:58.906 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.906 INFO:teuthology.orchestra.run.smithi157.stdout: osd recover clone overlap = true 2024-03-17T23:45:58.906 INFO:teuthology.orchestra.run.smithi157.stdout: osd recovery max chunk = 1048576 2024-03-17T23:45:58.907 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.907 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug shutdown = true 2024-03-17T23:45:58.907 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug op order = true 2024-03-17T23:45:58.907 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug verify stray on activate = true 2024-03-17T23:45:58.907 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug trim objects = true 2024-03-17T23:45:58.907 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.907 INFO:teuthology.orchestra.run.smithi157.stdout: osd open classes on start = true 2024-03-17T23:45:58.907 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug pg log writeout = true 2024-03-17T23:45:58.907 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.907 INFO:teuthology.orchestra.run.smithi157.stdout: osd deep scrub update digest min age = 30 2024-03-17T23:45:58.907 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.907 INFO:teuthology.orchestra.run.smithi157.stdout: osd map max advance = 10 2024-03-17T23:45:58.907 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.907 INFO:teuthology.orchestra.run.smithi157.stdout: journal zero on create = true 2024-03-17T23:45:58.907 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.908 INFO:teuthology.orchestra.run.smithi157.stdout: filestore ondisk finisher threads = 3 2024-03-17T23:45:58.908 INFO:teuthology.orchestra.run.smithi157.stdout: filestore apply finisher threads = 3 2024-03-17T23:45:58.908 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.908 INFO:teuthology.orchestra.run.smithi157.stdout: bdev debug aio = true 2024-03-17T23:45:58.908 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug misdirected ops = true 2024-03-17T23:45:58.908 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore block size = 96636764160 2024-03-17T23:45:58.908 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore fsck on mount = True 2024-03-17T23:45:58.908 INFO:teuthology.orchestra.run.smithi157.stdout: debug bluefs = 20 2024-03-17T23:45:58.908 INFO:teuthology.orchestra.run.smithi157.stdout: debug bluestore = 20 2024-03-17T23:45:58.908 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-03-17T23:45:58.908 INFO:teuthology.orchestra.run.smithi157.stdout: debug osd = 20 2024-03-17T23:45:58.908 INFO:teuthology.orchestra.run.smithi157.stdout: debug rocksdb = 10 2024-03-17T23:45:58.908 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd backfillfull_ratio = 0.85 2024-03-17T23:45:58.908 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd full ratio = 0.9 2024-03-17T23:45:58.908 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd nearfull ratio = 0.8 2024-03-17T23:45:58.909 INFO:teuthology.orchestra.run.smithi157.stdout: osd failsafe full ratio = 0.95 2024-03-17T23:45:58.909 INFO:teuthology.orchestra.run.smithi157.stdout: osd objectstore = bluestore 2024-03-17T23:45:58.909 INFO:teuthology.orchestra.run.smithi157.stdout: osd op complaint time = 180 2024-03-17T23:45:58.909 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.909 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr] 2024-03-17T23:45:58.909 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-03-17T23:45:58.909 INFO:teuthology.orchestra.run.smithi157.stdout: debug mgr = 20 2024-03-17T23:45:58.909 INFO:teuthology.orchestra.run.smithi157.stdout: debug mon = 20 2024-03-17T23:45:58.909 INFO:teuthology.orchestra.run.smithi157.stdout: debug auth = 20 2024-03-17T23:45:58.909 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min pgs per osd = 4 2024-03-17T23:45:58.909 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min bytes per osd = 10 2024-03-17T23:45:58.909 INFO:teuthology.orchestra.run.smithi157.stdout: mgr/telemetry/nag = false 2024-03-17T23:45:58.909 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.909 INFO:teuthology.orchestra.run.smithi157.stdout:[mon] 2024-03-17T23:45:58.909 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-03-17T23:45:58.910 INFO:teuthology.orchestra.run.smithi157.stdout: debug mon = 20 2024-03-17T23:45:58.910 INFO:teuthology.orchestra.run.smithi157.stdout: debug paxos = 20 2024-03-17T23:45:58.910 INFO:teuthology.orchestra.run.smithi157.stdout: debug auth = 20 2024-03-17T23:45:58.910 INFO:teuthology.orchestra.run.smithi157.stdout: mon data avail warn = 5 2024-03-17T23:45:58.910 INFO:teuthology.orchestra.run.smithi157.stdout: mon mgr mkfs grace = 240 2024-03-17T23:45:58.910 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min pgs per osd = 4 2024-03-17T23:45:58.910 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd reporter subtree level = osd 2024-03-17T23:45:58.910 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd prime pg temp = true 2024-03-17T23:45:58.910 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min bytes per osd = 10 2024-03-17T23:45:58.910 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.910 INFO:teuthology.orchestra.run.smithi157.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-17T23:45:58.910 INFO:teuthology.orchestra.run.smithi157.stdout: auth mon ticket ttl = 660 # 11m 2024-03-17T23:45:58.910 INFO:teuthology.orchestra.run.smithi157.stdout: auth service ticket ttl = 240 # 4m 2024-03-17T23:45:58.910 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.911 INFO:teuthology.orchestra.run.smithi157.stdout: # don't complain about insecure global_id in the test suite 2024-03-17T23:45:58.911 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-17T23:45:58.911 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-17T23:45:58.911 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.911 INFO:teuthology.orchestra.run.smithi157.stdout: # 1m isn't quite enough 2024-03-17T23:45:58.911 INFO:teuthology.orchestra.run.smithi157.stdout: mon_down_mkfs_grace = 2m 2024-03-17T23:45:58.911 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.911 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_filestore_osds = false 2024-03-17T23:45:58.911 INFO:teuthology.orchestra.run.smithi157.stdout: mon op complaint time = 120 2024-03-17T23:45:58.911 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-03-17T23:45:58.911 INFO:teuthology.orchestra.run.smithi157.stdout:[client] 2024-03-17T23:45:58.911 INFO:teuthology.orchestra.run.smithi157.stdout: rgw cache enabled = true 2024-03-17T23:45:58.911 INFO:teuthology.orchestra.run.smithi157.stdout: rgw enable ops log = true 2024-03-17T23:45:58.911 INFO:teuthology.orchestra.run.smithi157.stdout: rgw enable usage log = true 2024-03-17T23:45:58.911 INFO:teuthology.orchestra.run.smithi157.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-17T23:45:58.912 INFO:teuthology.orchestra.run.smithi157.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-17T23:45:58.912 INFO:teuthology.orchestra.run.smithi157.stdout: client mount timeout = 600 2024-03-17T23:45:58.912 INFO:teuthology.orchestra.run.smithi157.stdout: debug client = 20 2024-03-17T23:45:58.912 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-03-17T23:45:58.912 INFO:teuthology.orchestra.run.smithi157.stdout: pid file = "" 2024-03-17T23:45:58.912 INFO:teuthology.orchestra.run.smithi157.stdout: rados mon op timeout = 900 2024-03-17T23:45:58.912 INFO:teuthology.orchestra.run.smithi157.stdout: rados osd op timeout = 900 2024-03-17T23:45:58.912 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.a] 2024-03-17T23:45:58.912 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.b] 2024-03-17T23:45:58.912 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.c] 2024-03-17T23:45:58.912 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.a] 2024-03-17T23:45:58.912 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.c] 2024-03-17T23:45:58.912 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.b] 2024-03-17T23:45:58.912 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.d] 2024-03-17T23:45:58.912 INFO:teuthology.orchestra.run.smithi157.stdout:[mds] 2024-03-17T23:45:58.913 INFO:teuthology.orchestra.run.smithi157.stdout: debug mds = 20 2024-03-17T23:45:58.913 INFO:teuthology.orchestra.run.smithi157.stdout: debug mds balancer = 20 2024-03-17T23:45:58.913 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-03-17T23:45:58.913 INFO:teuthology.orchestra.run.smithi157.stdout: mds debug frag = True 2024-03-17T23:45:58.913 INFO:teuthology.orchestra.run.smithi157.stdout: mds debug scatterstat = True 2024-03-17T23:45:58.913 INFO:teuthology.orchestra.run.smithi157.stdout: mds op complaint time = 180 2024-03-17T23:45:58.913 INFO:teuthology.orchestra.run.smithi157.stdout: mds verify scatter = True 2024-03-17T23:45:58.913 INFO:teuthology.orchestra.run.smithi157.stdout: osd op complaint time = 180 2024-03-17T23:45:58.913 INFO:teuthology.orchestra.run.smithi157.stdout: rados mon op timeout = 900 2024-03-17T23:45:58.913 INFO:teuthology.orchestra.run.smithi157.stdout: rados osd op timeout = 900 2024-03-17T23:45:58.914 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-17T23:45:58.915 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-17T23:45:59.011 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-17T23:45:59.011 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:45:59.011 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-17T23:45:59.026 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:45:59.026 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-17T23:45:59.081 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi150.front.sepia.ceph.com 2024-03-17T23:45:59.081 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:45:59.081 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-17T23:45:59.081 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-17T23:45:59.160 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:45:59.160 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-17T23:45:59.215 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi157.front.sepia.ceph.com 2024-03-17T23:45:59.215 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:45:59.215 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-17T23:45:59.215 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-17T23:45:59.255 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:45:59.256 DEBUG:teuthology.orchestra.run.smithi157:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-17T23:45:59.310 INFO:tasks.ceph:Setting up mon nodes... 2024-03-17T23:45:59.310 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-17T23:45:59.310 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-17T23:45:59.365 INFO:teuthology.orchestra.run.smithi150.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-17T23:45:59.367 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-17T23:45:59.439 INFO:teuthology.orchestra.run.smithi157.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-17T23:45:59.441 INFO:tasks.ceph:Setting up mds nodes... 2024-03-17T23:45:59.441 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-17T23:45:59.496 INFO:teuthology.orchestra.run.smithi150.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-17T23:45:59.499 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-17T23:45:59.525 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-17T23:45:59.620 INFO:teuthology.orchestra.run.smithi150.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-17T23:45:59.622 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-17T23:45:59.650 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-17T23:45:59.705 INFO:teuthology.orchestra.run.smithi157.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-17T23:45:59.707 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-17T23:45:59.734 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-17T23:45:59.830 INFO:teuthology.orchestra.run.smithi157.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-17T23:45:59.832 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-17T23:45:59.859 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-17T23:45:59.859 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-17T23:45:59.901 INFO:teuthology.orchestra.run.smithi150.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-17T23:45:59.917 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-17T23:46:00.000 INFO:teuthology.orchestra.run.smithi150.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-17T23:46:00.015 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-17T23:46:00.057 INFO:teuthology.orchestra.run.smithi157.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-17T23:46:00.072 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-17T23:46:00.154 INFO:teuthology.orchestra.run.smithi157.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-17T23:46:00.169 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-17T23:46:00.170 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi150.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi157.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-17T23:46:00.170 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-17T23:46:00.197 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-17T23:46:00.197 INFO:tasks.ceph:role: osd.0 2024-03-17T23:46:00.197 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi150.front.sepia.ceph.com 2024-03-17T23:46:00.197 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-17T23:46:00.271 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-17T23:46:00.271 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-17T23:46:00.271 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-17T23:46:00.271 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-17T23:46:00.271 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-17T23:46:00.272 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2024-03-17T23:46:00.272 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-17T23:46:00.272 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-17T23:46:00.272 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-17T23:46:00.272 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-17T23:46:00.792 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2024-03-17T23:46:00.793 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2024-03-17T23:46:00.794 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-17T23:46:00.923 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-17T23:46:00.959 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-17T23:46:01.027 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-17T23:46:01.027 INFO:tasks.ceph:role: osd.1 2024-03-17T23:46:01.027 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi150.front.sepia.ceph.com 2024-03-17T23:46:01.027 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-17T23:46:01.102 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-17T23:46:01.102 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-17T23:46:01.102 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-17T23:46:01.102 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-17T23:46:01.102 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-17T23:46:01.102 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2024-03-17T23:46:01.102 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-17T23:46:01.102 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-17T23:46:01.102 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-17T23:46:01.102 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-17T23:46:01.547 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2024-03-17T23:46:01.549 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2024-03-17T23:46:01.549 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-17T23:46:01.644 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-17T23:46:01.676 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-17T23:46:01.744 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-17T23:46:01.744 INFO:tasks.ceph:role: osd.2 2024-03-17T23:46:01.744 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi150.front.sepia.ceph.com 2024-03-17T23:46:01.744 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-17T23:46:01.819 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-17T23:46:01.819 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-17T23:46:01.819 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-17T23:46:01.819 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-17T23:46:01.819 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-17T23:46:01.819 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2024-03-17T23:46:01.819 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-17T23:46:01.819 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-17T23:46:01.819 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-17T23:46:01.819 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-17T23:46:02.315 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2024-03-17T23:46:02.315 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2024-03-17T23:46:02.315 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-17T23:46:02.470 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-17T23:46:02.506 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-17T23:46:02.574 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-17T23:46:02.575 INFO:tasks.ceph:role: osd.3 2024-03-17T23:46:02.575 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi150.front.sepia.ceph.com 2024-03-17T23:46:02.575 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-17T23:46:02.650 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-17T23:46:02.650 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-17T23:46:02.650 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-17T23:46:02.650 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-17T23:46:02.650 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-17T23:46:02.650 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2024-03-17T23:46:02.650 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-17T23:46:02.650 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-17T23:46:02.650 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-17T23:46:02.650 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-17T23:46:03.126 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2024-03-17T23:46:03.128 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2024-03-17T23:46:03.128 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-17T23:46:03.274 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-17T23:46:03.308 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-17T23:46:03.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:03.421+0000 7f6533e1e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-17T23:46:03.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:03.421+0000 7f6533e1e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-17T23:46:03.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:03.421+0000 7f6533e1e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-17T23:46:03.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:03.421+0000 7f6533e1e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-17T23:46:03.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:03.421+0000 7f6533e1e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-17T23:46:03.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:03.422+0000 7f6533e1e640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-17T23:46:06.275 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-17T23:46:06.307 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-17T23:46:06.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:06.395+0000 7faa80ac6640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-17T23:46:06.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:06.395+0000 7faa80ac6640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-17T23:46:06.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:06.395+0000 7faa80ac6640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-17T23:46:06.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:06.395+0000 7faa80ac6640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-17T23:46:06.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:06.395+0000 7faa80ac6640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-17T23:46:06.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:06.396+0000 7faa80ac6640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-17T23:46:09.263 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-17T23:46:09.294 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-17T23:46:09.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:09.384+0000 7f35cde9d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-17T23:46:09.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:09.384+0000 7f35cde9d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-17T23:46:09.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:09.384+0000 7f35cde9d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-17T23:46:09.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:09.384+0000 7f35cde9d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-17T23:46:09.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:09.384+0000 7f35cde9d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-17T23:46:09.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:09.385+0000 7f35cde9d640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-17T23:46:12.241 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-17T23:46:12.272 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-17T23:46:12.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:12.361+0000 7f33e6e1f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-17T23:46:12.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:12.361+0000 7f33e6e1f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-17T23:46:12.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:12.361+0000 7f33e6e1f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-17T23:46:12.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:12.361+0000 7f33e6e1f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-17T23:46:12.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:12.361+0000 7f33e6e1f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-17T23:46:12.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:12.362+0000 7f33e6e1f640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-17T23:46:15.224 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-17T23:46:15.254 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-17T23:46:15.286 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-17T23:46:15.287 INFO:tasks.ceph:role: osd.4 2024-03-17T23:46:15.287 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-17T23:46:15.287 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-17T23:46:15.357 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-17T23:46:15.358 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-17T23:46:15.358 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-17T23:46:15.358 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-17T23:46:15.358 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-17T23:46:15.358 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-03-17T23:46:15.358 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-17T23:46:15.358 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-17T23:46:15.358 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-17T23:46:15.358 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-17T23:46:15.389 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-03-17T23:46:15.392 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-03-17T23:46:15.392 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-17T23:46:15.465 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-17T23:46:15.536 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-17T23:46:15.605 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-17T23:46:15.605 INFO:tasks.ceph:role: osd.5 2024-03-17T23:46:15.605 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-17T23:46:15.605 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-17T23:46:15.672 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-17T23:46:15.672 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-17T23:46:15.672 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-17T23:46:15.672 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-17T23:46:15.673 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-17T23:46:15.673 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-03-17T23:46:15.673 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-17T23:46:15.673 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-17T23:46:15.673 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-17T23:46:15.673 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-17T23:46:15.704 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-03-17T23:46:15.706 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-03-17T23:46:15.706 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-17T23:46:15.782 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-17T23:46:15.856 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-17T23:46:15.923 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-17T23:46:15.923 INFO:tasks.ceph:role: osd.6 2024-03-17T23:46:15.923 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-17T23:46:15.924 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-17T23:46:15.990 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-17T23:46:15.990 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-17T23:46:15.990 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-17T23:46:15.990 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-17T23:46:15.990 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-17T23:46:15.990 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-03-17T23:46:15.990 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-17T23:46:15.990 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-17T23:46:15.990 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-17T23:46:15.990 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-17T23:46:16.022 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-03-17T23:46:16.023 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-03-17T23:46:16.024 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-17T23:46:16.094 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-17T23:46:16.169 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-17T23:46:16.236 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-17T23:46:16.236 INFO:tasks.ceph:role: osd.7 2024-03-17T23:46:16.236 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-17T23:46:16.236 DEBUG:teuthology.orchestra.run.smithi157:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-17T23:46:16.303 INFO:teuthology.orchestra.run.smithi157.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-17T23:46:16.303 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-17T23:46:16.303 INFO:teuthology.orchestra.run.smithi157.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-17T23:46:16.303 INFO:teuthology.orchestra.run.smithi157.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-17T23:46:16.303 INFO:teuthology.orchestra.run.smithi157.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-17T23:46:16.303 INFO:teuthology.orchestra.run.smithi157.stdout: = sunit=0 swidth=0 blks 2024-03-17T23:46:16.303 INFO:teuthology.orchestra.run.smithi157.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-17T23:46:16.303 INFO:teuthology.orchestra.run.smithi157.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-17T23:46:16.303 INFO:teuthology.orchestra.run.smithi157.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-17T23:46:16.303 INFO:teuthology.orchestra.run.smithi157.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-17T23:46:16.335 INFO:teuthology.orchestra.run.smithi157.stdout:Discarding blocks...Done. 2024-03-17T23:46:16.337 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi157.front.sepia.ceph.com -o noatime 2024-03-17T23:46:16.337 DEBUG:teuthology.orchestra.run.smithi157:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-17T23:46:16.412 DEBUG:teuthology.orchestra.run.smithi157:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-17T23:46:16.483 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-17T23:46:16.577 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:16.576+0000 7efd9599e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-17T23:46:16.577 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:16.576+0000 7efd9599e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-17T23:46:16.577 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:16.576+0000 7efd9599e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-17T23:46:16.577 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:16.576+0000 7efd9599e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-17T23:46:16.577 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:16.576+0000 7efd9599e640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-17T23:46:16.578 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:16.576+0000 7efd9599e640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-17T23:46:19.419 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-17T23:46:19.450 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-17T23:46:19.544 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:19.542+0000 7fc428017640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-17T23:46:19.544 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:19.542+0000 7fc428017640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-17T23:46:19.544 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:19.542+0000 7fc428017640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-17T23:46:19.544 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:19.542+0000 7fc428017640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-17T23:46:19.544 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:19.542+0000 7fc428017640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-17T23:46:19.544 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:19.543+0000 7fc428017640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-17T23:46:22.395 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-17T23:46:22.426 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-17T23:46:22.518 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:22.517+0000 7fc9acc9b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-17T23:46:22.518 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:22.517+0000 7fc9acc9b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-17T23:46:22.518 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:22.517+0000 7fc9acc9b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-17T23:46:22.518 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:22.517+0000 7fc9acc9b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-17T23:46:22.519 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:22.517+0000 7fc9acc9b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-17T23:46:22.519 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:22.517+0000 7fc9acc9b640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-17T23:46:25.374 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-17T23:46:25.405 DEBUG:teuthology.orchestra.run.smithi157:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-17T23:46:25.497 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:25.495+0000 7ff4826a0640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-17T23:46:25.497 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:25.496+0000 7ff4826a0640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-17T23:46:25.497 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:25.496+0000 7ff4826a0640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-17T23:46:25.497 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:25.496+0000 7ff4826a0640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-17T23:46:25.497 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:25.496+0000 7ff4826a0640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-17T23:46:25.497 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:25.496+0000 7ff4826a0640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-17T23:46:26.942 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-17T23:46:26.945 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-17T23:46:28.345 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-17T23:46:28.374 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-17T23:46:28.374 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:46:28.374 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-17T23:46:28.406 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:46:28.406 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-17T23:46:28.476 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:46:28.476 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-17T23:46:28.546 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:46:28.546 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-17T23:46:28.615 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:46:28.615 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-17T23:46:28.683 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:46:28.684 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-17T23:46:28.753 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:46:28.753 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-17T23:46:28.820 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:46:28.821 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-17T23:46:28.852 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:46:28.853 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-17T23:46:28.924 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:46:28.924 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-17T23:46:28.993 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:46:28.994 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-17T23:46:29.063 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:46:29.063 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-17T23:46:29.132 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:46:29.132 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-17T23:46:29.201 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:46:29.201 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-17T23:46:29.269 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:46:29.269 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-17T23:46:29.285 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:46:29.285 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-17T23:46:29.341 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:46:29.341 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-17T23:46:29.356 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:46:29.356 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-17T23:46:29.411 INFO:tasks.ceph:Adding keys to all mons... 2024-03-17T23:46:29.411 DEBUG:teuthology.orchestra.run.smithi150:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-17T23:46:29.414 DEBUG:teuthology.orchestra.run.smithi157:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-17T23:46:29.454 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr.y] 2024-03-17T23:46:29.455 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQC3gPdlQ9jOFRAAo0eZ4CMoH7heIX+sa4IuCw== 2024-03-17T23:46:29.455 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.a] 2024-03-17T23:46:29.455 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQC3gPdl0bOfHRAAqIf9ExwsoDeTZAbP6JhU/w== 2024-03-17T23:46:29.455 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.c] 2024-03-17T23:46:29.455 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQC3gPdlJtcBJRAAsxXTt1otrV0PIJnP9p6CBQ== 2024-03-17T23:46:29.455 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.0] 2024-03-17T23:46:29.455 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQC7gPdlymEuGRAA0wyRX+r9Pocq1nrDOjfTfQ== 2024-03-17T23:46:29.455 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.1] 2024-03-17T23:46:29.455 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQC+gPdl87WwFxAAO0iNw6UDAJ1I3WxmTusxFQ== 2024-03-17T23:46:29.455 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.2] 2024-03-17T23:46:29.455 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQDBgPdlYVQIFxAANc97g+jl1RHahJS9BS2Kgw== 2024-03-17T23:46:29.455 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.3] 2024-03-17T23:46:29.455 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQDEgPdlmaOqFRAAoYhWrnWdTC4LK0LB1SfSwg== 2024-03-17T23:46:29.456 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr.x] 2024-03-17T23:46:29.456 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQC3gPdlPDo1GhAAO8CxqquWsZg4j7EipcmOcQ== 2024-03-17T23:46:29.456 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.b] 2024-03-17T23:46:29.456 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQC3gPdlSlEVKhAAAScr6WsFt5/c2N1ns90pOg== 2024-03-17T23:46:29.456 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.d] 2024-03-17T23:46:29.456 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQC3gPdlt7+DMRAAzv9aOR4QsJBCNHpUgSNQPg== 2024-03-17T23:46:29.456 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.4] 2024-03-17T23:46:29.456 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQDIgPdlxX5rIhAAXHKRWgfegyAAfb/57c/xvQ== 2024-03-17T23:46:29.456 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.5] 2024-03-17T23:46:29.456 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQDLgPdlf6xmIBAAK1d5IdCTjOHaR6yf41n/pQ== 2024-03-17T23:46:29.456 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.6] 2024-03-17T23:46:29.456 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQDOgPdljVrnHhAAtIBm9CDQeN44WpOpUF3g3Q== 2024-03-17T23:46:29.456 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.7] 2024-03-17T23:46:29.456 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQDRgPdlZDqfHRAAkkbz3Blr04H2uQsXjrQgRA== 2024-03-17T23:46:29.456 INFO:teuthology.orchestra.run.smithi150.stdout:[client.0] 2024-03-17T23:46:29.457 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQC3gPdlvlPJNRAAkUfetPJK4hZQG5rNiSDozw== 2024-03-17T23:46:29.457 INFO:teuthology.orchestra.run.smithi150.stdout:[client.2] 2024-03-17T23:46:29.457 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQC4gPdlvBwIABAA9kdWclUPy/mGtbT4dxfoxA== 2024-03-17T23:46:29.457 INFO:teuthology.orchestra.run.smithi150.stdout:[client.1] 2024-03-17T23:46:29.457 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQC4gPdlNqN0AxAA4FezHbC/cqNvSbsaThwfNw== 2024-03-17T23:46:29.457 INFO:teuthology.orchestra.run.smithi150.stdout:[client.3] 2024-03-17T23:46:29.457 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQC4gPdlbNtBCRAADYXS1k+3rScOubs0QA7doQ== 2024-03-17T23:46:29.477 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr.y] 2024-03-17T23:46:29.477 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQC3gPdlQ9jOFRAAo0eZ4CMoH7heIX+sa4IuCw== 2024-03-17T23:46:29.477 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.a] 2024-03-17T23:46:29.477 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQC3gPdl0bOfHRAAqIf9ExwsoDeTZAbP6JhU/w== 2024-03-17T23:46:29.477 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.c] 2024-03-17T23:46:29.477 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQC3gPdlJtcBJRAAsxXTt1otrV0PIJnP9p6CBQ== 2024-03-17T23:46:29.478 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.0] 2024-03-17T23:46:29.478 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQC7gPdlymEuGRAA0wyRX+r9Pocq1nrDOjfTfQ== 2024-03-17T23:46:29.478 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.1] 2024-03-17T23:46:29.478 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQC+gPdl87WwFxAAO0iNw6UDAJ1I3WxmTusxFQ== 2024-03-17T23:46:29.478 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.2] 2024-03-17T23:46:29.478 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDBgPdlYVQIFxAANc97g+jl1RHahJS9BS2Kgw== 2024-03-17T23:46:29.478 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.3] 2024-03-17T23:46:29.478 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDEgPdlmaOqFRAAoYhWrnWdTC4LK0LB1SfSwg== 2024-03-17T23:46:29.478 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr.x] 2024-03-17T23:46:29.478 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQC3gPdlPDo1GhAAO8CxqquWsZg4j7EipcmOcQ== 2024-03-17T23:46:29.478 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.b] 2024-03-17T23:46:29.478 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQC3gPdlSlEVKhAAAScr6WsFt5/c2N1ns90pOg== 2024-03-17T23:46:29.478 INFO:teuthology.orchestra.run.smithi157.stdout:[mds.d] 2024-03-17T23:46:29.479 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQC3gPdlt7+DMRAAzv9aOR4QsJBCNHpUgSNQPg== 2024-03-17T23:46:29.479 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.4] 2024-03-17T23:46:29.479 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDIgPdlxX5rIhAAXHKRWgfegyAAfb/57c/xvQ== 2024-03-17T23:46:29.479 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.5] 2024-03-17T23:46:29.479 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDLgPdlf6xmIBAAK1d5IdCTjOHaR6yf41n/pQ== 2024-03-17T23:46:29.479 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.6] 2024-03-17T23:46:29.479 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDOgPdljVrnHhAAtIBm9CDQeN44WpOpUF3g3Q== 2024-03-17T23:46:29.479 INFO:teuthology.orchestra.run.smithi157.stdout:[osd.7] 2024-03-17T23:46:29.479 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQDRgPdlZDqfHRAAkkbz3Blr04H2uQsXjrQgRA== 2024-03-17T23:46:29.479 INFO:teuthology.orchestra.run.smithi157.stdout:[client.0] 2024-03-17T23:46:29.479 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQC3gPdlvlPJNRAAkUfetPJK4hZQG5rNiSDozw== 2024-03-17T23:46:29.479 INFO:teuthology.orchestra.run.smithi157.stdout:[client.2] 2024-03-17T23:46:29.479 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQC4gPdlvBwIABAA9kdWclUPy/mGtbT4dxfoxA== 2024-03-17T23:46:29.479 INFO:teuthology.orchestra.run.smithi157.stdout:[client.1] 2024-03-17T23:46:29.479 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQC4gPdlNqN0AxAA4FezHbC/cqNvSbsaThwfNw== 2024-03-17T23:46:29.479 INFO:teuthology.orchestra.run.smithi157.stdout:[client.3] 2024-03-17T23:46:29.480 INFO:teuthology.orchestra.run.smithi157.stdout: key = AQC4gPdlbNtBCRAADYXS1k+3rScOubs0QA7doQ== 2024-03-17T23:46:29.481 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-17T23:46:29.501 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-17T23:46:29.573 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-17T23:46:29.576 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-17T23:46:29.632 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-17T23:46:29.635 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-17T23:46:29.690 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-17T23:46:29.693 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-17T23:46:29.745 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-17T23:46:29.748 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-17T23:46:29.801 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-17T23:46:29.803 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-17T23:46:29.855 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-17T23:46:29.858 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-17T23:46:29.910 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-17T23:46:29.912 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-17T23:46:29.964 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-17T23:46:29.966 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-17T23:46:30.021 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-17T23:46:30.024 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-17T23:46:30.078 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-17T23:46:30.080 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-17T23:46:30.132 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-17T23:46:30.135 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-17T23:46:30.187 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-17T23:46:30.189 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-17T23:46:30.241 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-17T23:46:30.243 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-17T23:46:30.295 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-17T23:46:30.298 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-17T23:46:30.354 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-17T23:46:30.356 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-17T23:46:30.412 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-17T23:46:30.414 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-17T23:46:30.469 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-17T23:46:30.472 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-17T23:46:30.526 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-17T23:46:30.526 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-17T23:46:30.553 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-17T23:46:30.652 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-17T23:46:30.679 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-17T23:46:30.707 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-17T23:46:30.806 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-17T23:46:30.833 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-17T23:46:30.900 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-17T23:46:30.998 DEBUG:teuthology.orchestra.run.smithi157:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-17T23:46:31.025 DEBUG:teuthology.orchestra.run.smithi150:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-17T23:46:31.028 DEBUG:teuthology.orchestra.run.smithi157:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-17T23:46:31.080 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-17T23:46:31.081 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-17T23:46:31.081 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-17T23:46:31.084 INFO:tasks.ceph.mon.a:Started 2024-03-17T23:46:31.084 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-17T23:46:31.084 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-17T23:46:31.123 INFO:tasks.ceph.mon.b:Started 2024-03-17T23:46:31.123 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-17T23:46:31.124 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-17T23:46:31.126 INFO:tasks.ceph.mon.c:Started 2024-03-17T23:46:31.126 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-17T23:46:31.126 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-17T23:46:31.126 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-17T23:46:31.129 INFO:tasks.ceph.mgr.y:Started 2024-03-17T23:46:31.129 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-17T23:46:31.129 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-17T23:46:31.131 INFO:tasks.ceph.mgr.x:Started 2024-03-17T23:46:31.131 INFO:tasks.ceph:Setting crush tunables to default 2024-03-17T23:46:31.131 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd crush tunables default 2024-03-17T23:46:31.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.232+0000 7f65fd097640 1 Processor -- start 2024-03-17T23:46:31.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.232+0000 7f65fd097640 1 -- start start 2024-03-17T23:46:31.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.233+0000 7f65fd097640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65f8165a80 0x7f65f80b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:31.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.233+0000 7f65fd097640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f65f806b8a0 0x7f65f81633e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:31.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.233+0000 7f65fd097640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65f80b5810 con 0x7f65f8163a70 2024-03-17T23:46:31.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.233+0000 7f65fd097640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f65f8163ed0 con 0x7f65f806b8a0 2024-03-17T23:46:31.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.233+0000 7f65fd097640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f65f80b5a80 con 0x7f65f8165a80 2024-03-17T23:46:31.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.233+0000 7f65f6d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65f8165a80 0x7f65f80b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:31.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.233+0000 7f65f759e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f65f806b8a0 0x7f65f81633e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:31.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.233+0000 7f65f6d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65f8165a80 0x7f65f80b3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:37850/0 (socket says 172.21.15.150:37850) 2024-03-17T23:46:31.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.233+0000 7f65f659c640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f65f8163a70 0x7f65f806b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:53572/0 (socket says 172.21.15.150:53572) 2024-03-17T23:46:31.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.233+0000 7f65f6d9d640 1 -- 172.21.15.150:0/3047862389 learned_addr learned my addr 172.21.15.150:0/3047862389 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:31.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.233+0000 7f65f759e640 1 -- 172.21.15.150:0/3047862389 >> v2:172.21.15.157:3300/0 conn(0x7f65f806b8a0 msgr2=0x7f65f81633e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-17T23:46:31.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.233+0000 7f65f759e640 1 -- 172.21.15.150:0/3047862389 >> v2:172.21.15.157:3300/0 conn(0x7f65f806b8a0 msgr2=0x7f65f81633e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-17T23:46:31.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.233+0000 7f65f759e640 1 --2- 172.21.15.150:0/3047862389 >> v2:172.21.15.157:3300/0 conn(0x7f65f806b8a0 0x7f65f81633e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-17T23:46:31.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.233+0000 7f65f759e640 1 --2- 172.21.15.150:0/3047862389 >> v2:172.21.15.157:3300/0 conn(0x7f65f806b8a0 0x7f65f81633e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-17T23:46:31.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.234+0000 7f65f6d9d640 1 -- 172.21.15.150:0/3047862389 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65f8165a80 msgr2=0x7f65f80b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-17T23:46:31.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.234+0000 7f65f6d9d640 1 -- 172.21.15.150:0/3047862389 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65f8165a80 msgr2=0x7f65f80b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-17T23:46:31.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.234+0000 7f65f6d9d640 1 --2- 172.21.15.150:0/3047862389 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65f8165a80 0x7f65f80b3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-17T23:46:31.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.234+0000 7f65f6d9d640 1 --2- 172.21.15.150:0/3047862389 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65f8165a80 0x7f65f80b3a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-17T23:46:31.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.412+0000 7f65f5d9b640 1 -- 172.21.15.150:0/3047862389 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 80701421 0 0) 0x7f65f80b5810 con 0x7f65f8163a70 2024-03-17T23:46:31.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.412+0000 7f65f5d9b640 1 -- 172.21.15.150:0/3047862389 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65e0003520 con 0x7f65f8163a70 2024-03-17T23:46:31.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.413+0000 7f65f5d9b640 1 -- 172.21.15.150:0/3047862389 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1582012515 0 0) 0x7f65e0003520 con 0x7f65f8163a70 2024-03-17T23:46:31.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.413+0000 7f65f5d9b640 1 -- 172.21.15.150:0/3047862389 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65f8165a80 msgr2=0x7f65f80b3a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-17T23:46:31.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.413+0000 7f65f5d9b640 1 --2- 172.21.15.150:0/3047862389 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65f8165a80 0x7f65f80b3a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:31.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.413+0000 7f65f5d9b640 1 -- 172.21.15.150:0/3047862389 >> v2:172.21.15.157:3300/0 conn(0x7f65f806b8a0 msgr2=0x7f65f81633e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-17T23:46:31.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.413+0000 7f65f5d9b640 1 --2- 172.21.15.150:0/3047862389 >> v2:172.21.15.157:3300/0 conn(0x7f65f806b8a0 0x7f65f81633e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:31.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.413+0000 7f65f5d9b640 1 -- 172.21.15.150:0/3047862389 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65f80b5d60 con 0x7f65f8163a70 2024-03-17T23:46:31.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.413+0000 7f65f5d9b640 1 -- 172.21.15.150:0/3047862389 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f65ec001670 con 0x7f65f8163a70 2024-03-17T23:46:31.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.413+0000 7f65f5d9b640 1 -- 172.21.15.150:0/3047862389 >> v1:172.21.15.157:6789/0 conn(0x7f65f8163a70 legacy=0x7f65f806b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:31.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.413+0000 7f65f5d9b640 1 --2- 172.21.15.150:0/3047862389 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65e00039b0 0x7f65e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:31.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.413+0000 7f65f5d9b640 1 --2- 172.21.15.150:0/3047862389 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65e0004800 0x7f65e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:31.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.413+0000 7f65f5d9b640 1 --2- 172.21.15.150:0/3047862389 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65f8165a80 0x7f65e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:31.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.413+0000 7f65f5d9b640 1 -- 172.21.15.150:0/3047862389 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f65f8163ed0 con 0x7f65e0004800 2024-03-17T23:46:31.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.413+0000 7f65f5d9b640 1 -- 172.21.15.150:0/3047862389 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f65f80b5a80 con 0x7f65e00039b0 2024-03-17T23:46:31.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.413+0000 7f65f5d9b640 1 -- 172.21.15.150:0/3047862389 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f65e000bac0 con 0x7f65f8165a80 2024-03-17T23:46:31.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.413+0000 7f65f659c640 1 --2- 172.21.15.150:0/3047862389 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65e0004800 0x7f65e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:31.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.413+0000 7f65f6d9d640 1 --2- 172.21.15.150:0/3047862389 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65e00039b0 0x7f65e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:31.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.413+0000 7f65f759e640 1 --2- 172.21.15.150:0/3047862389 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65f8165a80 0x7f65e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:31.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.413+0000 7f65f659c640 1 -- 172.21.15.150:0/3047862389 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65f8165a80 msgr2=0x7f65e00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:31.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.413+0000 7f65f659c640 1 --2- 172.21.15.150:0/3047862389 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65f8165a80 0x7f65e00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:31.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.413+0000 7f65f659c640 1 -- 172.21.15.150:0/3047862389 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65e00039b0 msgr2=0x7f65e0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:31.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.413+0000 7f65f659c640 1 --2- 172.21.15.150:0/3047862389 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65e00039b0 0x7f65e0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:31.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.413+0000 7f65f659c640 1 -- 172.21.15.150:0/3047862389 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65e0003520 con 0x7f65e0004800 2024-03-17T23:46:31.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.414+0000 7f65f659c640 1 --2- 172.21.15.150:0/3047862389 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65e0004800 0x7f65e0004c80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f65ec001eb0 tx=0x7f65e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b63e2f1e804d12 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:31.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.415+0000 7f65f5d9b640 1 -- 172.21.15.150:0/3047862389 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ec007be0 con 0x7f65e0004800 2024-03-17T23:46:31.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.415+0000 7f65f5d9b640 1 -- 172.21.15.150:0/3047862389 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65ec030070 con 0x7f65e0004800 2024-03-17T23:46:31.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.415+0000 7f65f5d9b640 1 -- 172.21.15.150:0/3047862389 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ec0074b0 con 0x7f65e0004800 2024-03-17T23:46:31.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.416+0000 7f65fd097640 1 -- 172.21.15.150:0/3047862389 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65e0004800 msgr2=0x7f65e0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:31.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.416+0000 7f65fd097640 1 --2- 172.21.15.150:0/3047862389 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65e0004800 0x7f65e0004c80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f65ec001eb0 tx=0x7f65e0005830 comp rx=0 tx=0).stop 2024-03-17T23:46:31.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.416+0000 7f65f759e640 1 -- 172.21.15.150:0/3047862389 reap_dead start 2024-03-17T23:46:31.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.416+0000 7f65fd097640 1 -- 172.21.15.150:0/3047862389 shutdown_connections 2024-03-17T23:46:31.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.416+0000 7f65fd097640 1 -- 172.21.15.150:0/3047862389 >> 172.21.15.150:0/3047862389 conn(0x7f65f80b6170 msgr2=0x7f65f8099a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:31.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.416+0000 7f65fd097640 1 -- 172.21.15.150:0/3047862389 shutdown_connections 2024-03-17T23:46:31.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.416+0000 7f65fd097640 1 -- 172.21.15.150:0/3047862389 wait complete. 2024-03-17T23:46:31.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.418+0000 7f65fd097640 1 Processor -- start 2024-03-17T23:46:31.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.418+0000 7f65fd097640 1 -- start start 2024-03-17T23:46:31.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.419+0000 7f65fd097640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65e0004800 0x7f65f809d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:31.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.419+0000 7f65fd097640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65f8151150 0x7f65f8161490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:31.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.419+0000 7f65fd097640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65f8144890 0x7f65f8144d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:31.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.419+0000 7f65fd097640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f65f80b3bb0 con 0x7f65e0004800 2024-03-17T23:46:31.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.419+0000 7f65fd097640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f65f80b3a30 con 0x7f65f8144890 2024-03-17T23:46:31.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.419+0000 7f65fd097640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f65f80b3d30 con 0x7f65f8151150 2024-03-17T23:46:31.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.419+0000 7f65f659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65f8151150 0x7f65f8161490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:31.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.419+0000 7f65f759e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65f8144890 0x7f65f8144d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:31.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.419+0000 7f65f659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65f8151150 0x7f65f8161490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:37896/0 (socket says 172.21.15.150:37896) 2024-03-17T23:46:31.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.419+0000 7f65f659c640 1 -- 172.21.15.150:0/2485135596 learned_addr learned my addr 172.21.15.150:0/2485135596 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:31.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.419+0000 7f65f6d9d640 1 --2- 172.21.15.150:0/2485135596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65e0004800 0x7f65f809d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:31.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.419+0000 7f65f659c640 1 -- 172.21.15.150:0/2485135596 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65f8144890 msgr2=0x7f65f8144d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:31.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.419+0000 7f65f659c640 1 --2- 172.21.15.150:0/2485135596 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65f8144890 0x7f65f8144d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:31.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.419+0000 7f65f659c640 1 -- 172.21.15.150:0/2485135596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65e0004800 msgr2=0x7f65f809d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:31.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.419+0000 7f65f659c640 1 --2- 172.21.15.150:0/2485135596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65e0004800 0x7f65f809d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:31.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.419+0000 7f65f659c640 1 -- 172.21.15.150:0/2485135596 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65f809de30 con 0x7f65f8151150 2024-03-17T23:46:31.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.419+0000 7f65f759e640 1 --2- 172.21.15.150:0/2485135596 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65f8144890 0x7f65f8144d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:46:31.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.420+0000 7f65f659c640 1 --2- 172.21.15.150:0/2485135596 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65f8151150 0x7f65f8161490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f65ec00c1a0 tx=0x7f65ec000fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:31.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.420+0000 7f65df7fe640 1 -- 172.21.15.150:0/2485135596 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ec004bf0 con 0x7f65f8151150 2024-03-17T23:46:31.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.421+0000 7f65fd097640 1 -- 172.21.15.150:0/2485135596 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65f809e110 con 0x7f65f8151150 2024-03-17T23:46:31.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.421+0000 7f65fd097640 1 -- 172.21.15.150:0/2485135596 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f65f809e6d0 con 0x7f65f8151150 2024-03-17T23:46:31.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.421+0000 7f65df7fe640 1 -- 172.21.15.150:0/2485135596 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65ec007bc0 con 0x7f65f8151150 2024-03-17T23:46:31.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.421+0000 7f65df7fe640 1 -- 172.21.15.150:0/2485135596 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ec02c740 con 0x7f65f8151150 2024-03-17T23:46:31.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.421+0000 7f65df7fe640 1 -- 172.21.15.150:0/2485135596 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f65ec02cc70 con 0x7f65f8151150 2024-03-17T23:46:31.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.421+0000 7f65df7fe640 1 -- 172.21.15.150:0/2485135596 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f65ec04a770 con 0x7f65f8151150 2024-03-17T23:46:31.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.422+0000 7f65fd097640 1 -- 172.21.15.150:0/2485135596 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65f809eab0 con 0x7f65f8151150 2024-03-17T23:46:31.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.424+0000 7f65df7fe640 1 -- 172.21.15.150:0/2485135596 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f65ec04a9c0 con 0x7f65f8151150 2024-03-17T23:46:31.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.489+0000 7f65fd097640 1 -- 172.21.15.150:0/2485135596 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f65f80688b0 con 0x7f65f8151150 2024-03-17T23:46:31.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.492+0000 7f65df7fe640 1 -- 172.21.15.150:0/2485135596 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f65ec04aba0 con 0x7f65f8151150 2024-03-17T23:46:31.493 INFO:teuthology.orchestra.run.smithi150.stderr:adjusted tunables profile to default 2024-03-17T23:46:31.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.493+0000 7f65fd097640 1 -- 172.21.15.150:0/2485135596 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65f8151150 msgr2=0x7f65f8161490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:31.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.493+0000 7f65fd097640 1 --2- 172.21.15.150:0/2485135596 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65f8151150 0x7f65f8161490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f65ec00c1a0 tx=0x7f65ec000fe0 comp rx=0 tx=0).stop 2024-03-17T23:46:31.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.493+0000 7f65fd097640 1 -- 172.21.15.150:0/2485135596 shutdown_connections 2024-03-17T23:46:31.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.493+0000 7f65fd097640 1 --2- 172.21.15.150:0/2485135596 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f65f8144890 0x7f65f8144d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:31.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.493+0000 7f65fd097640 1 --2- 172.21.15.150:0/2485135596 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f65f8151150 0x7f65f8161490 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:31.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.493+0000 7f65fd097640 1 --2- 172.21.15.150:0/2485135596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65e0004800 0x7f65f809d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:31.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.493+0000 7f65fd097640 1 -- 172.21.15.150:0/2485135596 >> 172.21.15.150:0/2485135596 conn(0x7f65f80b6170 msgr2=0x7f65f8099a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:31.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.493+0000 7f65fd097640 1 -- 172.21.15.150:0/2485135596 shutdown_connections 2024-03-17T23:46:31.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:31.493+0000 7f65fd097640 1 -- 172.21.15.150:0/2485135596 wait complete. 2024-03-17T23:46:31.504 INFO:tasks.ceph:check_enable_crimson: False 2024-03-17T23:46:31.505 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-17T23:46:31.505 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:46:31.505 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-17T23:46:31.532 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:46:31.532 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-17T23:46:31.599 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:46:31.599 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-17T23:46:31.668 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:46:31.669 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-17T23:46:31.735 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:46:31.735 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-17T23:46:31.763 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:46:31.763 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-17T23:46:31.831 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:46:31.831 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-17T23:46:31.880 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-17T23:46:31.880+0000 7ff97924b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-17T23:46:31.895 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:31.895+0000 7f290723a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-17T23:46:31.897 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:46:31.897 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-17T23:46:31.937 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-17T23:46:31.936+0000 7ff97924b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-17T23:46:31.959 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:31.959+0000 7f290723a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-17T23:46:31.964 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 3b0ae141-f0f1-4834-9621-bf11818f4be4 0 2024-03-17T23:46:32.105 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.104+0000 7fa008de1640 1 Processor -- start 2024-03-17T23:46:32.106 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.105+0000 7fa008de1640 1 -- start start 2024-03-17T23:46:32.115 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.105+0000 7fa008de1640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa0040b0c40 0x7fa00409b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:32.115 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.105+0000 7fa008de1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa00406d430 0x7fa0040b13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:32.115 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.105+0000 7fa008de1640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0040d0870 con 0x7fa004099e20 2024-03-17T23:46:32.115 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.105+0000 7fa008de1640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fa00406cfa0 con 0x7fa0040b0c40 2024-03-17T23:46:32.116 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.105+0000 7fa008de1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa00406c950 con 0x7fa00406d430 2024-03-17T23:46:32.116 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.106+0000 7fa002d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa00406d430 0x7fa0040b13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:32.116 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.106+0000 7fa001d9b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa0040b0c40 0x7fa00409b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:32.116 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.106+0000 7fa00259c640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fa004099e20 0x7fa00409a200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:38436/0 (socket says 172.21.15.157:38436) 2024-03-17T23:46:32.116 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.106+0000 7fa00259c640 1 -- 172.21.15.157:0/2803168747 learned_addr learned my addr 172.21.15.157:0/2803168747 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-17T23:46:32.116 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.106+0000 7fa002d9d640 1 -- 172.21.15.157:0/2803168747 >> v2:172.21.15.157:3300/0 conn(0x7fa0040b0c40 msgr2=0x7fa00409b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:32.116 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.106+0000 7fa002d9d640 1 --2- 172.21.15.157:0/2803168747 >> v2:172.21.15.157:3300/0 conn(0x7fa0040b0c40 0x7fa00409b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.116 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.106+0000 7fa002d9d640 1 -- 172.21.15.157:0/2803168747 >> v1:172.21.15.157:6789/0 conn(0x7fa004099e20 legacy=0x7fa00409a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:32.116 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.106+0000 7fa002d9d640 1 -- 172.21.15.157:0/2803168747 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa00406cc30 con 0x7fa00406d430 2024-03-17T23:46:32.117 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.107+0000 7fa002d9d640 1 --2- 172.21.15.157:0/2803168747 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa00406d430 0x7fa0040b13f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc009d90 tx=0x7f9ffc02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9609ecf973249211 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:32.117 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.107+0000 7fa00159a640 1 -- 172.21.15.157:0/2803168747 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc039070 con 0x7fa00406d430 2024-03-17T23:46:32.117 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.107+0000 7fa00159a640 1 -- 172.21.15.157:0/2803168747 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ffc02cc50 con 0x7fa00406d430 2024-03-17T23:46:32.117 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.107+0000 7fa00159a640 1 -- 172.21.15.157:0/2803168747 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ffc034440 con 0x7fa00406d430 2024-03-17T23:46:32.118 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.107+0000 7fa008de1640 1 -- 172.21.15.157:0/2803168747 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa00406d430 msgr2=0x7fa0040b13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:32.118 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.107+0000 7fa008de1640 1 --2- 172.21.15.157:0/2803168747 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa00406d430 0x7fa0040b13f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc009d90 tx=0x7f9ffc02bea0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.118 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.108+0000 7fa008de1640 1 -- 172.21.15.157:0/2803168747 shutdown_connections 2024-03-17T23:46:32.118 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.108+0000 7fa008de1640 1 --2- 172.21.15.157:0/2803168747 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa00406d430 0x7fa0040b13f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.118 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.108+0000 7fa008de1640 1 --2- 172.21.15.157:0/2803168747 >> v2:172.21.15.157:3300/0 conn(0x7fa0040b0c40 0x7fa00409b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.118 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.108+0000 7fa008de1640 1 -- 172.21.15.157:0/2803168747 >> 172.21.15.157:0/2803168747 conn(0x7fa0040d1b80 msgr2=0x7fa0040d1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:32.118 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.108+0000 7fa008de1640 1 -- 172.21.15.157:0/2803168747 shutdown_connections 2024-03-17T23:46:32.119 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.108+0000 7fa008de1640 1 -- 172.21.15.157:0/2803168747 wait complete. 2024-03-17T23:46:32.119 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.109+0000 7fa008de1640 1 Processor -- start 2024-03-17T23:46:32.119 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.109+0000 7fa008de1640 1 -- start start 2024-03-17T23:46:32.119 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.110+0000 7fa008de1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa00406d430 0x7fa0041f8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:32.119 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.110+0000 7fa008de1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa004099e20 0x7fa004147690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:32.119 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.110+0000 7fa008de1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0040b0c40 0x7fa00414a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:32.120 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.110+0000 7fa008de1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa0040d0610 con 0x7fa0040b0c40 2024-03-17T23:46:32.120 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.110+0000 7fa008de1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa0040d0490 con 0x7fa00406d430 2024-03-17T23:46:32.120 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.110+0000 7fa008de1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa00406d8e0 con 0x7fa004099e20 2024-03-17T23:46:32.120 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.110+0000 7fa00259c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa00406d430 0x7fa0041f8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:32.120 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.110+0000 7fa001d9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa004099e20 0x7fa004147690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:32.120 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.110+0000 7fa00259c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa00406d430 0x7fa0041f8590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:45882/0 (socket says 172.21.15.157:45882) 2024-03-17T23:46:32.121 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.110+0000 7fa00259c640 1 -- 172.21.15.157:0/1525466411 learned_addr learned my addr 172.21.15.157:0/1525466411 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-17T23:46:32.121 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.110+0000 7fa002d9d640 1 --2- 172.21.15.157:0/1525466411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0040b0c40 0x7fa00414a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:32.121 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.110+0000 7fa00259c640 1 -- 172.21.15.157:0/1525466411 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa004099e20 msgr2=0x7fa004147690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:32.121 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.110+0000 7fa00259c640 1 --2- 172.21.15.157:0/1525466411 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa004099e20 0x7fa004147690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.121 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.110+0000 7fa00259c640 1 -- 172.21.15.157:0/1525466411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0040b0c40 msgr2=0x7fa00414a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:32.121 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.110+0000 7fa00259c640 1 --2- 172.21.15.157:0/1525466411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0040b0c40 0x7fa00414a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.122 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.110+0000 7fa00259c640 1 -- 172.21.15.157:0/1525466411 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa00414aff0 con 0x7fa00406d430 2024-03-17T23:46:32.122 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.110+0000 7fa002d9d640 1 --2- 172.21.15.157:0/1525466411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0040b0c40 0x7fa00414a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:46:32.122 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.110+0000 7fa00259c640 1 --2- 172.21.15.157:0/1525466411 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa00406d430 0x7fa0041f8590 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9fec00c400 tx=0x7f9fec00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:32.122 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.110+0000 7f9ffaffd640 1 -- 172.21.15.157:0/1525466411 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fec017070 con 0x7fa00406d430 2024-03-17T23:46:32.122 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.110+0000 7f9ffaffd640 1 -- 172.21.15.157:0/1525466411 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fec005ce0 con 0x7fa00406d430 2024-03-17T23:46:32.122 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.110+0000 7fa008de1640 1 -- 172.21.15.157:0/1525466411 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa00414aa30 con 0x7fa00406d430 2024-03-17T23:46:32.122 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.110+0000 7f9ffaffd640 1 -- 172.21.15.157:0/1525466411 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fec0072f0 con 0x7fa00406d430 2024-03-17T23:46:32.123 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.110+0000 7fa008de1640 1 -- 172.21.15.157:0/1525466411 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0041f7d90 con 0x7fa00406d430 2024-03-17T23:46:32.123 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.111+0000 7f9ffaffd640 1 -- 172.21.15.157:0/1525466411 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9fec01a070 con 0x7fa00406d430 2024-03-17T23:46:32.123 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.111+0000 7f9ffaffd640 1 -- 172.21.15.157:0/1525466411 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f9fec012740 con 0x7fa00406d430 2024-03-17T23:46:32.123 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.111+0000 7fa008de1640 1 -- 172.21.15.157:0/1525466411 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0040688b0 con 0x7fa00406d430 2024-03-17T23:46:32.123 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.113+0000 7f9ffaffd640 1 -- 172.21.15.157:0/1525466411 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f9fec021020 con 0x7fa00406d430 2024-03-17T23:46:32.184 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.183+0000 7fa008de1640 1 -- 172.21.15.157:0/1525466411 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3b0ae141-f0f1-4834-9621-bf11818f4be4", "id": 0} v 0) -- 0x7fa0040d2730 con 0x7fa00406d430 2024-03-17T23:46:32.187 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.186+0000 7f9ffaffd640 1 -- 172.21.15.157:0/1525466411 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3b0ae141-f0f1-4834-9621-bf11818f4be4", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f9fec0075e0 con 0x7fa00406d430 2024-03-17T23:46:32.187 INFO:teuthology.orchestra.run.smithi157.stdout:0 2024-03-17T23:46:32.188 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.187+0000 7fa008de1640 1 -- 172.21.15.157:0/1525466411 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa00406d430 msgr2=0x7fa0041f8590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:32.188 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.187+0000 7fa008de1640 1 --2- 172.21.15.157:0/1525466411 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa00406d430 0x7fa0041f8590 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9fec00c400 tx=0x7f9fec00c8c0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.188 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.187+0000 7fa008de1640 1 -- 172.21.15.157:0/1525466411 shutdown_connections 2024-03-17T23:46:32.188 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.187+0000 7fa008de1640 1 --2- 172.21.15.157:0/1525466411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0040b0c40 0x7fa00414a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.188 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.187+0000 7fa008de1640 1 --2- 172.21.15.157:0/1525466411 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa004099e20 0x7fa004147690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.188 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.187+0000 7fa008de1640 1 --2- 172.21.15.157:0/1525466411 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa00406d430 0x7fa0041f8590 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.188 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.187+0000 7fa008de1640 1 -- 172.21.15.157:0/1525466411 >> 172.21.15.157:0/1525466411 conn(0x7fa0040d1b80 msgr2=0x7fa0040af9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:32.189 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.187+0000 7fa008de1640 1 -- 172.21.15.157:0/1525466411 shutdown_connections 2024-03-17T23:46:32.189 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.187+0000 7fa008de1640 1 -- 172.21.15.157:0/1525466411 wait complete. 2024-03-17T23:46:32.198 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 51e9a837-3530-419b-b5af-a30a5882ae2b 1 2024-03-17T23:46:32.302 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.301+0000 7f3f0139d640 1 Processor -- start 2024-03-17T23:46:32.302 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.301+0000 7f3f0139d640 1 -- start start 2024-03-17T23:46:32.303 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.302+0000 7f3f0139d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3efc151c60 0x7f3efc0af7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:32.303 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.302+0000 7f3f0139d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3efc15bcd0 0x7f3efc0cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:32.303 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.302+0000 7f3f0139d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3efc0cfc50 con 0x7f3efc06c1a0 2024-03-17T23:46:32.303 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.302+0000 7f3f0139d640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3efc0cff30 con 0x7f3efc15bcd0 2024-03-17T23:46:32.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.302+0000 7f3f0139d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3efc15b1f0 con 0x7f3efc151c60 2024-03-17T23:46:32.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.302+0000 7f3efbfff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3efc151c60 0x7f3efc0af7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:32.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.302+0000 7f3efb7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3efc15bcd0 0x7f3efc0cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:32.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.302+0000 7f3f00b9c640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f3efc06c1a0 0x7f3efc15e5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:38444/0 (socket says 172.21.15.157:38444) 2024-03-17T23:46:32.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.302+0000 7f3f00b9c640 1 -- 172.21.15.157:0/926229807 learned_addr learned my addr 172.21.15.157:0/926229807 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-17T23:46:32.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.302+0000 7f3efb7fe640 1 -- 172.21.15.157:0/926229807 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3efc151c60 msgr2=0x7f3efc0af7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:32.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.302+0000 7f3efb7fe640 1 --2- 172.21.15.157:0/926229807 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3efc151c60 0x7f3efc0af7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.302+0000 7f3efb7fe640 1 -- 172.21.15.157:0/926229807 >> v1:172.21.15.157:6789/0 conn(0x7f3efc06c1a0 legacy=0x7f3efc15e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:32.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.302+0000 7f3efb7fe640 1 -- 172.21.15.157:0/926229807 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3efc15b4d0 con 0x7f3efc15bcd0 2024-03-17T23:46:32.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.302+0000 7f3efaffd640 1 -- 172.21.15.157:0/926229807 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2098323869 0 0) 0x7f3efc0cfc50 con 0x7f3efc06c1a0 2024-03-17T23:46:32.304 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.302+0000 7f3efbfff640 1 --2- 172.21.15.157:0/926229807 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3efc151c60 0x7f3efc0af7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:46:32.305 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.304+0000 7f3efb7fe640 1 --2- 172.21.15.157:0/926229807 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3efc15bcd0 0x7f3efc0cf3e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3ef000ba40 tx=0x7f3ef002e080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb6e2ca38eb35f35 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:32.306 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.304+0000 7f3efaffd640 1 -- 172.21.15.157:0/926229807 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef003b070 con 0x7f3efc15bcd0 2024-03-17T23:46:32.306 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.304+0000 7f3efaffd640 1 -- 172.21.15.157:0/926229807 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ef002ee30 con 0x7f3efc15bcd0 2024-03-17T23:46:32.306 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.304+0000 7f3efaffd640 1 -- 172.21.15.157:0/926229807 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef0036640 con 0x7f3efc15bcd0 2024-03-17T23:46:32.306 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.304+0000 7f3f0139d640 1 -- 172.21.15.157:0/926229807 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3efc15bcd0 msgr2=0x7f3efc0cf3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:32.306 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.304+0000 7f3f0139d640 1 --2- 172.21.15.157:0/926229807 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3efc15bcd0 0x7f3efc0cf3e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3ef000ba40 tx=0x7f3ef002e080 comp rx=0 tx=0).stop 2024-03-17T23:46:32.306 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.304+0000 7f3f0139d640 1 -- 172.21.15.157:0/926229807 shutdown_connections 2024-03-17T23:46:32.306 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.304+0000 7f3f0139d640 1 --2- 172.21.15.157:0/926229807 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3efc15bcd0 0x7f3efc0cf3e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.306 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.304+0000 7f3f0139d640 1 --2- 172.21.15.157:0/926229807 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3efc151c60 0x7f3efc0af7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.306 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.304+0000 7f3f0139d640 1 -- 172.21.15.157:0/926229807 >> 172.21.15.157:0/926229807 conn(0x7f3efc06f160 msgr2=0x7f3efc0722f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:32.306 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.304+0000 7f3f0139d640 1 -- 172.21.15.157:0/926229807 shutdown_connections 2024-03-17T23:46:32.306 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.304+0000 7f3f0139d640 1 -- 172.21.15.157:0/926229807 wait complete. 2024-03-17T23:46:32.309 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.306+0000 7f3f0139d640 1 Processor -- start 2024-03-17T23:46:32.309 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.306+0000 7f3f0139d640 1 -- start start 2024-03-17T23:46:32.309 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.306+0000 7f3f0139d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3efc06c1a0 0x7f3efc148f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:32.309 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.306+0000 7f3f0139d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3efc151c60 0x7f3efc1420f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:32.310 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.306+0000 7f3f0139d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3efc15bcd0 0x7f3efc0b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:32.310 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.306+0000 7f3f0139d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc06c630 con 0x7f3efc15bcd0 2024-03-17T23:46:32.310 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.306+0000 7f3f0139d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc0cf920 con 0x7f3efc06c1a0 2024-03-17T23:46:32.310 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.306+0000 7f3f0139d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3efc06c7b0 con 0x7f3efc151c60 2024-03-17T23:46:32.310 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.307+0000 7f3efbfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3efc06c1a0 0x7f3efc148f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:32.310 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.307+0000 7f3efbfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3efc06c1a0 0x7f3efc148f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:45904/0 (socket says 172.21.15.157:45904) 2024-03-17T23:46:32.310 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.307+0000 7f3efbfff640 1 -- 172.21.15.157:0/3029666958 learned_addr learned my addr 172.21.15.157:0/3029666958 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-17T23:46:32.311 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.307+0000 7f3efb7fe640 1 --2- 172.21.15.157:0/3029666958 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3efc151c60 0x7f3efc1420f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:32.311 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.307+0000 7f3f00b9c640 1 --2- 172.21.15.157:0/3029666958 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3efc15bcd0 0x7f3efc0b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:32.311 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.307+0000 7f3efb7fe640 1 -- 172.21.15.157:0/3029666958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3efc06c1a0 msgr2=0x7f3efc148f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:32.311 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.307+0000 7f3efb7fe640 1 --2- 172.21.15.157:0/3029666958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3efc06c1a0 0x7f3efc148f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.311 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.307+0000 7f3efb7fe640 1 -- 172.21.15.157:0/3029666958 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3efc15bcd0 msgr2=0x7f3efc0b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:32.311 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.307+0000 7f3efb7fe640 1 --2- 172.21.15.157:0/3029666958 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3efc15bcd0 0x7f3efc0b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.311 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.307+0000 7f3efb7fe640 1 -- 172.21.15.157:0/3029666958 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3efc149540 con 0x7f3efc151c60 2024-03-17T23:46:32.311 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.307+0000 7f3efbfff640 1 --2- 172.21.15.157:0/3029666958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3efc06c1a0 0x7f3efc148f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:46:32.311 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.307+0000 7f3f00b9c640 1 --2- 172.21.15.157:0/3029666958 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3efc15bcd0 0x7f3efc0b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:46:32.311 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.307+0000 7f3efb7fe640 1 --2- 172.21.15.157:0/3029666958 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3efc151c60 0x7f3efc1420f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3ef0035000 tx=0x7f3ef0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:32.311 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.307+0000 7f3ef8ff9640 1 -- 172.21.15.157:0/3029666958 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef0046070 con 0x7f3efc151c60 2024-03-17T23:46:32.311 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.307+0000 7f3ef8ff9640 1 -- 172.21.15.157:0/3029666958 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ef003d560 con 0x7f3efc151c60 2024-03-17T23:46:32.311 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.307+0000 7f3f0139d640 1 -- 172.21.15.157:0/3029666958 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3efc0ad2d0 con 0x7f3efc151c60 2024-03-17T23:46:32.311 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.307+0000 7f3ef8ff9640 1 -- 172.21.15.157:0/3029666958 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef003b070 con 0x7f3efc151c60 2024-03-17T23:46:32.312 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.307+0000 7f3f0139d640 1 -- 172.21.15.157:0/3029666958 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3efc0adb70 con 0x7f3efc151c60 2024-03-17T23:46:32.312 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.307+0000 7f3ef8ff9640 1 -- 172.21.15.157:0/3029666958 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3ef0049070 con 0x7f3efc151c60 2024-03-17T23:46:32.312 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.308+0000 7f3ef8ff9640 1 -- 172.21.15.157:0/3029666958 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f3ef004bc30 con 0x7f3efc151c60 2024-03-17T23:46:32.312 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.308+0000 7f3f0139d640 1 -- 172.21.15.157:0/3029666958 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3efc0ad4a0 con 0x7f3efc151c60 2024-03-17T23:46:32.312 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.310+0000 7f3ef8ff9640 1 -- 172.21.15.157:0/3029666958 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f3ef003d950 con 0x7f3efc151c60 2024-03-17T23:46:32.382 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.382+0000 7f3f0139d640 1 -- 172.21.15.157:0/3029666958 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd new", "uuid": "51e9a837-3530-419b-b5af-a30a5882ae2b", "id": 1} v 0) -- 0x7f3efc074c40 con 0x7f3efc151c60 2024-03-17T23:46:32.387 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.387+0000 7f3ef8ff9640 1 -- 172.21.15.157:0/3029666958 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "51e9a837-3530-419b-b5af-a30a5882ae2b", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f3ef003db30 con 0x7f3efc151c60 2024-03-17T23:46:32.387 INFO:teuthology.orchestra.run.smithi157.stdout:1 2024-03-17T23:46:32.389 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.388+0000 7f3f0139d640 1 -- 172.21.15.157:0/3029666958 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3efc151c60 msgr2=0x7f3efc1420f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:32.389 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.388+0000 7f3f0139d640 1 --2- 172.21.15.157:0/3029666958 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3efc151c60 0x7f3efc1420f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3ef0035000 tx=0x7f3ef0004730 comp rx=0 tx=0).stop 2024-03-17T23:46:32.389 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.388+0000 7f3f0139d640 1 -- 172.21.15.157:0/3029666958 shutdown_connections 2024-03-17T23:46:32.389 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.388+0000 7f3f0139d640 1 --2- 172.21.15.157:0/3029666958 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3efc15bcd0 0x7f3efc0b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.389 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.388+0000 7f3f0139d640 1 --2- 172.21.15.157:0/3029666958 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3efc151c60 0x7f3efc1420f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.389 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.388+0000 7f3f0139d640 1 --2- 172.21.15.157:0/3029666958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3efc06c1a0 0x7f3efc148f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.389 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.388+0000 7f3f0139d640 1 -- 172.21.15.157:0/3029666958 >> 172.21.15.157:0/3029666958 conn(0x7f3efc06f160 msgr2=0x7f3efc0cdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:32.389 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.388+0000 7f3f0139d640 1 -- 172.21.15.157:0/3029666958 shutdown_connections 2024-03-17T23:46:32.389 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.388+0000 7f3f0139d640 1 -- 172.21.15.157:0/3029666958 wait complete. 2024-03-17T23:46:32.399 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f 2 2024-03-17T23:46:32.499 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:32.498+0000 7f290723a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-17T23:46:32.500 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.500+0000 7f030a0f8640 1 Processor -- start 2024-03-17T23:46:32.500 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.500+0000 7f030a0f8640 1 -- start start 2024-03-17T23:46:32.502 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.500+0000 7f030a0f8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03040b2bf0 0x7f03040b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:32.502 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.500+0000 7f030a0f8640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0304161d30 con 0x7f03040d0ce0 2024-03-17T23:46:32.502 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.500+0000 7f030a0f8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0304162010 con 0x7f0304161520 2024-03-17T23:46:32.502 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.500+0000 7f030a0f8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f03040b9bf0 con 0x7f03040b2bf0 2024-03-17T23:46:32.502 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.501+0000 7f0303fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03040b2bf0 0x7f03040b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:32.502 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.501+0000 7f03037fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f0304161520 0x7f03040d05d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:38458/0 (socket says 172.21.15.157:38458) 2024-03-17T23:46:32.502 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.501+0000 7f03037fe640 1 -- 172.21.15.157:0/929162405 learned_addr learned my addr 172.21.15.157:0/929162405 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-17T23:46:32.502 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.501+0000 7f03027fc640 1 -- 172.21.15.157:0/929162405 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2548897038 0 0) 0x7f0304162010 con 0x7f0304161520 2024-03-17T23:46:32.502 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.501+0000 7f03027fc640 1 -- 172.21.15.157:0/929162405 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02ec003520 con 0x7f0304161520 2024-03-17T23:46:32.502 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.501+0000 7f0303fff640 1 -- 172.21.15.157:0/929162405 >> v1:172.21.15.157:6789/0 conn(0x7f0304161520 legacy=0x7f03040d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:32.503 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.501+0000 7f0303fff640 1 -- 172.21.15.157:0/929162405 >> v1:172.21.15.150:6789/0 conn(0x7f03040d0ce0 legacy=0x7f03040d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:32.503 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.501+0000 7f0303fff640 1 -- 172.21.15.157:0/929162405 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03041622f0 con 0x7f03040b2bf0 2024-03-17T23:46:32.504 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.502+0000 7f0303fff640 1 --2- 172.21.15.157:0/929162405 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03040b2bf0 0x7f03040b96b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f02fc009dc0 tx=0x7f02fc02be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7614f24cda398fd server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:32.504 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.502+0000 7f03027fc640 1 -- 172.21.15.157:0/929162405 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02fc039070 con 0x7f03040b2bf0 2024-03-17T23:46:32.504 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.502+0000 7f03027fc640 1 -- 172.21.15.157:0/929162405 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02fc02cbb0 con 0x7f03040b2bf0 2024-03-17T23:46:32.504 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.502+0000 7f03027fc640 1 -- 172.21.15.157:0/929162405 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02fc034440 con 0x7f03040b2bf0 2024-03-17T23:46:32.504 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.502+0000 7f030a0f8640 1 -- 172.21.15.157:0/929162405 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03040b2bf0 msgr2=0x7f03040b96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:32.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.502+0000 7f030a0f8640 1 --2- 172.21.15.157:0/929162405 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03040b2bf0 0x7f03040b96b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f02fc009dc0 tx=0x7f02fc02be00 comp rx=0 tx=0).stop 2024-03-17T23:46:32.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.502+0000 7f030a0f8640 1 -- 172.21.15.157:0/929162405 shutdown_connections 2024-03-17T23:46:32.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.502+0000 7f030a0f8640 1 --2- 172.21.15.157:0/929162405 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f03040b2bf0 0x7f03040b96b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.502+0000 7f030a0f8640 1 -- 172.21.15.157:0/929162405 >> 172.21.15.157:0/929162405 conn(0x7f030406ca00 msgr2=0x7f030406ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:32.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.502+0000 7f030a0f8640 1 -- 172.21.15.157:0/929162405 shutdown_connections 2024-03-17T23:46:32.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.503+0000 7f030a0f8640 1 -- 172.21.15.157:0/929162405 wait complete. 2024-03-17T23:46:32.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.504+0000 7f030a0f8640 1 Processor -- start 2024-03-17T23:46:32.505 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.504+0000 7f030a0f8640 1 -- start start 2024-03-17T23:46:32.508 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.504+0000 7f030a0f8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03040b2bf0 0x7f03040b4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:32.508 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.504+0000 7f030a0f8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03040d0ce0 0x7f030414a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:32.508 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.505+0000 7f030a0f8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0304161520 0x7f03040ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:32.508 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.505+0000 7f030a0f8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0304161900 con 0x7f03040d0ce0 2024-03-17T23:46:32.508 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.505+0000 7f030a0f8640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0304162940 con 0x7f03040b2bf0 2024-03-17T23:46:32.508 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.505+0000 7f030a0f8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f03040d26b0 con 0x7f0304161520 2024-03-17T23:46:32.508 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.505+0000 7f03037fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03040b2bf0 0x7f03040b4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:32.508 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.505+0000 7f03037fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03040b2bf0 0x7f03040b4160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:45920/0 (socket says 172.21.15.157:45920) 2024-03-17T23:46:32.508 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.505+0000 7f03037fe640 1 -- 172.21.15.157:0/144863064 learned_addr learned my addr 172.21.15.157:0/144863064 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-17T23:46:32.508 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.505+0000 7f0303fff640 1 --2- 172.21.15.157:0/144863064 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0304161520 0x7f03040ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:32.508 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.505+0000 7f0302ffd640 1 --2- 172.21.15.157:0/144863064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03040d0ce0 0x7f030414a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:32.509 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.505+0000 7f03037fe640 1 -- 172.21.15.157:0/144863064 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0304161520 msgr2=0x7f03040ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:32.509 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.505+0000 7f03037fe640 1 --2- 172.21.15.157:0/144863064 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0304161520 0x7f03040ccb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.509 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.505+0000 7f03037fe640 1 -- 172.21.15.157:0/144863064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03040d0ce0 msgr2=0x7f030414a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:32.509 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.505+0000 7f03037fe640 1 --2- 172.21.15.157:0/144863064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03040d0ce0 0x7f030414a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.509 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.505+0000 7f03037fe640 1 -- 172.21.15.157:0/144863064 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03040b46a0 con 0x7f03040b2bf0 2024-03-17T23:46:32.509 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.505+0000 7f0303fff640 1 --2- 172.21.15.157:0/144863064 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0304161520 0x7f03040ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:46:32.509 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.505+0000 7f03037fe640 1 --2- 172.21.15.157:0/144863064 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03040b2bf0 0x7f03040b4160 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f02f400c410 tx=0x7f02f400c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:32.509 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.505+0000 7f02ebfff640 1 -- 172.21.15.157:0/144863064 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f4017070 con 0x7f03040b2bf0 2024-03-17T23:46:32.509 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.505+0000 7f02ebfff640 1 -- 172.21.15.157:0/144863064 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02f4005ce0 con 0x7f03040b2bf0 2024-03-17T23:46:32.509 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.505+0000 7f030a0f8640 1 -- 172.21.15.157:0/144863064 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03041605b0 con 0x7f03040b2bf0 2024-03-17T23:46:32.509 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.505+0000 7f02ebfff640 1 -- 172.21.15.157:0/144863064 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f40072f0 con 0x7f03040b2bf0 2024-03-17T23:46:32.509 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.505+0000 7f030a0f8640 1 -- 172.21.15.157:0/144863064 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03040b3680 con 0x7f03040b2bf0 2024-03-17T23:46:32.509 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.505+0000 7f02ebfff640 1 -- 172.21.15.157:0/144863064 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f02f401a070 con 0x7f03040b2bf0 2024-03-17T23:46:32.509 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.506+0000 7f02ebfff640 1 -- 172.21.15.157:0/144863064 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f02f4012820 con 0x7f03040b2bf0 2024-03-17T23:46:32.510 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.506+0000 7f030a0f8640 1 -- 172.21.15.157:0/144863064 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03040b29b0 con 0x7f03040b2bf0 2024-03-17T23:46:32.510 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.509+0000 7f02ebfff640 1 -- 172.21.15.157:0/144863064 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f03040b29b0 con 0x7f03040b2bf0 2024-03-17T23:46:32.514 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-17T23:46:32.514+0000 7ff97924b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-17T23:46:32.553 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:32.552+0000 7f290723a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-17T23:46:32.570 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-17T23:46:32.569+0000 7ff97924b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-17T23:46:32.581 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.580+0000 7f030a0f8640 1 -- 172.21.15.157:0/144863064 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f", "id": 2} v 0) -- 0x7f03040681a0 con 0x7f03040b2bf0 2024-03-17T23:46:32.586 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.586+0000 7f02ebfff640 1 -- 172.21.15.157:0/144863064 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f02f4021070 con 0x7f03040b2bf0 2024-03-17T23:46:32.587 INFO:teuthology.orchestra.run.smithi157.stdout:2 2024-03-17T23:46:32.588 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.587+0000 7f030a0f8640 1 -- 172.21.15.157:0/144863064 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03040b2bf0 msgr2=0x7f03040b4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:32.588 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.587+0000 7f030a0f8640 1 --2- 172.21.15.157:0/144863064 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03040b2bf0 0x7f03040b4160 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f02f400c410 tx=0x7f02f400c8d0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.588 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.587+0000 7f030a0f8640 1 -- 172.21.15.157:0/144863064 shutdown_connections 2024-03-17T23:46:32.588 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.587+0000 7f030a0f8640 1 --2- 172.21.15.157:0/144863064 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0304161520 0x7f03040ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.588 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.587+0000 7f030a0f8640 1 --2- 172.21.15.157:0/144863064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03040d0ce0 0x7f030414a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.588 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.587+0000 7f030a0f8640 1 --2- 172.21.15.157:0/144863064 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03040b2bf0 0x7f03040b4160 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.588 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.587+0000 7f030a0f8640 1 -- 172.21.15.157:0/144863064 >> 172.21.15.157:0/144863064 conn(0x7f030406ca00 msgr2=0x7f030415c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:32.588 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.587+0000 7f030a0f8640 1 -- 172.21.15.157:0/144863064 shutdown_connections 2024-03-17T23:46:32.588 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.587+0000 7f030a0f8640 1 -- 172.21.15.157:0/144863064 wait complete. 2024-03-17T23:46:32.598 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:32.597+0000 7f290723a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-17T23:46:32.598 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new d28afbd0-215b-419c-9071-40c83096e9bf 3 2024-03-17T23:46:32.618 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-17T23:46:32.617+0000 7ff97924b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-17T23:46:32.651 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:32.650+0000 7f290723a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-17T23:46:32.675 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-17T23:46:32.675+0000 7ff97924b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-17T23:46:32.692 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:32.692+0000 7f290723a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-17T23:46:32.698 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.697+0000 7efdb2c85640 1 Processor -- start 2024-03-17T23:46:32.698 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.697+0000 7efdb2c85640 1 -- start start 2024-03-17T23:46:32.699 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.698+0000 7efdb2c85640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efdac0cfe20 0x7efdac09a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:32.699 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.698+0000 7efdb2c85640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdac09af40 con 0x7efdac0d0600 2024-03-17T23:46:32.699 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.698+0000 7efdb2c85640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdac09b1e0 con 0x7efdac15be50 2024-03-17T23:46:32.699 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.698+0000 7efdb2c85640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efdac0c5690 con 0x7efdac0cfe20 2024-03-17T23:46:32.699 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.698+0000 7efdabfff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efdac0cfe20 0x7efdac09a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:32.700 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.698+0000 7efdb0a21640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7efdac15be50 0x7efdac0cf710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:38466/0 (socket says 172.21.15.157:38466) 2024-03-17T23:46:32.700 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.698+0000 7efdb0a21640 1 -- 172.21.15.157:0/3549888794 learned_addr learned my addr 172.21.15.157:0/3549888794 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-17T23:46:32.700 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.699+0000 7efdab7fe640 1 -- 172.21.15.157:0/3549888794 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3545472531 0 0) 0x7efdac09b1e0 con 0x7efdac15be50 2024-03-17T23:46:32.700 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.699+0000 7efdabfff640 1 -- 172.21.15.157:0/3549888794 >> v1:172.21.15.157:6789/0 conn(0x7efdac15be50 legacy=0x7efdac0cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:32.700 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.699+0000 7efdabfff640 1 -- 172.21.15.157:0/3549888794 >> v1:172.21.15.150:6789/0 conn(0x7efdac0d0600 legacy=0x7efdac0c60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:32.700 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.699+0000 7efdabfff640 1 -- 172.21.15.157:0/3549888794 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdac0c58e0 con 0x7efdac0cfe20 2024-03-17T23:46:32.701 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.699+0000 7efdabfff640 1 --2- 172.21.15.157:0/3549888794 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efdac0cfe20 0x7efdac09a830 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7efd980097d0 tx=0x7efd9802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fef192cb4ffc0d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:32.701 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.700+0000 7efdab7fe640 1 -- 172.21.15.157:0/3549888794 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd98035020 con 0x7efdac0cfe20 2024-03-17T23:46:32.701 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.700+0000 7efdab7fe640 1 -- 172.21.15.157:0/3549888794 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd98031890 con 0x7efdac0cfe20 2024-03-17T23:46:32.702 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.700+0000 7efdab7fe640 1 -- 172.21.15.157:0/3549888794 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd98031c60 con 0x7efdac0cfe20 2024-03-17T23:46:32.702 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.700+0000 7efdb2c85640 1 -- 172.21.15.157:0/3549888794 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efdac0cfe20 msgr2=0x7efdac09a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:32.702 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.700+0000 7efdb2c85640 1 --2- 172.21.15.157:0/3549888794 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efdac0cfe20 0x7efdac09a830 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7efd980097d0 tx=0x7efd9802b930 comp rx=0 tx=0).stop 2024-03-17T23:46:32.702 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.700+0000 7efdb2c85640 1 -- 172.21.15.157:0/3549888794 shutdown_connections 2024-03-17T23:46:32.702 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.700+0000 7efdb2c85640 1 --2- 172.21.15.157:0/3549888794 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efdac0cfe20 0x7efdac09a830 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.702 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.700+0000 7efdb2c85640 1 -- 172.21.15.157:0/3549888794 >> 172.21.15.157:0/3549888794 conn(0x7efdac0b3630 msgr2=0x7efdac0b3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:32.702 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.700+0000 7efdb2c85640 1 -- 172.21.15.157:0/3549888794 shutdown_connections 2024-03-17T23:46:32.702 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.700+0000 7efdb2c85640 1 -- 172.21.15.157:0/3549888794 wait complete. 2024-03-17T23:46:32.704 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.701+0000 7efdb2c85640 1 Processor -- start 2024-03-17T23:46:32.704 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.701+0000 7efdb2c85640 1 -- start start 2024-03-17T23:46:32.704 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.702+0000 7efdb2c85640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdac0cfe20 0x7efdac0aec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:32.704 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.702+0000 7efdb2c85640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdac0d0600 0x7efdac0b7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:32.705 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.702+0000 7efdb2c85640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efdac15be50 0x7efdac0ca640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:32.705 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.702+0000 7efdb2c85640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7efdac15c230 con 0x7efdac0cfe20 2024-03-17T23:46:32.705 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.702+0000 7efdb2c85640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7efdac0d0a60 con 0x7efdac0d0600 2024-03-17T23:46:32.705 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.702+0000 7efdb2c85640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efdac0c67f0 con 0x7efdac15be50 2024-03-17T23:46:32.705 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.702+0000 7efdabfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdac0d0600 0x7efdac0b7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:32.705 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.702+0000 7efdb1222640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efdac15be50 0x7efdac0ca640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:32.705 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.702+0000 7efdabfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdac0d0600 0x7efdac0b7530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:45934/0 (socket says 172.21.15.157:45934) 2024-03-17T23:46:32.705 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.702+0000 7efdabfff640 1 -- 172.21.15.157:0/1412337033 learned_addr learned my addr 172.21.15.157:0/1412337033 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-17T23:46:32.705 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.702+0000 7efdb0a21640 1 --2- 172.21.15.157:0/1412337033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdac0cfe20 0x7efdac0aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:32.705 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.703+0000 7efdabfff640 1 -- 172.21.15.157:0/1412337033 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efdac15be50 msgr2=0x7efdac0ca640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:32.705 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.703+0000 7efdabfff640 1 --2- 172.21.15.157:0/1412337033 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efdac15be50 0x7efdac0ca640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.705 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.703+0000 7efdabfff640 1 -- 172.21.15.157:0/1412337033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdac0cfe20 msgr2=0x7efdac0aec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:32.705 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.703+0000 7efdabfff640 1 --2- 172.21.15.157:0/1412337033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdac0cfe20 0x7efdac0aec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.705 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.703+0000 7efdabfff640 1 -- 172.21.15.157:0/1412337033 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdac0b0550 con 0x7efdac0d0600 2024-03-17T23:46:32.706 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.703+0000 7efdabfff640 1 --2- 172.21.15.157:0/1412337033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdac0d0600 0x7efdac0b7530 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efd980098f0 tx=0x7efd980046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:32.706 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.703+0000 7efda97fa640 1 -- 172.21.15.157:0/1412337033 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd98035040 con 0x7efdac0d0600 2024-03-17T23:46:32.706 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.703+0000 7efda97fa640 1 -- 172.21.15.157:0/1412337033 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd98046480 con 0x7efdac0d0600 2024-03-17T23:46:32.706 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.703+0000 7efdb2c85640 1 -- 172.21.15.157:0/1412337033 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdac0b07d0 con 0x7efdac0d0600 2024-03-17T23:46:32.706 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.703+0000 7efda97fa640 1 -- 172.21.15.157:0/1412337033 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd98046870 con 0x7efdac0d0600 2024-03-17T23:46:32.706 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.703+0000 7efdb2c85640 1 -- 172.21.15.157:0/1412337033 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdac0cb0c0 con 0x7efdac0d0600 2024-03-17T23:46:32.706 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.703+0000 7efda97fa640 1 -- 172.21.15.157:0/1412337033 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7efd98038020 con 0x7efdac0d0600 2024-03-17T23:46:32.706 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.703+0000 7efda97fa640 1 -- 172.21.15.157:0/1412337033 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7efd9803bdc0 con 0x7efdac0d0600 2024-03-17T23:46:32.707 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.704+0000 7efdb2c85640 1 -- 172.21.15.157:0/1412337033 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdac0b09a0 con 0x7efdac0d0600 2024-03-17T23:46:32.707 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.706+0000 7efda97fa640 1 -- 172.21.15.157:0/1412337033 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7efdac0b0550 con 0x7efdac0d0600 2024-03-17T23:46:32.721 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-17T23:46:32.720+0000 7ff97924b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-17T23:46:32.777 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.777+0000 7efdb2c85640 1 -- 172.21.15.157:0/1412337033 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d28afbd0-215b-419c-9071-40c83096e9bf", "id": 3} v 0) -- 0x7efdac0688b0 con 0x7efdac0d0600 2024-03-17T23:46:32.783 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.782+0000 7efda97fa640 1 -- 172.21.15.157:0/1412337033 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d28afbd0-215b-419c-9071-40c83096e9bf", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7efd9802d040 con 0x7efdac0d0600 2024-03-17T23:46:32.783 INFO:teuthology.orchestra.run.smithi157.stdout:3 2024-03-17T23:46:32.784 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.783+0000 7efdb2c85640 1 -- 172.21.15.157:0/1412337033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdac0d0600 msgr2=0x7efdac0b7530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:32.784 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.783+0000 7efdb2c85640 1 --2- 172.21.15.157:0/1412337033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdac0d0600 0x7efdac0b7530 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efd980098f0 tx=0x7efd980046c0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.784 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.783+0000 7efdb2c85640 1 -- 172.21.15.157:0/1412337033 shutdown_connections 2024-03-17T23:46:32.784 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.783+0000 7efdb2c85640 1 --2- 172.21.15.157:0/1412337033 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efdac15be50 0x7efdac0ca640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.785 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.783+0000 7efdb2c85640 1 --2- 172.21.15.157:0/1412337033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdac0d0600 0x7efdac0b7530 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.785 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.783+0000 7efdb2c85640 1 --2- 172.21.15.157:0/1412337033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efdac0cfe20 0x7efdac0aec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.785 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.783+0000 7efdb2c85640 1 -- 172.21.15.157:0/1412337033 >> 172.21.15.157:0/1412337033 conn(0x7efdac0b3630 msgr2=0x7efdac15e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:32.785 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.783+0000 7efdb2c85640 1 -- 172.21.15.157:0/1412337033 shutdown_connections 2024-03-17T23:46:32.785 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.783+0000 7efdb2c85640 1 -- 172.21.15.157:0/1412337033 wait complete. 2024-03-17T23:46:32.794 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new 7e973c64-ea03-4b69-ad16-6f5e6ccfb389 4 2024-03-17T23:46:32.808 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:32.808+0000 7f290723a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-17T23:46:32.841 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-17T23:46:32.841+0000 7ff97924b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-17T23:46:32.878 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:32.878+0000 7f290723a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-17T23:46:32.914 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-17T23:46:32.914+0000 7ff97924b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-17T23:46:32.923 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:32.923+0000 7f290723a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-17T23:46:32.934 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.933+0000 7f2ab7281640 1 Processor -- start 2024-03-17T23:46:32.934 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.934+0000 7f2ab7281640 1 -- start start 2024-03-17T23:46:32.936 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.934+0000 7f2ab7281640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2ab00cfe20 0x7f2ab009a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:32.936 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.934+0000 7f2ab7281640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ab00d0600 0x7f2ab00c60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:32.936 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.935+0000 7f2ab7281640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ab009af40 con 0x7f2ab015be50 2024-03-17T23:46:32.936 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.935+0000 7f2ab7281640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f2ab00c5690 con 0x7f2ab00cfe20 2024-03-17T23:46:32.936 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.935+0000 7f2ab7281640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2ab00d0a90 con 0x7f2ab00d0600 2024-03-17T23:46:32.936 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.935+0000 7f2ab581e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ab00d0600 0x7f2ab00c60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:32.936 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.935+0000 7f2ab481c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2ab00cfe20 0x7f2ab009a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:32.936 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.935+0000 7f2ab481c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2ab00cfe20 0x7f2ab009a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:45936/0 (socket says 172.21.15.157:45936) 2024-03-17T23:46:32.936 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.935+0000 7f2ab581e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ab00d0600 0x7f2ab00c60e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.157:60916/0 (socket says 172.21.15.157:60916) 2024-03-17T23:46:32.936 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.935+0000 7f2ab481c640 1 -- 172.21.15.157:0/1047475986 learned_addr learned my addr 172.21.15.157:0/1047475986 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-17T23:46:32.936 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.935+0000 7f2ab481c640 1 -- 172.21.15.157:0/1047475986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ab00d0600 msgr2=0x7f2ab00c60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:32.937 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.935+0000 7f2ab481c640 1 --2- 172.21.15.157:0/1047475986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ab00d0600 0x7f2ab00c60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.937 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.935+0000 7f2ab481c640 1 -- 172.21.15.157:0/1047475986 >> v1:172.21.15.150:6789/0 conn(0x7f2ab015be50 legacy=0x7f2ab00cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:32.937 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.935+0000 7f2ab481c640 1 -- 172.21.15.157:0/1047475986 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab009b1e0 con 0x7f2ab00cfe20 2024-03-17T23:46:32.937 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.935+0000 7f2ab581e640 1 --2- 172.21.15.157:0/1047475986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ab00d0600 0x7f2ab00c60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:46:32.938 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.936+0000 7f2ab481c640 1 --2- 172.21.15.157:0/1047475986 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ab00cfe20 0x7f2ab009a830 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0009870 tx=0x7f2aa002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea4728fc228e8f4a server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:32.938 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.936+0000 7f2aa7fff640 1 -- 172.21.15.157:0/1047475986 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0034070 con 0x7f2ab00cfe20 2024-03-17T23:46:32.938 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.936+0000 7f2aa7fff640 1 -- 172.21.15.157:0/1047475986 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aa0030840 con 0x7f2ab00cfe20 2024-03-17T23:46:32.939 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.936+0000 7f2aa7fff640 1 -- 172.21.15.157:0/1047475986 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0030c10 con 0x7f2ab00cfe20 2024-03-17T23:46:32.939 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.936+0000 7f2ab7281640 1 -- 172.21.15.157:0/1047475986 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ab00cfe20 msgr2=0x7f2ab009a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:32.939 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.936+0000 7f2ab7281640 1 --2- 172.21.15.157:0/1047475986 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ab00cfe20 0x7f2ab009a830 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0009870 tx=0x7f2aa002b980 comp rx=0 tx=0).stop 2024-03-17T23:46:32.939 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.937+0000 7f2ab7281640 1 -- 172.21.15.157:0/1047475986 shutdown_connections 2024-03-17T23:46:32.939 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.937+0000 7f2ab7281640 1 --2- 172.21.15.157:0/1047475986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ab00d0600 0x7f2ab00c60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.939 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.937+0000 7f2ab7281640 1 --2- 172.21.15.157:0/1047475986 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ab00cfe20 0x7f2ab009a830 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.939 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.937+0000 7f2ab7281640 1 -- 172.21.15.157:0/1047475986 >> 172.21.15.157:0/1047475986 conn(0x7f2ab00b3630 msgr2=0x7f2ab00b3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:32.939 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.937+0000 7f2ab7281640 1 -- 172.21.15.157:0/1047475986 shutdown_connections 2024-03-17T23:46:32.939 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.937+0000 7f2ab7281640 1 -- 172.21.15.157:0/1047475986 wait complete. 2024-03-17T23:46:32.939 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.938+0000 7f2ab7281640 1 Processor -- start 2024-03-17T23:46:32.939 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.938+0000 7f2ab7281640 1 -- start start 2024-03-17T23:46:32.941 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.939+0000 7f2ab7281640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ab00cfe20 0x7f2ab00cb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:32.942 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.939+0000 7f2ab7281640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ab00d0600 0x7f2ab0150d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:32.942 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.939+0000 7f2ab7281640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ab015be50 0x7f2ab009d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:32.942 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.939+0000 7f2ab7281640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab015c230 con 0x7f2ab00d0600 2024-03-17T23:46:32.942 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.939+0000 7f2ab7281640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab00c5bc0 con 0x7f2ab00cfe20 2024-03-17T23:46:32.942 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.939+0000 7f2ab7281640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2ab00c6620 con 0x7f2ab015be50 2024-03-17T23:46:32.942 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.939+0000 7f2ab501d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ab00cfe20 0x7f2ab00cb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:32.942 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.939+0000 7f2ab581e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ab015be50 0x7f2ab009d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:32.942 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.939+0000 7f2ab501d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ab00cfe20 0x7f2ab00cb790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:45952/0 (socket says 172.21.15.157:45952) 2024-03-17T23:46:32.942 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.939+0000 7f2ab501d640 1 -- 172.21.15.157:0/771930575 learned_addr learned my addr 172.21.15.157:0/771930575 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-17T23:46:32.942 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.939+0000 7f2ab481c640 1 --2- 172.21.15.157:0/771930575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ab00d0600 0x7f2ab0150d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:32.942 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.939+0000 7f2ab581e640 1 -- 172.21.15.157:0/771930575 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ab00cfe20 msgr2=0x7f2ab00cb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:32.942 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.939+0000 7f2ab581e640 1 --2- 172.21.15.157:0/771930575 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ab00cfe20 0x7f2ab00cb790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.942 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.939+0000 7f2ab581e640 1 -- 172.21.15.157:0/771930575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ab00d0600 msgr2=0x7f2ab0150d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:32.942 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.939+0000 7f2ab581e640 1 --2- 172.21.15.157:0/771930575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ab00d0600 0x7f2ab0150d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:32.943 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.939+0000 7f2ab581e640 1 -- 172.21.15.157:0/771930575 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab00c9d90 con 0x7f2ab015be50 2024-03-17T23:46:32.943 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.939+0000 7f2ab581e640 1 --2- 172.21.15.157:0/771930575 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ab015be50 0x7f2ab009d1d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2aa800efc0 tx=0x7f2aa800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:32.943 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.939+0000 7f2aa5ffb640 1 -- 172.21.15.157:0/771930575 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa8018030 con 0x7f2ab015be50 2024-03-17T23:46:32.943 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.939+0000 7f2aa5ffb640 1 -- 172.21.15.157:0/771930575 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aa800cd30 con 0x7f2ab015be50 2024-03-17T23:46:32.943 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.939+0000 7f2ab7281640 1 -- 172.21.15.157:0/771930575 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ab00ca070 con 0x7f2ab015be50 2024-03-17T23:46:32.943 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.939+0000 7f2aa5ffb640 1 -- 172.21.15.157:0/771930575 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa8003bb0 con 0x7f2ab015be50 2024-03-17T23:46:32.943 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.940+0000 7f2aa5ffb640 1 -- 172.21.15.157:0/771930575 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2aa801b070 con 0x7f2ab015be50 2024-03-17T23:46:32.943 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.940+0000 7f2ab7281640 1 -- 172.21.15.157:0/771930575 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ab00ca630 con 0x7f2ab015be50 2024-03-17T23:46:32.943 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.940+0000 7f2aa5ffb640 1 -- 172.21.15.157:0/771930575 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f2aa8003d50 con 0x7f2ab015be50 2024-03-17T23:46:32.943 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.940+0000 7f2ab7281640 1 -- 172.21.15.157:0/771930575 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ab00688b0 con 0x7f2ab015be50 2024-03-17T23:46:32.945 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:32.944+0000 7f2aa5ffb640 1 -- 172.21.15.157:0/771930575 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2aa8010040 con 0x7f2ab015be50 2024-03-17T23:46:32.965 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-17T23:46:32.964+0000 7ff97924b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-17T23:46:32.976 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:32.976+0000 7f290723a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-17T23:46:33.017 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.016+0000 7f2ab7281640 1 -- 172.21.15.157:0/771930575 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd new", "uuid": "7e973c64-ea03-4b69-ad16-6f5e6ccfb389", "id": 4} v 0) -- 0x7f2ab00cbcd0 con 0x7f2ab015be50 2024-03-17T23:46:33.022 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-17T23:46:33.019+0000 7ff97924b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-17T23:46:33.022 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.022+0000 7f2aa5ffb640 1 -- 172.21.15.157:0/771930575 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7e973c64-ea03-4b69-ad16-6f5e6ccfb389", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f2aa8003f50 con 0x7f2ab015be50 2024-03-17T23:46:33.023 INFO:teuthology.orchestra.run.smithi157.stdout:4 2024-03-17T23:46:33.024 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.023+0000 7f2ab7281640 1 -- 172.21.15.157:0/771930575 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ab015be50 msgr2=0x7f2ab009d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:33.024 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.023+0000 7f2ab7281640 1 --2- 172.21.15.157:0/771930575 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ab015be50 0x7f2ab009d1d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2aa800efc0 tx=0x7f2aa800c490 comp rx=0 tx=0).stop 2024-03-17T23:46:33.024 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.023+0000 7f2ab7281640 1 -- 172.21.15.157:0/771930575 shutdown_connections 2024-03-17T23:46:33.024 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.023+0000 7f2ab7281640 1 --2- 172.21.15.157:0/771930575 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ab015be50 0x7f2ab009d1d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.024 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.023+0000 7f2ab7281640 1 --2- 172.21.15.157:0/771930575 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ab00d0600 0x7f2ab0150d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.024 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.023+0000 7f2ab7281640 1 --2- 172.21.15.157:0/771930575 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ab00cfe20 0x7f2ab00cb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.024 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.023+0000 7f2ab7281640 1 -- 172.21.15.157:0/771930575 >> 172.21.15.157:0/771930575 conn(0x7f2ab00b3630 msgr2=0x7f2ab00702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:33.024 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.023+0000 7f2ab7281640 1 -- 172.21.15.157:0/771930575 shutdown_connections 2024-03-17T23:46:33.024 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.023+0000 7f2ab7281640 1 -- 172.21.15.157:0/771930575 wait complete. 2024-03-17T23:46:33.025 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:33.025+0000 7f290723a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-17T23:46:33.034 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new ea4552bf-1e67-4e73-87b6-ea8f00131017 5 2024-03-17T23:46:33.069 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-17T23:46:33.069+0000 7ff97924b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-17T23:46:33.169 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.168+0000 7f8f0800b640 1 Processor -- start 2024-03-17T23:46:33.169 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.169+0000 7f8f0800b640 1 -- start start 2024-03-17T23:46:33.171 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.169+0000 7f8f0800b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f000cff90 0x7f8f000b3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:33.171 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.169+0000 7f8f0800b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f000d0570 con 0x7f8f001615b0 2024-03-17T23:46:33.171 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.169+0000 7f8f0800b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f000d0850 con 0x7f8f000b02e0 2024-03-17T23:46:33.171 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.169+0000 7f8f0800b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8f00161990 con 0x7f8f000cff90 2024-03-17T23:46:33.171 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.170+0000 7f8f055a6640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f8f000b02e0 0x7f8f000ad560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:38482/0 (socket says 172.21.15.157:38482) 2024-03-17T23:46:33.171 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.170+0000 7f8f065a8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f000cff90 0x7f8f000b3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:33.171 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.170+0000 7f8f055a6640 1 -- 172.21.15.157:0/3513565156 learned_addr learned my addr 172.21.15.157:0/3513565156 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-17T23:46:33.171 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.170+0000 7f8f04da5640 1 -- 172.21.15.157:0/3513565156 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1823078438 0 0) 0x7f8f000d0850 con 0x7f8f000b02e0 2024-03-17T23:46:33.171 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.170+0000 7f8f065a8640 1 -- 172.21.15.157:0/3513565156 >> v1:172.21.15.157:6789/0 conn(0x7f8f000b02e0 legacy=0x7f8f000ad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:33.171 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.170+0000 7f8f065a8640 1 -- 172.21.15.157:0/3513565156 >> v1:172.21.15.150:6789/0 conn(0x7f8f001615b0 legacy=0x7f8f000afbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:33.171 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.170+0000 7f8f065a8640 1 -- 172.21.15.157:0/3513565156 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f000d0b30 con 0x7f8f000cff90 2024-03-17T23:46:33.172 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.171+0000 7f8f065a8640 1 --2- 172.21.15.157:0/3513565156 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f000cff90 0x7f8f000b3d20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8009dc0 tx=0x7f8ef802bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e2ac31e7a365918 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:33.173 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.171+0000 7f8f04da5640 1 -- 172.21.15.157:0/3513565156 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8039070 con 0x7f8f000cff90 2024-03-17T23:46:33.173 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.171+0000 7f8f04da5640 1 -- 172.21.15.157:0/3513565156 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ef802ccd0 con 0x7f8f000cff90 2024-03-17T23:46:33.173 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.171+0000 7f8f04da5640 1 -- 172.21.15.157:0/3513565156 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8034660 con 0x7f8f000cff90 2024-03-17T23:46:33.173 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.171+0000 7f8f0800b640 1 -- 172.21.15.157:0/3513565156 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f000cff90 msgr2=0x7f8f000b3d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:33.173 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.171+0000 7f8f0800b640 1 --2- 172.21.15.157:0/3513565156 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f000cff90 0x7f8f000b3d20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8009dc0 tx=0x7f8ef802bf20 comp rx=0 tx=0).stop 2024-03-17T23:46:33.173 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.172+0000 7f8f0800b640 1 -- 172.21.15.157:0/3513565156 shutdown_connections 2024-03-17T23:46:33.173 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.172+0000 7f8f0800b640 1 --2- 172.21.15.157:0/3513565156 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f000cff90 0x7f8f000b3d20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.173 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.172+0000 7f8f0800b640 1 -- 172.21.15.157:0/3513565156 >> 172.21.15.157:0/3513565156 conn(0x7f8f00072a40 msgr2=0x7f8f000b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:33.173 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.172+0000 7f8f0800b640 1 -- 172.21.15.157:0/3513565156 shutdown_connections 2024-03-17T23:46:33.173 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.172+0000 7f8f0800b640 1 -- 172.21.15.157:0/3513565156 wait complete. 2024-03-17T23:46:33.176 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.173+0000 7f8f0800b640 1 Processor -- start 2024-03-17T23:46:33.176 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.173+0000 7f8f0800b640 1 -- start start 2024-03-17T23:46:33.176 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.173+0000 7f8f0800b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f000b02e0 0x7f8f0015cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:33.176 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.173+0000 7f8f0800b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f000cff90 0x7f8f0014a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:33.176 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.173+0000 7f8f0800b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f001615b0 0x7f8f0014ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:33.176 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.173+0000 7f8f0800b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8f000adc70 con 0x7f8f000b02e0 2024-03-17T23:46:33.176 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.173+0000 7f8f0800b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8f00099d70 con 0x7f8f000cff90 2024-03-17T23:46:33.176 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.173+0000 7f8f0800b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8f000addf0 con 0x7f8f001615b0 2024-03-17T23:46:33.176 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.174+0000 7f8f055a6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f000cff90 0x7f8f0014a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:33.176 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.174+0000 7f8f065a8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f001615b0 0x7f8f0014ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:33.177 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.174+0000 7f8f065a8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f001615b0 0x7f8f0014ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.157:60948/0 (socket says 172.21.15.157:60948) 2024-03-17T23:46:33.177 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.174+0000 7f8f055a6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f000cff90 0x7f8f0014a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:45960/0 (socket says 172.21.15.157:45960) 2024-03-17T23:46:33.177 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.174+0000 7f8f065a8640 1 -- 172.21.15.157:0/3046296502 learned_addr learned my addr 172.21.15.157:0/3046296502 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-17T23:46:33.177 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.174+0000 7f8f05da7640 1 --2- 172.21.15.157:0/3046296502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f000b02e0 0x7f8f0015cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:33.177 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.174+0000 7f8f055a6640 1 -- 172.21.15.157:0/3046296502 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f001615b0 msgr2=0x7f8f0014ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:33.177 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.174+0000 7f8f055a6640 1 --2- 172.21.15.157:0/3046296502 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f001615b0 0x7f8f0014ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.177 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.174+0000 7f8f055a6640 1 -- 172.21.15.157:0/3046296502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f000b02e0 msgr2=0x7f8f0015cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:33.177 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.174+0000 7f8f055a6640 1 --2- 172.21.15.157:0/3046296502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f000b02e0 0x7f8f0015cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.177 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.174+0000 7f8f055a6640 1 -- 172.21.15.157:0/3046296502 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f00156c40 con 0x7f8f000cff90 2024-03-17T23:46:33.177 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.174+0000 7f8f065a8640 1 --2- 172.21.15.157:0/3046296502 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f001615b0 0x7f8f0014ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:46:33.177 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.174+0000 7f8f055a6640 1 --2- 172.21.15.157:0/3046296502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f000cff90 0x7f8f0014a740 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8ef000c3e0 tx=0x7f8ef000c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:33.177 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.174+0000 7f8ef67fc640 1 -- 172.21.15.157:0/3046296502 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef00074e0 con 0x7f8f000cff90 2024-03-17T23:46:33.177 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.174+0000 7f8ef67fc640 1 -- 172.21.15.157:0/3046296502 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ef0007680 con 0x7f8f000cff90 2024-03-17T23:46:33.177 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.174+0000 7f8ef67fc640 1 -- 172.21.15.157:0/3046296502 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef0004a20 con 0x7f8f000cff90 2024-03-17T23:46:33.177 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.174+0000 7f8f0800b640 1 -- 172.21.15.157:0/3046296502 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f00156f20 con 0x7f8f000cff90 2024-03-17T23:46:33.178 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.174+0000 7f8f0800b640 1 -- 172.21.15.157:0/3046296502 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f001574e0 con 0x7f8f000cff90 2024-03-17T23:46:33.178 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.174+0000 7f8ef67fc640 1 -- 172.21.15.157:0/3046296502 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8ef0011070 con 0x7f8f000cff90 2024-03-17T23:46:33.178 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.174+0000 7f8ef67fc640 1 -- 172.21.15.157:0/3046296502 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f8ef0013940 con 0x7f8f000cff90 2024-03-17T23:46:33.178 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.176+0000 7f8f0800b640 1 -- 172.21.15.157:0/3046296502 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f000688b0 con 0x7f8f000cff90 2024-03-17T23:46:33.178 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.178+0000 7f8ef67fc640 1 -- 172.21.15.157:0/3046296502 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f8ef0004d20 con 0x7f8f000cff90 2024-03-17T23:46:33.248 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.248+0000 7f8f0800b640 1 -- 172.21.15.157:0/3046296502 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ea4552bf-1e67-4e73-87b6-ea8f00131017", "id": 5} v 0) -- 0x7f8f0006c3b0 con 0x7f8f000cff90 2024-03-17T23:46:33.253 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.253+0000 7f8ef67fc640 1 -- 172.21.15.157:0/3046296502 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ea4552bf-1e67-4e73-87b6-ea8f00131017", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f8ef0013d30 con 0x7f8f000cff90 2024-03-17T23:46:33.254 INFO:teuthology.orchestra.run.smithi157.stdout:5 2024-03-17T23:46:33.255 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.254+0000 7f8f0800b640 1 -- 172.21.15.157:0/3046296502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f000cff90 msgr2=0x7f8f0014a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:33.255 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.254+0000 7f8f0800b640 1 --2- 172.21.15.157:0/3046296502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f000cff90 0x7f8f0014a740 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8ef000c3e0 tx=0x7f8ef000c8a0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.255 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.254+0000 7f8f0800b640 1 -- 172.21.15.157:0/3046296502 shutdown_connections 2024-03-17T23:46:33.255 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.254+0000 7f8f0800b640 1 --2- 172.21.15.157:0/3046296502 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f001615b0 0x7f8f0014ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.255 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.254+0000 7f8f0800b640 1 --2- 172.21.15.157:0/3046296502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f000cff90 0x7f8f0014a740 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.255 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.254+0000 7f8f0800b640 1 --2- 172.21.15.157:0/3046296502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f000b02e0 0x7f8f0015cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.255 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.254+0000 7f8f0800b640 1 -- 172.21.15.157:0/3046296502 >> 172.21.15.157:0/3046296502 conn(0x7f8f00072a40 msgr2=0x7f8f0006fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:33.255 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.254+0000 7f8f0800b640 1 -- 172.21.15.157:0/3046296502 shutdown_connections 2024-03-17T23:46:33.255 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.254+0000 7f8f0800b640 1 -- 172.21.15.157:0/3046296502 wait complete. 2024-03-17T23:46:33.265 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new c7411d26-c78e-4ccb-be71-c13a7a91235e 6 2024-03-17T23:46:33.365 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.364+0000 7f95f0590640 1 Processor -- start 2024-03-17T23:46:33.365 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.364+0000 7f95f0590640 1 -- start start 2024-03-17T23:46:33.366 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.365+0000 7f95f0590640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95e8160450 0x7f95e815c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:33.366 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.365+0000 7f95f0590640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95e815cec0 con 0x7f95e815c700 2024-03-17T23:46:33.366 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.365+0000 7f95f0590640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95e815f970 con 0x7f95e80b52a0 2024-03-17T23:46:33.366 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.365+0000 7f95f0590640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f95e80adc40 con 0x7f95e8160450 2024-03-17T23:46:33.366 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.365+0000 7f95edb2b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95e8160450 0x7f95e815c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:33.366 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.365+0000 7f95ee32c640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f95e80b52a0 0x7f95e80b5680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:38496/0 (socket says 172.21.15.157:38496) 2024-03-17T23:46:33.366 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.365+0000 7f95ee32c640 1 -- 172.21.15.157:0/2122194523 learned_addr learned my addr 172.21.15.157:0/2122194523 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-17T23:46:33.367 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.365+0000 7f95edb2b640 1 -- 172.21.15.157:0/2122194523 >> v1:172.21.15.157:6789/0 conn(0x7f95e80b52a0 legacy=0x7f95e80b5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:33.367 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.365+0000 7f95edb2b640 1 -- 172.21.15.157:0/2122194523 >> v1:172.21.15.150:6789/0 conn(0x7f95e815c700 legacy=0x7f95e80af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:33.367 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.365+0000 7f95edb2b640 1 -- 172.21.15.157:0/2122194523 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e815fc50 con 0x7f95e8160450 2024-03-17T23:46:33.367 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.365+0000 7f95ed32a640 1 -- 172.21.15.157:0/2122194523 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 271425966 0 0) 0x7f95e815f970 con 0x7f95e80b52a0 2024-03-17T23:46:33.368 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.366+0000 7f95edb2b640 1 --2- 172.21.15.157:0/2122194523 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95e8160450 0x7f95e815c1c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f95e40097d0 tx=0x7f95e402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5fc51163871c276 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:33.368 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.366+0000 7f95ed32a640 1 -- 172.21.15.157:0/2122194523 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e4035020 con 0x7f95e8160450 2024-03-17T23:46:33.368 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.366+0000 7f95ed32a640 1 -- 172.21.15.157:0/2122194523 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95e4031890 con 0x7f95e8160450 2024-03-17T23:46:33.368 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.366+0000 7f95ed32a640 1 -- 172.21.15.157:0/2122194523 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e4031c60 con 0x7f95e8160450 2024-03-17T23:46:33.368 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.367+0000 7f95f0590640 1 -- 172.21.15.157:0/2122194523 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95e8160450 msgr2=0x7f95e815c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:33.368 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.367+0000 7f95f0590640 1 --2- 172.21.15.157:0/2122194523 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95e8160450 0x7f95e815c1c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f95e40097d0 tx=0x7f95e402b930 comp rx=0 tx=0).stop 2024-03-17T23:46:33.368 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.367+0000 7f95f0590640 1 -- 172.21.15.157:0/2122194523 shutdown_connections 2024-03-17T23:46:33.368 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.367+0000 7f95f0590640 1 --2- 172.21.15.157:0/2122194523 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95e8160450 0x7f95e815c1c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.369 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.367+0000 7f95f0590640 1 -- 172.21.15.157:0/2122194523 >> 172.21.15.157:0/2122194523 conn(0x7f95e8099a70 msgr2=0x7f95e8099e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:33.369 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.367+0000 7f95f0590640 1 -- 172.21.15.157:0/2122194523 shutdown_connections 2024-03-17T23:46:33.369 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.367+0000 7f95f0590640 1 -- 172.21.15.157:0/2122194523 wait complete. 2024-03-17T23:46:33.371 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.368+0000 7f95f0590640 1 Processor -- start 2024-03-17T23:46:33.371 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.368+0000 7f95f0590640 1 -- start start 2024-03-17T23:46:33.371 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.369+0000 7f95f0590640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e80b52a0 0x7f95e80c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:33.371 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.369+0000 7f95f0590640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95e815c700 0x7f95e80709e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:33.371 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.369+0000 7f95f0590640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95e8160450 0x7f95e806f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:33.371 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.369+0000 7f95f0590640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f95e81458a0 con 0x7f95e80b52a0 2024-03-17T23:46:33.371 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.369+0000 7f95f0590640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f95e815cb90 con 0x7f95e815c700 2024-03-17T23:46:33.371 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.369+0000 7f95f0590640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f95e80af910 con 0x7f95e8160450 2024-03-17T23:46:33.371 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.369+0000 7f95edb2b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95e815c700 0x7f95e80709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:33.371 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.369+0000 7f95eeb2d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95e8160450 0x7f95e806f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:33.372 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.369+0000 7f95edb2b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95e815c700 0x7f95e80709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:45968/0 (socket says 172.21.15.157:45968) 2024-03-17T23:46:33.372 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.369+0000 7f95edb2b640 1 -- 172.21.15.157:0/1575126303 learned_addr learned my addr 172.21.15.157:0/1575126303 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-17T23:46:33.372 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.369+0000 7f95ee32c640 1 --2- 172.21.15.157:0/1575126303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e80b52a0 0x7f95e80c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:33.372 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.369+0000 7f95eeb2d640 1 -- 172.21.15.157:0/1575126303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95e815c700 msgr2=0x7f95e80709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:33.372 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.369+0000 7f95eeb2d640 1 --2- 172.21.15.157:0/1575126303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95e815c700 0x7f95e80709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.372 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.369+0000 7f95eeb2d640 1 -- 172.21.15.157:0/1575126303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e80b52a0 msgr2=0x7f95e80c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:33.372 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.369+0000 7f95eeb2d640 1 --2- 172.21.15.157:0/1575126303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e80b52a0 0x7f95e80c9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.372 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.369+0000 7f95eeb2d640 1 -- 172.21.15.157:0/1575126303 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e80ca300 con 0x7f95e8160450 2024-03-17T23:46:33.372 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.369+0000 7f95eeb2d640 1 --2- 172.21.15.157:0/1575126303 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95e8160450 0x7f95e806f890 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f95e000c6c0 tx=0x7f95e000cb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:33.372 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.370+0000 7f95deffd640 1 -- 172.21.15.157:0/1575126303 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e0003b70 con 0x7f95e8160450 2024-03-17T23:46:33.372 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.370+0000 7f95deffd640 1 -- 172.21.15.157:0/1575126303 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95e0003d10 con 0x7f95e8160450 2024-03-17T23:46:33.372 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.370+0000 7f95f0590640 1 -- 172.21.15.157:0/1575126303 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95e80ca5e0 con 0x7f95e8160450 2024-03-17T23:46:33.372 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.370+0000 7f95deffd640 1 -- 172.21.15.157:0/1575126303 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e00052f0 con 0x7f95e8160450 2024-03-17T23:46:33.372 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.370+0000 7f95f0590640 1 -- 172.21.15.157:0/1575126303 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f95e806f370 con 0x7f95e8160450 2024-03-17T23:46:33.372 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.370+0000 7f95deffd640 1 -- 172.21.15.157:0/1575126303 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f95e0005490 con 0x7f95e8160450 2024-03-17T23:46:33.373 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.370+0000 7f95deffd640 1 -- 172.21.15.157:0/1575126303 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f95e0005780 con 0x7f95e8160450 2024-03-17T23:46:33.373 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.371+0000 7f95f0590640 1 -- 172.21.15.157:0/1575126303 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95e80688b0 con 0x7f95e8160450 2024-03-17T23:46:33.373 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.373+0000 7f95deffd640 1 -- 172.21.15.157:0/1575126303 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f95e000a640 con 0x7f95e8160450 2024-03-17T23:46:33.406 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:33.406+0000 7f290723a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-17T23:46:33.445 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.444+0000 7f95f0590640 1 -- 172.21.15.157:0/1575126303 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c7411d26-c78e-4ccb-be71-c13a7a91235e", "id": 6} v 0) -- 0x7f95e80d0ea0 con 0x7f95e8160450 2024-03-17T23:46:33.450 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.449+0000 7f95deffd640 1 -- 172.21.15.157:0/1575126303 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c7411d26-c78e-4ccb-be71-c13a7a91235e", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f95e000a820 con 0x7f95e8160450 2024-03-17T23:46:33.450 INFO:teuthology.orchestra.run.smithi157.stdout:6 2024-03-17T23:46:33.451 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.450+0000 7f95f0590640 1 -- 172.21.15.157:0/1575126303 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95e8160450 msgr2=0x7f95e806f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:33.452 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.450+0000 7f95f0590640 1 --2- 172.21.15.157:0/1575126303 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95e8160450 0x7f95e806f890 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f95e000c6c0 tx=0x7f95e000cb80 comp rx=0 tx=0).stop 2024-03-17T23:46:33.452 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.450+0000 7f95f0590640 1 -- 172.21.15.157:0/1575126303 shutdown_connections 2024-03-17T23:46:33.452 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.450+0000 7f95f0590640 1 --2- 172.21.15.157:0/1575126303 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95e8160450 0x7f95e806f890 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.452 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.450+0000 7f95f0590640 1 --2- 172.21.15.157:0/1575126303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95e815c700 0x7f95e80709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.452 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.450+0000 7f95f0590640 1 --2- 172.21.15.157:0/1575126303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e80b52a0 0x7f95e80c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.452 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.450+0000 7f95f0590640 1 -- 172.21.15.157:0/1575126303 >> 172.21.15.157:0/1575126303 conn(0x7f95e8099a70 msgr2=0x7f95e80b4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:33.452 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.450+0000 7f95f0590640 1 -- 172.21.15.157:0/1575126303 shutdown_connections 2024-03-17T23:46:33.452 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.450+0000 7f95f0590640 1 -- 172.21.15.157:0/1575126303 wait complete. 2024-03-17T23:46:33.461 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --cluster ceph osd new bb6837d5-de8b-4ded-a8eb-c17b930002ee 7 2024-03-17T23:46:33.473 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-17T23:46:33.472+0000 7ff97924b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-17T23:46:33.541 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:33.540+0000 7f290723a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-17T23:46:33.562 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.561+0000 7ff3fdd69640 1 Processor -- start 2024-03-17T23:46:33.563 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.561+0000 7ff3fdd69640 1 -- start start 2024-03-17T23:46:33.565 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.562+0000 7ff3fdd69640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3f80cd4f0 0x7ff3f80cd980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:33.565 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.562+0000 7ff3fdd69640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3f80b2990 con 0x7ff3f80d0940 2024-03-17T23:46:33.565 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.562+0000 7ff3fdd69640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3f80cf8c0 con 0x7ff3f815f5e0 2024-03-17T23:46:33.566 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.562+0000 7ff3fdd69640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff3f80cc950 con 0x7ff3f80cd4f0 2024-03-17T23:46:33.566 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.562+0000 7ff3f77fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7ff3f815f5e0 0x7ff3f80d0230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.157:38506/0 (socket says 172.21.15.157:38506) 2024-03-17T23:46:33.566 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.562+0000 7ff3f77fe640 1 -- 172.21.15.157:0/2157362755 learned_addr learned my addr 172.21.15.157:0/2157362755 (peer_addr_for_me v1:172.21.15.157:0/0) 2024-03-17T23:46:33.566 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.562+0000 7ff3f7fff640 1 --2- 172.21.15.157:0/2157362755 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3f80cd4f0 0x7ff3f80cd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:33.566 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.563+0000 7ff3f67fc640 1 -- 172.21.15.157:0/2157362755 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2268061718 0 0) 0x7ff3f80cf8c0 con 0x7ff3f815f5e0 2024-03-17T23:46:33.566 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.563+0000 7ff3f67fc640 1 -- 172.21.15.157:0/2157362755 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3e0003520 con 0x7ff3f815f5e0 2024-03-17T23:46:33.566 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.563+0000 7ff3f7fff640 1 -- 172.21.15.157:0/2157362755 >> v1:172.21.15.157:6789/0 conn(0x7ff3f815f5e0 legacy=0x7ff3f80d0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:33.566 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.563+0000 7ff3f7fff640 1 -- 172.21.15.157:0/2157362755 >> v1:172.21.15.150:6789/0 conn(0x7ff3f80d0940 legacy=0x7ff3f80ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:33.566 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.563+0000 7ff3f7fff640 1 -- 172.21.15.157:0/2157362755 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3f80cfba0 con 0x7ff3f80cd4f0 2024-03-17T23:46:33.566 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.564+0000 7ff3f7fff640 1 --2- 172.21.15.157:0/2157362755 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3f80cd4f0 0x7ff3f80cd980 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0009dc0 tx=0x7ff3f002be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49cd2b7fd29223a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:33.566 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.564+0000 7ff3f67fc640 1 -- 172.21.15.157:0/2157362755 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f0039070 con 0x7ff3f80cd4f0 2024-03-17T23:46:33.566 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.564+0000 7ff3f67fc640 1 -- 172.21.15.157:0/2157362755 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3f002cc40 con 0x7ff3f80cd4f0 2024-03-17T23:46:33.566 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.564+0000 7ff3f67fc640 1 -- 172.21.15.157:0/2157362755 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f0034440 con 0x7ff3f80cd4f0 2024-03-17T23:46:33.567 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.565+0000 7ff3fdd69640 1 -- 172.21.15.157:0/2157362755 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3f80cd4f0 msgr2=0x7ff3f80cd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:33.567 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.565+0000 7ff3fdd69640 1 --2- 172.21.15.157:0/2157362755 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3f80cd4f0 0x7ff3f80cd980 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0009dc0 tx=0x7ff3f002be90 comp rx=0 tx=0).stop 2024-03-17T23:46:33.567 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.565+0000 7ff3fdd69640 1 -- 172.21.15.157:0/2157362755 shutdown_connections 2024-03-17T23:46:33.567 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.565+0000 7ff3fdd69640 1 --2- 172.21.15.157:0/2157362755 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3f80cd4f0 0x7ff3f80cd980 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.567 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.565+0000 7ff3fdd69640 1 -- 172.21.15.157:0/2157362755 >> 172.21.15.157:0/2157362755 conn(0x7ff3f80b3740 msgr2=0x7ff3f80b3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:33.567 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.565+0000 7ff3fdd69640 1 -- 172.21.15.157:0/2157362755 shutdown_connections 2024-03-17T23:46:33.567 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.565+0000 7ff3fdd69640 1 -- 172.21.15.157:0/2157362755 wait complete. 2024-03-17T23:46:33.569 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.566+0000 7ff3fdd69640 1 Processor -- start 2024-03-17T23:46:33.570 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.566+0000 7ff3fdd69640 1 -- start start 2024-03-17T23:46:33.570 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.567+0000 7ff3fdd69640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3f80cd4f0 0x7ff3f815a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:33.570 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.567+0000 7ff3fdd69640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3f80d0940 0x7ff3f80cb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:33.570 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.567+0000 7ff3fdd69640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3f815f5e0 0x7ff3f80b4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:33.570 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.567+0000 7ff3fdd69640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f80b4550 con 0x7ff3f815f5e0 2024-03-17T23:46:33.570 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.567+0000 7ff3fdd69640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f80b2c30 con 0x7ff3f80d0940 2024-03-17T23:46:33.570 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.567+0000 7ff3fdd69640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff3f80b46d0 con 0x7ff3f80cd4f0 2024-03-17T23:46:33.570 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.567+0000 7ff3f77fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3f80cd4f0 0x7ff3f815a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:33.570 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.567+0000 7ff3f6ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3f80d0940 0x7ff3f80cb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:33.570 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.567+0000 7ff3f77fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3f80cd4f0 0x7ff3f815a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.157:60974/0 (socket says 172.21.15.157:60974) 2024-03-17T23:46:33.570 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.567+0000 7ff3f6ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3f80d0940 0x7ff3f80cb0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:45972/0 (socket says 172.21.15.157:45972) 2024-03-17T23:46:33.570 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.567+0000 7ff3f77fe640 1 -- 172.21.15.157:0/2682764778 learned_addr learned my addr 172.21.15.157:0/2682764778 (peer_addr_for_me v2:172.21.15.157:0/0) 2024-03-17T23:46:33.570 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.567+0000 7ff3f7fff640 1 --2- 172.21.15.157:0/2682764778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3f815f5e0 0x7ff3f80b4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:33.571 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.567+0000 7ff3f6ffd640 1 -- 172.21.15.157:0/2682764778 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3f80cd4f0 msgr2=0x7ff3f815a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:33.571 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.567+0000 7ff3f6ffd640 1 --2- 172.21.15.157:0/2682764778 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3f80cd4f0 0x7ff3f815a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.571 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.567+0000 7ff3f6ffd640 1 -- 172.21.15.157:0/2682764778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3f815f5e0 msgr2=0x7ff3f80b4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:33.571 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.567+0000 7ff3f6ffd640 1 --2- 172.21.15.157:0/2682764778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3f815f5e0 0x7ff3f80b4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.571 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.567+0000 7ff3f6ffd640 1 -- 172.21.15.157:0/2682764778 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3f80c98d0 con 0x7ff3f80d0940 2024-03-17T23:46:33.571 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.567+0000 7ff3f77fe640 1 --2- 172.21.15.157:0/2682764778 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3f80cd4f0 0x7ff3f815a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:46:33.571 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.567+0000 7ff3f6ffd640 1 --2- 172.21.15.157:0/2682764778 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3f80d0940 0x7ff3f80cb0c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff3e800c340 tx=0x7ff3e800c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:33.571 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.567+0000 7ff3dffff640 1 -- 172.21.15.157:0/2682764778 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e80052d0 con 0x7ff3f80d0940 2024-03-17T23:46:33.571 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.568+0000 7ff3dffff640 1 -- 172.21.15.157:0/2682764778 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3e8005490 con 0x7ff3f80d0940 2024-03-17T23:46:33.571 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.568+0000 7ff3fdd69640 1 -- 172.21.15.157:0/2682764778 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3f80c9bb0 con 0x7ff3f80d0940 2024-03-17T23:46:33.571 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.568+0000 7ff3dffff640 1 -- 172.21.15.157:0/2682764778 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e8007260 con 0x7ff3f80d0940 2024-03-17T23:46:33.571 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.568+0000 7ff3fdd69640 1 -- 172.21.15.157:0/2682764778 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3f80ca170 con 0x7ff3f80d0940 2024-03-17T23:46:33.571 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.568+0000 7ff3dffff640 1 -- 172.21.15.157:0/2682764778 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff3e8007400 con 0x7ff3f80d0940 2024-03-17T23:46:33.571 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.568+0000 7ff3dffff640 1 -- 172.21.15.157:0/2682764778 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7ff3e8011bf0 con 0x7ff3f80d0940 2024-03-17T23:46:33.572 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.569+0000 7ff3fdd69640 1 -- 172.21.15.157:0/2682764778 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3f80688b0 con 0x7ff3f80d0940 2024-03-17T23:46:33.572 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.571+0000 7ff3dffff640 1 -- 172.21.15.157:0/2682764778 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff3e80035f0 con 0x7ff3f80d0940 2024-03-17T23:46:33.614 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-17T23:46:33.614+0000 7ff97924b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-17T23:46:33.642 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.641+0000 7ff3fdd69640 1 -- 172.21.15.157:0/2682764778 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd new", "uuid": "bb6837d5-de8b-4ded-a8eb-c17b930002ee", "id": 7} v 0) -- 0x7ff3f80b7bd0 con 0x7ff3f80d0940 2024-03-17T23:46:33.647 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.646+0000 7ff3dffff640 1 -- 172.21.15.157:0/2682764778 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bb6837d5-de8b-4ded-a8eb-c17b930002ee", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7ff3e8003a50 con 0x7ff3f80d0940 2024-03-17T23:46:33.648 INFO:teuthology.orchestra.run.smithi157.stdout:7 2024-03-17T23:46:33.648 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.647+0000 7ff3fdd69640 1 -- 172.21.15.157:0/2682764778 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3f80d0940 msgr2=0x7ff3f80cb0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:33.649 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.647+0000 7ff3fdd69640 1 --2- 172.21.15.157:0/2682764778 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3f80d0940 0x7ff3f80cb0c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff3e800c340 tx=0x7ff3e800c800 comp rx=0 tx=0).stop 2024-03-17T23:46:33.649 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.647+0000 7ff3fdd69640 1 -- 172.21.15.157:0/2682764778 shutdown_connections 2024-03-17T23:46:33.649 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.647+0000 7ff3fdd69640 1 --2- 172.21.15.157:0/2682764778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3f815f5e0 0x7ff3f80b4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.649 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.647+0000 7ff3fdd69640 1 --2- 172.21.15.157:0/2682764778 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3f80d0940 0x7ff3f80cb0c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.649 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.647+0000 7ff3fdd69640 1 --2- 172.21.15.157:0/2682764778 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3f80cd4f0 0x7ff3f815a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.649 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.647+0000 7ff3fdd69640 1 -- 172.21.15.157:0/2682764778 >> 172.21.15.157:0/2682764778 conn(0x7ff3f80b3740 msgr2=0x7ff3f8070810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:33.649 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.647+0000 7ff3fdd69640 1 -- 172.21.15.157:0/2682764778 shutdown_connections 2024-03-17T23:46:33.649 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:46:33.647+0000 7ff3fdd69640 1 -- 172.21.15.157:0/2682764778 wait complete. 2024-03-17T23:46:33.657 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:33.656+0000 7f290723a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-17T23:46:33.658 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-17T23:46:33.658 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-17T23:46:33.661 INFO:tasks.ceph.osd.0:Started 2024-03-17T23:46:33.661 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-17T23:46:33.661 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-17T23:46:33.663 INFO:tasks.ceph.osd.1:Started 2024-03-17T23:46:33.663 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-17T23:46:33.663 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-17T23:46:33.665 INFO:tasks.ceph.osd.2:Started 2024-03-17T23:46:33.665 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-17T23:46:33.665 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-17T23:46:33.667 INFO:tasks.ceph.osd.3:Started 2024-03-17T23:46:33.667 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-17T23:46:33.668 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-17T23:46:33.701 INFO:tasks.ceph.osd.4:Started 2024-03-17T23:46:33.701 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-17T23:46:33.701 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-17T23:46:33.703 INFO:tasks.ceph.osd.5:Started 2024-03-17T23:46:33.703 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-17T23:46:33.703 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-17T23:46:33.705 INFO:tasks.ceph.osd.6:Started 2024-03-17T23:46:33.705 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-17T23:46:33.706 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-17T23:46:33.708 INFO:tasks.ceph.osd.7:Started 2024-03-17T23:46:33.708 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:46:33.715 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:33.715+0000 7f290723a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-17T23:46:33.741 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-17T23:46:33.740+0000 7ff97924b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-17T23:46:33.807 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-17T23:46:33.806+0000 7ff97924b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-17T23:46:33.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.819+0000 7fd3a3959640 1 Processor -- start 2024-03-17T23:46:33.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.819+0000 7fd3a3959640 1 -- start start 2024-03-17T23:46:33.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.820+0000 7fd3a3959640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd39c0cf120 0x7fd39c1619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:33.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.820+0000 7fd3a3959640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd39c0b9860 0x7fd39c0d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:33.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.820+0000 7fd3a3959640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd39c1620a0 con 0x7fd39c15de30 2024-03-17T23:46:33.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.820+0000 7fd3a3959640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fd39c162940 con 0x7fd39c0cf120 2024-03-17T23:46:33.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.820+0000 7fd3a3959640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd39c162380 con 0x7fd39c0b9860 2024-03-17T23:46:33.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.820+0000 7fd3a0ef4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd39c0cf120 0x7fd39c1619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:33.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.820+0000 7fd3a0ef4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd39c0cf120 0x7fd39c1619a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60376/0 (socket says 172.21.15.150:60376) 2024-03-17T23:46:33.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.820+0000 7fd3a16f5640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd39c15de30 0x7fd39c0cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:40374/0 (socket says 172.21.15.150:40374) 2024-03-17T23:46:33.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.820+0000 7fd3a0ef4640 1 -- 172.21.15.150:0/3601307456 learned_addr learned my addr 172.21.15.150:0/3601307456 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:33.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.820+0000 7fd3a1ef6640 1 --2- 172.21.15.150:0/3601307456 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd39c0b9860 0x7fd39c0d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:33.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.820+0000 7fd3a0ef4640 1 -- 172.21.15.150:0/3601307456 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd39c0b9860 msgr2=0x7fd39c0d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:33.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.820+0000 7fd3a0ef4640 1 --2- 172.21.15.150:0/3601307456 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd39c0b9860 0x7fd39c0d2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.820+0000 7fd3a0ef4640 1 -- 172.21.15.150:0/3601307456 >> v1:172.21.15.150:6789/0 conn(0x7fd39c15de30 legacy=0x7fd39c0cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:33.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.820+0000 7fd3a0ef4640 1 -- 172.21.15.150:0/3601307456 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd39c162660 con 0x7fd39c0cf120 2024-03-17T23:46:33.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.820+0000 7fd393fff640 1 -- 172.21.15.150:0/3601307456 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 734682552 0 0) 0x7fd39c1620a0 con 0x7fd39c15de30 2024-03-17T23:46:33.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.821+0000 7fd3a0ef4640 1 --2- 172.21.15.150:0/3601307456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd39c0cf120 0x7fd39c1619a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd398009870 tx=0x7fd39802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=251a72bcc25c4958 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:33.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.821+0000 7fd393fff640 1 -- 172.21.15.150:0/3601307456 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd398035020 con 0x7fd39c0cf120 2024-03-17T23:46:33.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.821+0000 7fd393fff640 1 -- 172.21.15.150:0/3601307456 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd398031640 con 0x7fd39c0cf120 2024-03-17T23:46:33.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.821+0000 7fd393fff640 1 -- 172.21.15.150:0/3601307456 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd39802d040 con 0x7fd39c0cf120 2024-03-17T23:46:33.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.822+0000 7fd3a3959640 1 -- 172.21.15.150:0/3601307456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd39c0cf120 msgr2=0x7fd39c1619a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:33.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.822+0000 7fd3a3959640 1 --2- 172.21.15.150:0/3601307456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd39c0cf120 0x7fd39c1619a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd398009870 tx=0x7fd39802b8f0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.822+0000 7fd3a3959640 1 -- 172.21.15.150:0/3601307456 shutdown_connections 2024-03-17T23:46:33.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.822+0000 7fd3a3959640 1 --2- 172.21.15.150:0/3601307456 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd39c0b9860 0x7fd39c0d2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.822+0000 7fd3a3959640 1 --2- 172.21.15.150:0/3601307456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd39c0cf120 0x7fd39c1619a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.822+0000 7fd3a3959640 1 -- 172.21.15.150:0/3601307456 >> 172.21.15.150:0/3601307456 conn(0x7fd39c0730b0 msgr2=0x7fd39c0d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:33.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.822+0000 7fd3a3959640 1 -- 172.21.15.150:0/3601307456 shutdown_connections 2024-03-17T23:46:33.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.822+0000 7fd3a3959640 1 -- 172.21.15.150:0/3601307456 wait complete. 2024-03-17T23:46:33.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.823+0000 7fd3a3959640 1 Processor -- start 2024-03-17T23:46:33.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.823+0000 7fd3a3959640 1 -- start start 2024-03-17T23:46:33.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.824+0000 7fd3a3959640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd39c0b9860 0x7fd39c14ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:33.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.824+0000 7fd3a3959640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd39c0cf120 0x7fd39c0aa730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:33.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.824+0000 7fd3a3959640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd39c15de30 0x7fd39c0cd170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:33.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.824+0000 7fd3a3959640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd39c15e210 con 0x7fd39c15de30 2024-03-17T23:46:33.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.824+0000 7fd3a3959640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd39c0b9c60 con 0x7fd39c0cf120 2024-03-17T23:46:33.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.824+0000 7fd3a3959640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd39c0d2bd0 con 0x7fd39c0b9860 2024-03-17T23:46:33.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.824+0000 7fd3a1ef6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd39c15de30 0x7fd39c0cd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:33.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.824+0000 7fd3a1ef6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd39c15de30 0x7fd39c0cd170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60378/0 (socket says 172.21.15.150:60378) 2024-03-17T23:46:33.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.824+0000 7fd3a1ef6640 1 -- 172.21.15.150:0/1174655485 learned_addr learned my addr 172.21.15.150:0/1174655485 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:33.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.824+0000 7fd3a16f5640 1 --2- 172.21.15.150:0/1174655485 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd39c0b9860 0x7fd39c14ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:33.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.824+0000 7fd3a0ef4640 1 --2- 172.21.15.150:0/1174655485 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd39c0cf120 0x7fd39c0aa730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:33.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.824+0000 7fd3a1ef6640 1 -- 172.21.15.150:0/1174655485 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd39c0b9860 msgr2=0x7fd39c14ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:33.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.824+0000 7fd3a1ef6640 1 --2- 172.21.15.150:0/1174655485 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd39c0b9860 0x7fd39c14ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.824+0000 7fd3a1ef6640 1 -- 172.21.15.150:0/1174655485 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd39c0cf120 msgr2=0x7fd39c0aa730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:33.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.824+0000 7fd3a1ef6640 1 --2- 172.21.15.150:0/1174655485 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd39c0cf120 0x7fd39c0aa730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.824+0000 7fd3a1ef6640 1 -- 172.21.15.150:0/1174655485 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd39c0cc0d0 con 0x7fd39c15de30 2024-03-17T23:46:33.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.824+0000 7fd3a0ef4640 1 --2- 172.21.15.150:0/1174655485 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd39c0cf120 0x7fd39c0aa730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:46:33.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.824+0000 7fd3a1ef6640 1 --2- 172.21.15.150:0/1174655485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd39c15de30 0x7fd39c0cd170 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd39400efc0 tx=0x7fd39400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:33.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.824+0000 7fd391ffb640 1 -- 172.21.15.150:0/1174655485 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd394018020 con 0x7fd39c15de30 2024-03-17T23:46:33.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.824+0000 7fd391ffb640 1 -- 172.21.15.150:0/1174655485 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd394014480 con 0x7fd39c15de30 2024-03-17T23:46:33.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.824+0000 7fd3a3959640 1 -- 172.21.15.150:0/1174655485 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd39c0cc3b0 con 0x7fd39c15de30 2024-03-17T23:46:33.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.825+0000 7fd391ffb640 1 -- 172.21.15.150:0/1174655485 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd394014870 con 0x7fd39c15de30 2024-03-17T23:46:33.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.825+0000 7fd3a3959640 1 -- 172.21.15.150:0/1174655485 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd39c0cc970 con 0x7fd39c15de30 2024-03-17T23:46:33.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.825+0000 7fd391ffb640 1 -- 172.21.15.150:0/1174655485 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd394005d10 con 0x7fd39c15de30 2024-03-17T23:46:33.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.825+0000 7fd391ffb640 1 -- 172.21.15.150:0/1174655485 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fd394013ab0 con 0x7fd39c15de30 2024-03-17T23:46:33.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.826+0000 7fd3a3959640 1 -- 172.21.15.150:0/1174655485 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd39c0681a0 con 0x7fd39c15de30 2024-03-17T23:46:33.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.828+0000 7fd391ffb640 1 -- 172.21.15.150:0/1174655485 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fd394022070 con 0x7fd39c15de30 2024-03-17T23:46:33.850 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:33.850+0000 7f290723a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-17T23:46:33.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.888+0000 7fd3a3959640 1 -- 172.21.15.150:0/1174655485 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd39c0b9620 con 0x7fd39c15de30 2024-03-17T23:46:33.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.889+0000 7fd391ffb640 1 -- 172.21.15.150:0/1174655485 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fd394010040 con 0x7fd39c15de30 2024-03-17T23:46:33.890 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:46:33.890 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":10,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:46:33.643210+0000","last_up_change":"0.000000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:46:33.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.890+0000 7fd3a3959640 1 -- 172.21.15.150:0/1174655485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd39c15de30 msgr2=0x7fd39c0cd170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:33.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.890+0000 7fd3a3959640 1 --2- 172.21.15.150:0/1174655485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd39c15de30 0x7fd39c0cd170 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd39400efc0 tx=0x7fd39400c490 comp rx=0 tx=0).stop 2024-03-17T23:46:33.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.890+0000 7fd3a3959640 1 -- 172.21.15.150:0/1174655485 shutdown_connections 2024-03-17T23:46:33.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.890+0000 7fd3a3959640 1 --2- 172.21.15.150:0/1174655485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd39c15de30 0x7fd39c0cd170 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.890+0000 7fd3a3959640 1 --2- 172.21.15.150:0/1174655485 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd39c0cf120 0x7fd39c0aa730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.890+0000 7fd3a3959640 1 --2- 172.21.15.150:0/1174655485 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd39c0b9860 0x7fd39c14ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:33.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.890+0000 7fd3a3959640 1 -- 172.21.15.150:0/1174655485 >> 172.21.15.150:0/1174655485 conn(0x7fd39c0730b0 msgr2=0x7fd39c0d43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:33.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.890+0000 7fd3a3959640 1 -- 172.21.15.150:0/1174655485 shutdown_connections 2024-03-17T23:46:33.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:33.890+0000 7fd3a3959640 1 -- 172.21.15.150:0/1174655485 wait complete. 2024-03-17T23:46:33.901 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-17T23:46:33.902 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-17T23:46:33.919 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-17T23:46:33.918+0000 7ff97924b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-17T23:46:33.948 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:33.948+0000 7f290723a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-17T23:46:34.013 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-17T23:46:34.013+0000 7ff97924b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-17T23:46:34.104 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:34.103+0000 7f290723a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-17T23:46:34.145 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:34.145+0000 7f290723a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-17T23:46:34.170 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-17T23:46:34.170+0000 7ff97924b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-17T23:46:34.213 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-17T23:46:34.212+0000 7ff97924b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-17T23:46:34.271 INFO:tasks.ceph.osd.0.smithi150.stderr:2024-03-17T23:46:34.270+0000 7f9f78a0f640 -1 Falling back to public interface 2024-03-17T23:46:34.280 INFO:tasks.ceph.osd.1.smithi150.stderr:2024-03-17T23:46:34.279+0000 7f1d65821640 -1 Falling back to public interface 2024-03-17T23:46:34.291 INFO:tasks.ceph.osd.3.smithi150.stderr:2024-03-17T23:46:34.290+0000 7f9dc7b73640 -1 Falling back to public interface 2024-03-17T23:46:34.291 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-03-17T23:46:34.290+0000 7efcba6c4640 -1 Falling back to public interface 2024-03-17T23:46:34.302 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:34.301+0000 7f290723a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-17T23:46:34.305 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-03-17T23:46:34.304+0000 7f97c97fd640 -1 Falling back to public interface 2024-03-17T23:46:34.309 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-03-17T23:46:34.308+0000 7f2cc3511640 -1 Falling back to public interface 2024-03-17T23:46:34.309 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-03-17T23:46:34.308+0000 7f03a0813640 -1 Falling back to public interface 2024-03-17T23:46:34.319 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-03-17T23:46:34.318+0000 7f2037949640 -1 Falling back to public interface 2024-03-17T23:46:34.375 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-17T23:46:34.375+0000 7ff97924b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-17T23:46:34.708 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-17T23:46:34.799 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.798+0000 7f5c2ebc6640 1 Processor -- start 2024-03-17T23:46:34.799 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.799+0000 7f5c2ebc6640 1 -- start start 2024-03-17T23:46:34.800 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.799+0000 7f5c2ebc6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c2815e0c0 0x7f5c28162730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:34.801 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.799+0000 7f5c2ebc6640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5c280afa70 0x7f5c280b5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:34.801 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.799+0000 7f5c2ebc6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c280affe0 con 0x7f5c2806b200 2024-03-17T23:46:34.801 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.799+0000 7f5c2ebc6640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f5c2815f4b0 con 0x7f5c280afa70 2024-03-17T23:46:34.801 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.799+0000 7f5c2ebc6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5c28161c50 con 0x7f5c2815e0c0 2024-03-17T23:46:34.801 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.800+0000 7f5c2c962640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c2815e0c0 0x7f5c28162730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:34.801 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.800+0000 7f5c27fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5c280afa70 0x7f5c280b5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:34.801 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.800+0000 7f5c27fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5c280afa70 0x7f5c280b5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:46032/0 (socket says 172.21.15.150:46032) 2024-03-17T23:46:34.801 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.800+0000 7f5c2c962640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c2815e0c0 0x7f5c28162730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:37962/0 (socket says 172.21.15.150:37962) 2024-03-17T23:46:34.801 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.800+0000 7f5c2d163640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f5c2806b200 0x7f5c280d0810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:53608/0 (socket says 172.21.15.150:53608) 2024-03-17T23:46:34.801 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.800+0000 7f5c27fff640 1 -- 172.21.15.150:0/3906187196 learned_addr learned my addr 172.21.15.150:0/3906187196 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:34.801 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.800+0000 7f5c2c962640 1 -- 172.21.15.150:0/3906187196 >> v2:172.21.15.157:3300/0 conn(0x7f5c280afa70 msgr2=0x7f5c280b5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:34.801 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.800+0000 7f5c2c962640 1 --2- 172.21.15.150:0/3906187196 >> v2:172.21.15.157:3300/0 conn(0x7f5c280afa70 0x7f5c280b5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:34.801 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.800+0000 7f5c2c962640 1 -- 172.21.15.150:0/3906187196 >> v1:172.21.15.157:6789/0 conn(0x7f5c2806b200 legacy=0x7f5c280d0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:34.801 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.800+0000 7f5c2c962640 1 -- 172.21.15.150:0/3906187196 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c28161f30 con 0x7f5c2815e0c0 2024-03-17T23:46:34.802 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.800+0000 7f5c27fff640 1 --2- 172.21.15.150:0/3906187196 >> v2:172.21.15.157:3300/0 conn(0x7f5c280afa70 0x7f5c280b5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:46:34.803 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.801+0000 7f5c2c962640 1 --2- 172.21.15.150:0/3906187196 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c2815e0c0 0x7f5c28162730 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5c180097a0 tx=0x7f5c1802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14afc4aeec74be75 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:34.803 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.801+0000 7f5c277fe640 1 -- 172.21.15.150:0/3906187196 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c180315a0 con 0x7f5c2815e0c0 2024-03-17T23:46:34.803 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.801+0000 7f5c277fe640 1 -- 172.21.15.150:0/3906187196 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c18037070 con 0x7f5c2815e0c0 2024-03-17T23:46:34.803 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.801+0000 7f5c277fe640 1 -- 172.21.15.150:0/3906187196 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c18031960 con 0x7f5c2815e0c0 2024-03-17T23:46:34.803 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.801+0000 7f5c2ebc6640 1 -- 172.21.15.150:0/3906187196 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c2815e0c0 msgr2=0x7f5c28162730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:34.803 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.801+0000 7f5c2ebc6640 1 --2- 172.21.15.150:0/3906187196 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c2815e0c0 0x7f5c28162730 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5c180097a0 tx=0x7f5c1802b840 comp rx=0 tx=0).stop 2024-03-17T23:46:34.803 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.801+0000 7f5c2ebc6640 1 -- 172.21.15.150:0/3906187196 shutdown_connections 2024-03-17T23:46:34.803 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.801+0000 7f5c2ebc6640 1 --2- 172.21.15.150:0/3906187196 >> v2:172.21.15.157:3300/0 conn(0x7f5c280afa70 0x7f5c280b5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:34.803 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.801+0000 7f5c2ebc6640 1 --2- 172.21.15.150:0/3906187196 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c2815e0c0 0x7f5c28162730 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:34.803 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.801+0000 7f5c2ebc6640 1 -- 172.21.15.150:0/3906187196 >> 172.21.15.150:0/3906187196 conn(0x7f5c280b61a0 msgr2=0x7f5c28099a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:34.804 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.801+0000 7f5c2ebc6640 1 -- 172.21.15.150:0/3906187196 shutdown_connections 2024-03-17T23:46:34.804 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.801+0000 7f5c2ebc6640 1 -- 172.21.15.150:0/3906187196 wait complete. 2024-03-17T23:46:34.804 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.802+0000 7f5c2ebc6640 1 Processor -- start 2024-03-17T23:46:34.804 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.803+0000 7f5c2ebc6640 1 -- start start 2024-03-17T23:46:34.806 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.803+0000 7f5c2ebc6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c2806b200 0x7f5c281485e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:34.806 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.803+0000 7f5c2ebc6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c280afa70 0x7f5c281531f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:34.806 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.803+0000 7f5c2ebc6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c2815e0c0 0x7f5c28153730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:34.806 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.803+0000 7f5c2ebc6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2806b690 con 0x7f5c2806b200 2024-03-17T23:46:34.806 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.803+0000 7f5c2ebc6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5c280b5e60 con 0x7f5c280afa70 2024-03-17T23:46:34.806 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.803+0000 7f5c2ebc6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5c2806b810 con 0x7f5c2815e0c0 2024-03-17T23:46:34.806 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.803+0000 7f5c2c962640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c2806b200 0x7f5c281485e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:34.806 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.803+0000 7f5c2c962640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c2806b200 0x7f5c281485e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60380/0 (socket says 172.21.15.150:60380) 2024-03-17T23:46:34.807 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.803+0000 7f5c2c962640 1 -- 172.21.15.150:0/1498961312 learned_addr learned my addr 172.21.15.150:0/1498961312 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:34.807 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.804+0000 7f5c27fff640 1 --2- 172.21.15.150:0/1498961312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c280afa70 0x7f5c281531f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:34.807 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.804+0000 7f5c2d163640 1 --2- 172.21.15.150:0/1498961312 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c2815e0c0 0x7f5c28153730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:34.807 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.804+0000 7f5c2c962640 1 -- 172.21.15.150:0/1498961312 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c2815e0c0 msgr2=0x7f5c28153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:34.807 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.804+0000 7f5c2c962640 1 --2- 172.21.15.150:0/1498961312 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c2815e0c0 0x7f5c28153730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:34.807 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.804+0000 7f5c2c962640 1 -- 172.21.15.150:0/1498961312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c280afa70 msgr2=0x7f5c281531f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:34.807 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.804+0000 7f5c2c962640 1 --2- 172.21.15.150:0/1498961312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c280afa70 0x7f5c281531f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:34.807 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.804+0000 7f5c2c962640 1 -- 172.21.15.150:0/1498961312 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c28157430 con 0x7f5c2806b200 2024-03-17T23:46:34.807 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.804+0000 7f5c2c962640 1 --2- 172.21.15.150:0/1498961312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c2806b200 0x7f5c281485e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5c18000c00 tx=0x7f5c180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:34.807 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.804+0000 7f5c257fa640 1 -- 172.21.15.150:0/1498961312 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c18032630 con 0x7f5c2806b200 2024-03-17T23:46:34.807 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.804+0000 7f5c257fa640 1 -- 172.21.15.150:0/1498961312 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c180327f0 con 0x7f5c2806b200 2024-03-17T23:46:34.807 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.804+0000 7f5c2ebc6640 1 -- 172.21.15.150:0/1498961312 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c28148b20 con 0x7f5c2806b200 2024-03-17T23:46:34.807 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.804+0000 7f5c257fa640 1 -- 172.21.15.150:0/1498961312 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c18032be0 con 0x7f5c2806b200 2024-03-17T23:46:34.808 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.804+0000 7f5c2ebc6640 1 -- 172.21.15.150:0/1498961312 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c28154070 con 0x7f5c2806b200 2024-03-17T23:46:34.808 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.804+0000 7f5c257fa640 1 -- 172.21.15.150:0/1498961312 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5c18045780 con 0x7f5c2806b200 2024-03-17T23:46:34.808 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.804+0000 7f5c257fa640 1 -- 172.21.15.150:0/1498961312 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f5c1802f030 con 0x7f5c2806b200 2024-03-17T23:46:34.808 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.805+0000 7f5c2ebc6640 1 -- 172.21.15.150:0/1498961312 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c28075280 con 0x7f5c2806b200 2024-03-17T23:46:34.808 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.807+0000 7f5c257fa640 1 -- 172.21.15.150:0/1498961312 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f5c28148b20 con 0x7f5c2806b200 2024-03-17T23:46:34.869 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.869+0000 7f5c2ebc6640 1 -- 172.21.15.150:0/1498961312 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5c280733a0 con 0x7f5c2806b200 2024-03-17T23:46:34.870 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.869+0000 7f5c257fa640 1 -- 172.21.15.150:0/1498961312 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f5c28157430 con 0x7f5c2806b200 2024-03-17T23:46:34.870 INFO:teuthology.misc.health.smithi150.stdout: 2024-03-17T23:46:34.870 INFO:teuthology.misc.health.smithi150.stdout:{"epoch":10,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:46:33.643210+0000","last_up_change":"0.000000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:46:34.871 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.870+0000 7f5c2ebc6640 1 -- 172.21.15.150:0/1498961312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c2806b200 msgr2=0x7f5c281485e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:34.871 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.870+0000 7f5c2ebc6640 1 --2- 172.21.15.150:0/1498961312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c2806b200 0x7f5c281485e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5c18000c00 tx=0x7f5c180046c0 comp rx=0 tx=0).stop 2024-03-17T23:46:34.871 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.870+0000 7f5c2ebc6640 1 -- 172.21.15.150:0/1498961312 shutdown_connections 2024-03-17T23:46:34.871 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.870+0000 7f5c2ebc6640 1 --2- 172.21.15.150:0/1498961312 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c2815e0c0 0x7f5c28153730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:34.871 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.870+0000 7f5c2ebc6640 1 --2- 172.21.15.150:0/1498961312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c280afa70 0x7f5c281531f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:34.871 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.870+0000 7f5c2ebc6640 1 --2- 172.21.15.150:0/1498961312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c2806b200 0x7f5c281485e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:34.871 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.870+0000 7f5c2ebc6640 1 -- 172.21.15.150:0/1498961312 >> 172.21.15.150:0/1498961312 conn(0x7f5c280b61a0 msgr2=0x7f5c280b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:34.871 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.870+0000 7f5c2ebc6640 1 -- 172.21.15.150:0/1498961312 shutdown_connections 2024-03-17T23:46:34.871 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:34.870+0000 7f5c2ebc6640 1 -- 172.21.15.150:0/1498961312 wait complete. 2024-03-17T23:46:34.880 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-17T23:46:35.132 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:35.132+0000 7f290723a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-17T23:46:35.192 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-17T23:46:35.192+0000 7ff97924b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-17T23:46:35.408 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:35.408+0000 7f290723a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-17T23:46:35.460 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:35.460+0000 7f290723a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-17T23:46:35.470 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-17T23:46:35.470+0000 7ff97924b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-17T23:46:35.521 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-17T23:46:35.520+0000 7ff97924b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-17T23:46:35.573 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:35.573+0000 7f290723a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-17T23:46:35.639 INFO:tasks.ceph.mgr.x.smithi157.stderr:2024-03-17T23:46:35.639+0000 7ff97924b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-17T23:46:36.598 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:36.597+0000 7f28a7ec2640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-17T23:46:36.598 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-03-17T23:46:36.598+0000 7f28a7ec2640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-17T23:46:37.721 INFO:tasks.ceph.osd.0.smithi150.stderr:2024-03-17T23:46:37.720+0000 7f9f78a0f640 -1 osd.0 0 log_to_monitors true 2024-03-17T23:46:37.760 INFO:tasks.ceph.osd.1.smithi150.stderr:2024-03-17T23:46:37.759+0000 7f1d65821640 -1 osd.1 0 log_to_monitors true 2024-03-17T23:46:37.760 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-03-17T23:46:37.759+0000 7f97c97fd640 -1 osd.7 0 log_to_monitors true 2024-03-17T23:46:37.799 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-03-17T23:46:37.799+0000 7f2cc3511640 -1 osd.4 0 log_to_monitors true 2024-03-17T23:46:37.805 INFO:tasks.ceph.osd.3.smithi150.stderr:2024-03-17T23:46:37.805+0000 7f9dc7b73640 -1 osd.3 0 log_to_monitors true 2024-03-17T23:46:37.807 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-03-17T23:46:37.806+0000 7f03a0813640 -1 osd.6 0 log_to_monitors true 2024-03-17T23:46:37.810 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-03-17T23:46:37.809+0000 7efcba6c4640 -1 osd.2 0 log_to_monitors true 2024-03-17T23:46:37.827 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-03-17T23:46:37.827+0000 7f2037949640 -1 osd.5 0 log_to_monitors true 2024-03-17T23:46:40.984 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-03-17T23:46:40.983+0000 7f039c148640 -1 osd.6 0 waiting for initial osdmap 2024-03-17T23:46:40.996 INFO:tasks.ceph.osd.6.smithi157.stderr:2024-03-17T23:46:40.995+0000 7f0393f68640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-17T23:46:41.028 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-03-17T23:46:41.028+0000 7f2cbf4a7640 -1 osd.4 0 waiting for initial osdmap 2024-03-17T23:46:41.036 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-03-17T23:46:41.036+0000 7f97c5fa7640 -1 osd.7 0 waiting for initial osdmap 2024-03-17T23:46:41.037 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-03-17T23:46:41.037+0000 7f20338df640 -1 osd.5 0 waiting for initial osdmap 2024-03-17T23:46:41.043 INFO:tasks.ceph.osd.4.smithi157.stderr:2024-03-17T23:46:41.043+0000 7f2cb72c7640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-17T23:46:41.047 INFO:tasks.ceph.osd.5.smithi157.stderr:2024-03-17T23:46:41.048+0000 7f202b6ff640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-17T23:46:41.047 INFO:tasks.ceph.osd.7.smithi157.stderr:2024-03-17T23:46:41.048+0000 7f97bd5b3640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-17T23:46:41.556 INFO:tasks.ceph.osd.1.smithi150.stderr:2024-03-17T23:46:41.556+0000 7f1d611f7640 -1 osd.1 0 waiting for initial osdmap 2024-03-17T23:46:41.569 INFO:tasks.ceph.osd.1.smithi150.stderr:2024-03-17T23:46:41.569+0000 7f1d59017640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-17T23:46:41.688 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-17T23:46:41.728 INFO:tasks.ceph.osd.3.smithi150.stderr:2024-03-17T23:46:41.727+0000 7f9dc3b09640 -1 osd.3 0 waiting for initial osdmap 2024-03-17T23:46:41.738 INFO:tasks.ceph.osd.0.smithi150.stderr:2024-03-17T23:46:41.738+0000 7f9f7430f640 -1 osd.0 0 waiting for initial osdmap 2024-03-17T23:46:41.741 INFO:tasks.ceph.osd.3.smithi150.stderr:2024-03-17T23:46:41.741+0000 7f9dbb929640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-17T23:46:41.743 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-03-17T23:46:41.743+0000 7efcb665a640 -1 osd.2 0 waiting for initial osdmap 2024-03-17T23:46:41.752 INFO:tasks.ceph.osd.0.smithi150.stderr:2024-03-17T23:46:41.752+0000 7f9f6c12f640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-17T23:46:41.761 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-03-17T23:46:41.761+0000 7efcae47a640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-17T23:46:41.791 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.791+0000 7fd4660dc640 1 Processor -- start 2024-03-17T23:46:41.791 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.791+0000 7fd4660dc640 1 -- start start 2024-03-17T23:46:41.796 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.791+0000 7fd4660dc640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd46015be10 0x7fd460099d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:41.796 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.791+0000 7fd4660dc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4600cd8e0 0x7fd46009a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:41.796 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.791+0000 7fd4660dc640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4600cc840 con 0x7fd4600d0100 2024-03-17T23:46:41.796 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.791+0000 7fd4660dc640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd4600ccb20 con 0x7fd46015be10 2024-03-17T23:46:41.796 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.791+0000 7fd4660dc640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd4600cce00 con 0x7fd4600cd8e0 2024-03-17T23:46:41.796 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.791+0000 7fd45ffff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd4600d0100 0x7fd4600d0590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:47002/0 (socket says 172.21.15.150:47002) 2024-03-17T23:46:41.796 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45ffff640 1 -- 172.21.15.150:0/3679065501 learned_addr learned my addr 172.21.15.150:0/3679065501 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:46:41.796 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45e7fc640 1 -- 172.21.15.150:0/3679065501 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4258823151 0 0) 0x7fd4600cc840 con 0x7fd4600d0100 2024-03-17T23:46:41.796 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45e7fc640 1 -- 172.21.15.150:0/3679065501 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd448003520 con 0x7fd4600d0100 2024-03-17T23:46:41.796 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45f7fe640 1 --2- 172.21.15.150:0/3679065501 >> v2:172.21.15.157:3300/0 conn(0x7fd46015be10 0x7fd460099d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:41.796 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45effd640 1 --2- 172.21.15.150:0/3679065501 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4600cd8e0 0x7fd46009a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:41.796 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45e7fc640 1 -- 172.21.15.150:0/3679065501 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 711646069 0 0) 0x7fd448003520 con 0x7fd4600d0100 2024-03-17T23:46:41.797 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45e7fc640 1 -- 172.21.15.150:0/3679065501 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4600cd8e0 msgr2=0x7fd46009a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:41.797 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45e7fc640 1 --2- 172.21.15.150:0/3679065501 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4600cd8e0 0x7fd46009a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:41.797 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45e7fc640 1 -- 172.21.15.150:0/3679065501 >> v2:172.21.15.157:3300/0 conn(0x7fd46015be10 msgr2=0x7fd460099d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:41.797 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45e7fc640 1 --2- 172.21.15.150:0/3679065501 >> v2:172.21.15.157:3300/0 conn(0x7fd46015be10 0x7fd460099d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:41.797 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45e7fc640 1 -- 172.21.15.150:0/3679065501 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4600cd0e0 con 0x7fd4600d0100 2024-03-17T23:46:41.797 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45e7fc640 1 -- 172.21.15.150:0/3679065501 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7fd4580019f0 con 0x7fd4600d0100 2024-03-17T23:46:41.797 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45f7fe640 1 --2- 172.21.15.150:0/3679065501 >> v2:172.21.15.157:3300/0 conn(0x7fd46015be10 0x7fd460099d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:46:41.797 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45e7fc640 1 -- 172.21.15.150:0/3679065501 >> v1:172.21.15.150:6789/0 conn(0x7fd4600d0100 legacy=0x7fd4600d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:41.797 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45e7fc640 1 --2- 172.21.15.150:0/3679065501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4480039b0 0x7fd448003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:41.797 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45e7fc640 1 --2- 172.21.15.150:0/3679065501 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd448004800 0x7fd448004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:41.797 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45e7fc640 1 --2- 172.21.15.150:0/3679065501 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4600cd8e0 0x7fd4480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:41.797 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45e7fc640 1 -- 172.21.15.150:0/3679065501 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd4600ccb20 con 0x7fd4480039b0 2024-03-17T23:46:41.797 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45e7fc640 1 -- 172.21.15.150:0/3679065501 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd4600cce00 con 0x7fd448004800 2024-03-17T23:46:41.798 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45e7fc640 1 -- 172.21.15.150:0/3679065501 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd44800ba70 con 0x7fd4600cd8e0 2024-03-17T23:46:41.798 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45f7fe640 1 --2- 172.21.15.150:0/3679065501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4480039b0 0x7fd448003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:41.798 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45effd640 1 --2- 172.21.15.150:0/3679065501 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd448004800 0x7fd448004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:41.798 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45ffff640 1 --2- 172.21.15.150:0/3679065501 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4600cd8e0 0x7fd4480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:41.798 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45f7fe640 1 -- 172.21.15.150:0/3679065501 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4600cd8e0 msgr2=0x7fd4480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:41.798 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45f7fe640 1 --2- 172.21.15.150:0/3679065501 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4600cd8e0 0x7fd4480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:41.798 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45f7fe640 1 -- 172.21.15.150:0/3679065501 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd448004800 msgr2=0x7fd448004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:41.798 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45f7fe640 1 --2- 172.21.15.150:0/3679065501 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd448004800 0x7fd448004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:41.798 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.792+0000 7fd45f7fe640 1 -- 172.21.15.150:0/3679065501 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd448003520 con 0x7fd4480039b0 2024-03-17T23:46:41.798 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.793+0000 7fd45f7fe640 1 --2- 172.21.15.150:0/3679065501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4480039b0 0x7fd448003dd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd454006fd0 tx=0x7fd448005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=945bcae6db5f3f7f server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:41.798 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.793+0000 7fd45e7fc640 1 -- 172.21.15.150:0/3679065501 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45400f040 con 0x7fd4480039b0 2024-03-17T23:46:41.798 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.793+0000 7fd45e7fc640 1 -- 172.21.15.150:0/3679065501 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd454004230 con 0x7fd4480039b0 2024-03-17T23:46:41.798 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.793+0000 7fd45e7fc640 1 -- 172.21.15.150:0/3679065501 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd454003610 con 0x7fd4480039b0 2024-03-17T23:46:41.798 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.794+0000 7fd4660dc640 1 -- 172.21.15.150:0/3679065501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4480039b0 msgr2=0x7fd448003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:41.799 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.794+0000 7fd4660dc640 1 --2- 172.21.15.150:0/3679065501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4480039b0 0x7fd448003dd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd454006fd0 tx=0x7fd448005830 comp rx=0 tx=0).stop 2024-03-17T23:46:41.799 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.794+0000 7fd45ffff640 1 -- 172.21.15.150:0/3679065501 reap_dead start 2024-03-17T23:46:41.799 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.794+0000 7fd4660dc640 1 -- 172.21.15.150:0/3679065501 shutdown_connections 2024-03-17T23:46:41.799 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.794+0000 7fd4660dc640 1 -- 172.21.15.150:0/3679065501 >> 172.21.15.150:0/3679065501 conn(0x7fd4600b3610 msgr2=0x7fd4600b3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:41.799 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.794+0000 7fd4660dc640 1 -- 172.21.15.150:0/3679065501 shutdown_connections 2024-03-17T23:46:41.799 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.794+0000 7fd4660dc640 1 -- 172.21.15.150:0/3679065501 wait complete. 2024-03-17T23:46:41.799 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.795+0000 7fd4660dc640 1 Processor -- start 2024-03-17T23:46:41.799 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.795+0000 7fd4660dc640 1 -- start start 2024-03-17T23:46:41.801 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.796+0000 7fd4660dc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4480039b0 0x7fd4601510b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:41.801 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.796+0000 7fd4660dc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4601f2390 0x7fd4600b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:41.801 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.796+0000 7fd4660dc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4601f2810 0x7fd4600b1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:41.802 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.796+0000 7fd4660dc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd46009a2b0 con 0x7fd4480039b0 2024-03-17T23:46:41.802 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.796+0000 7fd4660dc640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd46015c1f0 con 0x7fd4601f2810 2024-03-17T23:46:41.802 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.796+0000 7fd4660dc640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd46009a430 con 0x7fd4601f2390 2024-03-17T23:46:41.802 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.796+0000 7fd45f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4480039b0 0x7fd4601510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:41.802 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.796+0000 7fd45f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4480039b0 0x7fd4601510b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44048/0 (socket says 172.21.15.150:44048) 2024-03-17T23:46:41.802 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.796+0000 7fd45f7fe640 1 -- 172.21.15.150:0/4279763926 learned_addr learned my addr 172.21.15.150:0/4279763926 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:41.802 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.796+0000 7fd45effd640 1 --2- 172.21.15.150:0/4279763926 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4601f2390 0x7fd4600b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:41.802 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.796+0000 7fd45ffff640 1 --2- 172.21.15.150:0/4279763926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4601f2810 0x7fd4600b1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:41.802 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.796+0000 7fd45f7fe640 1 -- 172.21.15.150:0/4279763926 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4601f2390 msgr2=0x7fd4600b0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:41.802 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.796+0000 7fd45f7fe640 1 --2- 172.21.15.150:0/4279763926 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4601f2390 0x7fd4600b0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:41.802 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.796+0000 7fd45f7fe640 1 -- 172.21.15.150:0/4279763926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4601f2810 msgr2=0x7fd4600b1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:41.802 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.796+0000 7fd45f7fe640 1 --2- 172.21.15.150:0/4279763926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4601f2810 0x7fd4600b1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:41.802 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.796+0000 7fd45f7fe640 1 -- 172.21.15.150:0/4279763926 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4600aedf0 con 0x7fd4480039b0 2024-03-17T23:46:41.802 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.796+0000 7fd45ffff640 1 --2- 172.21.15.150:0/4279763926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4601f2810 0x7fd4600b1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:46:41.803 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.796+0000 7fd45f7fe640 1 --2- 172.21.15.150:0/4279763926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4480039b0 0x7fd4601510b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd45402bfc0 tx=0x7fd454004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:41.803 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.796+0000 7fd447fff640 1 -- 172.21.15.150:0/4279763926 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd454048070 con 0x7fd4480039b0 2024-03-17T23:46:41.803 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.797+0000 7fd447fff640 1 -- 172.21.15.150:0/4279763926 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd454030460 con 0x7fd4480039b0 2024-03-17T23:46:41.803 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.797+0000 7fd4660dc640 1 -- 172.21.15.150:0/4279763926 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4600af0d0 con 0x7fd4480039b0 2024-03-17T23:46:41.803 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.797+0000 7fd447fff640 1 -- 172.21.15.150:0/4279763926 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd454030830 con 0x7fd4480039b0 2024-03-17T23:46:41.803 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.797+0000 7fd4660dc640 1 -- 172.21.15.150:0/4279763926 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4600af690 con 0x7fd4480039b0 2024-03-17T23:46:41.803 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.798+0000 7fd4660dc640 1 -- 172.21.15.150:0/4279763926 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd460075280 con 0x7fd4480039b0 2024-03-17T23:46:41.803 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.798+0000 7fd447fff640 1 -- 172.21.15.150:0/4279763926 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 4) ==== 90080+0+0 (secure 0 0 0) 0x7fd45404b070 con 0x7fd4480039b0 2024-03-17T23:46:41.803 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.798+0000 7fd447fff640 1 --2- 172.21.15.150:0/4279763926 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd44c06bb40 0x7fd44c06e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:41.803 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.798+0000 7fd447fff640 1 -- 172.21.15.150:0/4279763926 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 3631+0+0 (secure 0 0 0) 0x7fd4540abbb0 con 0x7fd4480039b0 2024-03-17T23:46:41.803 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.798+0000 7fd45effd640 1 --2- 172.21.15.150:0/4279763926 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd44c06bb40 0x7fd44c06e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:41.803 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.799+0000 7fd45effd640 1 --2- 172.21.15.150:0/4279763926 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd44c06bb40 0x7fd44c06e000 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd460154f20 tx=0x7fd450008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:41.804 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.803+0000 7fd447fff640 1 -- 172.21.15.150:0/4279763926 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd454034020 con 0x7fd4480039b0 2024-03-17T23:46:41.978 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.978+0000 7fd4660dc640 1 -- 172.21.15.150:0/4279763926 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd4601588b0 con 0x7fd4480039b0 2024-03-17T23:46:41.979 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.978+0000 7fd447fff640 1 -- 172.21.15.150:0/4279763926 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+9499 (secure 0 0 0) 0x7fd45407afb0 con 0x7fd4480039b0 2024-03-17T23:46:41.979 INFO:teuthology.misc.health.smithi150.stdout: 2024-03-17T23:46:41.980 INFO:teuthology.misc.health.smithi150.stdout:{"epoch":13,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:46:41.707940+0000","last_up_change":"2024-03-17T23:46:41.707940+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:46:41.981 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.980+0000 7fd4660dc640 1 -- 172.21.15.150:0/4279763926 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd44c06bb40 msgr2=0x7fd44c06e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:41.981 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.980+0000 7fd4660dc640 1 --2- 172.21.15.150:0/4279763926 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd44c06bb40 0x7fd44c06e000 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd460154f20 tx=0x7fd450008040 comp rx=0 tx=0).stop 2024-03-17T23:46:41.981 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.980+0000 7fd4660dc640 1 -- 172.21.15.150:0/4279763926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4480039b0 msgr2=0x7fd4601510b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:41.981 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.980+0000 7fd4660dc640 1 --2- 172.21.15.150:0/4279763926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4480039b0 0x7fd4601510b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd45402bfc0 tx=0x7fd454004730 comp rx=0 tx=0).stop 2024-03-17T23:46:41.982 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.981+0000 7fd4660dc640 1 -- 172.21.15.150:0/4279763926 shutdown_connections 2024-03-17T23:46:41.982 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.981+0000 7fd4660dc640 1 --2- 172.21.15.150:0/4279763926 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd44c06bb40 0x7fd44c06e000 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:41.982 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.981+0000 7fd4660dc640 1 --2- 172.21.15.150:0/4279763926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4601f2810 0x7fd4600b1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:41.982 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.981+0000 7fd4660dc640 1 --2- 172.21.15.150:0/4279763926 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4601f2390 0x7fd4600b0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:41.982 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.981+0000 7fd4660dc640 1 --2- 172.21.15.150:0/4279763926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4480039b0 0x7fd4601510b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:41.982 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.981+0000 7fd4660dc640 1 -- 172.21.15.150:0/4279763926 >> 172.21.15.150:0/4279763926 conn(0x7fd4600b3610 msgr2=0x7fd4600d04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:41.982 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.981+0000 7fd4660dc640 1 -- 172.21.15.150:0/4279763926 shutdown_connections 2024-03-17T23:46:41.982 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:41.981+0000 7fd4660dc640 1 -- 172.21.15.150:0/4279763926 wait complete. 2024-03-17T23:46:41.991 DEBUG:teuthology.misc:5 of 8 OSDs are up 2024-03-17T23:46:48.799 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-17T23:46:48.899 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.898+0000 7f6967e7e640 1 Processor -- start 2024-03-17T23:46:48.899 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.899+0000 7f6967e7e640 1 -- start start 2024-03-17T23:46:48.903 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.899+0000 7f6967e7e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6960070bd0 0x7f6960168b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:48.903 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.899+0000 7f6967e7e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f696006c100 con 0x7f696015e020 2024-03-17T23:46:48.903 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.899+0000 7f6967e7e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69600d0590 con 0x7f696006bca0 2024-03-17T23:46:48.903 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.899+0000 7f6967e7e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f696015e400 con 0x7f6960070bd0 2024-03-17T23:46:48.903 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.900+0000 7f6965c1a640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f696015e020 0x7f69600cfd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39006/0 (socket says 172.21.15.150:39006) 2024-03-17T23:46:48.903 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.900+0000 7f6965c1a640 1 -- 172.21.15.150:0/1902637425 learned_addr learned my addr 172.21.15.150:0/1902637425 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:46:48.903 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.900+0000 7f6964c18640 1 -- 172.21.15.150:0/1902637425 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2474848628 0 0) 0x7f696006c100 con 0x7f696015e020 2024-03-17T23:46:48.903 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.900+0000 7f6964c18640 1 -- 172.21.15.150:0/1902637425 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6948003520 con 0x7f696015e020 2024-03-17T23:46:48.903 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.900+0000 7f696641b640 1 --2- 172.21.15.150:0/1902637425 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6960070bd0 0x7f6960168b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:48.904 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.900+0000 7f6964c18640 1 -- 172.21.15.150:0/1902637425 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1127885373 0 0) 0x7f6948003520 con 0x7f696015e020 2024-03-17T23:46:48.904 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.900+0000 7f6964c18640 1 -- 172.21.15.150:0/1902637425 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6960070bd0 msgr2=0x7f6960168b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:48.904 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.900+0000 7f6964c18640 1 --2- 172.21.15.150:0/1902637425 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6960070bd0 0x7f6960168b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:48.904 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.900+0000 7f6964c18640 1 -- 172.21.15.150:0/1902637425 >> v1:172.21.15.157:6789/0 conn(0x7f696006bca0 legacy=0x7f69600704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:48.904 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.900+0000 7f6964c18640 1 -- 172.21.15.150:0/1902637425 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69600ada40 con 0x7f696015e020 2024-03-17T23:46:48.904 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.900+0000 7f6964c18640 1 -- 172.21.15.150:0/1902637425 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f69580019f0 con 0x7f696015e020 2024-03-17T23:46:48.904 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.900+0000 7f6964c18640 1 -- 172.21.15.150:0/1902637425 >> v1:172.21.15.150:6789/0 conn(0x7f696015e020 legacy=0x7f69600cfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:48.904 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.900+0000 7f6964c18640 1 --2- 172.21.15.150:0/1902637425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69480039e0 0x7f6948003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:48.904 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.900+0000 7f6964c18640 1 --2- 172.21.15.150:0/1902637425 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6948004800 0x7f6948004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:48.904 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.900+0000 7f6964c18640 1 --2- 172.21.15.150:0/1902637425 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6960070bd0 0x7f6948005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:48.904 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.900+0000 7f6964c18640 1 -- 172.21.15.150:0/1902637425 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f696015e400 con 0x7f69480039e0 2024-03-17T23:46:48.904 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.900+0000 7f6964c18640 1 -- 172.21.15.150:0/1902637425 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f694800bb20 con 0x7f6948004800 2024-03-17T23:46:48.905 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.900+0000 7f6964c18640 1 -- 172.21.15.150:0/1902637425 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f694800bd20 con 0x7f6960070bd0 2024-03-17T23:46:48.905 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.900+0000 7f6965c1a640 1 --2- 172.21.15.150:0/1902637425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69480039e0 0x7f6948003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:48.905 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.900+0000 7f696641b640 1 --2- 172.21.15.150:0/1902637425 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6960070bd0 0x7f6948005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:48.905 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.900+0000 7f6965419640 1 --2- 172.21.15.150:0/1902637425 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6948004800 0x7f6948004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:48.905 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.901+0000 7f6965c1a640 1 -- 172.21.15.150:0/1902637425 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6960070bd0 msgr2=0x7f6948005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:48.905 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.901+0000 7f6965c1a640 1 --2- 172.21.15.150:0/1902637425 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6960070bd0 0x7f6948005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:48.905 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.901+0000 7f6965c1a640 1 -- 172.21.15.150:0/1902637425 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6948004800 msgr2=0x7f6948004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:48.905 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.901+0000 7f6965c1a640 1 --2- 172.21.15.150:0/1902637425 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6948004800 0x7f6948004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:48.905 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.901+0000 7f6965c1a640 1 -- 172.21.15.150:0/1902637425 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6948003520 con 0x7f69480039e0 2024-03-17T23:46:48.905 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.901+0000 7f6965419640 1 --2- 172.21.15.150:0/1902637425 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6948004800 0x7f6948004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:46:48.905 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.901+0000 7f6965c1a640 1 --2- 172.21.15.150:0/1902637425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69480039e0 0x7f6948003e00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f695800a850 tx=0x7f6948005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62387d1dcc5bf005 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:48.905 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.901+0000 7f6964c18640 1 -- 172.21.15.150:0/1902637425 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695800a4a0 con 0x7f69480039e0 2024-03-17T23:46:48.905 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.901+0000 7f6964c18640 1 -- 172.21.15.150:0/1902637425 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6958034650 con 0x7f69480039e0 2024-03-17T23:46:48.905 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.901+0000 7f6964c18640 1 -- 172.21.15.150:0/1902637425 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6958034a40 con 0x7f69480039e0 2024-03-17T23:46:48.906 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.902+0000 7f6967e7e640 1 -- 172.21.15.150:0/1902637425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69480039e0 msgr2=0x7f6948003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:48.906 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.902+0000 7f6967e7e640 1 --2- 172.21.15.150:0/1902637425 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69480039e0 0x7f6948003e00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f695800a850 tx=0x7f6948005830 comp rx=0 tx=0).stop 2024-03-17T23:46:48.906 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.902+0000 7f696641b640 1 -- 172.21.15.150:0/1902637425 reap_dead start 2024-03-17T23:46:48.906 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.902+0000 7f6967e7e640 1 -- 172.21.15.150:0/1902637425 shutdown_connections 2024-03-17T23:46:48.906 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.902+0000 7f6967e7e640 1 -- 172.21.15.150:0/1902637425 >> 172.21.15.150:0/1902637425 conn(0x7f69600b25c0 msgr2=0x7f69600b29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:48.906 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.902+0000 7f6967e7e640 1 -- 172.21.15.150:0/1902637425 shutdown_connections 2024-03-17T23:46:48.906 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.902+0000 7f6967e7e640 1 -- 172.21.15.150:0/1902637425 wait complete. 2024-03-17T23:46:48.909 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.903+0000 7f6967e7e640 1 Processor -- start 2024-03-17T23:46:48.909 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.903+0000 7f6967e7e640 1 -- start start 2024-03-17T23:46:48.909 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.904+0000 7f6967e7e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69480039e0 0x7f696015fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:48.909 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.904+0000 7f6967e7e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6960160ca0 0x7f696006f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:48.909 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.904+0000 7f6967e7e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6960153220 0x7f696006fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:48.909 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.904+0000 7f6967e7e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f69600af530 con 0x7f6960160ca0 2024-03-17T23:46:48.909 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.904+0000 7f6967e7e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f69600ae000 con 0x7f6960153220 2024-03-17T23:46:48.909 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.904+0000 7f6967e7e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f69600af6b0 con 0x7f69480039e0 2024-03-17T23:46:48.910 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.904+0000 7f6965419640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6960160ca0 0x7f696006f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:48.910 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.904+0000 7f6965419640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6960160ca0 0x7f696006f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45006/0 (socket says 172.21.15.150:45006) 2024-03-17T23:46:48.910 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.904+0000 7f6965419640 1 -- 172.21.15.150:0/2696191461 learned_addr learned my addr 172.21.15.150:0/2696191461 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:48.910 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.904+0000 7f6965c1a640 1 --2- 172.21.15.150:0/2696191461 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69480039e0 0x7f696015fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:48.910 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.904+0000 7f6965419640 1 -- 172.21.15.150:0/2696191461 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69480039e0 msgr2=0x7f696015fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:48.910 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.904+0000 7f696641b640 1 --2- 172.21.15.150:0/2696191461 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6960153220 0x7f696006fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:48.910 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.904+0000 7f6965419640 1 --2- 172.21.15.150:0/2696191461 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69480039e0 0x7f696015fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:48.910 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.904+0000 7f6965419640 1 -- 172.21.15.150:0/2696191461 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6960153220 msgr2=0x7f696006fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:48.910 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.904+0000 7f6965419640 1 --2- 172.21.15.150:0/2696191461 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6960153220 0x7f696006fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:48.910 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.904+0000 7f6965419640 1 -- 172.21.15.150:0/2696191461 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f696015f6b0 con 0x7f6960160ca0 2024-03-17T23:46:48.910 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.905+0000 7f6965c1a640 1 --2- 172.21.15.150:0/2696191461 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69480039e0 0x7f696015fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:46:48.910 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.905+0000 7f6965419640 1 --2- 172.21.15.150:0/2696191461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6960160ca0 0x7f696006f7d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f695400e580 tx=0x7f695400ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:48.910 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.905+0000 7f695e7fc640 1 -- 172.21.15.150:0/2696191461 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69540052e0 con 0x7f6960160ca0 2024-03-17T23:46:48.910 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.905+0000 7f695e7fc640 1 -- 172.21.15.150:0/2696191461 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6954018070 con 0x7f6960160ca0 2024-03-17T23:46:48.911 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.905+0000 7f695e7fc640 1 -- 172.21.15.150:0/2696191461 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6954003830 con 0x7f6960160ca0 2024-03-17T23:46:48.911 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.905+0000 7f6967e7e640 1 -- 172.21.15.150:0/2696191461 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f696015f0f0 con 0x7f6960160ca0 2024-03-17T23:46:48.911 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.905+0000 7f6967e7e640 1 -- 172.21.15.150:0/2696191461 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69600cb160 con 0x7f6960160ca0 2024-03-17T23:46:48.911 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.906+0000 7f6967e7e640 1 -- 172.21.15.150:0/2696191461 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6960075280 con 0x7f6960160ca0 2024-03-17T23:46:48.911 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.906+0000 7f695e7fc640 1 -- 172.21.15.150:0/2696191461 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f69540039d0 con 0x7f6960160ca0 2024-03-17T23:46:48.911 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.907+0000 7f695e7fc640 1 --2- 172.21.15.150:0/2696191461 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f694c06bbd0 0x7f694c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:48.911 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.907+0000 7f695e7fc640 1 -- 172.21.15.150:0/2696191461 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f695408a360 con 0x7f6960160ca0 2024-03-17T23:46:48.911 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.907+0000 7f6965c1a640 1 --2- 172.21.15.150:0/2696191461 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f694c06bbd0 0x7f694c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:48.911 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.907+0000 7f6965c1a640 1 --2- 172.21.15.150:0/2696191461 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f694c06bbd0 0x7f694c06e090 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6958034000 tx=0x7f695802f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:48.912 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:48.911+0000 7f695e7fc640 1 -- 172.21.15.150:0/2696191461 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f69540591b0 con 0x7f6960160ca0 2024-03-17T23:46:49.085 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:49.084+0000 7f6967e7e640 1 -- 172.21.15.150:0/2696191461 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f696006b410 con 0x7f6960160ca0 2024-03-17T23:46:49.085 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:49.085+0000 7f695e7fc640 1 -- 172.21.15.150:0/2696191461 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12920 (secure 0 0 0) 0x7f6954058ab0 con 0x7f6960160ca0 2024-03-17T23:46:49.085 INFO:teuthology.misc.health.smithi150.stdout: 2024-03-17T23:46:49.086 INFO:teuthology.misc.health.smithi150.stdout:{"epoch":17,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:46:45.785911+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:46:49.088 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:49.087+0000 7f6967e7e640 1 -- 172.21.15.150:0/2696191461 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f694c06bbd0 msgr2=0x7f694c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:49.088 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:49.087+0000 7f6967e7e640 1 --2- 172.21.15.150:0/2696191461 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f694c06bbd0 0x7f694c06e090 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6958034000 tx=0x7f695802f000 comp rx=0 tx=0).stop 2024-03-17T23:46:49.088 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:49.087+0000 7f6967e7e640 1 -- 172.21.15.150:0/2696191461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6960160ca0 msgr2=0x7f696006f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:49.088 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:49.087+0000 7f6967e7e640 1 --2- 172.21.15.150:0/2696191461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6960160ca0 0x7f696006f7d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f695400e580 tx=0x7f695400ea40 comp rx=0 tx=0).stop 2024-03-17T23:46:49.088 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:49.087+0000 7f6967e7e640 1 -- 172.21.15.150:0/2696191461 shutdown_connections 2024-03-17T23:46:49.088 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:49.087+0000 7f6967e7e640 1 --2- 172.21.15.150:0/2696191461 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f694c06bbd0 0x7f694c06e090 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.088 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:49.087+0000 7f6967e7e640 1 --2- 172.21.15.150:0/2696191461 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6960153220 0x7f696006fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.088 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:49.087+0000 7f6967e7e640 1 --2- 172.21.15.150:0/2696191461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6960160ca0 0x7f696006f7d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.088 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:49.087+0000 7f6967e7e640 1 --2- 172.21.15.150:0/2696191461 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f69480039e0 0x7f696015fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.088 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:49.087+0000 7f6967e7e640 1 -- 172.21.15.150:0/2696191461 >> 172.21.15.150:0/2696191461 conn(0x7f69600b25c0 msgr2=0x7f696015d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:49.088 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:49.087+0000 7f6967e7e640 1 -- 172.21.15.150:0/2696191461 shutdown_connections 2024-03-17T23:46:49.088 INFO:teuthology.misc.health.smithi150.stderr:2024-03-17T23:46:49.087+0000 7f6967e7e640 1 -- 172.21.15.150:0/2696191461 wait complete. 2024-03-17T23:46:49.097 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-17T23:46:49.097 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-17T23:46:49.098 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-17T23:46:49.098 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-17T23:46:49.100 INFO:tasks.ceph.mds.a:Started 2024-03-17T23:46:49.100 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-17T23:46:49.100 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-17T23:46:49.102 INFO:tasks.ceph.mds.c:Started 2024-03-17T23:46:49.102 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-17T23:46:49.103 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-17T23:46:49.105 INFO:tasks.ceph.mds.b:Started 2024-03-17T23:46:49.105 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-17T23:46:49.105 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-17T23:46:49.107 INFO:tasks.ceph.mds.d:Started 2024-03-17T23:46:49.107 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-17T23:46:49.110 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:46:49.177 INFO:tasks.ceph.mds.a.smithi150.stdout:starting mds.a at 2024-03-17T23:46:49.178 INFO:tasks.ceph.mds.c.smithi150.stdout:starting mds.c at 2024-03-17T23:46:49.194 INFO:tasks.ceph.mds.b.smithi157.stdout:starting mds.b at 2024-03-17T23:46:49.194 INFO:tasks.ceph.mds.d.smithi157.stdout:starting mds.d at 2024-03-17T23:46:49.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.215+0000 7f2851f90640 1 Processor -- start 2024-03-17T23:46:49.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.215+0000 7f2851f90640 1 -- start start 2024-03-17T23:46:49.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.216+0000 7f2851f90640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f284c0cf120 0x7f284c1619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:49.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.216+0000 7f2851f90640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f284c0b9860 0x7f284c0d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:49.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.216+0000 7f2851f90640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f284c1620a0 con 0x7f284c15de30 2024-03-17T23:46:49.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.216+0000 7f2851f90640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f284c162940 con 0x7f284c0b9860 2024-03-17T23:46:49.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.216+0000 7f2851f90640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f284c162380 con 0x7f284c0cf120 2024-03-17T23:46:49.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.216+0000 7f284bfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f284c0b9860 0x7f284c0d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:49.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.216+0000 7f284bfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f284c0b9860 0x7f284c0d2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45034/0 (socket says 172.21.15.150:45034) 2024-03-17T23:46:49.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.216+0000 7f284bfff640 1 -- 172.21.15.150:0/3544664113 learned_addr learned my addr 172.21.15.150:0/3544664113 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:49.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.216+0000 7f284affd640 1 --2- 172.21.15.150:0/3544664113 >> v2:172.21.15.157:3300/0 conn(0x7f284c0cf120 0x7f284c1619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:49.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.216+0000 7f284bfff640 1 -- 172.21.15.150:0/3544664113 >> v2:172.21.15.157:3300/0 conn(0x7f284c0cf120 msgr2=0x7f284c1619a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:49.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.216+0000 7f284bfff640 1 --2- 172.21.15.150:0/3544664113 >> v2:172.21.15.157:3300/0 conn(0x7f284c0cf120 0x7f284c1619a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.216+0000 7f284bfff640 1 -- 172.21.15.150:0/3544664113 >> v1:172.21.15.157:6789/0 conn(0x7f284c15de30 legacy=0x7f284c0cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:49.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.216+0000 7f284bfff640 1 -- 172.21.15.150:0/3544664113 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f284c162660 con 0x7f284c0b9860 2024-03-17T23:46:49.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.217+0000 7f284bfff640 1 --2- 172.21.15.150:0/3544664113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f284c0b9860 0x7f284c0d2690 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f284c1618d0 tx=0x7f284402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e183a4488d224fea server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:49.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.217+0000 7f284a7fc640 1 -- 172.21.15.150:0/3544664113 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284402c9f0 con 0x7f284c0b9860 2024-03-17T23:46:49.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.217+0000 7f284a7fc640 1 -- 172.21.15.150:0/3544664113 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f284402cb90 con 0x7f284c0b9860 2024-03-17T23:46:49.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.217+0000 7f284a7fc640 1 -- 172.21.15.150:0/3544664113 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28440343f0 con 0x7f284c0b9860 2024-03-17T23:46:49.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.217+0000 7f2851f90640 1 -- 172.21.15.150:0/3544664113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f284c0b9860 msgr2=0x7f284c0d2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:49.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.217+0000 7f2851f90640 1 --2- 172.21.15.150:0/3544664113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f284c0b9860 0x7f284c0d2690 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f284c1618d0 tx=0x7f284402be50 comp rx=0 tx=0).stop 2024-03-17T23:46:49.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.217+0000 7f2851f90640 1 -- 172.21.15.150:0/3544664113 shutdown_connections 2024-03-17T23:46:49.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.217+0000 7f2851f90640 1 --2- 172.21.15.150:0/3544664113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f284c0b9860 0x7f284c0d2690 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.217+0000 7f2851f90640 1 --2- 172.21.15.150:0/3544664113 >> v2:172.21.15.157:3300/0 conn(0x7f284c0cf120 0x7f284c1619a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.217+0000 7f2851f90640 1 -- 172.21.15.150:0/3544664113 >> 172.21.15.150:0/3544664113 conn(0x7f284c0730b0 msgr2=0x7f284c0d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:49.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.217+0000 7f2851f90640 1 -- 172.21.15.150:0/3544664113 shutdown_connections 2024-03-17T23:46:49.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.217+0000 7f2851f90640 1 -- 172.21.15.150:0/3544664113 wait complete. 2024-03-17T23:46:49.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.218+0000 7f2851f90640 1 Processor -- start 2024-03-17T23:46:49.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.219+0000 7f2851f90640 1 -- start start 2024-03-17T23:46:49.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.219+0000 7f2851f90640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f284c0b9860 0x7f284c159670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:49.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.219+0000 7f2851f90640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f284c0cf120 0x7f284c158f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:49.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.219+0000 7f2851f90640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f284c15de30 0x7f284c0cd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:49.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.219+0000 7f2851f90640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f284c15e210 con 0x7f284c0cf120 2024-03-17T23:46:49.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.219+0000 7f2851f90640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f284c0b9c90 con 0x7f284c15de30 2024-03-17T23:46:49.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.219+0000 7f2851f90640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f284c0d2bd0 con 0x7f284c0b9860 2024-03-17T23:46:49.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.219+0000 7f284affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f284c0cf120 0x7f284c158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:49.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.219+0000 7f284affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f284c0cf120 0x7f284c158f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45048/0 (socket says 172.21.15.150:45048) 2024-03-17T23:46:49.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.219+0000 7f284affd640 1 -- 172.21.15.150:0/1953610339 learned_addr learned my addr 172.21.15.150:0/1953610339 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:49.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.219+0000 7f284b7fe640 1 --2- 172.21.15.150:0/1953610339 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f284c0b9860 0x7f284c159670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:49.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.219+0000 7f284bfff640 1 --2- 172.21.15.150:0/1953610339 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f284c15de30 0x7f284c0cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:49.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.220+0000 7f284affd640 1 -- 172.21.15.150:0/1953610339 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f284c0b9860 msgr2=0x7f284c159670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:49.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.220+0000 7f284affd640 1 --2- 172.21.15.150:0/1953610339 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f284c0b9860 0x7f284c159670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.220+0000 7f284affd640 1 -- 172.21.15.150:0/1953610339 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f284c15de30 msgr2=0x7f284c0cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:49.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.220+0000 7f284affd640 1 --2- 172.21.15.150:0/1953610339 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f284c15de30 0x7f284c0cd390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.220+0000 7f284affd640 1 -- 172.21.15.150:0/1953610339 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f284c0cc360 con 0x7f284c0cf120 2024-03-17T23:46:49.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.220+0000 7f284b7fe640 1 --2- 172.21.15.150:0/1953610339 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f284c0b9860 0x7f284c159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:46:49.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.220+0000 7f284affd640 1 --2- 172.21.15.150:0/1953610339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f284c0cf120 0x7f284c158f00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f283c00d830 tx=0x7f283c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:49.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.220+0000 7f2833fff640 1 -- 172.21.15.150:0/1953610339 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f283c00b660 con 0x7f284c0cf120 2024-03-17T23:46:49.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.220+0000 7f2833fff640 1 -- 172.21.15.150:0/1953610339 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f283c019070 con 0x7f284c0cf120 2024-03-17T23:46:49.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.220+0000 7f2851f90640 1 -- 172.21.15.150:0/1953610339 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f284c159440 con 0x7f284c0cf120 2024-03-17T23:46:49.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.220+0000 7f2833fff640 1 -- 172.21.15.150:0/1953610339 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f283c00ba20 con 0x7f284c0cf120 2024-03-17T23:46:49.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.220+0000 7f2851f90640 1 -- 172.21.15.150:0/1953610339 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f284c0cc8b0 con 0x7f284c0cf120 2024-03-17T23:46:49.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.221+0000 7f2851f90640 1 -- 172.21.15.150:0/1953610339 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f284c0681a0 con 0x7f284c0cf120 2024-03-17T23:46:49.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.221+0000 7f2833fff640 1 -- 172.21.15.150:0/1953610339 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f283c01e460 con 0x7f284c0cf120 2024-03-17T23:46:49.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.222+0000 7f2833fff640 1 --2- 172.21.15.150:0/1953610339 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f283806bbd0 0x7f283806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:49.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.222+0000 7f284b7fe640 1 --2- 172.21.15.150:0/1953610339 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f283806bbd0 0x7f283806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:49.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.222+0000 7f2833fff640 1 -- 172.21.15.150:0/1953610339 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f283c08ac40 con 0x7f284c0cf120 2024-03-17T23:46:49.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.222+0000 7f284b7fe640 1 --2- 172.21.15.150:0/1953610339 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f283806bbd0 0x7f283806e090 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2840006fd0 tx=0x7f2840008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:49.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.226+0000 7f2833fff640 1 -- 172.21.15.150:0/1953610339 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f283c059b10 con 0x7f284c0cf120 2024-03-17T23:46:49.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.401+0000 7f2851f90640 1 -- 172.21.15.150:0/1953610339 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f284c0d1910 con 0x7f284c0cf120 2024-03-17T23:46:49.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.402+0000 7f2833fff640 1 -- 172.21.15.150:0/1953610339 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12920 (secure 0 0 0) 0x7f283c059410 con 0x7f284c0cf120 2024-03-17T23:46:49.403 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:46:49.403 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":17,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:46:45.785911+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:46:49.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.404+0000 7f2851f90640 1 -- 172.21.15.150:0/1953610339 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f283806bbd0 msgr2=0x7f283806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:49.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.404+0000 7f2851f90640 1 --2- 172.21.15.150:0/1953610339 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f283806bbd0 0x7f283806e090 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2840006fd0 tx=0x7f2840008040 comp rx=0 tx=0).stop 2024-03-17T23:46:49.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.404+0000 7f2851f90640 1 -- 172.21.15.150:0/1953610339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f284c0cf120 msgr2=0x7f284c158f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:49.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.404+0000 7f2851f90640 1 --2- 172.21.15.150:0/1953610339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f284c0cf120 0x7f284c158f00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f283c00d830 tx=0x7f283c00dcf0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.404+0000 7f2851f90640 1 -- 172.21.15.150:0/1953610339 shutdown_connections 2024-03-17T23:46:49.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.404+0000 7f2851f90640 1 --2- 172.21.15.150:0/1953610339 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f283806bbd0 0x7f283806e090 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.404+0000 7f2851f90640 1 --2- 172.21.15.150:0/1953610339 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f284c15de30 0x7f284c0cd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.404+0000 7f2851f90640 1 --2- 172.21.15.150:0/1953610339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f284c0cf120 0x7f284c158f00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.404+0000 7f2851f90640 1 --2- 172.21.15.150:0/1953610339 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f284c0b9860 0x7f284c159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.404+0000 7f2851f90640 1 -- 172.21.15.150:0/1953610339 >> 172.21.15.150:0/1953610339 conn(0x7f284c0730b0 msgr2=0x7f284c15ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:49.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.404+0000 7f2851f90640 1 -- 172.21.15.150:0/1953610339 shutdown_connections 2024-03-17T23:46:49.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.404+0000 7f2851f90640 1 -- 172.21.15.150:0/1953610339 wait complete. 2024-03-17T23:46:49.415 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-17T23:46:49.415 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:46:49.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.518+0000 7f1e0875b640 1 Processor -- start 2024-03-17T23:46:49.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.518+0000 7f1e0875b640 1 -- start start 2024-03-17T23:46:49.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.519+0000 7f1e0875b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1e00161590 0x7f1e00067690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:49.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.519+0000 7f1e0875b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1e00072a20 0x7f1e000ad180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:49.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.519+0000 7f1e0875b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1e000d0140 0x7f1e000d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:49.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.519+0000 7f1e0875b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f1e000d0af0 con 0x7f1e00161590 2024-03-17T23:46:49.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.519+0000 7f1e0875b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f1e0006e580 con 0x7f1e000d0140 2024-03-17T23:46:49.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.519+0000 7f1e0875b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1e00161970 con 0x7f1e00072a20 2024-03-17T23:46:49.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.519+0000 7f1e064f7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1e00161590 0x7f1e00067690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:49.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.519+0000 7f1e064f7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1e00161590 0x7f1e00067690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45056/0 (socket says 172.21.15.150:45056) 2024-03-17T23:46:49.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.519+0000 7f1e064f7640 1 -- 172.21.15.150:0/2374546418 learned_addr learned my addr 172.21.15.150:0/2374546418 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:49.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.519+0000 7f1e05cf6640 1 --2- 172.21.15.150:0/2374546418 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1e00072a20 0x7f1e000ad180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:49.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.519+0000 7f1e064f7640 1 -- 172.21.15.150:0/2374546418 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1e00072a20 msgr2=0x7f1e000ad180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:49.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.519+0000 7f1e064f7640 1 --2- 172.21.15.150:0/2374546418 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1e00072a20 0x7f1e000ad180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.519+0000 7f1e06cf8640 1 --2- 172.21.15.150:0/2374546418 >> v2:172.21.15.157:3300/0 conn(0x7f1e000d0140 0x7f1e000d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:49.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.519+0000 7f1e064f7640 1 -- 172.21.15.150:0/2374546418 >> v2:172.21.15.157:3300/0 conn(0x7f1e000d0140 msgr2=0x7f1e000d05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:49.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.519+0000 7f1e064f7640 1 --2- 172.21.15.150:0/2374546418 >> v2:172.21.15.157:3300/0 conn(0x7f1e000d0140 0x7f1e000d05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.519+0000 7f1e064f7640 1 -- 172.21.15.150:0/2374546418 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e0006e7d0 con 0x7f1e00161590 2024-03-17T23:46:49.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.520+0000 7f1e06cf8640 1 --2- 172.21.15.150:0/2374546418 >> v2:172.21.15.157:3300/0 conn(0x7f1e000d0140 0x7f1e000d05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:46:49.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.520+0000 7f1e064f7640 1 --2- 172.21.15.150:0/2374546418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e00161590 0x7f1e00067690 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1e00072470 tx=0x7f1dfc02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a6e0e533526a477 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:49.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.520+0000 7f1e054f5640 1 -- 172.21.15.150:0/2374546418 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc02c950 con 0x7f1e00161590 2024-03-17T23:46:49.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.520+0000 7f1e054f5640 1 -- 172.21.15.150:0/2374546418 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dfc02caf0 con 0x7f1e00161590 2024-03-17T23:46:49.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.520+0000 7f1e054f5640 1 -- 172.21.15.150:0/2374546418 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc0343f0 con 0x7f1e00161590 2024-03-17T23:46:49.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.521+0000 7f1e0875b640 1 -- 172.21.15.150:0/2374546418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e00161590 msgr2=0x7f1e00067690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:49.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.521+0000 7f1e0875b640 1 --2- 172.21.15.150:0/2374546418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e00161590 0x7f1e00067690 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1e00072470 tx=0x7f1dfc02bdb0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.521+0000 7f1e0875b640 1 -- 172.21.15.150:0/2374546418 shutdown_connections 2024-03-17T23:46:49.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.521+0000 7f1e0875b640 1 --2- 172.21.15.150:0/2374546418 >> v2:172.21.15.157:3300/0 conn(0x7f1e000d0140 0x7f1e000d05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.521+0000 7f1e0875b640 1 --2- 172.21.15.150:0/2374546418 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1e00072a20 0x7f1e000ad180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.521+0000 7f1e0875b640 1 --2- 172.21.15.150:0/2374546418 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e00161590 0x7f1e00067690 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.521+0000 7f1e0875b640 1 -- 172.21.15.150:0/2374546418 >> 172.21.15.150:0/2374546418 conn(0x7f1e000b3da0 msgr2=0x7f1e000b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:49.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.521+0000 7f1e0875b640 1 -- 172.21.15.150:0/2374546418 shutdown_connections 2024-03-17T23:46:49.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.521+0000 7f1e0875b640 1 -- 172.21.15.150:0/2374546418 wait complete. 2024-03-17T23:46:49.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.522+0000 7f1e0875b640 1 Processor -- start 2024-03-17T23:46:49.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.522+0000 7f1e0875b640 1 -- start start 2024-03-17T23:46:49.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.523+0000 7f1e0875b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e00072a20 0x7f1e000b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:49.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.523+0000 7f1e0875b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1e000d0140 0x7f1e0013ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:49.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.523+0000 7f1e0875b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e00161590 0x7f1e00151a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:49.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.523+0000 7f1e0875b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1e00099c20 con 0x7f1e00072a20 2024-03-17T23:46:49.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.523+0000 7f1e0875b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1e0006ed90 con 0x7f1e00161590 2024-03-17T23:46:49.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.523+0000 7f1e0875b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1e000b1240 con 0x7f1e000d0140 2024-03-17T23:46:49.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.523+0000 7f1e064f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e00072a20 0x7f1e000b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:49.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.523+0000 7f1e064f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e00072a20 0x7f1e000b2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45058/0 (socket says 172.21.15.150:45058) 2024-03-17T23:46:49.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.523+0000 7f1e064f7640 1 -- 172.21.15.150:0/961743238 learned_addr learned my addr 172.21.15.150:0/961743238 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:49.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.523+0000 7f1e06cf8640 1 --2- 172.21.15.150:0/961743238 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e00161590 0x7f1e00151a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:49.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.523+0000 7f1e05cf6640 1 --2- 172.21.15.150:0/961743238 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1e000d0140 0x7f1e0013ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:49.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.523+0000 7f1e064f7640 1 -- 172.21.15.150:0/961743238 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1e000d0140 msgr2=0x7f1e0013ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:49.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.523+0000 7f1e064f7640 1 --2- 172.21.15.150:0/961743238 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1e000d0140 0x7f1e0013ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.523+0000 7f1e064f7640 1 -- 172.21.15.150:0/961743238 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e00161590 msgr2=0x7f1e00151a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:49.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.523+0000 7f1e064f7640 1 --2- 172.21.15.150:0/961743238 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e00161590 0x7f1e00151a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.523+0000 7f1e064f7640 1 -- 172.21.15.150:0/961743238 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e000b1520 con 0x7f1e00072a20 2024-03-17T23:46:49.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.523+0000 7f1e05cf6640 1 --2- 172.21.15.150:0/961743238 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1e000d0140 0x7f1e0013ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:46:49.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.523+0000 7f1e064f7640 1 --2- 172.21.15.150:0/961743238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e00072a20 0x7f1e000b2000 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009f40 tx=0x7f1dfc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:49.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.523+0000 7f1df6ffd640 1 -- 172.21.15.150:0/961743238 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc043020 con 0x7f1e00072a20 2024-03-17T23:46:49.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.523+0000 7f1df6ffd640 1 -- 172.21.15.150:0/961743238 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dfc03e450 con 0x7f1e00072a20 2024-03-17T23:46:49.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.524+0000 7f1df6ffd640 1 -- 172.21.15.150:0/961743238 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc03e840 con 0x7f1e00072a20 2024-03-17T23:46:49.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.524+0000 7f1e0875b640 1 -- 172.21.15.150:0/961743238 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e000b1800 con 0x7f1e00072a20 2024-03-17T23:46:49.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.524+0000 7f1e0875b640 1 -- 172.21.15.150:0/961743238 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e00142140 con 0x7f1e00072a20 2024-03-17T23:46:49.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.525+0000 7f1e0875b640 1 -- 172.21.15.150:0/961743238 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e000681a0 con 0x7f1e00072a20 2024-03-17T23:46:49.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.525+0000 7f1df6ffd640 1 -- 172.21.15.150:0/961743238 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1dfc046020 con 0x7f1e00072a20 2024-03-17T23:46:49.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.525+0000 7f1df6ffd640 1 --2- 172.21.15.150:0/961743238 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1de406bbd0 0x7f1de406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:49.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.525+0000 7f1e05cf6640 1 --2- 172.21.15.150:0/961743238 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1de406bbd0 0x7f1de406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:49.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.525+0000 7f1df6ffd640 1 -- 172.21.15.150:0/961743238 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f1dfc0acfd0 con 0x7f1e00072a20 2024-03-17T23:46:49.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.526+0000 7f1e05cf6640 1 --2- 172.21.15.150:0/961743238 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1de406bbd0 0x7f1de406e090 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1e000cfe60 tx=0x7f1df0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:49.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.530+0000 7f1df6ffd640 1 -- 172.21.15.150:0/961743238 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1dfc04d030 con 0x7f1e00072a20 2024-03-17T23:46:49.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.701+0000 7f1e0875b640 1 -- 172.21.15.150:0/961743238 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1e00160310 con 0x7f1e00072a20 2024-03-17T23:46:49.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.701+0000 7f1df6ffd640 1 -- 172.21.15.150:0/961743238 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7f1dfc07bea0 con 0x7f1e00072a20 2024-03-17T23:46:49.702 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:46:49.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.703+0000 7f1e0875b640 1 -- 172.21.15.150:0/961743238 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1de406bbd0 msgr2=0x7f1de406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:49.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.703+0000 7f1e0875b640 1 --2- 172.21.15.150:0/961743238 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1de406bbd0 0x7f1de406e090 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1e000cfe60 tx=0x7f1df0008040 comp rx=0 tx=0).stop 2024-03-17T23:46:49.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.703+0000 7f1e0875b640 1 -- 172.21.15.150:0/961743238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e00072a20 msgr2=0x7f1e000b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:49.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.703+0000 7f1e0875b640 1 --2- 172.21.15.150:0/961743238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e00072a20 0x7f1e000b2000 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009f40 tx=0x7f1dfc004a40 comp rx=0 tx=0).stop 2024-03-17T23:46:49.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.703+0000 7f1e0875b640 1 -- 172.21.15.150:0/961743238 shutdown_connections 2024-03-17T23:46:49.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.703+0000 7f1e0875b640 1 --2- 172.21.15.150:0/961743238 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1de406bbd0 0x7f1de406e090 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.703+0000 7f1e0875b640 1 --2- 172.21.15.150:0/961743238 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e00161590 0x7f1e00151a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.703+0000 7f1e0875b640 1 --2- 172.21.15.150:0/961743238 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1e000d0140 0x7f1e0013ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.703+0000 7f1e0875b640 1 --2- 172.21.15.150:0/961743238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e00072a20 0x7f1e000b2000 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.704+0000 7f1e0875b640 1 -- 172.21.15.150:0/961743238 >> 172.21.15.150:0/961743238 conn(0x7f1e000b3da0 msgr2=0x7f1e001578a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:49.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.704+0000 7f1e0875b640 1 -- 172.21.15.150:0/961743238 shutdown_connections 2024-03-17T23:46:49.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.704+0000 7f1e0875b640 1 -- 172.21.15.150:0/961743238 wait complete. 2024-03-17T23:46:49.715 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:46:49.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.814+0000 7f0455db9640 1 Processor -- start 2024-03-17T23:46:49.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.814+0000 7f0455db9640 1 -- start start 2024-03-17T23:46:49.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.815+0000 7f0455db9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f045015bce0 0x7f04500cfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:49.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.815+0000 7f0455db9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04500d02a0 0x7f045006b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:49.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.815+0000 7f0455db9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04500b3a00 con 0x7f045006b6d0 2024-03-17T23:46:49.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.815+0000 7f0455db9640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f045006bb00 con 0x7f045015bce0 2024-03-17T23:46:49.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.815+0000 7f0455db9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f04500cb2c0 con 0x7f04500d02a0 2024-03-17T23:46:49.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.815+0000 7f044ffff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f045006b6d0 0x7f04500d0890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39024/0 (socket says 172.21.15.150:39024) 2024-03-17T23:46:49.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.815+0000 7f044ffff640 1 -- 172.21.15.150:0/410449506 learned_addr learned my addr 172.21.15.150:0/410449506 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:46:49.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.816+0000 7f0446ffd640 1 --2- 172.21.15.150:0/410449506 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04500d02a0 0x7f045006b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:49.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.816+0000 7f044f7fe640 1 --2- 172.21.15.150:0/410449506 >> v2:172.21.15.157:3300/0 conn(0x7f045015bce0 0x7f04500cfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:49.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.816+0000 7f044effd640 1 -- 172.21.15.150:0/410449506 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3027771737 0 0) 0x7f04500b3a00 con 0x7f045006b6d0 2024-03-17T23:46:49.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.816+0000 7f044effd640 1 -- 172.21.15.150:0/410449506 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f043c003520 con 0x7f045006b6d0 2024-03-17T23:46:49.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.816+0000 7f044effd640 1 -- 172.21.15.150:0/410449506 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3943058325 0 0) 0x7f043c003520 con 0x7f045006b6d0 2024-03-17T23:46:49.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.816+0000 7f044effd640 1 -- 172.21.15.150:0/410449506 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04500d02a0 msgr2=0x7f045006b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:49.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.816+0000 7f044effd640 1 --2- 172.21.15.150:0/410449506 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04500d02a0 0x7f045006b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.816+0000 7f044effd640 1 -- 172.21.15.150:0/410449506 >> v2:172.21.15.157:3300/0 conn(0x7f045015bce0 msgr2=0x7f04500cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:49.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.816+0000 7f044effd640 1 --2- 172.21.15.150:0/410449506 >> v2:172.21.15.157:3300/0 conn(0x7f045015bce0 0x7f04500cfd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.816+0000 7f044effd640 1 -- 172.21.15.150:0/410449506 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04500cb5a0 con 0x7f045006b6d0 2024-03-17T23:46:49.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.816+0000 7f044effd640 1 -- 172.21.15.150:0/410449506 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f04480019f0 con 0x7f045006b6d0 2024-03-17T23:46:49.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.816+0000 7f044effd640 1 -- 172.21.15.150:0/410449506 >> v1:172.21.15.150:6789/0 conn(0x7f045006b6d0 legacy=0x7f04500d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:49.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.816+0000 7f044effd640 1 --2- 172.21.15.150:0/410449506 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04500d02a0 0x7f043c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:49.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.816+0000 7f044effd640 1 --2- 172.21.15.150:0/410449506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f043c003f30 0x7f043c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:49.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.816+0000 7f044effd640 1 --2- 172.21.15.150:0/410449506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f043c004d40 0x7f043c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:49.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.816+0000 7f044effd640 1 -- 172.21.15.150:0/410449506 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f045006bb00 con 0x7f043c004d40 2024-03-17T23:46:49.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.816+0000 7f044effd640 1 -- 172.21.15.150:0/410449506 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f04500cb2c0 con 0x7f043c003f30 2024-03-17T23:46:49.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.816+0000 7f044effd640 1 -- 172.21.15.150:0/410449506 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f043c00b9c0 con 0x7f04500d02a0 2024-03-17T23:46:49.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.816+0000 7f044ffff640 1 --2- 172.21.15.150:0/410449506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f043c004d40 0x7f043c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:49.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.816+0000 7f044f7fe640 1 --2- 172.21.15.150:0/410449506 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04500d02a0 0x7f043c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:49.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.816+0000 7f0446ffd640 1 --2- 172.21.15.150:0/410449506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f043c003f30 0x7f043c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:49.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.816+0000 7f044ffff640 1 -- 172.21.15.150:0/410449506 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04500d02a0 msgr2=0x7f043c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:49.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.816+0000 7f044ffff640 1 --2- 172.21.15.150:0/410449506 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04500d02a0 0x7f043c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.816+0000 7f044ffff640 1 -- 172.21.15.150:0/410449506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f043c003f30 msgr2=0x7f043c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:49.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.816+0000 7f044ffff640 1 --2- 172.21.15.150:0/410449506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f043c003f30 0x7f043c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.816+0000 7f044ffff640 1 -- 172.21.15.150:0/410449506 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f043c003520 con 0x7f043c004d40 2024-03-17T23:46:49.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.817+0000 7f044ffff640 1 --2- 172.21.15.150:0/410449506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f043c004d40 0x7f043c0051c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f044800a880 tx=0x7f043c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f4d8eda5c881d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:49.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.817+0000 7f044effd640 1 -- 172.21.15.150:0/410449506 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044800a4a0 con 0x7f043c004d40 2024-03-17T23:46:49.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.817+0000 7f044effd640 1 -- 172.21.15.150:0/410449506 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0448034650 con 0x7f043c004d40 2024-03-17T23:46:49.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.817+0000 7f044effd640 1 -- 172.21.15.150:0/410449506 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0448034a40 con 0x7f043c004d40 2024-03-17T23:46:49.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.818+0000 7f0455db9640 1 -- 172.21.15.150:0/410449506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f043c004d40 msgr2=0x7f043c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:49.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.818+0000 7f0455db9640 1 --2- 172.21.15.150:0/410449506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f043c004d40 0x7f043c0051c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f044800a880 tx=0x7f043c005830 comp rx=0 tx=0).stop 2024-03-17T23:46:49.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.818+0000 7f044ffff640 1 -- 172.21.15.150:0/410449506 reap_dead start 2024-03-17T23:46:49.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.818+0000 7f0455db9640 1 -- 172.21.15.150:0/410449506 shutdown_connections 2024-03-17T23:46:49.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.818+0000 7f0455db9640 1 -- 172.21.15.150:0/410449506 >> 172.21.15.150:0/410449506 conn(0x7f0450099a70 msgr2=0x7f0450099e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:49.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.818+0000 7f0455db9640 1 -- 172.21.15.150:0/410449506 shutdown_connections 2024-03-17T23:46:49.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.818+0000 7f0455db9640 1 -- 172.21.15.150:0/410449506 wait complete. 2024-03-17T23:46:49.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.819+0000 7f0455db9640 1 Processor -- start 2024-03-17T23:46:49.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.819+0000 7f0455db9640 1 -- start start 2024-03-17T23:46:49.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.820+0000 7f0455db9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0450159d30 0x7f04500b6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:49.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.820+0000 7f0455db9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f045015a110 0x7f04500b4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:49.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.820+0000 7f0455db9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0450156d60 0x7f0450157210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:49.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.820+0000 7f0455db9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f04500cd090 con 0x7f0450159d30 2024-03-17T23:46:49.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.820+0000 7f0455db9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f04500cbb60 con 0x7f045015a110 2024-03-17T23:46:49.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.820+0000 7f0455db9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f045015c0c0 con 0x7f0450156d60 2024-03-17T23:46:49.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.820+0000 7f044f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0450159d30 0x7f04500b6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:49.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.820+0000 7f044f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0450159d30 0x7f04500b6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45072/0 (socket says 172.21.15.150:45072) 2024-03-17T23:46:49.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.820+0000 7f044f7fe640 1 -- 172.21.15.150:0/3586016378 learned_addr learned my addr 172.21.15.150:0/3586016378 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:49.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.820+0000 7f044ffff640 1 --2- 172.21.15.150:0/3586016378 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0450156d60 0x7f0450157210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:49.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.820+0000 7f0446ffd640 1 --2- 172.21.15.150:0/3586016378 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f045015a110 0x7f04500b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:49.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.820+0000 7f044f7fe640 1 -- 172.21.15.150:0/3586016378 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0450156d60 msgr2=0x7f0450157210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:49.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.820+0000 7f044f7fe640 1 --2- 172.21.15.150:0/3586016378 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0450156d60 0x7f0450157210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.820+0000 7f044f7fe640 1 -- 172.21.15.150:0/3586016378 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f045015a110 msgr2=0x7f04500b4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:49.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.820+0000 7f044f7fe640 1 --2- 172.21.15.150:0/3586016378 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f045015a110 0x7f04500b4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:49.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.820+0000 7f044f7fe640 1 -- 172.21.15.150:0/3586016378 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04500b4b40 con 0x7f0450159d30 2024-03-17T23:46:49.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.820+0000 7f044ffff640 1 --2- 172.21.15.150:0/3586016378 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0450156d60 0x7f0450157210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:46:49.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.820+0000 7f044f7fe640 1 --2- 172.21.15.150:0/3586016378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0450159d30 0x7f04500b6970 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0440011a70 tx=0x7f0440011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:49.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.820+0000 7f044cff9640 1 -- 172.21.15.150:0/3586016378 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044000fb50 con 0x7f0450159d30 2024-03-17T23:46:49.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.821+0000 7f044cff9640 1 -- 172.21.15.150:0/3586016378 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f044000fcf0 con 0x7f0450159d30 2024-03-17T23:46:49.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.821+0000 7f0455db9640 1 -- 172.21.15.150:0/3586016378 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0450157750 con 0x7f0450159d30 2024-03-17T23:46:49.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.821+0000 7f044cff9640 1 -- 172.21.15.150:0/3586016378 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044000fea0 con 0x7f0450159d30 2024-03-17T23:46:49.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.821+0000 7f0455db9640 1 -- 172.21.15.150:0/3586016378 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04500b50f0 con 0x7f0450159d30 2024-03-17T23:46:49.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.821+0000 7f0455db9640 1 -- 172.21.15.150:0/3586016378 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04500688b0 con 0x7f0450159d30 2024-03-17T23:46:49.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.822+0000 7f044cff9640 1 -- 172.21.15.150:0/3586016378 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0440012040 con 0x7f0450159d30 2024-03-17T23:46:49.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.822+0000 7f044cff9640 1 --2- 172.21.15.150:0/3586016378 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f043006bc00 0x7f043006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:49.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.822+0000 7f044cff9640 1 -- 172.21.15.150:0/3586016378 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f044008ae50 con 0x7f0450159d30 2024-03-17T23:46:49.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.822+0000 7f0446ffd640 1 --2- 172.21.15.150:0/3586016378 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f043006bc00 0x7f043006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:49.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.823+0000 7f0446ffd640 1 --2- 172.21.15.150:0/3586016378 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f043006bc00 0x7f043006e0c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f04500ca560 tx=0x7f0438008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:49.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:49.826+0000 7f044cff9640 1 -- 172.21.15.150:0/3586016378 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0440019030 con 0x7f0450159d30 2024-03-17T23:46:50.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:50.037+0000 7f0455db9640 1 -- 172.21.15.150:0/3586016378 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0450073000 con 0x7f0450159d30 2024-03-17T23:46:50.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:50.037+0000 7f044cff9640 1 -- 172.21.15.150:0/3586016378 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1395 (secure 0 0 0) 0x7f0440059c90 con 0x7f0450159d30 2024-03-17T23:46:50.038 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:46:50.038 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3655589010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3655589010},{"type":"v1","addr":"172.21.15.150:6837","nonce":3655589010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-17T23:46:50.039 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 2 2024-03-17T23:46:50.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:50.039+0000 7f0455db9640 1 -- 172.21.15.150:0/3586016378 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f043006bc00 msgr2=0x7f043006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:50.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:50.039+0000 7f0455db9640 1 --2- 172.21.15.150:0/3586016378 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f043006bc00 0x7f043006e0c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f04500ca560 tx=0x7f0438008040 comp rx=0 tx=0).stop 2024-03-17T23:46:50.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:50.039+0000 7f0455db9640 1 -- 172.21.15.150:0/3586016378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0450159d30 msgr2=0x7f04500b6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:50.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:50.039+0000 7f0455db9640 1 --2- 172.21.15.150:0/3586016378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0450159d30 0x7f04500b6970 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0440011a70 tx=0x7f0440011f30 comp rx=0 tx=0).stop 2024-03-17T23:46:50.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:50.039+0000 7f0455db9640 1 -- 172.21.15.150:0/3586016378 shutdown_connections 2024-03-17T23:46:50.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:50.039+0000 7f0455db9640 1 --2- 172.21.15.150:0/3586016378 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0450156d60 0x7f0450157210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:50.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:50.039+0000 7f0455db9640 1 --2- 172.21.15.150:0/3586016378 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f043006bc00 0x7f043006e0c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:50.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:50.039+0000 7f0455db9640 1 --2- 172.21.15.150:0/3586016378 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f045015a110 0x7f04500b4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:50.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:50.039+0000 7f0455db9640 1 --2- 172.21.15.150:0/3586016378 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0450159d30 0x7f04500b6970 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:50.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:50.039+0000 7f0455db9640 1 -- 172.21.15.150:0/3586016378 >> 172.21.15.150:0/3586016378 conn(0x7f0450099a70 msgr2=0x7f045015f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:50.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:50.040+0000 7f0455db9640 1 -- 172.21.15.150:0/3586016378 shutdown_connections 2024-03-17T23:46:50.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:50.040+0000 7f0455db9640 1 -- 172.21.15.150:0/3586016378 wait complete. 2024-03-17T23:46:52.052 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:46:52.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.164+0000 7f1f351d4640 1 Processor -- start 2024-03-17T23:46:52.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.164+0000 7f1f351d4640 1 -- start start 2024-03-17T23:46:52.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.165+0000 7f1f351d4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1f3015f460 0x7f1f30099e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:52.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.165+0000 7f1f351d4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1f300cd9d0 0x7f1f3009a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:52.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.165+0000 7f1f351d4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f300cc4e0 con 0x7f1f300b2dc0 2024-03-17T23:46:52.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.165+0000 7f1f351d4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f1f300d09c0 con 0x7f1f3015f460 2024-03-17T23:46:52.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.165+0000 7f1f351d4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1f300b3250 con 0x7f1f300cd9d0 2024-03-17T23:46:52.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.165+0000 7f1f2ed9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1f3015f460 0x7f1f30099e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:52.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.165+0000 7f1f2ed9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1f3015f460 0x7f1f30099e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45084/0 (socket says 172.21.15.150:45084) 2024-03-17T23:46:52.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.165+0000 7f1f2ed9d640 1 -- 172.21.15.150:0/901458263 learned_addr learned my addr 172.21.15.150:0/901458263 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:52.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.165+0000 7f1f2659c640 1 --1- 172.21.15.150:0/901458263 >> v1:172.21.15.150:6789/0 conn(0x7f1f300b2dc0 0x7f1f300b1330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39028/0 (socket says 172.21.15.150:39028) 2024-03-17T23:46:52.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.165+0000 7f1f2ed9d640 1 -- 172.21.15.150:0/901458263 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1f300cd9d0 msgr2=0x7f1f3009a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:52.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.165+0000 7f1f2f59e640 1 --2- 172.21.15.150:0/901458263 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1f300cd9d0 0x7f1f3009a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:52.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.165+0000 7f1f2ed9d640 1 --2- 172.21.15.150:0/901458263 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1f300cd9d0 0x7f1f3009a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.165+0000 7f1f2ed9d640 1 -- 172.21.15.150:0/901458263 >> v1:172.21.15.150:6789/0 conn(0x7f1f300b2dc0 legacy=0x7f1f300b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:52.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.165+0000 7f1f2e59c640 1 -- 172.21.15.150:0/901458263 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 259102248 0 0) 0x7f1f300cc4e0 con 0x7f1f300b2dc0 2024-03-17T23:46:52.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.165+0000 7f1f2ed9d640 1 -- 172.21.15.150:0/901458263 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f300d0120 con 0x7f1f3015f460 2024-03-17T23:46:52.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.166+0000 7f1f2ed9d640 1 --2- 172.21.15.150:0/901458263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f3015f460 0x7f1f30099e20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1f20009870 tx=0x7f1f2002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb93c607371096cd server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:52.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.166+0000 7f1f2e59c640 1 -- 172.21.15.150:0/901458263 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f20035020 con 0x7f1f3015f460 2024-03-17T23:46:52.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.166+0000 7f1f2e59c640 1 -- 172.21.15.150:0/901458263 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f20031640 con 0x7f1f3015f460 2024-03-17T23:46:52.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.166+0000 7f1f2e59c640 1 -- 172.21.15.150:0/901458263 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f2002d040 con 0x7f1f3015f460 2024-03-17T23:46:52.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.166+0000 7f1f351d4640 1 -- 172.21.15.150:0/901458263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f3015f460 msgr2=0x7f1f30099e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:52.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.166+0000 7f1f351d4640 1 --2- 172.21.15.150:0/901458263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f3015f460 0x7f1f30099e20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1f20009870 tx=0x7f1f2002b8f0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.166+0000 7f1f351d4640 1 -- 172.21.15.150:0/901458263 shutdown_connections 2024-03-17T23:46:52.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.166+0000 7f1f351d4640 1 --2- 172.21.15.150:0/901458263 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1f300cd9d0 0x7f1f3009a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.167+0000 7f1f351d4640 1 --2- 172.21.15.150:0/901458263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f3015f460 0x7f1f30099e20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.167+0000 7f1f351d4640 1 -- 172.21.15.150:0/901458263 >> 172.21.15.150:0/901458263 conn(0x7f1f300b3630 msgr2=0x7f1f300b3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:52.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.167+0000 7f1f351d4640 1 -- 172.21.15.150:0/901458263 shutdown_connections 2024-03-17T23:46:52.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.167+0000 7f1f351d4640 1 -- 172.21.15.150:0/901458263 wait complete. 2024-03-17T23:46:52.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.168+0000 7f1f351d4640 1 Processor -- start 2024-03-17T23:46:52.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.168+0000 7f1f351d4640 1 -- start start 2024-03-17T23:46:52.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.169+0000 7f1f351d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f300cd9d0 0x7f1f30154990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:52.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.169+0000 7f1f351d4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1f3015f460 0x7f1f300c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:52.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.169+0000 7f1f351d4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f30155220 0x7f1f300c5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:52.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.169+0000 7f1f351d4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1f3015f840 con 0x7f1f300cd9d0 2024-03-17T23:46:52.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.169+0000 7f1f351d4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1f300d06e0 con 0x7f1f30155220 2024-03-17T23:46:52.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.169+0000 7f1f351d4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1f30154ed0 con 0x7f1f3015f460 2024-03-17T23:46:52.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.169+0000 7f1f2ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f300cd9d0 0x7f1f30154990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:52.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.169+0000 7f1f2ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f300cd9d0 0x7f1f30154990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45092/0 (socket says 172.21.15.150:45092) 2024-03-17T23:46:52.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.169+0000 7f1f2ed9d640 1 -- 172.21.15.150:0/4182936203 learned_addr learned my addr 172.21.15.150:0/4182936203 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:52.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.169+0000 7f1f2659c640 1 --2- 172.21.15.150:0/4182936203 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1f3015f460 0x7f1f300c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:52.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.169+0000 7f1f2f59e640 1 --2- 172.21.15.150:0/4182936203 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f30155220 0x7f1f300c5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:52.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.169+0000 7f1f2ed9d640 1 -- 172.21.15.150:0/4182936203 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1f3015f460 msgr2=0x7f1f300c72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:52.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.169+0000 7f1f2ed9d640 1 --2- 172.21.15.150:0/4182936203 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1f3015f460 0x7f1f300c72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.169+0000 7f1f2ed9d640 1 -- 172.21.15.150:0/4182936203 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f30155220 msgr2=0x7f1f300c5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:52.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.169+0000 7f1f2ed9d640 1 --2- 172.21.15.150:0/4182936203 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f30155220 0x7f1f300c5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.169+0000 7f1f2ed9d640 1 -- 172.21.15.150:0/4182936203 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f300c6250 con 0x7f1f300cd9d0 2024-03-17T23:46:52.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.169+0000 7f1f2ed9d640 1 --2- 172.21.15.150:0/4182936203 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f300cd9d0 0x7f1f30154990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1f20000c00 tx=0x7f1f200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:52.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.170+0000 7f1f27fff640 1 -- 172.21.15.150:0/4182936203 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f20040070 con 0x7f1f300cd9d0 2024-03-17T23:46:52.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.170+0000 7f1f27fff640 1 -- 172.21.15.150:0/4182936203 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f20039c10 con 0x7f1f300cd9d0 2024-03-17T23:46:52.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.170+0000 7f1f351d4640 1 -- 172.21.15.150:0/4182936203 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f300c6530 con 0x7f1f300cd9d0 2024-03-17T23:46:52.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.170+0000 7f1f27fff640 1 -- 172.21.15.150:0/4182936203 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f20035040 con 0x7f1f300cd9d0 2024-03-17T23:46:52.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.170+0000 7f1f351d4640 1 -- 172.21.15.150:0/4182936203 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f300c6af0 con 0x7f1f300cd9d0 2024-03-17T23:46:52.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.171+0000 7f1f351d4640 1 -- 172.21.15.150:0/4182936203 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f300681a0 con 0x7f1f300cd9d0 2024-03-17T23:46:52.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.171+0000 7f1f27fff640 1 -- 172.21.15.150:0/4182936203 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1f20038020 con 0x7f1f300cd9d0 2024-03-17T23:46:52.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.171+0000 7f1f27fff640 1 --2- 172.21.15.150:0/4182936203 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1f1006bbd0 0x7f1f1006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:52.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.172+0000 7f1f27fff640 1 -- 172.21.15.150:0/4182936203 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f1f200ac220 con 0x7f1f300cd9d0 2024-03-17T23:46:52.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.172+0000 7f1f2659c640 1 --2- 172.21.15.150:0/4182936203 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1f1006bbd0 0x7f1f1006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:52.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.172+0000 7f1f2659c640 1 --2- 172.21.15.150:0/4182936203 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1f1006bbd0 0x7f1f1006e090 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1f300c7ca0 tx=0x7f1f18007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:52.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.176+0000 7f1f27fff640 1 -- 172.21.15.150:0/4182936203 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1f2007a980 con 0x7f1f300cd9d0 2024-03-17T23:46:52.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.384+0000 7f1f351d4640 1 -- 172.21.15.150:0/4182936203 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f3015ec80 con 0x7f1f300cd9d0 2024-03-17T23:46:52.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.385+0000 7f1f27fff640 1 -- 172.21.15.150:0/4182936203 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3873 (secure 0 0 0) 0x7f1f2007ea20 con 0x7f1f300cd9d0 2024-03-17T23:46:52.385 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:46:52.385 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/1949609985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6833","nonce":1949609985},{"type":"v1","addr":"172.21.15.157:6835","nonce":1949609985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/758264728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":758264728},{"type":"v1","addr":"172.21.15.150:6835","nonce":758264728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3655589010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3655589010},{"type":"v1","addr":"172.21.15.150:6837","nonce":3655589010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4297,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6834/1625058792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1625058792},{"type":"v1","addr":"172.21.15.157:6834","nonce":1625058792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-17T23:46:52.386 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 3 2024-03-17T23:46:52.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.386+0000 7f1f351d4640 1 -- 172.21.15.150:0/4182936203 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1f1006bbd0 msgr2=0x7f1f1006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:52.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.386+0000 7f1f351d4640 1 --2- 172.21.15.150:0/4182936203 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1f1006bbd0 0x7f1f1006e090 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1f300c7ca0 tx=0x7f1f18007d50 comp rx=0 tx=0).stop 2024-03-17T23:46:52.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.386+0000 7f1f351d4640 1 -- 172.21.15.150:0/4182936203 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f300cd9d0 msgr2=0x7f1f30154990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:52.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.386+0000 7f1f351d4640 1 --2- 172.21.15.150:0/4182936203 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f300cd9d0 0x7f1f30154990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1f20000c00 tx=0x7f1f200046c0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.387+0000 7f1f351d4640 1 -- 172.21.15.150:0/4182936203 shutdown_connections 2024-03-17T23:46:52.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.387+0000 7f1f351d4640 1 --2- 172.21.15.150:0/4182936203 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1f1006bbd0 0x7f1f1006e090 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.387+0000 7f1f351d4640 1 --2- 172.21.15.150:0/4182936203 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1f30155220 0x7f1f300c5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.387+0000 7f1f351d4640 1 --2- 172.21.15.150:0/4182936203 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1f3015f460 0x7f1f300c72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.387+0000 7f1f351d4640 1 --2- 172.21.15.150:0/4182936203 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1f300cd9d0 0x7f1f30154990 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.387+0000 7f1f351d4640 1 -- 172.21.15.150:0/4182936203 >> 172.21.15.150:0/4182936203 conn(0x7f1f300b3630 msgr2=0x7f1f300cf5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:52.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.387+0000 7f1f351d4640 1 -- 172.21.15.150:0/4182936203 shutdown_connections 2024-03-17T23:46:52.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.387+0000 7f1f351d4640 1 -- 172.21.15.150:0/4182936203 wait complete. 2024-03-17T23:46:52.398 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:46:52.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.498+0000 7f4a07db2640 1 Processor -- start 2024-03-17T23:46:52.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.498+0000 7f4a07db2640 1 -- start start 2024-03-17T23:46:52.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.499+0000 7f4a07db2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4a000cf120 0x7f4a001619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:52.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.499+0000 7f4a07db2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4a000b9860 0x7f4a000d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:52.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.499+0000 7f4a07db2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a000b9cb0 con 0x7f4a0015de30 2024-03-17T23:46:52.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.499+0000 7f4a07db2640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f4a00162940 con 0x7f4a000b9860 2024-03-17T23:46:52.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.499+0000 7f4a07db2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4a001620a0 con 0x7f4a000cf120 2024-03-17T23:46:52.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.499+0000 7f4a05b4e640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f4a0015de30 0x7f4a000cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39042/0 (socket says 172.21.15.150:39042) 2024-03-17T23:46:52.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.499+0000 7f4a05b4e640 1 -- 172.21.15.150:0/3564953084 learned_addr learned my addr 172.21.15.150:0/3564953084 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:46:52.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.499+0000 7f4a0634f640 1 --2- 172.21.15.150:0/3564953084 >> v2:172.21.15.157:3300/0 conn(0x7f4a000b9860 0x7f4a000d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:52.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.499+0000 7f4a0534d640 1 --2- 172.21.15.150:0/3564953084 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4a000cf120 0x7f4a001619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:52.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.499+0000 7f4a04b4c640 1 -- 172.21.15.150:0/3564953084 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3919747414 0 0) 0x7f4a000b9cb0 con 0x7f4a0015de30 2024-03-17T23:46:52.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.499+0000 7f4a04b4c640 1 -- 172.21.15.150:0/3564953084 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49ec003520 con 0x7f4a0015de30 2024-03-17T23:46:52.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.499+0000 7f4a04b4c640 1 -- 172.21.15.150:0/3564953084 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2711835701 0 0) 0x7f49ec003520 con 0x7f4a0015de30 2024-03-17T23:46:52.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.499+0000 7f4a0534d640 1 -- 172.21.15.150:0/3564953084 >> v2:172.21.15.157:3300/0 conn(0x7f4a000b9860 msgr2=0x7f4a000d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:52.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.499+0000 7f4a0534d640 1 --2- 172.21.15.150:0/3564953084 >> v2:172.21.15.157:3300/0 conn(0x7f4a000b9860 0x7f4a000d2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.499+0000 7f4a0534d640 1 -- 172.21.15.150:0/3564953084 >> v1:172.21.15.150:6789/0 conn(0x7f4a0015de30 legacy=0x7f4a000cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:52.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.499+0000 7f4a0534d640 1 -- 172.21.15.150:0/3564953084 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a00162380 con 0x7f4a000cf120 2024-03-17T23:46:52.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.500+0000 7f4a0534d640 1 --2- 172.21.15.150:0/3564953084 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4a000cf120 0x7f4a001619a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f49f00097a0 tx=0x7f49f002b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d449fd913a2e10c server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:52.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.500+0000 7f4a04b4c640 1 -- 172.21.15.150:0/3564953084 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f0037070 con 0x7f4a000cf120 2024-03-17T23:46:52.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.500+0000 7f4a04b4c640 1 -- 172.21.15.150:0/3564953084 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49f0030510 con 0x7f4a000cf120 2024-03-17T23:46:52.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.500+0000 7f4a04b4c640 1 -- 172.21.15.150:0/3564953084 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc001670 con 0x7f4a000cf120 2024-03-17T23:46:52.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.501+0000 7f4a07db2640 1 -- 172.21.15.150:0/3564953084 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4a000cf120 msgr2=0x7f4a001619a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:52.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.501+0000 7f4a07db2640 1 --2- 172.21.15.150:0/3564953084 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4a000cf120 0x7f4a001619a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f49f00097a0 tx=0x7f49f002b710 comp rx=0 tx=0).stop 2024-03-17T23:46:52.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.501+0000 7f4a07db2640 1 -- 172.21.15.150:0/3564953084 shutdown_connections 2024-03-17T23:46:52.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.501+0000 7f4a07db2640 1 --2- 172.21.15.150:0/3564953084 >> v2:172.21.15.157:3300/0 conn(0x7f4a000b9860 0x7f4a000d2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.501+0000 7f4a07db2640 1 --2- 172.21.15.150:0/3564953084 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4a000cf120 0x7f4a001619a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.501+0000 7f4a07db2640 1 -- 172.21.15.150:0/3564953084 >> 172.21.15.150:0/3564953084 conn(0x7f4a000730b0 msgr2=0x7f4a000d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:52.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.501+0000 7f4a07db2640 1 -- 172.21.15.150:0/3564953084 shutdown_connections 2024-03-17T23:46:52.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.501+0000 7f4a07db2640 1 -- 172.21.15.150:0/3564953084 wait complete. 2024-03-17T23:46:52.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.502+0000 7f4a07db2640 1 Processor -- start 2024-03-17T23:46:52.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.502+0000 7f4a07db2640 1 -- start start 2024-03-17T23:46:52.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.503+0000 7f4a07db2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a000b9860 0x7f4a000d4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:52.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.503+0000 7f4a07db2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4a000cf120 0x7f4a000b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:52.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.503+0000 7f4a07db2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4a0015de30 0x7f4a0015b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:52.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.503+0000 7f4a07db2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0015e210 con 0x7f4a000b9860 2024-03-17T23:46:52.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.503+0000 7f4a07db2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4a000d2170 con 0x7f4a0015de30 2024-03-17T23:46:52.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.503+0000 7f4a07db2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4a000d2bd0 con 0x7f4a000cf120 2024-03-17T23:46:52.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.503+0000 7f4a05b4e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a000b9860 0x7f4a000d4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:52.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.503+0000 7f4a05b4e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a000b9860 0x7f4a000d4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45104/0 (socket says 172.21.15.150:45104) 2024-03-17T23:46:52.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.503+0000 7f4a05b4e640 1 -- 172.21.15.150:0/1242820704 learned_addr learned my addr 172.21.15.150:0/1242820704 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:52.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.503+0000 7f4a0534d640 1 --2- 172.21.15.150:0/1242820704 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4a000cf120 0x7f4a000b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:52.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.503+0000 7f4a0634f640 1 --2- 172.21.15.150:0/1242820704 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4a0015de30 0x7f4a0015b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:52.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.503+0000 7f4a05b4e640 1 -- 172.21.15.150:0/1242820704 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4a000cf120 msgr2=0x7f4a000b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:52.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.503+0000 7f4a05b4e640 1 --2- 172.21.15.150:0/1242820704 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4a000cf120 0x7f4a000b21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.503+0000 7f4a05b4e640 1 -- 172.21.15.150:0/1242820704 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4a0015de30 msgr2=0x7f4a0015b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:52.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.503+0000 7f4a05b4e640 1 --2- 172.21.15.150:0/1242820704 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4a0015de30 0x7f4a0015b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.503+0000 7f4a05b4e640 1 -- 172.21.15.150:0/1242820704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a0015e530 con 0x7f4a000b9860 2024-03-17T23:46:52.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.503+0000 7f4a05b4e640 1 --2- 172.21.15.150:0/1242820704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a000b9860 0x7f4a000d4010 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f49fc0017e0 tx=0x7f49fc00cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:52.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.503+0000 7f49f67fc640 1 -- 172.21.15.150:0/1242820704 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc005ad0 con 0x7f4a000b9860 2024-03-17T23:46:52.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.503+0000 7f49f67fc640 1 -- 172.21.15.150:0/1242820704 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49fc005c90 con 0x7f4a000b9860 2024-03-17T23:46:52.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.503+0000 7f49f67fc640 1 -- 172.21.15.150:0/1242820704 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc0072b0 con 0x7f4a000b9860 2024-03-17T23:46:52.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.503+0000 7f4a07db2640 1 -- 172.21.15.150:0/1242820704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a0015e810 con 0x7f4a000b9860 2024-03-17T23:46:52.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.503+0000 7f4a07db2640 1 -- 172.21.15.150:0/1242820704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a0015edd0 con 0x7f4a000b9860 2024-03-17T23:46:52.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.504+0000 7f4a07db2640 1 -- 172.21.15.150:0/1242820704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a000d2990 con 0x7f4a000b9860 2024-03-17T23:46:52.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.505+0000 7f49f67fc640 1 -- 172.21.15.150:0/1242820704 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f49fc007450 con 0x7f4a000b9860 2024-03-17T23:46:52.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.505+0000 7f49f67fc640 1 --2- 172.21.15.150:0/1242820704 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f49e406bbd0 0x7f49e406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:52.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.505+0000 7f49f67fc640 1 -- 172.21.15.150:0/1242820704 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f49fc08a640 con 0x7f4a000b9860 2024-03-17T23:46:52.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.505+0000 7f4a0534d640 1 --2- 172.21.15.150:0/1242820704 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f49e406bbd0 0x7f49e406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:52.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.506+0000 7f4a0534d640 1 --2- 172.21.15.150:0/1242820704 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f49e406bbd0 0x7f49e406e090 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f49f0034000 tx=0x7f49f0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:52.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.509+0000 7f49f67fc640 1 -- 172.21.15.150:0/1242820704 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4a000d2990 con 0x7f4a000b9860 2024-03-17T23:46:52.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.686+0000 7f4a07db2640 1 -- 172.21.15.150:0/1242820704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4a000681a0 con 0x7f4a000b9860 2024-03-17T23:46:52.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.687+0000 7f49f67fc640 1 -- 172.21.15.150:0/1242820704 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12920 (secure 0 0 0) 0x7f49fc059360 con 0x7f4a000b9860 2024-03-17T23:46:52.688 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:46:52.688 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":17,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:46:45.785911+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:46:52.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.689+0000 7f4a07db2640 1 -- 172.21.15.150:0/1242820704 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f49e406bbd0 msgr2=0x7f49e406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:52.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.689+0000 7f4a07db2640 1 --2- 172.21.15.150:0/1242820704 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f49e406bbd0 0x7f49e406e090 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f49f0034000 tx=0x7f49f0008030 comp rx=0 tx=0).stop 2024-03-17T23:46:52.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.690+0000 7f4a07db2640 1 -- 172.21.15.150:0/1242820704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a000b9860 msgr2=0x7f4a000d4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:52.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.690+0000 7f4a07db2640 1 --2- 172.21.15.150:0/1242820704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a000b9860 0x7f4a000d4010 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f49fc0017e0 tx=0x7f49fc00cbe0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.690+0000 7f4a07db2640 1 -- 172.21.15.150:0/1242820704 shutdown_connections 2024-03-17T23:46:52.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.690+0000 7f4a07db2640 1 --2- 172.21.15.150:0/1242820704 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f49e406bbd0 0x7f49e406e090 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.690+0000 7f4a07db2640 1 --2- 172.21.15.150:0/1242820704 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4a0015de30 0x7f4a0015b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.690+0000 7f4a07db2640 1 --2- 172.21.15.150:0/1242820704 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4a000cf120 0x7f4a000b21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.690+0000 7f4a07db2640 1 --2- 172.21.15.150:0/1242820704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a000b9860 0x7f4a000d4010 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.690+0000 7f4a07db2640 1 -- 172.21.15.150:0/1242820704 >> 172.21.15.150:0/1242820704 conn(0x7f4a000730b0 msgr2=0x7f4a000d2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:52.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.690+0000 7f4a07db2640 1 -- 172.21.15.150:0/1242820704 shutdown_connections 2024-03-17T23:46:52.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.690+0000 7f4a07db2640 1 -- 172.21.15.150:0/1242820704 wait complete. 2024-03-17T23:46:52.701 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-17T23:46:52.701 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:46:52.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.804+0000 7f948b353640 1 Processor -- start 2024-03-17T23:46:52.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.804+0000 7f948b353640 1 -- start start 2024-03-17T23:46:52.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.805+0000 7f948b353640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f948406b5f0 0x7f94840ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:52.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.805+0000 7f948b353640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94840d0850 con 0x7f948406b190 2024-03-17T23:46:52.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.805+0000 7f948b353640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94840d0b30 con 0x7f948415e170 2024-03-17T23:46:52.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.805+0000 7f948b353640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f94840d0e10 con 0x7f948406b5f0 2024-03-17T23:46:52.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.805+0000 7f94888ee640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f948406b190 0x7f94840ce620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39050/0 (socket says 172.21.15.150:39050) 2024-03-17T23:46:52.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.805+0000 7f94888ee640 1 -- 172.21.15.150:0/4172898300 learned_addr learned my addr 172.21.15.150:0/4172898300 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:46:52.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.806+0000 7f947bfff640 1 -- 172.21.15.150:0/4172898300 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1742459917 0 0) 0x7f94840d0850 con 0x7f948406b190 2024-03-17T23:46:52.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.806+0000 7f94898f0640 1 --2- 172.21.15.150:0/4172898300 >> v2:172.21.15.157:3300/0 conn(0x7f948406b5f0 0x7f94840ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:52.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.806+0000 7f947bfff640 1 -- 172.21.15.150:0/4172898300 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9470003520 con 0x7f948406b190 2024-03-17T23:46:52.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.806+0000 7f947bfff640 1 -- 172.21.15.150:0/4172898300 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3453935916 0 0) 0x7f9470003520 con 0x7f948406b190 2024-03-17T23:46:52.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.806+0000 7f947bfff640 1 -- 172.21.15.150:0/4172898300 >> v2:172.21.15.157:3300/0 conn(0x7f948406b5f0 msgr2=0x7f94840ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:52.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.806+0000 7f947bfff640 1 --2- 172.21.15.150:0/4172898300 >> v2:172.21.15.157:3300/0 conn(0x7f948406b5f0 0x7f94840ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.806+0000 7f947bfff640 1 -- 172.21.15.150:0/4172898300 >> v1:172.21.15.157:6789/0 conn(0x7f948415e170 legacy=0x7f94840cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:52.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.806+0000 7f947bfff640 1 -- 172.21.15.150:0/4172898300 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94840d10f0 con 0x7f948406b190 2024-03-17T23:46:52.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.806+0000 7f947bfff640 1 -- 172.21.15.150:0/4172898300 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f9474001670 con 0x7f948406b190 2024-03-17T23:46:52.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.806+0000 7f947bfff640 1 -- 172.21.15.150:0/4172898300 >> v1:172.21.15.150:6789/0 conn(0x7f948406b190 legacy=0x7f94840ce620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:52.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.806+0000 7f947bfff640 1 --2- 172.21.15.150:0/4172898300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94700039b0 0x7f9470003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:52.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.806+0000 7f947bfff640 1 --2- 172.21.15.150:0/4172898300 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9470004800 0x7f9470004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:52.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.806+0000 7f947bfff640 1 --2- 172.21.15.150:0/4172898300 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94700051c0 0x7f947000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:52.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.806+0000 7f947bfff640 1 -- 172.21.15.150:0/4172898300 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f94840d0e10 con 0x7f94700039b0 2024-03-17T23:46:52.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.806+0000 7f947bfff640 1 -- 172.21.15.150:0/4172898300 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f947000c160 con 0x7f94700051c0 2024-03-17T23:46:52.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.806+0000 7f947bfff640 1 -- 172.21.15.150:0/4172898300 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f947000c360 con 0x7f9470004800 2024-03-17T23:46:52.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.806+0000 7f94890ef640 1 --2- 172.21.15.150:0/4172898300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94700039b0 0x7f9470003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:52.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.806+0000 7f94888ee640 1 --2- 172.21.15.150:0/4172898300 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9470004800 0x7f9470004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:52.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.806+0000 7f94898f0640 1 --2- 172.21.15.150:0/4172898300 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94700051c0 0x7f947000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:52.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.806+0000 7f94890ef640 1 -- 172.21.15.150:0/4172898300 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9470004800 msgr2=0x7f9470004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:52.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.806+0000 7f94890ef640 1 --2- 172.21.15.150:0/4172898300 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9470004800 0x7f9470004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.806+0000 7f94890ef640 1 -- 172.21.15.150:0/4172898300 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94700051c0 msgr2=0x7f947000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:52.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.806+0000 7f94890ef640 1 --2- 172.21.15.150:0/4172898300 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94700051c0 0x7f947000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.806+0000 7f94898f0640 1 -- 172.21.15.150:0/4172898300 reap_dead start 2024-03-17T23:46:52.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.806+0000 7f94890ef640 1 -- 172.21.15.150:0/4172898300 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9470003520 con 0x7f94700039b0 2024-03-17T23:46:52.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.807+0000 7f94890ef640 1 --2- 172.21.15.150:0/4172898300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94700039b0 0x7f9470003dd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9480001e50 tx=0x7f9470005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cbfc4aa9113d094 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:52.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.807+0000 7f947bfff640 1 -- 172.21.15.150:0/4172898300 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9480001a70 con 0x7f94700039b0 2024-03-17T23:46:52.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.807+0000 7f947bfff640 1 -- 172.21.15.150:0/4172898300 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9480003b70 con 0x7f94700039b0 2024-03-17T23:46:52.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.807+0000 7f947bfff640 1 -- 172.21.15.150:0/4172898300 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94800050d0 con 0x7f94700039b0 2024-03-17T23:46:52.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.808+0000 7f948b353640 1 -- 172.21.15.150:0/4172898300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94700039b0 msgr2=0x7f9470003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:52.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.808+0000 7f948b353640 1 --2- 172.21.15.150:0/4172898300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94700039b0 0x7f9470003dd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9480001e50 tx=0x7f9470005830 comp rx=0 tx=0).stop 2024-03-17T23:46:52.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.808+0000 7f948b353640 1 -- 172.21.15.150:0/4172898300 shutdown_connections 2024-03-17T23:46:52.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.808+0000 7f948b353640 1 --2- 172.21.15.150:0/4172898300 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94700039b0 0x7f9470003dd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.808+0000 7f948b353640 1 -- 172.21.15.150:0/4172898300 >> 172.21.15.150:0/4172898300 conn(0x7f9484099a70 msgr2=0x7f9484099e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:52.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.808+0000 7f948b353640 1 -- 172.21.15.150:0/4172898300 shutdown_connections 2024-03-17T23:46:52.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.808+0000 7f948b353640 1 -- 172.21.15.150:0/4172898300 wait complete. 2024-03-17T23:46:52.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.809+0000 7f948b353640 1 Processor -- start 2024-03-17T23:46:52.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.809+0000 7f948b353640 1 -- start start 2024-03-17T23:46:52.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.810+0000 7f948b353640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f94700039b0 0x7f9484155480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:52.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.810+0000 7f948b353640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f948415a030 0x7f9484143650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:52.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.810+0000 7f948b353640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f948415a4b0 0x7f9484143b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:52.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.810+0000 7f948b353640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f94840afcf0 con 0x7f948415a030 2024-03-17T23:46:52.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.810+0000 7f948b353640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f948406ba60 con 0x7f948415a4b0 2024-03-17T23:46:52.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.810+0000 7f948b353640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f94840afe70 con 0x7f94700039b0 2024-03-17T23:46:52.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.810+0000 7f94888ee640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f948415a030 0x7f9484143650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:52.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.810+0000 7f94888ee640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f948415a030 0x7f9484143650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45130/0 (socket says 172.21.15.150:45130) 2024-03-17T23:46:52.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.810+0000 7f94888ee640 1 -- 172.21.15.150:0/411866881 learned_addr learned my addr 172.21.15.150:0/411866881 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:52.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.810+0000 7f94890ef640 1 --2- 172.21.15.150:0/411866881 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f94700039b0 0x7f9484155480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.810+0000 7f94898f0640 1 --2- 172.21.15.150:0/411866881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f948415a4b0 0x7f9484143b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.810+0000 7f94888ee640 1 -- 172.21.15.150:0/411866881 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f94700039b0 msgr2=0x7f9484155480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.810+0000 7f94888ee640 1 --2- 172.21.15.150:0/411866881 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f94700039b0 0x7f9484155480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.810+0000 7f94888ee640 1 -- 172.21.15.150:0/411866881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f948415a4b0 msgr2=0x7f9484143b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:52.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.810+0000 7f94888ee640 1 --2- 172.21.15.150:0/411866881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f948415a4b0 0x7f9484143b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.810+0000 7f94888ee640 1 -- 172.21.15.150:0/411866881 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94840b49b0 con 0x7f948415a030 2024-03-17T23:46:52.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.810+0000 7f94888ee640 1 --2- 172.21.15.150:0/411866881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f948415a030 0x7f9484143650 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f94740073a0 tx=0x7f9474007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:52.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.810+0000 7f9479ffb640 1 -- 172.21.15.150:0/411866881 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947401a070 con 0x7f948415a030 2024-03-17T23:46:52.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.810+0000 7f9479ffb640 1 -- 172.21.15.150:0/411866881 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f947400f960 con 0x7f948415a030 2024-03-17T23:46:52.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.810+0000 7f948b353640 1 -- 172.21.15.150:0/411866881 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94840b4c90 con 0x7f948415a030 2024-03-17T23:46:52.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.810+0000 7f9479ffb640 1 -- 172.21.15.150:0/411866881 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947400fd30 con 0x7f948415a030 2024-03-17T23:46:52.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.811+0000 7f948b353640 1 -- 172.21.15.150:0/411866881 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94840b5250 con 0x7f948415a030 2024-03-17T23:46:52.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.812+0000 7f948b353640 1 -- 172.21.15.150:0/411866881 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94841412f0 con 0x7f948415a030 2024-03-17T23:46:52.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.812+0000 7f9479ffb640 1 -- 172.21.15.150:0/411866881 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f947401d070 con 0x7f948415a030 2024-03-17T23:46:52.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.812+0000 7f9479ffb640 1 --2- 172.21.15.150:0/411866881 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f946806bbd0 0x7f946806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:52.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.812+0000 7f9479ffb640 1 -- 172.21.15.150:0/411866881 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f9474089fe0 con 0x7f948415a030 2024-03-17T23:46:52.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.812+0000 7f94890ef640 1 --2- 172.21.15.150:0/411866881 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f946806bbd0 0x7f946806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:52.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.813+0000 7f94890ef640 1 --2- 172.21.15.150:0/411866881 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f946806bbd0 0x7f946806e090 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9480034000 tx=0x7f9480009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:52.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.817+0000 7f9479ffb640 1 -- 172.21.15.150:0/411866881 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9474058eb0 con 0x7f948415a030 2024-03-17T23:46:52.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.991+0000 7f948b353640 1 -- 172.21.15.150:0/411866881 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f94840688b0 con 0x7f948415a030 2024-03-17T23:46:52.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.991+0000 7f9479ffb640 1 -- 172.21.15.150:0/411866881 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7f94740587b0 con 0x7f948415a030 2024-03-17T23:46:52.992 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:46:52.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.993+0000 7f948b353640 1 -- 172.21.15.150:0/411866881 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f946806bbd0 msgr2=0x7f946806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:52.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.993+0000 7f948b353640 1 --2- 172.21.15.150:0/411866881 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f946806bbd0 0x7f946806e090 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9480034000 tx=0x7f9480009030 comp rx=0 tx=0).stop 2024-03-17T23:46:52.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.993+0000 7f948b353640 1 -- 172.21.15.150:0/411866881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f948415a030 msgr2=0x7f9484143650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:52.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.993+0000 7f948b353640 1 --2- 172.21.15.150:0/411866881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f948415a030 0x7f9484143650 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f94740073a0 tx=0x7f9474007860 comp rx=0 tx=0).stop 2024-03-17T23:46:52.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.993+0000 7f948b353640 1 -- 172.21.15.150:0/411866881 shutdown_connections 2024-03-17T23:46:52.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.993+0000 7f948b353640 1 --2- 172.21.15.150:0/411866881 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f946806bbd0 0x7f946806e090 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.993+0000 7f948b353640 1 --2- 172.21.15.150:0/411866881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f948415a4b0 0x7f9484143b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.993+0000 7f948b353640 1 --2- 172.21.15.150:0/411866881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f948415a030 0x7f9484143650 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.993+0000 7f948b353640 1 --2- 172.21.15.150:0/411866881 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f94700039b0 0x7f9484155480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:52.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.993+0000 7f948b353640 1 -- 172.21.15.150:0/411866881 >> 172.21.15.150:0/411866881 conn(0x7f9484099a70 msgr2=0x7f9484070500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:52.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.993+0000 7f948b353640 1 -- 172.21.15.150:0/411866881 shutdown_connections 2024-03-17T23:46:52.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:52.993+0000 7f948b353640 1 -- 172.21.15.150:0/411866881 wait complete. 2024-03-17T23:46:53.004 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-17T23:46:53.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.104+0000 7fbc4da36640 1 Processor -- start 2024-03-17T23:46:53.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.104+0000 7fbc4da36640 1 -- start start 2024-03-17T23:46:53.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.105+0000 7fbc4da36640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbc4815de30 0x7fbc481619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:53.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.105+0000 7fbc4da36640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fbc480b9860 0x7fbc480cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:53.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.105+0000 7fbc4da36640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc480cef50 0x7fbc4806c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:53.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.105+0000 7fbc4da36640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fbc480cf3a0 con 0x7fbc4815de30 2024-03-17T23:46:53.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.105+0000 7fbc4da36640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fbc48162940 con 0x7fbc480b9860 2024-03-17T23:46:53.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.105+0000 7fbc4da36640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbc480b9cf0 con 0x7fbc480cef50 2024-03-17T23:46:53.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.105+0000 7fbc46ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbc4815de30 0x7fbc481619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:53.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.105+0000 7fbc46ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbc4815de30 0x7fbc481619a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45138/0 (socket says 172.21.15.150:45138) 2024-03-17T23:46:53.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.105+0000 7fbc46ffd640 1 -- 172.21.15.150:0/3820028537 learned_addr learned my addr 172.21.15.150:0/3820028537 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:53.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.105+0000 7fbc467fc640 1 --2- 172.21.15.150:0/3820028537 >> v2:172.21.15.157:3300/0 conn(0x7fbc480b9860 0x7fbc480cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:53.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.105+0000 7fbc477fe640 1 --2- 172.21.15.150:0/3820028537 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc480cef50 0x7fbc4806c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:53.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.105+0000 7fbc46ffd640 1 -- 172.21.15.150:0/3820028537 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc480cef50 msgr2=0x7fbc4806c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:53.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.105+0000 7fbc46ffd640 1 --2- 172.21.15.150:0/3820028537 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc480cef50 0x7fbc4806c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:53.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.105+0000 7fbc46ffd640 1 -- 172.21.15.150:0/3820028537 >> v2:172.21.15.157:3300/0 conn(0x7fbc480b9860 msgr2=0x7fbc480cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:53.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.105+0000 7fbc46ffd640 1 --2- 172.21.15.150:0/3820028537 >> v2:172.21.15.157:3300/0 conn(0x7fbc480b9860 0x7fbc480cea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:53.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.105+0000 7fbc46ffd640 1 -- 172.21.15.150:0/3820028537 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc481620a0 con 0x7fbc4815de30 2024-03-17T23:46:53.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.105+0000 7fbc477fe640 1 --2- 172.21.15.150:0/3820028537 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc480cef50 0x7fbc4806c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:46:53.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.106+0000 7fbc46ffd640 1 --2- 172.21.15.150:0/3820028537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc4815de30 0x7fbc481619a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbc480cada0 tx=0x7fbc3802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fff20922d6a7243 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:53.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.106+0000 7fbc45ffb640 1 -- 172.21.15.150:0/3820028537 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3802e070 con 0x7fbc4815de30 2024-03-17T23:46:53.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.106+0000 7fbc45ffb640 1 -- 172.21.15.150:0/3820028537 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc38037070 con 0x7fbc4815de30 2024-03-17T23:46:53.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.106+0000 7fbc45ffb640 1 -- 172.21.15.150:0/3820028537 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc38031a00 con 0x7fbc4815de30 2024-03-17T23:46:53.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.106+0000 7fbc4da36640 1 -- 172.21.15.150:0/3820028537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc4815de30 msgr2=0x7fbc481619a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:53.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.106+0000 7fbc4da36640 1 --2- 172.21.15.150:0/3820028537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc4815de30 0x7fbc481619a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbc480cada0 tx=0x7fbc3802b880 comp rx=0 tx=0).stop 2024-03-17T23:46:53.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.107+0000 7fbc4da36640 1 -- 172.21.15.150:0/3820028537 shutdown_connections 2024-03-17T23:46:53.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.107+0000 7fbc4da36640 1 --2- 172.21.15.150:0/3820028537 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc480cef50 0x7fbc4806c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:53.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.107+0000 7fbc4da36640 1 --2- 172.21.15.150:0/3820028537 >> v2:172.21.15.157:3300/0 conn(0x7fbc480b9860 0x7fbc480cea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:53.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.107+0000 7fbc4da36640 1 --2- 172.21.15.150:0/3820028537 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc4815de30 0x7fbc481619a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:53.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.107+0000 7fbc4da36640 1 -- 172.21.15.150:0/3820028537 >> 172.21.15.150:0/3820028537 conn(0x7fbc480730b0 msgr2=0x7fbc480d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:53.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.107+0000 7fbc4da36640 1 -- 172.21.15.150:0/3820028537 shutdown_connections 2024-03-17T23:46:53.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.107+0000 7fbc4da36640 1 -- 172.21.15.150:0/3820028537 wait complete. 2024-03-17T23:46:53.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.108+0000 7fbc4da36640 1 Processor -- start 2024-03-17T23:46:53.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.108+0000 7fbc4da36640 1 -- start start 2024-03-17T23:46:53.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.108+0000 7fbc4da36640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc480b9860 0x7fbc481f4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:53.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.109+0000 7fbc4da36640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc480cef50 0x7fbc48160570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:53.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.109+0000 7fbc4da36640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc4815de30 0x7fbc48151f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:53.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.109+0000 7fbc4da36640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbc480d2170 con 0x7fbc480cef50 2024-03-17T23:46:53.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.109+0000 7fbc4da36640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbc48162660 con 0x7fbc480b9860 2024-03-17T23:46:53.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.109+0000 7fbc4da36640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbc4815e210 con 0x7fbc4815de30 2024-03-17T23:46:53.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.109+0000 7fbc467fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc480cef50 0x7fbc48160570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:53.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.109+0000 7fbc467fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc480cef50 0x7fbc48160570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45144/0 (socket says 172.21.15.150:45144) 2024-03-17T23:46:53.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.109+0000 7fbc467fc640 1 -- 172.21.15.150:0/916637383 learned_addr learned my addr 172.21.15.150:0/916637383 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:53.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.109+0000 7fbc467fc640 1 -- 172.21.15.150:0/916637383 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc4815de30 msgr2=0x7fbc48151f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:53.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.109+0000 7fbc467fc640 1 --2- 172.21.15.150:0/916637383 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc4815de30 0x7fbc48151f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:53.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.109+0000 7fbc467fc640 1 -- 172.21.15.150:0/916637383 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc480b9860 msgr2=0x7fbc481f4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:53.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.109+0000 7fbc467fc640 1 --2- 172.21.15.150:0/916637383 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc480b9860 0x7fbc481f4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:53.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.109+0000 7fbc467fc640 1 -- 172.21.15.150:0/916637383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc481f7440 con 0x7fbc480cef50 2024-03-17T23:46:53.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.109+0000 7fbc467fc640 1 --2- 172.21.15.150:0/916637383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc480cef50 0x7fbc48160570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c00c570 tx=0x7fbc3c00ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:53.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.109+0000 7fbc2ffff640 1 -- 172.21.15.150:0/916637383 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c015070 con 0x7fbc480cef50 2024-03-17T23:46:53.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.109+0000 7fbc2ffff640 1 -- 172.21.15.150:0/916637383 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc3c011800 con 0x7fbc480cef50 2024-03-17T23:46:53.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.109+0000 7fbc4da36640 1 -- 172.21.15.150:0/916637383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc481f7720 con 0x7fbc480cef50 2024-03-17T23:46:53.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.109+0000 7fbc2ffff640 1 -- 172.21.15.150:0/916637383 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c011bf0 con 0x7fbc480cef50 2024-03-17T23:46:53.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.109+0000 7fbc4da36640 1 -- 172.21.15.150:0/916637383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc481f7ce0 con 0x7fbc480cef50 2024-03-17T23:46:53.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.110+0000 7fbc4da36640 1 -- 172.21.15.150:0/916637383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc480681a0 con 0x7fbc480cef50 2024-03-17T23:46:53.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.111+0000 7fbc2ffff640 1 -- 172.21.15.150:0/916637383 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbc3c018070 con 0x7fbc480cef50 2024-03-17T23:46:53.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.111+0000 7fbc2ffff640 1 --2- 172.21.15.150:0/916637383 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbc3406bc00 0x7fbc3406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:53.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.111+0000 7fbc2ffff640 1 -- 172.21.15.150:0/916637383 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fbc3c08a4b0 con 0x7fbc480cef50 2024-03-17T23:46:53.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.111+0000 7fbc46ffd640 1 --2- 172.21.15.150:0/916637383 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbc3406bc00 0x7fbc3406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:53.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.111+0000 7fbc46ffd640 1 --2- 172.21.15.150:0/916637383 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbc3406bc00 0x7fbc3406e0c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbc38036000 tx=0x7fbc38030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:53.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.115+0000 7fbc2ffff640 1 -- 172.21.15.150:0/916637383 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbc3c0592f0 con 0x7fbc480cef50 2024-03-17T23:46:53.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.291+0000 7fbc4da36640 1 -- 172.21.15.150:0/916637383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fbc480d1910 con 0x7fbc480cef50 2024-03-17T23:46:53.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.291+0000 7fbc2ffff640 1 -- 172.21.15.150:0/916637383 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7fbc3c058bf0 con 0x7fbc480cef50 2024-03-17T23:46:53.292 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:46:53.292 INFO:teuthology.orchestra.run.smithi150.stdout:[ 2024-03-17T23:46:53.292 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:46:53.292 INFO:teuthology.orchestra.run.smithi150.stdout: "poolnum": 1, 2024-03-17T23:46:53.292 INFO:teuthology.orchestra.run.smithi150.stdout: "poolname": ".mgr" 2024-03-17T23:46:53.293 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:46:53.293 INFO:teuthology.orchestra.run.smithi150.stdout:] 2024-03-17T23:46:53.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.293+0000 7fbc4da36640 1 -- 172.21.15.150:0/916637383 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbc3406bc00 msgr2=0x7fbc3406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:53.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.293+0000 7fbc4da36640 1 --2- 172.21.15.150:0/916637383 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbc3406bc00 0x7fbc3406e0c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbc38036000 tx=0x7fbc38030000 comp rx=0 tx=0).stop 2024-03-17T23:46:53.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.293+0000 7fbc4da36640 1 -- 172.21.15.150:0/916637383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc480cef50 msgr2=0x7fbc48160570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:53.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.293+0000 7fbc4da36640 1 --2- 172.21.15.150:0/916637383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc480cef50 0x7fbc48160570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c00c570 tx=0x7fbc3c00ca30 comp rx=0 tx=0).stop 2024-03-17T23:46:53.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.293+0000 7fbc4da36640 1 -- 172.21.15.150:0/916637383 shutdown_connections 2024-03-17T23:46:53.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.293+0000 7fbc4da36640 1 --2- 172.21.15.150:0/916637383 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbc4815de30 0x7fbc48151f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:53.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.293+0000 7fbc4da36640 1 --2- 172.21.15.150:0/916637383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc480cef50 0x7fbc48160570 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:53.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.293+0000 7fbc4da36640 1 --2- 172.21.15.150:0/916637383 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbc3406bc00 0x7fbc3406e0c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:53.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.293+0000 7fbc4da36640 1 --2- 172.21.15.150:0/916637383 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc480b9860 0x7fbc481f4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:53.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.293+0000 7fbc4da36640 1 -- 172.21.15.150:0/916637383 >> 172.21.15.150:0/916637383 conn(0x7fbc480730b0 msgr2=0x7fbc480b37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:53.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.294+0000 7fbc4da36640 1 -- 172.21.15.150:0/916637383 shutdown_connections 2024-03-17T23:46:53.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.294+0000 7fbc4da36640 1 -- 172.21.15.150:0/916637383 wait complete. 2024-03-17T23:46:53.305 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-17T23:46:53.305 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-17T23:46:53.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.408+0000 7fa3a70a7640 1 Processor -- start 2024-03-17T23:46:53.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.408+0000 7fa3a70a7640 1 -- start start 2024-03-17T23:46:53.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.408+0000 7fa3a70a7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa3a00afc80 0x7fa3a00ad280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:53.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.408+0000 7fa3a70a7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3a0067570 0x7fa3a006bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:53.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.409+0000 7fa3a70a7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3a0067a00 con 0x7fa3a00d09b0 2024-03-17T23:46:53.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.409+0000 7fa3a70a7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fa3a0099d20 con 0x7fa3a00afc80 2024-03-17T23:46:53.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.409+0000 7fa3a70a7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa3a00d0eb0 con 0x7fa3a0067570 2024-03-17T23:46:53.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.409+0000 7fa3a4e43640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa3a00afc80 0x7fa3a00ad280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:53.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.409+0000 7fa39ffff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa3a00d09b0 0x7fa3a00b0060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39056/0 (socket says 172.21.15.150:39056) 2024-03-17T23:46:53.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.409+0000 7fa39ffff640 1 -- 172.21.15.150:0/1970003126 learned_addr learned my addr 172.21.15.150:0/1970003126 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:46:53.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.409+0000 7fa3a5644640 1 --2- 172.21.15.150:0/1970003126 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3a0067570 0x7fa3a006bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:53.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.409+0000 7fa3a4e43640 1 -- 172.21.15.150:0/1970003126 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3a0067570 msgr2=0x7fa3a006bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:53.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.409+0000 7fa3a4e43640 1 --2- 172.21.15.150:0/1970003126 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3a0067570 0x7fa3a006bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:53.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.409+0000 7fa3a4e43640 1 -- 172.21.15.150:0/1970003126 >> v1:172.21.15.150:6789/0 conn(0x7fa3a00d09b0 legacy=0x7fa3a00b0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:53.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.409+0000 7fa3a4e43640 1 -- 172.21.15.150:0/1970003126 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a00d1190 con 0x7fa3a00afc80 2024-03-17T23:46:53.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.410+0000 7fa3a4e43640 1 --2- 172.21.15.150:0/1970003126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a00afc80 0x7fa3a00ad280 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa3940097a0 tx=0x7fa39402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=621c462f54180377 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:53.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.410+0000 7fa39f7fe640 1 -- 172.21.15.150:0/1970003126 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa394035020 con 0x7fa3a00afc80 2024-03-17T23:46:53.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.410+0000 7fa39f7fe640 1 -- 172.21.15.150:0/1970003126 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa394031830 con 0x7fa3a00afc80 2024-03-17T23:46:53.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.410+0000 7fa39f7fe640 1 -- 172.21.15.150:0/1970003126 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa394031c00 con 0x7fa3a00afc80 2024-03-17T23:46:53.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.410+0000 7fa3a70a7640 1 -- 172.21.15.150:0/1970003126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a00afc80 msgr2=0x7fa3a00ad280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:53.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.410+0000 7fa3a70a7640 1 --2- 172.21.15.150:0/1970003126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a00afc80 0x7fa3a00ad280 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa3940097a0 tx=0x7fa39402b8d0 comp rx=0 tx=0).stop 2024-03-17T23:46:53.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.410+0000 7fa3a70a7640 1 -- 172.21.15.150:0/1970003126 shutdown_connections 2024-03-17T23:46:53.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.410+0000 7fa3a70a7640 1 --2- 172.21.15.150:0/1970003126 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3a0067570 0x7fa3a006bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:53.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.410+0000 7fa3a70a7640 1 --2- 172.21.15.150:0/1970003126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a00afc80 0x7fa3a00ad280 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:53.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.410+0000 7fa3a70a7640 1 -- 172.21.15.150:0/1970003126 >> 172.21.15.150:0/1970003126 conn(0x7fa3a0072a90 msgr2=0x7fa3a00b5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:53.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.410+0000 7fa3a70a7640 1 -- 172.21.15.150:0/1970003126 shutdown_connections 2024-03-17T23:46:53.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.410+0000 7fa3a70a7640 1 -- 172.21.15.150:0/1970003126 wait complete. 2024-03-17T23:46:53.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.411+0000 7fa3a70a7640 1 Processor -- start 2024-03-17T23:46:53.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.411+0000 7fa3a70a7640 1 -- start start 2024-03-17T23:46:53.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.412+0000 7fa3a70a7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a0067570 0x7fa3a01552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:53.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.412+0000 7fa3a70a7640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3a00afc80 0x7fa3a009dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:53.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.412+0000 7fa3a70a7640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3a00d09b0 0x7fa3a015fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:53.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.412+0000 7fa3a70a7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a006b7f0 con 0x7fa3a0067570 2024-03-17T23:46:53.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.412+0000 7fa3a70a7640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a006b510 con 0x7fa3a00d09b0 2024-03-17T23:46:53.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.412+0000 7fa3a70a7640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa3a015e2d0 con 0x7fa3a00afc80 2024-03-17T23:46:53.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.412+0000 7fa3a4e43640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a0067570 0x7fa3a01552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:53.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.412+0000 7fa3a4e43640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a0067570 0x7fa3a01552e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45164/0 (socket says 172.21.15.150:45164) 2024-03-17T23:46:53.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.412+0000 7fa3a4e43640 1 -- 172.21.15.150:0/2652661990 learned_addr learned my addr 172.21.15.150:0/2652661990 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:53.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.413+0000 7fa39ffff640 1 --2- 172.21.15.150:0/2652661990 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3a00afc80 0x7fa3a009dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:53.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.413+0000 7fa3a5644640 1 --2- 172.21.15.150:0/2652661990 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3a00d09b0 0x7fa3a015fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:53.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.413+0000 7fa3a4e43640 1 -- 172.21.15.150:0/2652661990 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3a00afc80 msgr2=0x7fa3a009dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:53.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.413+0000 7fa3a4e43640 1 --2- 172.21.15.150:0/2652661990 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3a00afc80 0x7fa3a009dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:53.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.413+0000 7fa3a4e43640 1 -- 172.21.15.150:0/2652661990 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3a00d09b0 msgr2=0x7fa3a015fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:53.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.413+0000 7fa3a4e43640 1 --2- 172.21.15.150:0/2652661990 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3a00d09b0 0x7fa3a015fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:53.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.413+0000 7fa3a4e43640 1 -- 172.21.15.150:0/2652661990 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a009be50 con 0x7fa3a0067570 2024-03-17T23:46:53.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.413+0000 7fa39ffff640 1 --2- 172.21.15.150:0/2652661990 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3a00afc80 0x7fa3a009dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:46:53.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.413+0000 7fa3a4e43640 1 --2- 172.21.15.150:0/2652661990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a0067570 0x7fa3a01552e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa3940096f0 tx=0x7fa3940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:53.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.413+0000 7fa39d7fa640 1 -- 172.21.15.150:0/2652661990 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa394041070 con 0x7fa3a0067570 2024-03-17T23:46:53.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.413+0000 7fa39d7fa640 1 -- 172.21.15.150:0/2652661990 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa39403b860 con 0x7fa3a0067570 2024-03-17T23:46:53.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.413+0000 7fa3a70a7640 1 -- 172.21.15.150:0/2652661990 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3a009c130 con 0x7fa3a0067570 2024-03-17T23:46:53.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.413+0000 7fa39d7fa640 1 -- 172.21.15.150:0/2652661990 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa394035040 con 0x7fa3a0067570 2024-03-17T23:46:53.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.413+0000 7fa3a70a7640 1 -- 172.21.15.150:0/2652661990 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3a009c9d0 con 0x7fa3a0067570 2024-03-17T23:46:53.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.414+0000 7fa3a70a7640 1 -- 172.21.15.150:0/2652661990 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3a00681a0 con 0x7fa3a0067570 2024-03-17T23:46:53.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.414+0000 7fa39d7fa640 1 -- 172.21.15.150:0/2652661990 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa39404c060 con 0x7fa3a0067570 2024-03-17T23:46:53.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.415+0000 7fa39d7fa640 1 --2- 172.21.15.150:0/2652661990 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa38406bbd0 0x7fa38406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:53.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.415+0000 7fa39d7fa640 1 -- 172.21.15.150:0/2652661990 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fa3940ac0c0 con 0x7fa3a0067570 2024-03-17T23:46:53.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.415+0000 7fa39ffff640 1 --2- 172.21.15.150:0/2652661990 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa38406bbd0 0x7fa38406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:53.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.415+0000 7fa39ffff640 1 --2- 172.21.15.150:0/2652661990 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa38406bbd0 0x7fa38406e090 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa3a01563f0 tx=0x7fa38c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:53.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.419+0000 7fa39d7fa640 1 -- 172.21.15.150:0/2652661990 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa39407af10 con 0x7fa3a0067570 2024-03-17T23:46:53.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.592+0000 7fa3a70a7640 1 -- 172.21.15.150:0/2652661990 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fa3a0070220 con 0x7fa3a0067570 2024-03-17T23:46:53.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.898+0000 7fa39d7fa640 1 -- 172.21.15.150:0/2652661990 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7fa39407a810 con 0x7fa3a0067570 2024-03-17T23:46:53.899 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_metadata' created 2024-03-17T23:46:53.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.901+0000 7fa3a70a7640 1 -- 172.21.15.150:0/2652661990 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa38406bbd0 msgr2=0x7fa38406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:53.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.901+0000 7fa3a70a7640 1 --2- 172.21.15.150:0/2652661990 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa38406bbd0 0x7fa38406e090 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa3a01563f0 tx=0x7fa38c009040 comp rx=0 tx=0).stop 2024-03-17T23:46:53.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.901+0000 7fa3a70a7640 1 -- 172.21.15.150:0/2652661990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a0067570 msgr2=0x7fa3a01552e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:53.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.901+0000 7fa3a70a7640 1 --2- 172.21.15.150:0/2652661990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a0067570 0x7fa3a01552e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa3940096f0 tx=0x7fa3940046c0 comp rx=0 tx=0).stop 2024-03-17T23:46:53.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.901+0000 7fa3a70a7640 1 -- 172.21.15.150:0/2652661990 shutdown_connections 2024-03-17T23:46:53.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.901+0000 7fa3a70a7640 1 --2- 172.21.15.150:0/2652661990 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa38406bbd0 0x7fa38406e090 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:53.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.901+0000 7fa3a70a7640 1 --2- 172.21.15.150:0/2652661990 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3a00d09b0 0x7fa3a015fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:53.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.901+0000 7fa3a70a7640 1 --2- 172.21.15.150:0/2652661990 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3a00afc80 0x7fa3a009dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:53.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.901+0000 7fa3a70a7640 1 --2- 172.21.15.150:0/2652661990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a0067570 0x7fa3a01552e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:53.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.901+0000 7fa3a70a7640 1 -- 172.21.15.150:0/2652661990 >> 172.21.15.150:0/2652661990 conn(0x7fa3a0072a90 msgr2=0x7fa3a006e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:53.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.901+0000 7fa3a70a7640 1 -- 172.21.15.150:0/2652661990 shutdown_connections 2024-03-17T23:46:53.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:53.901+0000 7fa3a70a7640 1 -- 172.21.15.150:0/2652661990 wait complete. 2024-03-17T23:46:53.915 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-17T23:46:54.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.022+0000 7fe40a812640 1 Processor -- start 2024-03-17T23:46:54.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.023+0000 7fe40a812640 1 -- start start 2024-03-17T23:46:54.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.023+0000 7fe40a812640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe40406b8a0 0x7fe40415e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:54.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.023+0000 7fe40a812640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe40406f360 con 0x7fe40415dce0 2024-03-17T23:46:54.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.023+0000 7fe40a812640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe40406f640 con 0x7fe40406ee10 2024-03-17T23:46:54.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.023+0000 7fe40a812640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fe4040cd480 con 0x7fe40406b8a0 2024-03-17T23:46:54.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.023+0000 7fe4037fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe40406b8a0 0x7fe40415e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:54.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.023+0000 7fe403fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fe40415dce0 0x7fe40406b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39064/0 (socket says 172.21.15.150:39064) 2024-03-17T23:46:54.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.023+0000 7fe4037fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe40406b8a0 0x7fe40415e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45168/0 (socket says 172.21.15.150:45168) 2024-03-17T23:46:54.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.023+0000 7fe403fff640 1 -- 172.21.15.150:0/727457047 learned_addr learned my addr 172.21.15.150:0/727457047 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:46:54.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.024+0000 7fe4037fe640 1 -- 172.21.15.150:0/727457047 >> v1:172.21.15.157:6789/0 conn(0x7fe40406ee10 legacy=0x7fe4040cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:54.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.024+0000 7fe4037fe640 1 -- 172.21.15.150:0/727457047 >> v1:172.21.15.150:6789/0 conn(0x7fe40415dce0 legacy=0x7fe40406b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:54.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.024+0000 7fe4037fe640 1 -- 172.21.15.150:0/727457047 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe40406f920 con 0x7fe40406b8a0 2024-03-17T23:46:54.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.024+0000 7fe402ffd640 1 -- 172.21.15.150:0/727457047 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2436656967 0 0) 0x7fe40406f360 con 0x7fe40415dce0 2024-03-17T23:46:54.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.024+0000 7fe4037fe640 1 --2- 172.21.15.150:0/727457047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe40406b8a0 0x7fe40415e0c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe3f80097a0 tx=0x7fe3f802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a099bd5f187f768 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:54.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.025+0000 7fe402ffd640 1 -- 172.21.15.150:0/727457047 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8035020 con 0x7fe40406b8a0 2024-03-17T23:46:54.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.025+0000 7fe402ffd640 1 -- 172.21.15.150:0/727457047 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3f8031830 con 0x7fe40406b8a0 2024-03-17T23:46:54.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.025+0000 7fe402ffd640 1 -- 172.21.15.150:0/727457047 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8031c00 con 0x7fe40406b8a0 2024-03-17T23:46:54.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.025+0000 7fe40a812640 1 -- 172.21.15.150:0/727457047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe40406b8a0 msgr2=0x7fe40415e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:54.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.025+0000 7fe40a812640 1 --2- 172.21.15.150:0/727457047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe40406b8a0 0x7fe40415e0c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe3f80097a0 tx=0x7fe3f802b8d0 comp rx=0 tx=0).stop 2024-03-17T23:46:54.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.025+0000 7fe40a812640 1 -- 172.21.15.150:0/727457047 shutdown_connections 2024-03-17T23:46:54.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.025+0000 7fe40a812640 1 --2- 172.21.15.150:0/727457047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe40406b8a0 0x7fe40415e0c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:54.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.025+0000 7fe40a812640 1 -- 172.21.15.150:0/727457047 >> 172.21.15.150:0/727457047 conn(0x7fe404099a70 msgr2=0x7fe404099e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:54.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.025+0000 7fe40a812640 1 -- 172.21.15.150:0/727457047 shutdown_connections 2024-03-17T23:46:54.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.025+0000 7fe40a812640 1 -- 172.21.15.150:0/727457047 wait complete. 2024-03-17T23:46:54.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.026+0000 7fe40a812640 1 Processor -- start 2024-03-17T23:46:54.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.026+0000 7fe40a812640 1 -- start start 2024-03-17T23:46:54.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.027+0000 7fe40a812640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe40406b8a0 0x7fe4040c7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:54.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.027+0000 7fe40a812640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe40406ee10 0x7fe40414ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:54.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.027+0000 7fe40a812640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe40415dce0 0x7fe40415f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:54.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.027+0000 7fe40a812640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe4040afb10 con 0x7fe40406ee10 2024-03-17T23:46:54.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.027+0000 7fe40a812640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe40406ff70 con 0x7fe40406b8a0 2024-03-17T23:46:54.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.027+0000 7fe40a812640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe4040afc90 con 0x7fe40415dce0 2024-03-17T23:46:54.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.027+0000 7fe4037fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe40406ee10 0x7fe40414ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:54.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.027+0000 7fe4037fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe40406ee10 0x7fe40414ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45172/0 (socket says 172.21.15.150:45172) 2024-03-17T23:46:54.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.027+0000 7fe4037fe640 1 -- 172.21.15.150:0/149945662 learned_addr learned my addr 172.21.15.150:0/149945662 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:54.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.027+0000 7fe403fff640 1 --2- 172.21.15.150:0/149945662 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe40406b8a0 0x7fe4040c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:54.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.027+0000 7fe408daf640 1 --2- 172.21.15.150:0/149945662 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe40415dce0 0x7fe40415f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:54.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.028+0000 7fe4037fe640 1 -- 172.21.15.150:0/149945662 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe40415dce0 msgr2=0x7fe40415f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:54.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.028+0000 7fe4037fe640 1 --2- 172.21.15.150:0/149945662 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe40415dce0 0x7fe40415f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:54.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.028+0000 7fe4037fe640 1 -- 172.21.15.150:0/149945662 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe40406b8a0 msgr2=0x7fe4040c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:54.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.028+0000 7fe4037fe640 1 --2- 172.21.15.150:0/149945662 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe40406b8a0 0x7fe4040c7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:54.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.028+0000 7fe4037fe640 1 -- 172.21.15.150:0/149945662 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4040c7890 con 0x7fe40406ee10 2024-03-17T23:46:54.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.028+0000 7fe408daf640 1 --2- 172.21.15.150:0/149945662 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe40415dce0 0x7fe40415f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:46:54.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.028+0000 7fe4037fe640 1 --2- 172.21.15.150:0/149945662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe40406ee10 0x7fe40414ed80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe3f80097a0 tx=0x7fe3f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:54.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.028+0000 7fe400ff9640 1 -- 172.21.15.150:0/149945662 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8004230 con 0x7fe40406ee10 2024-03-17T23:46:54.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.028+0000 7fe400ff9640 1 -- 172.21.15.150:0/149945662 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3f80043d0 con 0x7fe40406ee10 2024-03-17T23:46:54.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.028+0000 7fe40a812640 1 -- 172.21.15.150:0/149945662 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe40415e5a0 con 0x7fe40406ee10 2024-03-17T23:46:54.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.028+0000 7fe400ff9640 1 -- 172.21.15.150:0/149945662 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f803cad0 con 0x7fe40406ee10 2024-03-17T23:46:54.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.028+0000 7fe40a812640 1 -- 172.21.15.150:0/149945662 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe40415eb60 con 0x7fe40406ee10 2024-03-17T23:46:54.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.029+0000 7fe40a812640 1 -- 172.21.15.150:0/149945662 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4040688b0 con 0x7fe40406ee10 2024-03-17T23:46:54.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.029+0000 7fe400ff9640 1 -- 172.21.15.150:0/149945662 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe3f8038020 con 0x7fe40406ee10 2024-03-17T23:46:54.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.030+0000 7fe400ff9640 1 --2- 172.21.15.150:0/149945662 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe3f006bc00 0x7fe3f006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:54.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.030+0000 7fe403fff640 1 --2- 172.21.15.150:0/149945662 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe3f006bc00 0x7fe3f006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:54.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.030+0000 7fe400ff9640 1 -- 172.21.15.150:0/149945662 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7fe3f80aca50 con 0x7fe40406ee10 2024-03-17T23:46:54.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.030+0000 7fe403fff640 1 --2- 172.21.15.150:0/149945662 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe3f006bc00 0x7fe3f006e0c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4007fd0 tx=0x7fe3f4007de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:54.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.034+0000 7fe400ff9640 1 -- 172.21.15.150:0/149945662 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe3f8041070 con 0x7fe40406ee10 2024-03-17T23:46:54.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.209+0000 7fe40a812640 1 -- 172.21.15.150:0/149945662 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fe4040cd660 con 0x7fe40406ee10 2024-03-17T23:46:54.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.900+0000 7fe400ff9640 1 -- 172.21.15.150:0/149945662 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v19) ==== 171+0+0 (secure 0 0 0) 0x7fe3f807b720 con 0x7fe40406ee10 2024-03-17T23:46:54.901 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_data' created 2024-03-17T23:46:54.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.903+0000 7fe40a812640 1 -- 172.21.15.150:0/149945662 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe3f006bc00 msgr2=0x7fe3f006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:54.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.903+0000 7fe40a812640 1 --2- 172.21.15.150:0/149945662 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe3f006bc00 0x7fe3f006e0c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4007fd0 tx=0x7fe3f4007de0 comp rx=0 tx=0).stop 2024-03-17T23:46:54.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.903+0000 7fe40a812640 1 -- 172.21.15.150:0/149945662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe40406ee10 msgr2=0x7fe40414ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:54.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.903+0000 7fe40a812640 1 --2- 172.21.15.150:0/149945662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe40406ee10 0x7fe40414ed80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe3f80097a0 tx=0x7fe3f80046c0 comp rx=0 tx=0).stop 2024-03-17T23:46:54.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.903+0000 7fe40a812640 1 -- 172.21.15.150:0/149945662 shutdown_connections 2024-03-17T23:46:54.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.903+0000 7fe40a812640 1 --2- 172.21.15.150:0/149945662 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe40415dce0 0x7fe40415f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:54.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.903+0000 7fe40a812640 1 --2- 172.21.15.150:0/149945662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe40406ee10 0x7fe40414ed80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:54.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.904+0000 7fe40a812640 1 --2- 172.21.15.150:0/149945662 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe3f006bc00 0x7fe3f006e0c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:54.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.904+0000 7fe40a812640 1 --2- 172.21.15.150:0/149945662 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe40406b8a0 0x7fe4040c7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:54.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.904+0000 7fe40a812640 1 -- 172.21.15.150:0/149945662 >> 172.21.15.150:0/149945662 conn(0x7fe404099a70 msgr2=0x7fe4040b3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:54.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.904+0000 7fe40a812640 1 -- 172.21.15.150:0/149945662 shutdown_connections 2024-03-17T23:46:54.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:54.904+0000 7fe40a812640 1 -- 172.21.15.150:0/149945662 wait complete. 2024-03-17T23:46:54.929 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-17T23:46:55.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.032+0000 7f4e031b3640 1 Processor -- start 2024-03-17T23:46:55.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.032+0000 7f4e031b3640 1 -- start start 2024-03-17T23:46:55.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.032+0000 7f4e031b3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4dfc099e20 0x7f4dfc09a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:55.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.032+0000 7f4e031b3640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4dfc15e460 0x7f4dfc06c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:55.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.032+0000 7f4e031b3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4dfc06c8c0 0x7f4dfc0d0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:55.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.032+0000 7f4e031b3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f4dfc09a740 con 0x7f4dfc099e20 2024-03-17T23:46:55.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.032+0000 7f4e031b3640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f4dfc15e990 con 0x7f4dfc15e460 2024-03-17T23:46:55.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.032+0000 7f4e031b3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4dfc15ec70 con 0x7f4dfc06c8c0 2024-03-17T23:46:55.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.033+0000 7f4e00f4f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4dfc099e20 0x7f4dfc09a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:55.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.033+0000 7f4e00f4f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4dfc099e20 0x7f4dfc09a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45182/0 (socket says 172.21.15.150:45182) 2024-03-17T23:46:55.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.033+0000 7f4e00f4f640 1 -- 172.21.15.150:0/1073241943 learned_addr learned my addr 172.21.15.150:0/1073241943 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:55.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.033+0000 7f4e01750640 1 --2- 172.21.15.150:0/1073241943 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4dfc06c8c0 0x7f4dfc0d0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:55.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.033+0000 7f4dfbfff640 1 --2- 172.21.15.150:0/1073241943 >> v2:172.21.15.157:3300/0 conn(0x7f4dfc15e460 0x7f4dfc06c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:55.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.033+0000 7f4e00f4f640 1 -- 172.21.15.150:0/1073241943 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4dfc06c8c0 msgr2=0x7f4dfc0d0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:55.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.033+0000 7f4e00f4f640 1 --2- 172.21.15.150:0/1073241943 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4dfc06c8c0 0x7f4dfc0d0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:55.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.033+0000 7f4e00f4f640 1 -- 172.21.15.150:0/1073241943 >> v2:172.21.15.157:3300/0 conn(0x7f4dfc15e460 msgr2=0x7f4dfc06c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:55.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.033+0000 7f4e00f4f640 1 --2- 172.21.15.150:0/1073241943 >> v2:172.21.15.157:3300/0 conn(0x7f4dfc15e460 0x7f4dfc06c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:55.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.033+0000 7f4e00f4f640 1 -- 172.21.15.150:0/1073241943 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dfc15ef50 con 0x7f4dfc099e20 2024-03-17T23:46:55.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.033+0000 7f4dfbfff640 1 --2- 172.21.15.150:0/1073241943 >> v2:172.21.15.157:3300/0 conn(0x7f4dfc15e460 0x7f4dfc06c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:46:55.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.034+0000 7f4e00f4f640 1 --2- 172.21.15.150:0/1073241943 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4dfc099e20 0x7f4dfc09a200 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc0733b0 tx=0x7f4df002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca876a9bd6b60554 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:55.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.034+0000 7f4dfb7fe640 1 -- 172.21.15.150:0/1073241943 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df0035020 con 0x7f4dfc099e20 2024-03-17T23:46:55.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.034+0000 7f4dfb7fe640 1 -- 172.21.15.150:0/1073241943 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4df0031870 con 0x7f4dfc099e20 2024-03-17T23:46:55.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.034+0000 7f4dfb7fe640 1 -- 172.21.15.150:0/1073241943 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df0031c40 con 0x7f4dfc099e20 2024-03-17T23:46:55.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.034+0000 7f4e031b3640 1 -- 172.21.15.150:0/1073241943 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4dfc099e20 msgr2=0x7f4dfc09a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:55.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.034+0000 7f4e031b3640 1 --2- 172.21.15.150:0/1073241943 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4dfc099e20 0x7f4dfc09a200 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc0733b0 tx=0x7f4df002b910 comp rx=0 tx=0).stop 2024-03-17T23:46:55.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.034+0000 7f4e031b3640 1 -- 172.21.15.150:0/1073241943 shutdown_connections 2024-03-17T23:46:55.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.034+0000 7f4e031b3640 1 --2- 172.21.15.150:0/1073241943 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4dfc06c8c0 0x7f4dfc0d0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:55.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.034+0000 7f4e031b3640 1 --2- 172.21.15.150:0/1073241943 >> v2:172.21.15.157:3300/0 conn(0x7f4dfc15e460 0x7f4dfc06c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:55.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.034+0000 7f4e031b3640 1 --2- 172.21.15.150:0/1073241943 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4dfc099e20 0x7f4dfc09a200 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:55.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.034+0000 7f4e031b3640 1 -- 172.21.15.150:0/1073241943 >> 172.21.15.150:0/1073241943 conn(0x7f4dfc0b3630 msgr2=0x7f4dfc0b3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:55.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.034+0000 7f4e031b3640 1 -- 172.21.15.150:0/1073241943 shutdown_connections 2024-03-17T23:46:55.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.034+0000 7f4e031b3640 1 -- 172.21.15.150:0/1073241943 wait complete. 2024-03-17T23:46:55.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.035+0000 7f4e031b3640 1 Processor -- start 2024-03-17T23:46:55.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.036+0000 7f4e031b3640 1 -- start start 2024-03-17T23:46:55.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.036+0000 7f4e031b3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4dfc099e20 0x7f4dfc154d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:55.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.036+0000 7f4e031b3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4dfc15e460 0x7f4dfc1e6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:55.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.036+0000 7f4e031b3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4dfc1e83b0 0x7f4dfc0af8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:55.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.036+0000 7f4e031b3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4dfc06d570 con 0x7f4dfc1e83b0 2024-03-17T23:46:55.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.036+0000 7f4e031b3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4dfc0cfcd0 con 0x7f4dfc15e460 2024-03-17T23:46:55.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.036+0000 7f4e031b3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4dfc0d0740 con 0x7f4dfc099e20 2024-03-17T23:46:55.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.036+0000 7f4e01750640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4dfc1e83b0 0x7f4dfc0af8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:55.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.036+0000 7f4e01750640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4dfc1e83b0 0x7f4dfc0af8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45194/0 (socket says 172.21.15.150:45194) 2024-03-17T23:46:55.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.036+0000 7f4e01750640 1 -- 172.21.15.150:0/3436202661 learned_addr learned my addr 172.21.15.150:0/3436202661 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:55.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.037+0000 7f4e00f4f640 1 --2- 172.21.15.150:0/3436202661 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4dfc099e20 0x7f4dfc154d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:55.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.037+0000 7f4dfbfff640 1 --2- 172.21.15.150:0/3436202661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4dfc15e460 0x7f4dfc1e6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:55.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.037+0000 7f4e01750640 1 -- 172.21.15.150:0/3436202661 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4dfc099e20 msgr2=0x7f4dfc154d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:55.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.037+0000 7f4e01750640 1 --2- 172.21.15.150:0/3436202661 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4dfc099e20 0x7f4dfc154d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:55.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.037+0000 7f4e01750640 1 -- 172.21.15.150:0/3436202661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4dfc15e460 msgr2=0x7f4dfc1e6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:55.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.037+0000 7f4e01750640 1 --2- 172.21.15.150:0/3436202661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4dfc15e460 0x7f4dfc1e6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:55.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.037+0000 7f4e01750640 1 -- 172.21.15.150:0/3436202661 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dfc1552b0 con 0x7f4dfc1e83b0 2024-03-17T23:46:55.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.037+0000 7f4e01750640 1 --2- 172.21.15.150:0/3436202661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4dfc1e83b0 0x7f4dfc0af8c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4df400eed0 tx=0x7f4df400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:55.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.037+0000 7f4df97fa640 1 -- 172.21.15.150:0/3436202661 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df4018020 con 0x7f4dfc1e83b0 2024-03-17T23:46:55.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.037+0000 7f4df97fa640 1 -- 172.21.15.150:0/3436202661 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4df4014400 con 0x7f4dfc1e83b0 2024-03-17T23:46:55.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.037+0000 7f4e031b3640 1 -- 172.21.15.150:0/3436202661 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dfc155e40 con 0x7f4dfc1e83b0 2024-03-17T23:46:55.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.037+0000 7f4df97fa640 1 -- 172.21.15.150:0/3436202661 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df40147f0 con 0x7f4dfc1e83b0 2024-03-17T23:46:55.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.037+0000 7f4e031b3640 1 -- 172.21.15.150:0/3436202661 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dfc156400 con 0x7f4dfc1e83b0 2024-03-17T23:46:55.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.038+0000 7f4e031b3640 1 -- 172.21.15.150:0/3436202661 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dfc0681a0 con 0x7f4dfc1e83b0 2024-03-17T23:46:55.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.038+0000 7f4df97fa640 1 -- 172.21.15.150:0/3436202661 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4df4005d10 con 0x7f4dfc1e83b0 2024-03-17T23:46:55.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.039+0000 7f4df97fa640 1 --2- 172.21.15.150:0/3436202661 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4de006bbd0 0x7f4de006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:55.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.039+0000 7f4df97fa640 1 -- 172.21.15.150:0/3436202661 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7f4df408b2e0 con 0x7f4dfc1e83b0 2024-03-17T23:46:55.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.039+0000 7f4e00f4f640 1 --2- 172.21.15.150:0/3436202661 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4de006bbd0 0x7f4de006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:55.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.039+0000 7f4e00f4f640 1 --2- 172.21.15.150:0/3436202661 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4de006bbd0 0x7f4de006e090 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4df0036000 tx=0x7f4df003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:55.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.043+0000 7f4df97fa640 1 -- 172.21.15.150:0/3436202661 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4df4022030 con 0x7f4dfc1e83b0 2024-03-17T23:46:55.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.252+0000 7f4e031b3640 1 -- 172.21.15.150:0/3436202661 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f4dfc06cec0 con 0x7f4dfc1e83b0 2024-03-17T23:46:55.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.919+0000 7f4df97fa640 1 -- 172.21.15.150:0/3436202661 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f4df40597e0 con 0x7f4dfc1e83b0 2024-03-17T23:46:55.921 INFO:teuthology.orchestra.run.smithi150.stderr:new fs with metadata pool 2 and data pool 3 2024-03-17T23:46:55.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.923+0000 7f4e031b3640 1 -- 172.21.15.150:0/3436202661 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4de006bbd0 msgr2=0x7f4de006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:55.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.923+0000 7f4e031b3640 1 --2- 172.21.15.150:0/3436202661 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4de006bbd0 0x7f4de006e090 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4df0036000 tx=0x7f4df003c040 comp rx=0 tx=0).stop 2024-03-17T23:46:55.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.923+0000 7f4e031b3640 1 -- 172.21.15.150:0/3436202661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4dfc1e83b0 msgr2=0x7f4dfc0af8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:55.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.923+0000 7f4e031b3640 1 --2- 172.21.15.150:0/3436202661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4dfc1e83b0 0x7f4dfc0af8c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4df400eed0 tx=0x7f4df400b480 comp rx=0 tx=0).stop 2024-03-17T23:46:55.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.924+0000 7f4e031b3640 1 -- 172.21.15.150:0/3436202661 shutdown_connections 2024-03-17T23:46:55.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.924+0000 7f4e031b3640 1 --2- 172.21.15.150:0/3436202661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4dfc1e83b0 0x7f4dfc0af8c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:55.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.924+0000 7f4e031b3640 1 --2- 172.21.15.150:0/3436202661 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4de006bbd0 0x7f4de006e090 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:55.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.924+0000 7f4e031b3640 1 --2- 172.21.15.150:0/3436202661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4dfc15e460 0x7f4dfc1e6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:55.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.924+0000 7f4e031b3640 1 --2- 172.21.15.150:0/3436202661 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4dfc099e20 0x7f4dfc154d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:55.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.924+0000 7f4e031b3640 1 -- 172.21.15.150:0/3436202661 >> 172.21.15.150:0/3436202661 conn(0x7f4dfc0b3630 msgr2=0x7f4dfc0b3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:55.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.926+0000 7f4e031b3640 1 -- 172.21.15.150:0/3436202661 shutdown_connections 2024-03-17T23:46:55.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:55.928+0000 7f4e031b3640 1 -- 172.21.15.150:0/3436202661 wait complete. 2024-03-17T23:46:55.946 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-17T23:46:55.946 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-17T23:46:56.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.049+0000 7f9264722640 1 Processor -- start 2024-03-17T23:46:56.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.050+0000 7f9264722640 1 -- start start 2024-03-17T23:46:56.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.051+0000 7f9264722640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f925c0b2090 0x7f925c06b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:56.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.051+0000 7f9264722640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f925c06b6d0 0x7f925c0b0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:56.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.051+0000 7f9264722640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f925c0b3aa0 con 0x7f925c15de90 2024-03-17T23:46:56.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.051+0000 7f9264722640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f925c06bb30 con 0x7f925c0b2090 2024-03-17T23:46:56.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.051+0000 7f9264722640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f925c0b0570 con 0x7f925c06b6d0 2024-03-17T23:46:56.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.051+0000 7f9261cbd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f925c0b2090 0x7f925c06b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:56.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.051+0000 7f9262cbf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f925c06b6d0 0x7f925c0b0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:56.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.051+0000 7f9261cbd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f925c0b2090 0x7f925c06b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:34148/0 (socket says 172.21.15.150:34148) 2024-03-17T23:46:56.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.051+0000 7f9261cbd640 1 -- 172.21.15.150:0/2325933208 learned_addr learned my addr 172.21.15.150:0/2325933208 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:56.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.051+0000 7f9261cbd640 1 -- 172.21.15.150:0/2325933208 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f925c06b6d0 msgr2=0x7f925c0b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:56.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.051+0000 7f9261cbd640 1 --2- 172.21.15.150:0/2325933208 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f925c06b6d0 0x7f925c0b0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:56.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.051+0000 7f9261cbd640 1 -- 172.21.15.150:0/2325933208 >> v1:172.21.15.157:6789/0 conn(0x7f925c15de90 legacy=0x7f925c0b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:56.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.051+0000 7f9261cbd640 1 -- 172.21.15.150:0/2325933208 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f925c0b5eb0 con 0x7f925c0b2090 2024-03-17T23:46:56.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.051+0000 7f9262cbf640 1 --2- 172.21.15.150:0/2325933208 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f925c06b6d0 0x7f925c0b0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:46:56.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.051+0000 7f92614bc640 1 -- 172.21.15.150:0/2325933208 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1815406133 0 0) 0x7f925c0b3aa0 con 0x7f925c15de90 2024-03-17T23:46:56.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.052+0000 7f9261cbd640 1 --2- 172.21.15.150:0/2325933208 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f925c0b2090 0x7f925c06b190 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9258009870 tx=0x7f925802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6124f862ffcc2182 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:56.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.052+0000 7f92614bc640 1 -- 172.21.15.150:0/2325933208 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9258035020 con 0x7f925c0b2090 2024-03-17T23:46:56.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.052+0000 7f92614bc640 1 -- 172.21.15.150:0/2325933208 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9258031640 con 0x7f925c0b2090 2024-03-17T23:46:56.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.052+0000 7f92614bc640 1 -- 172.21.15.150:0/2325933208 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925802d040 con 0x7f925c0b2090 2024-03-17T23:46:56.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.053+0000 7f9264722640 1 -- 172.21.15.150:0/2325933208 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f925c0b2090 msgr2=0x7f925c06b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:56.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.053+0000 7f9264722640 1 --2- 172.21.15.150:0/2325933208 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f925c0b2090 0x7f925c06b190 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9258009870 tx=0x7f925802b8f0 comp rx=0 tx=0).stop 2024-03-17T23:46:56.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.053+0000 7f9264722640 1 -- 172.21.15.150:0/2325933208 shutdown_connections 2024-03-17T23:46:56.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.053+0000 7f9264722640 1 --2- 172.21.15.150:0/2325933208 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f925c06b6d0 0x7f925c0b0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:56.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.053+0000 7f9264722640 1 --2- 172.21.15.150:0/2325933208 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f925c0b2090 0x7f925c06b190 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:56.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.053+0000 7f9264722640 1 -- 172.21.15.150:0/2325933208 >> 172.21.15.150:0/2325933208 conn(0x7f925c0b6140 msgr2=0x7f925c099a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:56.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.053+0000 7f9264722640 1 -- 172.21.15.150:0/2325933208 shutdown_connections 2024-03-17T23:46:56.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.053+0000 7f9264722640 1 -- 172.21.15.150:0/2325933208 wait complete. 2024-03-17T23:46:56.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.054+0000 7f9264722640 1 Processor -- start 2024-03-17T23:46:56.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.054+0000 7f9264722640 1 -- start start 2024-03-17T23:46:56.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.055+0000 7f9264722640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f925c06b6d0 0x7f925c147390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:56.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.055+0000 7f9264722640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f925c0b2090 0x7f925c15f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:56.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.055+0000 7f9264722640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f925c15de90 0x7f925c1f3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:56.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.055+0000 7f9264722640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f925c0b4040 con 0x7f925c06b6d0 2024-03-17T23:46:56.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.055+0000 7f9264722640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f925c0b3d60 con 0x7f925c0b2090 2024-03-17T23:46:56.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.055+0000 7f9264722640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f925c0ad970 con 0x7f925c15de90 2024-03-17T23:46:56.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.055+0000 7f92624be640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f925c06b6d0 0x7f925c147390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:56.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.055+0000 7f92624be640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f925c06b6d0 0x7f925c147390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60974/0 (socket says 172.21.15.150:60974) 2024-03-17T23:46:56.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.055+0000 7f92624be640 1 -- 172.21.15.150:0/1124758725 learned_addr learned my addr 172.21.15.150:0/1124758725 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:56.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.055+0000 7f9262cbf640 1 --2- 172.21.15.150:0/1124758725 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f925c15de90 0x7f925c1f3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:56.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.055+0000 7f9261cbd640 1 --2- 172.21.15.150:0/1124758725 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f925c0b2090 0x7f925c15f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:56.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.055+0000 7f92624be640 1 -- 172.21.15.150:0/1124758725 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f925c15de90 msgr2=0x7f925c1f3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:56.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.055+0000 7f92624be640 1 --2- 172.21.15.150:0/1124758725 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f925c15de90 0x7f925c1f3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:56.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.055+0000 7f92624be640 1 -- 172.21.15.150:0/1124758725 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f925c0b2090 msgr2=0x7f925c15f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:56.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.055+0000 7f92624be640 1 --2- 172.21.15.150:0/1124758725 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f925c0b2090 0x7f925c15f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:56.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.055+0000 7f92624be640 1 -- 172.21.15.150:0/1124758725 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f925c15fc30 con 0x7f925c06b6d0 2024-03-17T23:46:56.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.055+0000 7f92624be640 1 --2- 172.21.15.150:0/1124758725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f925c06b6d0 0x7f925c147390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f924c00c410 tx=0x7f924c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:56.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.055+0000 7f9252ffd640 1 -- 172.21.15.150:0/1124758725 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924c017070 con 0x7f925c06b6d0 2024-03-17T23:46:56.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.055+0000 7f9252ffd640 1 -- 172.21.15.150:0/1124758725 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f924c0035f0 con 0x7f925c06b6d0 2024-03-17T23:46:56.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.055+0000 7f9264722640 1 -- 172.21.15.150:0/1124758725 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f925c15ff10 con 0x7f925c06b6d0 2024-03-17T23:46:56.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.055+0000 7f9252ffd640 1 -- 172.21.15.150:0/1124758725 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924c003a20 con 0x7f925c06b6d0 2024-03-17T23:46:56.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.056+0000 7f9264722640 1 -- 172.21.15.150:0/1124758725 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f925c1604d0 con 0x7f925c06b6d0 2024-03-17T23:46:56.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.056+0000 7f9264722640 1 -- 172.21.15.150:0/1124758725 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f925c0681a0 con 0x7f925c06b6d0 2024-03-17T23:46:56.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.057+0000 7f9252ffd640 1 -- 172.21.15.150:0/1124758725 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f924c01a070 con 0x7f925c06b6d0 2024-03-17T23:46:56.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.057+0000 7f9252ffd640 1 --2- 172.21.15.150:0/1124758725 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f924006bbd0 0x7f924006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:56.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.057+0000 7f9252ffd640 1 -- 172.21.15.150:0/1124758725 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f924c089d70 con 0x7f925c06b6d0 2024-03-17T23:46:56.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.057+0000 7f9261cbd640 1 --2- 172.21.15.150:0/1124758725 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f924006bbd0 0x7f924006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:56.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.058+0000 7f9261cbd640 1 --2- 172.21.15.150:0/1124758725 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f924006bbd0 0x7f924006e090 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9258036000 tx=0x7f925803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:56.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.062+0000 7f9252ffd640 1 -- 172.21.15.150:0/1124758725 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f924c058210 con 0x7f925c06b6d0 2024-03-17T23:46:56.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.243+0000 7f9264722640 1 -- 172.21.15.150:0/1124758725 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f925c0734a0 con 0x7f925c06b6d0 2024-03-17T23:46:56.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.920+0000 7f9252ffd640 1 -- 172.21.15.150:0/1124758725 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v21) ==== 157+0+0 (secure 0 0 0) 0x7f924c05c2b0 con 0x7f925c06b6d0 2024-03-17T23:46:56.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.924+0000 7f9264722640 1 -- 172.21.15.150:0/1124758725 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f924006bbd0 msgr2=0x7f924006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:56.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.924+0000 7f9264722640 1 --2- 172.21.15.150:0/1124758725 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f924006bbd0 0x7f924006e090 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9258036000 tx=0x7f925803c040 comp rx=0 tx=0).stop 2024-03-17T23:46:56.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.924+0000 7f9264722640 1 -- 172.21.15.150:0/1124758725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f925c06b6d0 msgr2=0x7f925c147390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:56.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.924+0000 7f9264722640 1 --2- 172.21.15.150:0/1124758725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f925c06b6d0 0x7f925c147390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f924c00c410 tx=0x7f924c00c810 comp rx=0 tx=0).stop 2024-03-17T23:46:56.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.924+0000 7f9264722640 1 -- 172.21.15.150:0/1124758725 shutdown_connections 2024-03-17T23:46:56.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.924+0000 7f9264722640 1 --2- 172.21.15.150:0/1124758725 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f925c15de90 0x7f925c1f3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:56.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.924+0000 7f9264722640 1 --2- 172.21.15.150:0/1124758725 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f924006bbd0 0x7f924006e090 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:56.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.924+0000 7f9264722640 1 --2- 172.21.15.150:0/1124758725 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f925c0b2090 0x7f925c15f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:56.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.924+0000 7f9264722640 1 --2- 172.21.15.150:0/1124758725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f925c06b6d0 0x7f925c147390 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:56.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.924+0000 7f9264722640 1 -- 172.21.15.150:0/1124758725 >> 172.21.15.150:0/1124758725 conn(0x7f925c0b6140 msgr2=0x7f925c0b1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:56.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.924+0000 7f9264722640 1 -- 172.21.15.150:0/1124758725 shutdown_connections 2024-03-17T23:46:56.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:56.924+0000 7f9264722640 1 -- 172.21.15.150:0/1124758725 wait complete. 2024-03-17T23:46:56.940 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-17T23:46:57.035 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-17T23:46:57.038 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-17T23:46:57.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.054+0000 7f31fffff640 1 Processor -- start 2024-03-17T23:46:57.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.054+0000 7f31fffff640 1 -- start start 2024-03-17T23:46:57.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.055+0000 7f31fffff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f32000700e0 0x7f32000704c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:57.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.055+0000 7f31fffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f32000cd8f0 0x7f32000cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:57.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.055+0000 7f31fffff640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3200070a00 con 0x7f320006bba0 2024-03-17T23:46:57.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.055+0000 7f31fffff640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f32000ccb30 con 0x7f32000cd8f0 2024-03-17T23:46:57.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.055+0000 7f31fffff640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f32000cce10 con 0x7f32000700e0 2024-03-17T23:46:57.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.055+0000 7f31ff7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f32000cd8f0 0x7f32000cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:57.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.055+0000 7f31f7fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f320006bba0 0x7f32000cfbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39888/0 (socket says 172.21.15.150:39888) 2024-03-17T23:46:57.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.055+0000 7f31ff7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f32000cd8f0 0x7f32000cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60984/0 (socket says 172.21.15.150:60984) 2024-03-17T23:46:57.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.055+0000 7f31f7fff640 1 -- 172.21.15.150:0/1338344553 learned_addr learned my addr 172.21.15.150:0/1338344553 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:46:57.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.055+0000 7f31feffd640 1 --2- 172.21.15.150:0/1338344553 >> v2:172.21.15.157:3300/0 conn(0x7f32000700e0 0x7f32000704c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:57.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.055+0000 7f31ff7fe640 1 -- 172.21.15.150:0/1338344553 >> v2:172.21.15.157:3300/0 conn(0x7f32000700e0 msgr2=0x7f32000704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:46:57.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.055+0000 7f31ff7fe640 1 --2- 172.21.15.150:0/1338344553 >> v2:172.21.15.157:3300/0 conn(0x7f32000700e0 0x7f32000704c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:57.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.055+0000 7f31ff7fe640 1 -- 172.21.15.150:0/1338344553 >> v1:172.21.15.150:6789/0 conn(0x7f320006bba0 legacy=0x7f32000cfbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:57.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.055+0000 7f31ff7fe640 1 -- 172.21.15.150:0/1338344553 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32000cd0f0 con 0x7f32000cd8f0 2024-03-17T23:46:57.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.055+0000 7f31fe7fc640 1 -- 172.21.15.150:0/1338344553 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3179455702 0 0) 0x7f3200070a00 con 0x7f320006bba0 2024-03-17T23:46:57.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.056+0000 7f31ff7fe640 1 --2- 172.21.15.150:0/1338344553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32000cd8f0 0x7f32000cdd00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f32000b33f0 tx=0x7f31f802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2c70a46e7cec6a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:57.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.056+0000 7f31fe7fc640 1 -- 172.21.15.150:0/1338344553 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f8039070 con 0x7f32000cd8f0 2024-03-17T23:46:57.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.056+0000 7f31fe7fc640 1 -- 172.21.15.150:0/1338344553 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31f802cbf0 con 0x7f32000cd8f0 2024-03-17T23:46:57.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.056+0000 7f31fe7fc640 1 -- 172.21.15.150:0/1338344553 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f8034440 con 0x7f32000cd8f0 2024-03-17T23:46:57.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.057+0000 7f31fffff640 1 -- 172.21.15.150:0/1338344553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32000cd8f0 msgr2=0x7f32000cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:57.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.057+0000 7f31fffff640 1 --2- 172.21.15.150:0/1338344553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32000cd8f0 0x7f32000cdd00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f32000b33f0 tx=0x7f31f802be40 comp rx=0 tx=0).stop 2024-03-17T23:46:57.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.057+0000 7f31fffff640 1 -- 172.21.15.150:0/1338344553 shutdown_connections 2024-03-17T23:46:57.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.057+0000 7f31fffff640 1 --2- 172.21.15.150:0/1338344553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32000cd8f0 0x7f32000cdd00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:57.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.057+0000 7f31fffff640 1 --2- 172.21.15.150:0/1338344553 >> v2:172.21.15.157:3300/0 conn(0x7f32000700e0 0x7f32000704c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:57.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.057+0000 7f31fffff640 1 -- 172.21.15.150:0/1338344553 >> 172.21.15.150:0/1338344553 conn(0x7f32000b2330 msgr2=0x7f32000b2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:57.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.057+0000 7f31fffff640 1 -- 172.21.15.150:0/1338344553 shutdown_connections 2024-03-17T23:46:57.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.057+0000 7f31fffff640 1 -- 172.21.15.150:0/1338344553 wait complete. 2024-03-17T23:46:57.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.058+0000 7f31fffff640 1 Processor -- start 2024-03-17T23:46:57.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.058+0000 7f31fffff640 1 -- start start 2024-03-17T23:46:57.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.059+0000 7f31fffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f320006bba0 0x7f320014e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:57.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.059+0000 7f31fffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32000700e0 0x7f32000b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:57.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.059+0000 7f31fffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32000cd8f0 0x7f32000b4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:57.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.059+0000 7f31fffff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f32000cc500 con 0x7f32000700e0 2024-03-17T23:46:57.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.059+0000 7f31fffff640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f32000d04c0 con 0x7f320006bba0 2024-03-17T23:46:57.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.059+0000 7f31fffff640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f32000cc680 con 0x7f32000cd8f0 2024-03-17T23:46:57.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.059+0000 7f31f7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32000700e0 0x7f32000b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:57.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.059+0000 7f31f7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32000700e0 0x7f32000b46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60994/0 (socket says 172.21.15.150:60994) 2024-03-17T23:46:57.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.059+0000 7f31feffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f320006bba0 0x7f320014e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:57.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.059+0000 7f31f7fff640 1 -- 172.21.15.150:0/809783499 learned_addr learned my addr 172.21.15.150:0/809783499 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:57.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.059+0000 7f31ff7fe640 1 --2- 172.21.15.150:0/809783499 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32000cd8f0 0x7f32000b4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:57.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.059+0000 7f31f7fff640 1 -- 172.21.15.150:0/809783499 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32000cd8f0 msgr2=0x7f32000b4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:57.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.059+0000 7f31f7fff640 1 --2- 172.21.15.150:0/809783499 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32000cd8f0 0x7f32000b4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:57.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.059+0000 7f31f7fff640 1 -- 172.21.15.150:0/809783499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f320006bba0 msgr2=0x7f320014e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:57.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.059+0000 7f31f7fff640 1 --2- 172.21.15.150:0/809783499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f320006bba0 0x7f320014e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:57.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.059+0000 7f31f7fff640 1 -- 172.21.15.150:0/809783499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f320009c180 con 0x7f32000700e0 2024-03-17T23:46:57.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.059+0000 7f31feffd640 1 --2- 172.21.15.150:0/809783499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f320006bba0 0x7f320014e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:46:57.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.059+0000 7f31f7fff640 1 --2- 172.21.15.150:0/809783499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32000700e0 0x7f32000b46a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f31e800c430 tx=0x7f31e800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:57.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.059+0000 7f31f77fe640 1 -- 172.21.15.150:0/809783499 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e80074e0 con 0x7f32000700e0 2024-03-17T23:46:57.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.060+0000 7f31f77fe640 1 -- 172.21.15.150:0/809783499 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31e80047e0 con 0x7f32000700e0 2024-03-17T23:46:57.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.060+0000 7f31fffff640 1 -- 172.21.15.150:0/809783499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f320009ac50 con 0x7f32000700e0 2024-03-17T23:46:57.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.060+0000 7f31f77fe640 1 -- 172.21.15.150:0/809783499 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e8004bd0 con 0x7f32000700e0 2024-03-17T23:46:57.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.060+0000 7f31fffff640 1 -- 172.21.15.150:0/809783499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f320014ee60 con 0x7f32000700e0 2024-03-17T23:46:57.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.061+0000 7f31fffff640 1 -- 172.21.15.150:0/809783499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3200068ac0 con 0x7f32000700e0 2024-03-17T23:46:57.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.061+0000 7f31f77fe640 1 -- 172.21.15.150:0/809783499 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f31e8011070 con 0x7f32000700e0 2024-03-17T23:46:57.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.061+0000 7f31f77fe640 1 --2- 172.21.15.150:0/809783499 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f31e006bbd0 0x7f31e006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:57.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.062+0000 7f31f77fe640 1 -- 172.21.15.150:0/809783499 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 6007+0+0 (secure 0 0 0) 0x7f31e808a6c0 con 0x7f32000700e0 2024-03-17T23:46:57.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.062+0000 7f31feffd640 1 --2- 172.21.15.150:0/809783499 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f31e006bbd0 0x7f31e006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:57.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.062+0000 7f31feffd640 1 --2- 172.21.15.150:0/809783499 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f31e006bbd0 0x7f31e006e090 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f31f0009f30 tx=0x7f31f0009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:57.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.066+0000 7f31f77fe640 1 -- 172.21.15.150:0/809783499 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3200068ac0 con 0x7f32000700e0 2024-03-17T23:46:57.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:57.242+0000 7f31fffff640 1 -- 172.21.15.150:0/809783499 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f3200074c40 con 0x7f32000700e0 2024-03-17T23:46:58.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:58.933+0000 7f31f77fe640 1 -- 172.21.15.150:0/809783499 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v23) ==== 228+0+0 (secure 0 0 0) 0x7f320009c180 con 0x7f32000700e0 2024-03-17T23:46:58.934 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_data_ec' created 2024-03-17T23:46:58.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:58.936+0000 7f31fffff640 1 -- 172.21.15.150:0/809783499 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f31e006bbd0 msgr2=0x7f31e006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:58.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:58.936+0000 7f31fffff640 1 --2- 172.21.15.150:0/809783499 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f31e006bbd0 0x7f31e006e090 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f31f0009f30 tx=0x7f31f0009190 comp rx=0 tx=0).stop 2024-03-17T23:46:58.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:58.936+0000 7f31fffff640 1 -- 172.21.15.150:0/809783499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32000700e0 msgr2=0x7f32000b46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:58.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:58.936+0000 7f31fffff640 1 --2- 172.21.15.150:0/809783499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32000700e0 0x7f32000b46a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f31e800c430 tx=0x7f31e800c8f0 comp rx=0 tx=0).stop 2024-03-17T23:46:58.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:58.937+0000 7f31fffff640 1 -- 172.21.15.150:0/809783499 shutdown_connections 2024-03-17T23:46:58.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:58.937+0000 7f31fffff640 1 --2- 172.21.15.150:0/809783499 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32000cd8f0 0x7f32000b4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:58.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:58.937+0000 7f31fffff640 1 --2- 172.21.15.150:0/809783499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32000700e0 0x7f32000b46a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:58.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:58.937+0000 7f31fffff640 1 --2- 172.21.15.150:0/809783499 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f31e006bbd0 0x7f31e006e090 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:58.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:58.937+0000 7f31fffff640 1 --2- 172.21.15.150:0/809783499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f320006bba0 0x7f320014e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:58.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:58.937+0000 7f31fffff640 1 -- 172.21.15.150:0/809783499 >> 172.21.15.150:0/809783499 conn(0x7f32000b2330 msgr2=0x7f32000704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:58.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:58.938+0000 7f31fffff640 1 -- 172.21.15.150:0/809783499 shutdown_connections 2024-03-17T23:46:58.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:58.939+0000 7f31fffff640 1 -- 172.21.15.150:0/809783499 wait complete. 2024-03-17T23:46:58.958 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-17T23:46:59.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.064+0000 7f352c11b640 1 Processor -- start 2024-03-17T23:46:59.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.065+0000 7f352c11b640 1 -- start start 2024-03-17T23:46:59.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.065+0000 7f352c11b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35240724d0 0x7f352415de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:59.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.065+0000 7f352c11b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f352415fbe0 con 0x7f3524145450 2024-03-17T23:46:59.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.065+0000 7f352c11b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f352415fec0 con 0x7f3524072070 2024-03-17T23:46:59.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.065+0000 7f352c11b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f35241607f0 con 0x7f35240724d0 2024-03-17T23:46:59.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.066+0000 7f3529eb7640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f3524145450 0x7f352415cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39902/0 (socket says 172.21.15.150:39902) 2024-03-17T23:46:59.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.066+0000 7f3529eb7640 1 -- 172.21.15.150:0/2708533166 learned_addr learned my addr 172.21.15.150:0/2708533166 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:46:59.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.066+0000 7f352a6b8640 1 --2- 172.21.15.150:0/2708533166 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35240724d0 0x7f352415de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:59.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.066+0000 7f3528eb5640 1 -- 172.21.15.150:0/2708533166 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 814071159 0 0) 0x7f352415fbe0 con 0x7f3524145450 2024-03-17T23:46:59.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.066+0000 7f3528eb5640 1 -- 172.21.15.150:0/2708533166 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3510003520 con 0x7f3524145450 2024-03-17T23:46:59.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.066+0000 7f3528eb5640 1 -- 172.21.15.150:0/2708533166 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2392881237 0 0) 0x7f352415fec0 con 0x7f3524072070 2024-03-17T23:46:59.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.066+0000 7f3528eb5640 1 -- 172.21.15.150:0/2708533166 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f352415fbe0 con 0x7f3524072070 2024-03-17T23:46:59.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.066+0000 7f352a6b8640 1 -- 172.21.15.150:0/2708533166 >> v1:172.21.15.157:6789/0 conn(0x7f3524072070 legacy=0x7f352415d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:59.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.066+0000 7f352a6b8640 1 -- 172.21.15.150:0/2708533166 >> v1:172.21.15.150:6789/0 conn(0x7f3524145450 legacy=0x7f352415cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:59.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.066+0000 7f352a6b8640 1 -- 172.21.15.150:0/2708533166 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35241601a0 con 0x7f35240724d0 2024-03-17T23:46:59.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.067+0000 7f352a6b8640 1 --2- 172.21.15.150:0/2708533166 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35240724d0 0x7f352415de20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f351c009dc0 tx=0x7f351c02bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bf9e57356a8c903 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:59.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.067+0000 7f3528eb5640 1 -- 172.21.15.150:0/2708533166 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351c039070 con 0x7f35240724d0 2024-03-17T23:46:59.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.067+0000 7f3528eb5640 1 -- 172.21.15.150:0/2708533166 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f351c02ccd0 con 0x7f35240724d0 2024-03-17T23:46:59.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.067+0000 7f3528eb5640 1 -- 172.21.15.150:0/2708533166 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351c034640 con 0x7f35240724d0 2024-03-17T23:46:59.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.067+0000 7f352c11b640 1 -- 172.21.15.150:0/2708533166 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35240724d0 msgr2=0x7f352415de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:59.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.067+0000 7f352c11b640 1 --2- 172.21.15.150:0/2708533166 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35240724d0 0x7f352415de20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f351c009dc0 tx=0x7f351c02bf20 comp rx=0 tx=0).stop 2024-03-17T23:46:59.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.067+0000 7f352c11b640 1 -- 172.21.15.150:0/2708533166 shutdown_connections 2024-03-17T23:46:59.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.067+0000 7f352c11b640 1 --2- 172.21.15.150:0/2708533166 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35240724d0 0x7f352415de20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:59.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.067+0000 7f352c11b640 1 -- 172.21.15.150:0/2708533166 >> 172.21.15.150:0/2708533166 conn(0x7f35240b1be0 msgr2=0x7f35240b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:59.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.068+0000 7f352c11b640 1 -- 172.21.15.150:0/2708533166 shutdown_connections 2024-03-17T23:46:59.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.068+0000 7f352c11b640 1 -- 172.21.15.150:0/2708533166 wait complete. 2024-03-17T23:46:59.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.069+0000 7f352c11b640 1 Processor -- start 2024-03-17T23:46:59.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.069+0000 7f352c11b640 1 -- start start 2024-03-17T23:46:59.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.069+0000 7f352c11b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3524072070 0x7f3524143580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:59.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.069+0000 7f352c11b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35240724d0 0x7f35240b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:59.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.069+0000 7f352c11b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3524145450 0x7f3524158f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:59.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.069+0000 7f352c11b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3524145830 con 0x7f35240724d0 2024-03-17T23:46:59.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.069+0000 7f352c11b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3524072960 con 0x7f3524072070 2024-03-17T23:46:59.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.069+0000 7f352c11b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f352406f810 con 0x7f3524145450 2024-03-17T23:46:59.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.069+0000 7f35296b6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35240724d0 0x7f35240b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:59.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.069+0000 7f35296b6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35240724d0 0x7f35240b35d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32776/0 (socket says 172.21.15.150:32776) 2024-03-17T23:46:59.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.069+0000 7f35296b6640 1 -- 172.21.15.150:0/2064556368 learned_addr learned my addr 172.21.15.150:0/2064556368 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:46:59.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.069+0000 7f352a6b8640 1 --2- 172.21.15.150:0/2064556368 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3524145450 0x7f3524158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:59.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.069+0000 7f3529eb7640 1 --2- 172.21.15.150:0/2064556368 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3524072070 0x7f3524143580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:59.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.070+0000 7f35296b6640 1 -- 172.21.15.150:0/2064556368 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3524145450 msgr2=0x7f3524158f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:59.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.070+0000 7f35296b6640 1 --2- 172.21.15.150:0/2064556368 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3524145450 0x7f3524158f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:59.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.070+0000 7f35296b6640 1 -- 172.21.15.150:0/2064556368 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3524072070 msgr2=0x7f3524143580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:59.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.070+0000 7f35296b6640 1 --2- 172.21.15.150:0/2064556368 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3524072070 0x7f3524143580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:59.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.070+0000 7f35296b6640 1 -- 172.21.15.150:0/2064556368 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35240b4070 con 0x7f35240724d0 2024-03-17T23:46:59.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.070+0000 7f352a6b8640 1 --2- 172.21.15.150:0/2064556368 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3524145450 0x7f3524158f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:46:59.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.070+0000 7f35296b6640 1 --2- 172.21.15.150:0/2064556368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35240724d0 0x7f35240b35d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f351400c430 tx=0x7f351400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:59.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.070+0000 7f351a7fc640 1 -- 172.21.15.150:0/2064556368 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3514017070 con 0x7f35240724d0 2024-03-17T23:46:59.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.070+0000 7f351a7fc640 1 -- 172.21.15.150:0/2064556368 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35140052f0 con 0x7f35240724d0 2024-03-17T23:46:59.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.070+0000 7f352c11b640 1 -- 172.21.15.150:0/2064556368 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35240b4350 con 0x7f35240724d0 2024-03-17T23:46:59.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.070+0000 7f351a7fc640 1 -- 172.21.15.150:0/2064556368 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3514003830 con 0x7f35240724d0 2024-03-17T23:46:59.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.070+0000 7f352c11b640 1 -- 172.21.15.150:0/2064556368 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35240b4910 con 0x7f35240724d0 2024-03-17T23:46:59.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.071+0000 7f352c11b640 1 -- 172.21.15.150:0/2064556368 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35240681a0 con 0x7f35240724d0 2024-03-17T23:46:59.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.072+0000 7f351a7fc640 1 -- 172.21.15.150:0/2064556368 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f351401a070 con 0x7f35240724d0 2024-03-17T23:46:59.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.072+0000 7f351a7fc640 1 --2- 172.21.15.150:0/2064556368 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f350806bbd0 0x7f350806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:46:59.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.072+0000 7f351a7fc640 1 -- 172.21.15.150:0/2064556368 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 6500+0+0 (secure 0 0 0) 0x7f351408a120 con 0x7f35240724d0 2024-03-17T23:46:59.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.072+0000 7f3529eb7640 1 --2- 172.21.15.150:0/2064556368 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f350806bbd0 0x7f350806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:46:59.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.073+0000 7f3529eb7640 1 --2- 172.21.15.150:0/2064556368 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f350806bbd0 0x7f350806e090 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f35200011e0 tx=0x7f3520007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:46:59.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.077+0000 7f351a7fc640 1 -- 172.21.15.150:0/2064556368 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3514058200 con 0x7f35240724d0 2024-03-17T23:46:59.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.251+0000 7f352c11b640 1 -- 172.21.15.150:0/2064556368 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f352413a470 con 0x7f35240724d0 2024-03-17T23:46:59.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.937+0000 7f351a7fc640 1 -- 172.21.15.150:0/2064556368 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v24) ==== 169+0+0 (secure 0 0 0) 0x7f351405c2a0 con 0x7f35240724d0 2024-03-17T23:46:59.939 INFO:teuthology.orchestra.run.smithi150.stderr:set pool 4 allow_ec_overwrites to true 2024-03-17T23:46:59.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.940+0000 7f352c11b640 1 -- 172.21.15.150:0/2064556368 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f350806bbd0 msgr2=0x7f350806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:59.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.940+0000 7f352c11b640 1 --2- 172.21.15.150:0/2064556368 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f350806bbd0 0x7f350806e090 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f35200011e0 tx=0x7f3520007d30 comp rx=0 tx=0).stop 2024-03-17T23:46:59.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.940+0000 7f352c11b640 1 -- 172.21.15.150:0/2064556368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35240724d0 msgr2=0x7f35240b35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:46:59.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.940+0000 7f352c11b640 1 --2- 172.21.15.150:0/2064556368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35240724d0 0x7f35240b35d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f351400c430 tx=0x7f351400c8f0 comp rx=0 tx=0).stop 2024-03-17T23:46:59.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.941+0000 7f352c11b640 1 -- 172.21.15.150:0/2064556368 shutdown_connections 2024-03-17T23:46:59.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.941+0000 7f352c11b640 1 --2- 172.21.15.150:0/2064556368 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3524145450 0x7f3524158f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:59.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.941+0000 7f352c11b640 1 --2- 172.21.15.150:0/2064556368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35240724d0 0x7f35240b35d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:59.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.941+0000 7f352c11b640 1 --2- 172.21.15.150:0/2064556368 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f350806bbd0 0x7f350806e090 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:59.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.941+0000 7f352c11b640 1 --2- 172.21.15.150:0/2064556368 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3524072070 0x7f3524143580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:46:59.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.941+0000 7f352c11b640 1 -- 172.21.15.150:0/2064556368 >> 172.21.15.150:0/2064556368 conn(0x7f35240b1be0 msgr2=0x7f352415ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:46:59.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.941+0000 7f352c11b640 1 -- 172.21.15.150:0/2064556368 shutdown_connections 2024-03-17T23:46:59.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:46:59.941+0000 7f352c11b640 1 -- 172.21.15.150:0/2064556368 wait complete. 2024-03-17T23:46:59.967 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-17T23:47:00.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.068+0000 7f36a53df640 1 Processor -- start 2024-03-17T23:47:00.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.068+0000 7f36a53df640 1 -- start start 2024-03-17T23:47:00.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.069+0000 7f36a53df640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f36a0161520 0x7f36a00b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:00.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.069+0000 7f36a53df640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f36a00b2bf0 0x7f36a00d05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:00.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.069+0000 7f36a53df640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36a0161d30 con 0x7f36a00d0b10 2024-03-17T23:47:00.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.069+0000 7f36a53df640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f36a00b9bf0 con 0x7f36a00b2bf0 2024-03-17T23:47:00.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.069+0000 7f36a53df640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f36a0162940 con 0x7f36a0161520 2024-03-17T23:47:00.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.069+0000 7f369f7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f36a00d0b10 0x7f36a00b57a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39906/0 (socket says 172.21.15.150:39906) 2024-03-17T23:47:00.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.069+0000 7f369f7fe640 1 -- 172.21.15.150:0/171405079 learned_addr learned my addr 172.21.15.150:0/171405079 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:00.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.069+0000 7f369e7fc640 1 --2- 172.21.15.150:0/171405079 >> v2:172.21.15.157:3300/0 conn(0x7f36a00b2bf0 0x7f36a00d05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:00.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.069+0000 7f369effd640 1 --2- 172.21.15.150:0/171405079 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f36a0161520 0x7f36a00b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:00.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.069+0000 7f369dffb640 1 -- 172.21.15.150:0/171405079 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2928856825 0 0) 0x7f36a0161d30 con 0x7f36a00d0b10 2024-03-17T23:47:00.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.069+0000 7f369dffb640 1 -- 172.21.15.150:0/171405079 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3688003520 con 0x7f36a00d0b10 2024-03-17T23:47:00.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.069+0000 7f369e7fc640 1 -- 172.21.15.150:0/171405079 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f36a0161520 msgr2=0x7f36a00b96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:00.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.069+0000 7f369e7fc640 1 --2- 172.21.15.150:0/171405079 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f36a0161520 0x7f36a00b96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:00.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.069+0000 7f369e7fc640 1 -- 172.21.15.150:0/171405079 >> v1:172.21.15.150:6789/0 conn(0x7f36a00d0b10 legacy=0x7f36a00b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:00.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.069+0000 7f369e7fc640 1 -- 172.21.15.150:0/171405079 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36a0162010 con 0x7f36a00b2bf0 2024-03-17T23:47:00.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.070+0000 7f369e7fc640 1 --2- 172.21.15.150:0/171405079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36a00b2bf0 0x7f36a00d05d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f36940097a0 tx=0x7f369402b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a8934be61a81e08 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:00.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.070+0000 7f369dffb640 1 -- 172.21.15.150:0/171405079 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3694038070 con 0x7f36a00b2bf0 2024-03-17T23:47:00.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.070+0000 7f369dffb640 1 -- 172.21.15.150:0/171405079 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3694031700 con 0x7f36a00b2bf0 2024-03-17T23:47:00.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.070+0000 7f369dffb640 1 -- 172.21.15.150:0/171405079 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3694031ab0 con 0x7f36a00b2bf0 2024-03-17T23:47:00.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.071+0000 7f36a53df640 1 -- 172.21.15.150:0/171405079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36a00b2bf0 msgr2=0x7f36a00d05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:00.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.071+0000 7f36a53df640 1 --2- 172.21.15.150:0/171405079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36a00b2bf0 0x7f36a00d05d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f36940097a0 tx=0x7f369402b7a0 comp rx=0 tx=0).stop 2024-03-17T23:47:00.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.071+0000 7f36a53df640 1 -- 172.21.15.150:0/171405079 shutdown_connections 2024-03-17T23:47:00.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.071+0000 7f36a53df640 1 --2- 172.21.15.150:0/171405079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36a00b2bf0 0x7f36a00d05d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:00.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.071+0000 7f36a53df640 1 --2- 172.21.15.150:0/171405079 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f36a0161520 0x7f36a00b96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:00.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.071+0000 7f36a53df640 1 -- 172.21.15.150:0/171405079 >> 172.21.15.150:0/171405079 conn(0x7f36a006ca00 msgr2=0x7f36a006ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:00.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.071+0000 7f36a53df640 1 -- 172.21.15.150:0/171405079 shutdown_connections 2024-03-17T23:47:00.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.071+0000 7f36a53df640 1 -- 172.21.15.150:0/171405079 wait complete. 2024-03-17T23:47:00.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.072+0000 7f36a53df640 1 Processor -- start 2024-03-17T23:47:00.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.072+0000 7f36a53df640 1 -- start start 2024-03-17T23:47:00.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.073+0000 7f36a53df640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f36a00b2bf0 0x7f36a0151b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:00.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.073+0000 7f36a53df640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36a00d0b10 0x7f36a015a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:00.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.073+0000 7f36a53df640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f36a0161520 0x7f36a015ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:00.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.073+0000 7f36a53df640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f36a01625d0 con 0x7f36a00b2bf0 2024-03-17T23:47:00.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.073+0000 7f36a53df640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f36a00d0f60 con 0x7f36a00d0b10 2024-03-17T23:47:00.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.073+0000 7f36a53df640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f36a0161900 con 0x7f36a0161520 2024-03-17T23:47:00.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.073+0000 7f369effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f36a00b2bf0 0x7f36a0151b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:00.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.073+0000 7f369effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f36a00b2bf0 0x7f36a0151b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32778/0 (socket says 172.21.15.150:32778) 2024-03-17T23:47:00.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.073+0000 7f369effd640 1 -- 172.21.15.150:0/2979863137 learned_addr learned my addr 172.21.15.150:0/2979863137 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:00.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.073+0000 7f369f7fe640 1 --2- 172.21.15.150:0/2979863137 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f36a0161520 0x7f36a015ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:00.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.073+0000 7f369e7fc640 1 --2- 172.21.15.150:0/2979863137 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36a00d0b10 0x7f36a015a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:00.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.073+0000 7f369effd640 1 -- 172.21.15.150:0/2979863137 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f36a0161520 msgr2=0x7f36a015ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:00.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.073+0000 7f369effd640 1 --2- 172.21.15.150:0/2979863137 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f36a0161520 0x7f36a015ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:00.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.073+0000 7f369effd640 1 -- 172.21.15.150:0/2979863137 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36a00d0b10 msgr2=0x7f36a015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:00.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.073+0000 7f369effd640 1 --2- 172.21.15.150:0/2979863137 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36a00d0b10 0x7f36a015a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:00.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.073+0000 7f369effd640 1 -- 172.21.15.150:0/2979863137 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36a00cdd30 con 0x7f36a00b2bf0 2024-03-17T23:47:00.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.073+0000 7f369e7fc640 1 --2- 172.21.15.150:0/2979863137 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36a00d0b10 0x7f36a015a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:00.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.073+0000 7f369f7fe640 1 --2- 172.21.15.150:0/2979863137 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f36a0161520 0x7f36a015ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:00.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.073+0000 7f369effd640 1 --2- 172.21.15.150:0/2979863137 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f36a00b2bf0 0x7f36a0151b20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f369000e9e0 tx=0x7f369000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:00.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.074+0000 7f36877fe640 1 -- 172.21.15.150:0/2979863137 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3690017070 con 0x7f36a00b2bf0 2024-03-17T23:47:00.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.074+0000 7f36877fe640 1 -- 172.21.15.150:0/2979863137 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f369000c9d0 con 0x7f36a00b2bf0 2024-03-17T23:47:00.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.074+0000 7f36a53df640 1 -- 172.21.15.150:0/2979863137 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36a00cbf70 con 0x7f36a00b2bf0 2024-03-17T23:47:00.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.074+0000 7f36877fe640 1 -- 172.21.15.150:0/2979863137 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f369000cdc0 con 0x7f36a00b2bf0 2024-03-17T23:47:00.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.074+0000 7f36a53df640 1 -- 172.21.15.150:0/2979863137 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36a00cc4f0 con 0x7f36a00b2bf0 2024-03-17T23:47:00.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.075+0000 7f36a53df640 1 -- 172.21.15.150:0/2979863137 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36a00681a0 con 0x7f36a00b2bf0 2024-03-17T23:47:00.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.075+0000 7f36877fe640 1 -- 172.21.15.150:0/2979863137 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f369001a070 con 0x7f36a00b2bf0 2024-03-17T23:47:00.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.075+0000 7f36877fe640 1 --2- 172.21.15.150:0/2979863137 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f368c06bc00 0x7f368c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:00.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.075+0000 7f36877fe640 1 -- 172.21.15.150:0/2979863137 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6500+0+0 (secure 0 0 0) 0x7f369008ad40 con 0x7f36a00b2bf0 2024-03-17T23:47:00.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.076+0000 7f369e7fc640 1 --2- 172.21.15.150:0/2979863137 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f368c06bc00 0x7f368c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:00.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.076+0000 7f369e7fc640 1 --2- 172.21.15.150:0/2979863137 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f368c06bc00 0x7f368c06e0c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3694035000 tx=0x7f3694008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:00.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.080+0000 7f36877fe640 1 -- 172.21.15.150:0/2979863137 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3690058eb0 con 0x7f36a00b2bf0 2024-03-17T23:47:00.203 INFO:tasks.ceph.mds.b.smithi157.stderr:2024-03-17T23:47:00.203+0000 7f61d3871640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-17T23:47:00.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.290+0000 7f36a53df640 1 -- 172.21.15.150:0/2979863137 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f36a00c75b0 con 0x7f36a00b2bf0 2024-03-17T23:47:00.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.942+0000 7f36877fe640 1 -- 172.21.15.150:0/2979863137 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v6) ==== 137+0+0 (secure 0 0 0) 0x7f369005cf50 con 0x7f36a00b2bf0 2024-03-17T23:47:00.944 INFO:teuthology.orchestra.run.smithi150.stderr:added data pool 4 to fsmap 2024-03-17T23:47:00.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.945+0000 7f36a53df640 1 -- 172.21.15.150:0/2979863137 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f368c06bc00 msgr2=0x7f368c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:00.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.945+0000 7f36a53df640 1 --2- 172.21.15.150:0/2979863137 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f368c06bc00 0x7f368c06e0c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3694035000 tx=0x7f3694008030 comp rx=0 tx=0).stop 2024-03-17T23:47:00.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.945+0000 7f36a53df640 1 -- 172.21.15.150:0/2979863137 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f36a00b2bf0 msgr2=0x7f36a0151b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:00.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.945+0000 7f36a53df640 1 --2- 172.21.15.150:0/2979863137 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f36a00b2bf0 0x7f36a0151b20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f369000e9e0 tx=0x7f369000eea0 comp rx=0 tx=0).stop 2024-03-17T23:47:00.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.945+0000 7f36a53df640 1 -- 172.21.15.150:0/2979863137 shutdown_connections 2024-03-17T23:47:00.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.945+0000 7f36a53df640 1 --2- 172.21.15.150:0/2979863137 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f36a0161520 0x7f36a015ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:00.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.945+0000 7f36a53df640 1 --2- 172.21.15.150:0/2979863137 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f368c06bc00 0x7f368c06e0c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:00.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.945+0000 7f36a53df640 1 --2- 172.21.15.150:0/2979863137 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36a00d0b10 0x7f36a015a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:00.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.945+0000 7f36a53df640 1 --2- 172.21.15.150:0/2979863137 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f36a00b2bf0 0x7f36a0151b20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:00.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.945+0000 7f36a53df640 1 -- 172.21.15.150:0/2979863137 >> 172.21.15.150:0/2979863137 conn(0x7f36a006ca00 msgr2=0x7f36a00b83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:00.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.945+0000 7f36a53df640 1 -- 172.21.15.150:0/2979863137 shutdown_connections 2024-03-17T23:47:00.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:00.945+0000 7f36a53df640 1 -- 172.21.15.150:0/2979863137 wait complete. 2024-03-17T23:47:00.958 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:47:01.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.060+0000 7f8d94081640 1 Processor -- start 2024-03-17T23:47:01.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.060+0000 7f8d94081640 1 -- start start 2024-03-17T23:47:01.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.061+0000 7f8d94081640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8d8c0c8210 0x7f8d8c0cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:01.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.061+0000 7f8d94081640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d8c09b260 con 0x7f8d8c0ccb80 2024-03-17T23:47:01.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.061+0000 7f8d94081640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d8c09af80 con 0x7f8d8c15be40 2024-03-17T23:47:01.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.061+0000 7f8d94081640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f8d8c0c5d70 con 0x7f8d8c0c8210 2024-03-17T23:47:01.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.061+0000 7f8d9261e640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f8d8c0ccb80 0x7f8d8c0cd010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39908/0 (socket says 172.21.15.150:39908) 2024-03-17T23:47:01.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.061+0000 7f8d9261e640 1 -- 172.21.15.150:0/985168683 learned_addr learned my addr 172.21.15.150:0/985168683 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:01.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.061+0000 7f8d9161c640 1 --2- 172.21.15.150:0/985168683 >> v2:172.21.15.157:3300/0 conn(0x7f8d8c0c8210 0x7f8d8c0cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:01.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.062+0000 7f8d90e1b640 1 -- 172.21.15.150:0/985168683 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2508235758 0 0) 0x7f8d8c09b260 con 0x7f8d8c0ccb80 2024-03-17T23:47:01.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.062+0000 7f8d90e1b640 1 -- 172.21.15.150:0/985168683 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d78003520 con 0x7f8d8c0ccb80 2024-03-17T23:47:01.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.062+0000 7f8d9161c640 1 -- 172.21.15.150:0/985168683 >> v1:172.21.15.157:6789/0 conn(0x7f8d8c15be40 legacy=0x7f8d8c09a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:01.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.062+0000 7f8d9161c640 1 -- 172.21.15.150:0/985168683 >> v1:172.21.15.150:6789/0 conn(0x7f8d8c0ccb80 legacy=0x7f8d8c0cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:01.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.062+0000 7f8d9161c640 1 -- 172.21.15.150:0/985168683 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d8c0c5720 con 0x7f8d8c0c8210 2024-03-17T23:47:01.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.062+0000 7f8d9161c640 1 --2- 172.21.15.150:0/985168683 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d8c0c8210 0x7f8d8c0cc640 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c0097a0 tx=0x7f8d7c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51271b1c782ec594 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:01.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.063+0000 7f8d90e1b640 1 -- 172.21.15.150:0/985168683 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c034070 con 0x7f8d8c0c8210 2024-03-17T23:47:01.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.063+0000 7f8d90e1b640 1 -- 172.21.15.150:0/985168683 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d7c030610 con 0x7f8d8c0c8210 2024-03-17T23:47:01.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.063+0000 7f8d90e1b640 1 -- 172.21.15.150:0/985168683 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c0309e0 con 0x7f8d8c0c8210 2024-03-17T23:47:01.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.063+0000 7f8d94081640 1 -- 172.21.15.150:0/985168683 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d8c0c8210 msgr2=0x7f8d8c0cc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:01.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.063+0000 7f8d94081640 1 --2- 172.21.15.150:0/985168683 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d8c0c8210 0x7f8d8c0cc640 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c0097a0 tx=0x7f8d7c02b960 comp rx=0 tx=0).stop 2024-03-17T23:47:01.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.063+0000 7f8d94081640 1 -- 172.21.15.150:0/985168683 shutdown_connections 2024-03-17T23:47:01.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.063+0000 7f8d94081640 1 --2- 172.21.15.150:0/985168683 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d8c0c8210 0x7f8d8c0cc640 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:01.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.063+0000 7f8d94081640 1 -- 172.21.15.150:0/985168683 >> 172.21.15.150:0/985168683 conn(0x7f8d8c0b3620 msgr2=0x7f8d8c0b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:01.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.063+0000 7f8d94081640 1 -- 172.21.15.150:0/985168683 shutdown_connections 2024-03-17T23:47:01.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.063+0000 7f8d94081640 1 -- 172.21.15.150:0/985168683 wait complete. 2024-03-17T23:47:01.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.064+0000 7f8d94081640 1 Processor -- start 2024-03-17T23:47:01.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.064+0000 7f8d94081640 1 -- start start 2024-03-17T23:47:01.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.065+0000 7f8d94081640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d8c0c8210 0x7f8d8c15efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:01.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.065+0000 7f8d94081640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d8c0ccb80 0x7f8d8c149360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:01.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.065+0000 7f8d94081640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d8c15be40 0x7f8d8c0caf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:01.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.065+0000 7f8d94081640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c1669f0 con 0x7f8d8c15be40 2024-03-17T23:47:01.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.065+0000 7f8d94081640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c15c220 con 0x7f8d8c0ccb80 2024-03-17T23:47:01.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.065+0000 7f8d94081640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8d8c166b70 con 0x7f8d8c0c8210 2024-03-17T23:47:01.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.065+0000 7f8d9261e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d8c15be40 0x7f8d8c0caf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:01.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.065+0000 7f8d9261e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d8c15be40 0x7f8d8c0caf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32788/0 (socket says 172.21.15.150:32788) 2024-03-17T23:47:01.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.065+0000 7f8d9161c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d8c0ccb80 0x7f8d8c149360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:01.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.065+0000 7f8d9261e640 1 -- 172.21.15.150:0/2913372843 learned_addr learned my addr 172.21.15.150:0/2913372843 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:01.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.065+0000 7f8d91e1d640 1 --2- 172.21.15.150:0/2913372843 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d8c0c8210 0x7f8d8c15efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:01.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.065+0000 7f8d9261e640 1 -- 172.21.15.150:0/2913372843 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d8c0c8210 msgr2=0x7f8d8c15efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:01.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.065+0000 7f8d9261e640 1 --2- 172.21.15.150:0/2913372843 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d8c0c8210 0x7f8d8c15efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:01.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.065+0000 7f8d9261e640 1 -- 172.21.15.150:0/2913372843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d8c0ccb80 msgr2=0x7f8d8c149360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:01.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.065+0000 7f8d9261e640 1 --2- 172.21.15.150:0/2913372843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d8c0ccb80 0x7f8d8c149360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:01.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.065+0000 7f8d9261e640 1 -- 172.21.15.150:0/2913372843 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d8c1498a0 con 0x7f8d8c15be40 2024-03-17T23:47:01.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.066+0000 7f8d9261e640 1 --2- 172.21.15.150:0/2913372843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d8c15be40 0x7f8d8c0caf70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8d8400c400 tx=0x7f8d8400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:01.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.066+0000 7f8d827fc640 1 -- 172.21.15.150:0/2913372843 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d84017070 con 0x7f8d8c15be40 2024-03-17T23:47:01.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.066+0000 7f8d827fc640 1 -- 172.21.15.150:0/2913372843 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d84005ce0 con 0x7f8d8c15be40 2024-03-17T23:47:01.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.066+0000 7f8d827fc640 1 -- 172.21.15.150:0/2913372843 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d840072d0 con 0x7f8d8c15be40 2024-03-17T23:47:01.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.066+0000 7f8d94081640 1 -- 172.21.15.150:0/2913372843 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d8c14ac00 con 0x7f8d8c15be40 2024-03-17T23:47:01.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.066+0000 7f8d94081640 1 -- 172.21.15.150:0/2913372843 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d8c14b1c0 con 0x7f8d8c15be40 2024-03-17T23:47:01.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.067+0000 7f8d94081640 1 -- 172.21.15.150:0/2913372843 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d8c0688b0 con 0x7f8d8c15be40 2024-03-17T23:47:01.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.067+0000 7f8d827fc640 1 -- 172.21.15.150:0/2913372843 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8d8401a070 con 0x7f8d8c15be40 2024-03-17T23:47:01.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.068+0000 7f8d827fc640 1 --2- 172.21.15.150:0/2913372843 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8d7006bbd0 0x7f8d7006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:01.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.068+0000 7f8d827fc640 1 -- 172.21.15.150:0/2913372843 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f8d8408a090 con 0x7f8d8c15be40 2024-03-17T23:47:01.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.068+0000 7f8d91e1d640 1 --2- 172.21.15.150:0/2913372843 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8d7006bbd0 0x7f8d7006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:01.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.068+0000 7f8d91e1d640 1 --2- 172.21.15.150:0/2913372843 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8d7006bbd0 0x7f8d7006e090 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c15e560 tx=0x7f8d88009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:01.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.072+0000 7f8d827fc640 1 -- 172.21.15.150:0/2913372843 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8d840581f0 con 0x7f8d8c15be40 2024-03-17T23:47:01.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.284+0000 7f8d94081640 1 -- 172.21.15.150:0/2913372843 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d8c0c6100 con 0x7f8d8c15be40 2024-03-17T23:47:01.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.284+0000 7f8d827fc640 1 -- 172.21.15.150:0/2913372843 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5186 (secure 0 0 0) 0x7f8d8405e070 con 0x7f8d8c15be40 2024-03-17T23:47:01.286 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:01.286 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/1949609985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6833","nonce":1949609985},{"type":"v1","addr":"172.21.15.157:6835","nonce":1949609985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/758264728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":758264728},{"type":"v1","addr":"172.21.15.150:6835","nonce":758264728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3655589010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3655589010},{"type":"v1","addr":"172.21.15.150:6837","nonce":3655589010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-17T23:46:55.253739+0000","modified":"2024-03-17T23:47:00.292677+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6834/1625058792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1625058792},{"type":"v1","addr":"172.21.15.157:6834","nonce":1625058792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4297,"qdb_cluster":[4297]},"id":1}]} 2024-03-17T23:47:01.286 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 6 2024-03-17T23:47:01.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.286+0000 7f8d94081640 1 -- 172.21.15.150:0/2913372843 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8d7006bbd0 msgr2=0x7f8d7006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:01.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.286+0000 7f8d94081640 1 --2- 172.21.15.150:0/2913372843 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8d7006bbd0 0x7f8d7006e090 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c15e560 tx=0x7f8d88009040 comp rx=0 tx=0).stop 2024-03-17T23:47:01.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.286+0000 7f8d94081640 1 -- 172.21.15.150:0/2913372843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d8c15be40 msgr2=0x7f8d8c0caf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:01.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.286+0000 7f8d94081640 1 --2- 172.21.15.150:0/2913372843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d8c15be40 0x7f8d8c0caf70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8d8400c400 tx=0x7f8d8400c8c0 comp rx=0 tx=0).stop 2024-03-17T23:47:01.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.286+0000 7f8d94081640 1 -- 172.21.15.150:0/2913372843 shutdown_connections 2024-03-17T23:47:01.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.286+0000 7f8d94081640 1 --2- 172.21.15.150:0/2913372843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d8c15be40 0x7f8d8c0caf70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:01.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.286+0000 7f8d94081640 1 --2- 172.21.15.150:0/2913372843 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8d7006bbd0 0x7f8d7006e090 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:01.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.286+0000 7f8d94081640 1 --2- 172.21.15.150:0/2913372843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d8c0ccb80 0x7f8d8c149360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:01.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.286+0000 7f8d94081640 1 --2- 172.21.15.150:0/2913372843 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d8c0c8210 0x7f8d8c15efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:01.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.286+0000 7f8d94081640 1 -- 172.21.15.150:0/2913372843 >> 172.21.15.150:0/2913372843 conn(0x7f8d8c0b3620 msgr2=0x7f8d8c0c64d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:01.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.286+0000 7f8d94081640 1 -- 172.21.15.150:0/2913372843 shutdown_connections 2024-03-17T23:47:01.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.286+0000 7f8d94081640 1 -- 172.21.15.150:0/2913372843 wait complete. 2024-03-17T23:47:01.299 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:01.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.397+0000 7f8416234640 1 Processor -- start 2024-03-17T23:47:01.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.397+0000 7f8416234640 1 -- start start 2024-03-17T23:47:01.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.397+0000 7f8416234640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f84100b5730 0x7f84100b2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:01.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.397+0000 7f8416234640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f84100b3c30 0x7f84100b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:01.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.397+0000 7f8416234640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84100d1230 con 0x7f841006b190 2024-03-17T23:47:01.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.397+0000 7f8416234640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f84100b2560 con 0x7f84100b3c30 2024-03-17T23:47:01.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.397+0000 7f8416234640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f84100b5c10 con 0x7f84100b5730 2024-03-17T23:47:01.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.398+0000 7f840ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f84100b3c30 0x7f84100b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:01.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.398+0000 7f840ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f84100b3c30 0x7f84100b01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32794/0 (socket says 172.21.15.150:32794) 2024-03-17T23:47:01.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.398+0000 7f840ffff640 1 -- 172.21.15.150:0/1694655668 learned_addr learned my addr 172.21.15.150:0/1694655668 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:01.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.398+0000 7f8406ffd640 1 --2- 172.21.15.150:0/1694655668 >> v2:172.21.15.157:3300/0 conn(0x7f84100b5730 0x7f84100b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:01.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.398+0000 7f840ffff640 1 -- 172.21.15.150:0/1694655668 >> v2:172.21.15.157:3300/0 conn(0x7f84100b5730 msgr2=0x7f84100b2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:01.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.398+0000 7f840ffff640 1 --2- 172.21.15.150:0/1694655668 >> v2:172.21.15.157:3300/0 conn(0x7f84100b5730 0x7f84100b2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:01.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.398+0000 7f840ffff640 1 -- 172.21.15.150:0/1694655668 >> v1:172.21.15.157:6789/0 conn(0x7f841006b190 legacy=0x7f841006b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:01.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.398+0000 7f840ffff640 1 -- 172.21.15.150:0/1694655668 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84100d14d0 con 0x7f84100b3c30 2024-03-17T23:47:01.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.398+0000 7f840ffff640 1 --2- 172.21.15.150:0/1694655668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84100b3c30 0x7f84100b01a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f84100d1b60 tx=0x7f840802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=339a6055e9ec9317 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:01.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.399+0000 7f840effd640 1 -- 172.21.15.150:0/1694655668 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f840802ca80 con 0x7f84100b3c30 2024-03-17T23:47:01.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.399+0000 7f840effd640 1 -- 172.21.15.150:0/1694655668 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f840802cc20 con 0x7f84100b3c30 2024-03-17T23:47:01.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.399+0000 7f840effd640 1 -- 172.21.15.150:0/1694655668 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84080343f0 con 0x7f84100b3c30 2024-03-17T23:47:01.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.399+0000 7f8416234640 1 -- 172.21.15.150:0/1694655668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84100b3c30 msgr2=0x7f84100b01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:01.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.399+0000 7f8416234640 1 --2- 172.21.15.150:0/1694655668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84100b3c30 0x7f84100b01a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f84100d1b60 tx=0x7f840802bee0 comp rx=0 tx=0).stop 2024-03-17T23:47:01.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.399+0000 7f8416234640 1 -- 172.21.15.150:0/1694655668 shutdown_connections 2024-03-17T23:47:01.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.399+0000 7f8416234640 1 --2- 172.21.15.150:0/1694655668 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f84100b3c30 0x7f84100b01a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:01.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.399+0000 7f8416234640 1 --2- 172.21.15.150:0/1694655668 >> v2:172.21.15.157:3300/0 conn(0x7f84100b5730 0x7f84100b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:01.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.399+0000 7f8416234640 1 -- 172.21.15.150:0/1694655668 >> 172.21.15.150:0/1694655668 conn(0x7f84100b6170 msgr2=0x7f8410099a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:01.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.399+0000 7f8416234640 1 -- 172.21.15.150:0/1694655668 shutdown_connections 2024-03-17T23:47:01.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.399+0000 7f8416234640 1 -- 172.21.15.150:0/1694655668 wait complete. 2024-03-17T23:47:01.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.400+0000 7f8416234640 1 Processor -- start 2024-03-17T23:47:01.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.401+0000 7f8416234640 1 -- start start 2024-03-17T23:47:01.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.401+0000 7f8416234640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f841006b190 0x7f8410146630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:01.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.401+0000 7f8416234640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f84100b3c30 0x7f841015e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:01.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.401+0000 7f8416234640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f84100b5730 0x7f841015ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:01.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.401+0000 7f8416234640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f841009a380 con 0x7f841006b190 2024-03-17T23:47:01.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.401+0000 7f8416234640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f84100b5ef0 con 0x7f84100b5730 2024-03-17T23:47:01.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.401+0000 7f8416234640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f841015e260 con 0x7f84100b3c30 2024-03-17T23:47:01.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.401+0000 7f840f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f841006b190 0x7f8410146630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:01.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.401+0000 7f840f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f841006b190 0x7f8410146630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32810/0 (socket says 172.21.15.150:32810) 2024-03-17T23:47:01.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.401+0000 7f840f7fe640 1 -- 172.21.15.150:0/200493530 learned_addr learned my addr 172.21.15.150:0/200493530 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:01.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.401+0000 7f840ffff640 1 --2- 172.21.15.150:0/200493530 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f84100b5730 0x7f841015ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.401+0000 7f8406ffd640 1 --2- 172.21.15.150:0/200493530 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f84100b3c30 0x7f841015e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.402+0000 7f840f7fe640 1 -- 172.21.15.150:0/200493530 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f84100b3c30 msgr2=0x7f841015e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.402+0000 7f840f7fe640 1 --2- 172.21.15.150:0/200493530 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f84100b3c30 0x7f841015e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.402+0000 7f840f7fe640 1 -- 172.21.15.150:0/200493530 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f84100b5730 msgr2=0x7f841015ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.402+0000 7f840f7fe640 1 --2- 172.21.15.150:0/200493530 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f84100b5730 0x7f841015ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.402+0000 7f840f7fe640 1 -- 172.21.15.150:0/200493530 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8410155b60 con 0x7f841006b190 2024-03-17T23:47:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.402+0000 7f840f7fe640 1 --2- 172.21.15.150:0/200493530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f841006b190 0x7f8410146630 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f840000c430 tx=0x7f840000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.402+0000 7f840cff9640 1 -- 172.21.15.150:0/200493530 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8400017070 con 0x7f841006b190 2024-03-17T23:47:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.402+0000 7f840cff9640 1 -- 172.21.15.150:0/200493530 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84000052d0 con 0x7f841006b190 2024-03-17T23:47:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.402+0000 7f8416234640 1 -- 172.21.15.150:0/200493530 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8410159270 con 0x7f841006b190 2024-03-17T23:47:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.402+0000 7f840cff9640 1 -- 172.21.15.150:0/200493530 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8400003850 con 0x7f841006b190 2024-03-17T23:47:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.402+0000 7f8416234640 1 -- 172.21.15.150:0/200493530 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8410156960 con 0x7f841006b190 2024-03-17T23:47:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.403+0000 7f8416234640 1 -- 172.21.15.150:0/200493530 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84100681a0 con 0x7f841006b190 2024-03-17T23:47:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.403+0000 7f840cff9640 1 -- 172.21.15.150:0/200493530 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f840001a070 con 0x7f841006b190 2024-03-17T23:47:01.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.404+0000 7f840cff9640 1 --2- 172.21.15.150:0/200493530 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f83f006bbd0 0x7f83f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:01.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.404+0000 7f840cff9640 1 -- 172.21.15.150:0/200493530 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f840008a050 con 0x7f841006b190 2024-03-17T23:47:01.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.404+0000 7f8406ffd640 1 --2- 172.21.15.150:0/200493530 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f83f006bbd0 0x7f83f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:01.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.404+0000 7f8406ffd640 1 --2- 172.21.15.150:0/200493530 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f83f006bbd0 0x7f83f006e090 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f84100d02f0 tx=0x7f83f8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:01.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.408+0000 7f840cff9640 1 -- 172.21.15.150:0/200493530 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8400058240 con 0x7f841006b190 2024-03-17T23:47:01.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.580+0000 7f8416234640 1 -- 172.21.15.150:0/200493530 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f841015db80 con 0x7f841006b190 2024-03-17T23:47:01.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.582+0000 7f840cff9640 1 -- 172.21.15.150:0/200493530 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18563 (secure 0 0 0) 0x7f840005c2e0 con 0x7f841006b190 2024-03-17T23:47:01.583 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:01.584 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":25,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:00.292487+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":32773,"flags_names":"hashpspool,ec_overwrites,creating","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:01.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.584+0000 7f8416234640 1 -- 172.21.15.150:0/200493530 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f83f006bbd0 msgr2=0x7f83f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:01.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.584+0000 7f8416234640 1 --2- 172.21.15.150:0/200493530 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f83f006bbd0 0x7f83f006e090 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f84100d02f0 tx=0x7f83f8009190 comp rx=0 tx=0).stop 2024-03-17T23:47:01.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.584+0000 7f8416234640 1 -- 172.21.15.150:0/200493530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f841006b190 msgr2=0x7f8410146630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:01.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.584+0000 7f8416234640 1 --2- 172.21.15.150:0/200493530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f841006b190 0x7f8410146630 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f840000c430 tx=0x7f840000c8f0 comp rx=0 tx=0).stop 2024-03-17T23:47:01.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.584+0000 7f8416234640 1 -- 172.21.15.150:0/200493530 shutdown_connections 2024-03-17T23:47:01.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.584+0000 7f8416234640 1 --2- 172.21.15.150:0/200493530 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f83f006bbd0 0x7f83f006e090 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:01.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.584+0000 7f8416234640 1 --2- 172.21.15.150:0/200493530 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f84100b5730 0x7f841015ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:01.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.584+0000 7f8416234640 1 --2- 172.21.15.150:0/200493530 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f84100b3c30 0x7f841015e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:01.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.584+0000 7f8416234640 1 --2- 172.21.15.150:0/200493530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f841006b190 0x7f8410146630 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:01.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.584+0000 7f8416234640 1 -- 172.21.15.150:0/200493530 >> 172.21.15.150:0/200493530 conn(0x7f84100b6170 msgr2=0x7f841006eff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:01.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.584+0000 7f8416234640 1 -- 172.21.15.150:0/200493530 shutdown_connections 2024-03-17T23:47:01.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.584+0000 7f8416234640 1 -- 172.21.15.150:0/200493530 wait complete. 2024-03-17T23:47:01.596 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:01.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.695+0000 7f14bd4c0640 1 Processor -- start 2024-03-17T23:47:01.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.695+0000 7f14bd4c0640 1 -- start start 2024-03-17T23:47:01.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.696+0000 7f14bd4c0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f14b81619a0 0x7f14b80b9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:01.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.696+0000 7f14bd4c0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14b8161dd0 con 0x7f14b80cf120 2024-03-17T23:47:01.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.696+0000 7f14bd4c0640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14b81620a0 con 0x7f14b815de30 2024-03-17T23:47:01.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.696+0000 7f14bd4c0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f14b8162940 con 0x7f14b81619a0 2024-03-17T23:47:01.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.696+0000 7f14b77fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f14b81619a0 0x7f14b80b9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:01.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.696+0000 7f14b6ffd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f14b815de30 0x7f14b80cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:42540/0 (socket says 172.21.15.150:42540) 2024-03-17T23:47:01.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.696+0000 7f14b6ffd640 1 -- 172.21.15.150:0/1036409957 learned_addr learned my addr 172.21.15.150:0/1036409957 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:01.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.696+0000 7f14b67fc640 1 --1- 172.21.15.150:0/1036409957 >> v1:172.21.15.150:6789/0 conn(0x7f14b80cf120 0x7f14b80d2690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39910/0 (socket says 172.21.15.150:39910) 2024-03-17T23:47:01.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.696+0000 7f14b77fe640 1 -- 172.21.15.150:0/1036409957 >> v1:172.21.15.157:6789/0 conn(0x7f14b815de30 legacy=0x7f14b80cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:01.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.696+0000 7f14b5ffb640 1 -- 172.21.15.150:0/1036409957 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2181299082 0 0) 0x7f14b8161dd0 con 0x7f14b80cf120 2024-03-17T23:47:01.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.696+0000 7f14b77fe640 1 -- 172.21.15.150:0/1036409957 >> v1:172.21.15.150:6789/0 conn(0x7f14b80cf120 legacy=0x7f14b80d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:01.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.696+0000 7f14b77fe640 1 -- 172.21.15.150:0/1036409957 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14b8162380 con 0x7f14b81619a0 2024-03-17T23:47:01.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.697+0000 7f14b77fe640 1 --2- 172.21.15.150:0/1036409957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b81619a0 0x7f14b80b9860 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009d90 tx=0x7f14b002bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b574f7ad4eab151 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:01.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.697+0000 7f14b5ffb640 1 -- 172.21.15.150:0/1036409957 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b0039070 con 0x7f14b81619a0 2024-03-17T23:47:01.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.697+0000 7f14b5ffb640 1 -- 172.21.15.150:0/1036409957 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14b002ca50 con 0x7f14b81619a0 2024-03-17T23:47:01.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.697+0000 7f14b5ffb640 1 -- 172.21.15.150:0/1036409957 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b002ce20 con 0x7f14b81619a0 2024-03-17T23:47:01.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.698+0000 7f14bd4c0640 1 -- 172.21.15.150:0/1036409957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b81619a0 msgr2=0x7f14b80b9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:01.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.698+0000 7f14bd4c0640 1 --2- 172.21.15.150:0/1036409957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b81619a0 0x7f14b80b9860 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009d90 tx=0x7f14b002bd20 comp rx=0 tx=0).stop 2024-03-17T23:47:01.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.698+0000 7f14bd4c0640 1 -- 172.21.15.150:0/1036409957 shutdown_connections 2024-03-17T23:47:01.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.698+0000 7f14bd4c0640 1 --2- 172.21.15.150:0/1036409957 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b81619a0 0x7f14b80b9860 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:01.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.698+0000 7f14bd4c0640 1 -- 172.21.15.150:0/1036409957 >> 172.21.15.150:0/1036409957 conn(0x7f14b80730b0 msgr2=0x7f14b80d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:01.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.698+0000 7f14bd4c0640 1 -- 172.21.15.150:0/1036409957 shutdown_connections 2024-03-17T23:47:01.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.698+0000 7f14bd4c0640 1 -- 172.21.15.150:0/1036409957 wait complete. 2024-03-17T23:47:01.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.699+0000 7f14bd4c0640 1 Processor -- start 2024-03-17T23:47:01.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.699+0000 7f14bd4c0640 1 -- start start 2024-03-17T23:47:01.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.700+0000 7f14bd4c0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b80cf120 0x7f14b80c9750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:01.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.700+0000 7f14bd4c0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14b815de30 0x7f14b8146fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:01.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.700+0000 7f14bd4c0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14b81619a0 0x7f14b815a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:01.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.700+0000 7f14bd4c0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f14b815e210 con 0x7f14b80cf120 2024-03-17T23:47:01.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.700+0000 7f14bd4c0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f14b80d2170 con 0x7f14b815de30 2024-03-17T23:47:01.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.700+0000 7f14bd4c0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f14b80d2da0 con 0x7f14b81619a0 2024-03-17T23:47:01.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.700+0000 7f14b6ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b80cf120 0x7f14b80c9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:01.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.700+0000 7f14b6ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b80cf120 0x7f14b80c9750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32816/0 (socket says 172.21.15.150:32816) 2024-03-17T23:47:01.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.700+0000 7f14b6ffd640 1 -- 172.21.15.150:0/420666874 learned_addr learned my addr 172.21.15.150:0/420666874 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:01.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.700+0000 7f14b67fc640 1 --2- 172.21.15.150:0/420666874 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14b815de30 0x7f14b8146fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:01.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.700+0000 7f14b77fe640 1 --2- 172.21.15.150:0/420666874 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14b81619a0 0x7f14b815a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:01.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.700+0000 7f14b6ffd640 1 -- 172.21.15.150:0/420666874 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14b81619a0 msgr2=0x7f14b815a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:01.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.700+0000 7f14b6ffd640 1 --2- 172.21.15.150:0/420666874 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14b81619a0 0x7f14b815a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:01.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.700+0000 7f14b6ffd640 1 -- 172.21.15.150:0/420666874 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14b815de30 msgr2=0x7f14b8146fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:01.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.700+0000 7f14b6ffd640 1 --2- 172.21.15.150:0/420666874 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14b815de30 0x7f14b8146fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:01.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.700+0000 7f14b6ffd640 1 -- 172.21.15.150:0/420666874 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14b815ae70 con 0x7f14b80cf120 2024-03-17T23:47:01.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.700+0000 7f14b6ffd640 1 --2- 172.21.15.150:0/420666874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b80cf120 0x7f14b80c9750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f14a800c780 tx=0x7f14a800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:01.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.700+0000 7f14af7fe640 1 -- 172.21.15.150:0/420666874 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a8018070 con 0x7f14b80cf120 2024-03-17T23:47:01.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.700+0000 7f14af7fe640 1 -- 172.21.15.150:0/420666874 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14a8007500 con 0x7f14b80cf120 2024-03-17T23:47:01.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.700+0000 7f14af7fe640 1 -- 172.21.15.150:0/420666874 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a8004a20 con 0x7f14b80cf120 2024-03-17T23:47:01.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.700+0000 7f14bd4c0640 1 -- 172.21.15.150:0/420666874 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14b80cbf70 con 0x7f14b80cf120 2024-03-17T23:47:01.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.700+0000 7f14bd4c0640 1 -- 172.21.15.150:0/420666874 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14b80cc530 con 0x7f14b80cf120 2024-03-17T23:47:01.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.702+0000 7f14bd4c0640 1 -- 172.21.15.150:0/420666874 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14b80681a0 con 0x7f14b80cf120 2024-03-17T23:47:01.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.702+0000 7f14af7fe640 1 -- 172.21.15.150:0/420666874 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f14a801b070 con 0x7f14b80cf120 2024-03-17T23:47:01.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.702+0000 7f14af7fe640 1 --2- 172.21.15.150:0/420666874 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f149806bbd0 0x7f149806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:01.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.702+0000 7f14b67fc640 1 --2- 172.21.15.150:0/420666874 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f149806bbd0 0x7f149806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:01.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.702+0000 7f14af7fe640 1 -- 172.21.15.150:0/420666874 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f14a808a110 con 0x7f14b80cf120 2024-03-17T23:47:01.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.703+0000 7f14b67fc640 1 --2- 172.21.15.150:0/420666874 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f149806bbd0 0x7f149806e090 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f14a0007f70 tx=0x7f14a0007f00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:01.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.707+0000 7f14af7fe640 1 -- 172.21.15.150:0/420666874 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f14a8058300 con 0x7f14b80cf120 2024-03-17T23:47:01.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.883+0000 7f14bd4c0640 1 -- 172.21.15.150:0/420666874 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f14b80b29c0 con 0x7f14b80cf120 2024-03-17T23:47:01.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.884+0000 7f14af7fe640 1 -- 172.21.15.150:0/420666874 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18563 (secure 0 0 0) 0x7f14a805c3a0 con 0x7f14b80cf120 2024-03-17T23:47:01.885 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:01.886 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":25,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:00.292487+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":32773,"flags_names":"hashpspool,ec_overwrites,creating","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:01.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.886+0000 7f14bd4c0640 1 -- 172.21.15.150:0/420666874 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f149806bbd0 msgr2=0x7f149806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:01.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.886+0000 7f14bd4c0640 1 --2- 172.21.15.150:0/420666874 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f149806bbd0 0x7f149806e090 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f14a0007f70 tx=0x7f14a0007f00 comp rx=0 tx=0).stop 2024-03-17T23:47:01.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.886+0000 7f14bd4c0640 1 -- 172.21.15.150:0/420666874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b80cf120 msgr2=0x7f14b80c9750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:01.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.886+0000 7f14bd4c0640 1 --2- 172.21.15.150:0/420666874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b80cf120 0x7f14b80c9750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f14a800c780 tx=0x7f14a800cc40 comp rx=0 tx=0).stop 2024-03-17T23:47:01.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.886+0000 7f14bd4c0640 1 -- 172.21.15.150:0/420666874 shutdown_connections 2024-03-17T23:47:01.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.886+0000 7f14bd4c0640 1 --2- 172.21.15.150:0/420666874 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f14b81619a0 0x7f14b815a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:01.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.886+0000 7f14bd4c0640 1 --2- 172.21.15.150:0/420666874 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f149806bbd0 0x7f149806e090 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:01.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.886+0000 7f14bd4c0640 1 --2- 172.21.15.150:0/420666874 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f14b815de30 0x7f14b8146fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:01.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.886+0000 7f14bd4c0640 1 --2- 172.21.15.150:0/420666874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b80cf120 0x7f14b80c9750 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:01.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.886+0000 7f14bd4c0640 1 -- 172.21.15.150:0/420666874 >> 172.21.15.150:0/420666874 conn(0x7f14b80730b0 msgr2=0x7f14b80b37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:01.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.886+0000 7f14bd4c0640 1 -- 172.21.15.150:0/420666874 shutdown_connections 2024-03-17T23:47:01.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:01.886+0000 7f14bd4c0640 1 -- 172.21.15.150:0/420666874 wait complete. 2024-03-17T23:47:01.912 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:47:02.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.012+0000 7ff1aacd1640 1 Processor -- start 2024-03-17T23:47:02.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.012+0000 7ff1aacd1640 1 -- start start 2024-03-17T23:47:02.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.013+0000 7ff1aacd1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff1a415bc50 0x7ff1a40cf910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:02.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.013+0000 7ff1aacd1640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff1a415c030 0x7ff1a40ad180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:02.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.013+0000 7ff1aacd1640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1a40cfff0 con 0x7ff1a4099a70 2024-03-17T23:47:02.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.013+0000 7ff1aacd1640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7ff1a4067340 con 0x7ff1a415bc50 2024-03-17T23:47:02.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.013+0000 7ff1aacd1640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff1a40d0340 con 0x7ff1a415c030 2024-03-17T23:47:02.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.013+0000 7ff1a8a6d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff1a415bc50 0x7ff1a40cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:02.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.013+0000 7ff1a8a6d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff1a415bc50 0x7ff1a40cf910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32820/0 (socket says 172.21.15.150:32820) 2024-03-17T23:47:02.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.013+0000 7ff1a8a6d640 1 -- 172.21.15.150:0/3892479479 learned_addr learned my addr 172.21.15.150:0/3892479479 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:02.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.013+0000 7ff1a3fff640 1 --2- 172.21.15.150:0/3892479479 >> v2:172.21.15.157:3300/0 conn(0x7ff1a415c030 0x7ff1a40ad180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:02.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.013+0000 7ff1a8a6d640 1 -- 172.21.15.150:0/3892479479 >> v2:172.21.15.157:3300/0 conn(0x7ff1a415c030 msgr2=0x7ff1a40ad180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:02.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.013+0000 7ff1a8a6d640 1 --2- 172.21.15.150:0/3892479479 >> v2:172.21.15.157:3300/0 conn(0x7ff1a415c030 0x7ff1a40ad180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:02.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.013+0000 7ff1a8a6d640 1 -- 172.21.15.150:0/3892479479 >> v1:172.21.15.157:6789/0 conn(0x7ff1a4099a70 legacy=0x7ff1a406b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:02.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.013+0000 7ff1a8a6d640 1 -- 172.21.15.150:0/3892479479 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1a4067590 con 0x7ff1a415bc50 2024-03-17T23:47:02.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.014+0000 7ff1a8a6d640 1 --2- 172.21.15.150:0/3892479479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1a415bc50 0x7ff1a40cf910 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff1a40d1210 tx=0x7ff19802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e27c07046f74cd9b server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:02.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.014+0000 7ff1a37fe640 1 -- 172.21.15.150:0/3892479479 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff198035020 con 0x7ff1a415bc50 2024-03-17T23:47:02.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.014+0000 7ff1a37fe640 1 -- 172.21.15.150:0/3892479479 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff198031870 con 0x7ff1a415bc50 2024-03-17T23:47:02.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.014+0000 7ff1a37fe640 1 -- 172.21.15.150:0/3892479479 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff198031c40 con 0x7ff1a415bc50 2024-03-17T23:47:02.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.014+0000 7ff1aacd1640 1 -- 172.21.15.150:0/3892479479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1a415bc50 msgr2=0x7ff1a40cf910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:02.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.014+0000 7ff1aacd1640 1 --2- 172.21.15.150:0/3892479479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1a415bc50 0x7ff1a40cf910 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff1a40d1210 tx=0x7ff19802b910 comp rx=0 tx=0).stop 2024-03-17T23:47:02.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.015+0000 7ff1aacd1640 1 -- 172.21.15.150:0/3892479479 shutdown_connections 2024-03-17T23:47:02.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.015+0000 7ff1aacd1640 1 --2- 172.21.15.150:0/3892479479 >> v2:172.21.15.157:3300/0 conn(0x7ff1a415c030 0x7ff1a40ad180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:02.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.015+0000 7ff1aacd1640 1 --2- 172.21.15.150:0/3892479479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1a415bc50 0x7ff1a40cf910 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:02.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.015+0000 7ff1aacd1640 1 -- 172.21.15.150:0/3892479479 >> 172.21.15.150:0/3892479479 conn(0x7ff1a40b3da0 msgr2=0x7ff1a40b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:02.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.015+0000 7ff1aacd1640 1 -- 172.21.15.150:0/3892479479 shutdown_connections 2024-03-17T23:47:02.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.015+0000 7ff1aacd1640 1 -- 172.21.15.150:0/3892479479 wait complete. 2024-03-17T23:47:02.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.016+0000 7ff1aacd1640 1 Processor -- start 2024-03-17T23:47:02.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.016+0000 7ff1aacd1640 1 -- start start 2024-03-17T23:47:02.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.017+0000 7ff1aacd1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1a4099a70 0x7ff1a409ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:02.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.017+0000 7ff1aacd1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1a415bc50 0x7ff1a41457d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:02.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.017+0000 7ff1aacd1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1a409ce20 0x7ff1a409d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:02.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.017+0000 7ff1aacd1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a40d07d0 con 0x7ff1a415bc50 2024-03-17T23:47:02.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.017+0000 7ff1aacd1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a4072d30 con 0x7ff1a4099a70 2024-03-17T23:47:02.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.017+0000 7ff1aacd1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff1a40d0950 con 0x7ff1a409ce20 2024-03-17T23:47:02.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.017+0000 7ff1a3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1a415bc50 0x7ff1a41457d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:02.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.017+0000 7ff1a3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1a415bc50 0x7ff1a41457d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32832/0 (socket says 172.21.15.150:32832) 2024-03-17T23:47:02.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.017+0000 7ff1a3fff640 1 -- 172.21.15.150:0/1670022696 learned_addr learned my addr 172.21.15.150:0/1670022696 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:02.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.017+0000 7ff1a8a6d640 1 --2- 172.21.15.150:0/1670022696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1a4099a70 0x7ff1a409ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:02.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.017+0000 7ff1a926e640 1 --2- 172.21.15.150:0/1670022696 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1a409ce20 0x7ff1a409d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:02.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.017+0000 7ff1a3fff640 1 -- 172.21.15.150:0/1670022696 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1a409ce20 msgr2=0x7ff1a409d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:02.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.017+0000 7ff1a3fff640 1 --2- 172.21.15.150:0/1670022696 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1a409ce20 0x7ff1a409d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:02.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.017+0000 7ff1a3fff640 1 -- 172.21.15.150:0/1670022696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1a4099a70 msgr2=0x7ff1a409ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:02.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.017+0000 7ff1a3fff640 1 --2- 172.21.15.150:0/1670022696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1a4099a70 0x7ff1a409ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:02.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.017+0000 7ff1a3fff640 1 -- 172.21.15.150:0/1670022696 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1a409bc40 con 0x7ff1a415bc50 2024-03-17T23:47:02.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.017+0000 7ff1a3fff640 1 --2- 172.21.15.150:0/1670022696 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1a415bc50 0x7ff1a41457d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff19000e870 tx=0x7ff19000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:02.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.017+0000 7ff1a17fa640 1 -- 172.21.15.150:0/1670022696 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff190017070 con 0x7ff1a415bc50 2024-03-17T23:47:02.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.017+0000 7ff1a17fa640 1 -- 172.21.15.150:0/1670022696 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff19000c940 con 0x7ff1a415bc50 2024-03-17T23:47:02.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.017+0000 7ff1aacd1640 1 -- 172.21.15.150:0/1670022696 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1a409b3a0 con 0x7ff1a415bc50 2024-03-17T23:47:02.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.017+0000 7ff1a17fa640 1 -- 172.21.15.150:0/1670022696 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19000cd10 con 0x7ff1a415bc50 2024-03-17T23:47:02.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.017+0000 7ff1aacd1640 1 -- 172.21.15.150:0/1670022696 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1a409b960 con 0x7ff1a415bc50 2024-03-17T23:47:02.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.018+0000 7ff1aacd1640 1 -- 172.21.15.150:0/1670022696 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1a40681a0 con 0x7ff1a415bc50 2024-03-17T23:47:02.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.019+0000 7ff1a17fa640 1 -- 172.21.15.150:0/1670022696 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff19001a070 con 0x7ff1a415bc50 2024-03-17T23:47:02.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.019+0000 7ff1a17fa640 1 --2- 172.21.15.150:0/1670022696 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff18806bbd0 0x7ff18806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:02.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.019+0000 7ff1a8a6d640 1 --2- 172.21.15.150:0/1670022696 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff18806bbd0 0x7ff18806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:02.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.019+0000 7ff1a17fa640 1 -- 172.21.15.150:0/1670022696 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7ff19008ab80 con 0x7ff1a415bc50 2024-03-17T23:47:02.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.019+0000 7ff1a8a6d640 1 --2- 172.21.15.150:0/1670022696 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff18806bbd0 0x7ff18806e090 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff198036000 tx=0x7ff19803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:02.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.023+0000 7ff1a17fa640 1 -- 172.21.15.150:0/1670022696 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff190058ce0 con 0x7ff1a415bc50 2024-03-17T23:47:02.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.233+0000 7ff1aacd1640 1 -- 172.21.15.150:0/1670022696 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1a40cf4e0 con 0x7ff1a415bc50 2024-03-17T23:47:02.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.233+0000 7ff1a17fa640 1 -- 172.21.15.150:0/1670022696 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5186 (secure 0 0 0) 0x7ff19005cd80 con 0x7ff1a415bc50 2024-03-17T23:47:02.235 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:02.235 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/1949609985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6833","nonce":1949609985},{"type":"v1","addr":"172.21.15.157:6835","nonce":1949609985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/758264728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":758264728},{"type":"v1","addr":"172.21.15.150:6835","nonce":758264728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3655589010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3655589010},{"type":"v1","addr":"172.21.15.150:6837","nonce":3655589010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-17T23:46:55.253739+0000","modified":"2024-03-17T23:47:00.292677+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6834/1625058792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1625058792},{"type":"v1","addr":"172.21.15.157:6834","nonce":1625058792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4297,"qdb_cluster":[4297]},"id":1}]} 2024-03-17T23:47:02.235 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 6 2024-03-17T23:47:02.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.235+0000 7ff1aacd1640 1 -- 172.21.15.150:0/1670022696 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff18806bbd0 msgr2=0x7ff18806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:02.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.235+0000 7ff1aacd1640 1 --2- 172.21.15.150:0/1670022696 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff18806bbd0 0x7ff18806e090 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff198036000 tx=0x7ff19803c040 comp rx=0 tx=0).stop 2024-03-17T23:47:02.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.235+0000 7ff1aacd1640 1 -- 172.21.15.150:0/1670022696 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1a415bc50 msgr2=0x7ff1a41457d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:02.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.235+0000 7ff1aacd1640 1 --2- 172.21.15.150:0/1670022696 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1a415bc50 0x7ff1a41457d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff19000e870 tx=0x7ff19000ed30 comp rx=0 tx=0).stop 2024-03-17T23:47:02.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.235+0000 7ff1aacd1640 1 -- 172.21.15.150:0/1670022696 shutdown_connections 2024-03-17T23:47:02.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.235+0000 7ff1aacd1640 1 --2- 172.21.15.150:0/1670022696 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff1a409ce20 0x7ff1a409d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:02.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.235+0000 7ff1aacd1640 1 --2- 172.21.15.150:0/1670022696 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff1a415bc50 0x7ff1a41457d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:02.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.235+0000 7ff1aacd1640 1 --2- 172.21.15.150:0/1670022696 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff18806bbd0 0x7ff18806e090 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:02.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.235+0000 7ff1aacd1640 1 --2- 172.21.15.150:0/1670022696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff1a4099a70 0x7ff1a409ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:02.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.235+0000 7ff1aacd1640 1 -- 172.21.15.150:0/1670022696 >> 172.21.15.150:0/1670022696 conn(0x7ff1a40b3da0 msgr2=0x7ff1a40b2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:02.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.235+0000 7ff1aacd1640 1 -- 172.21.15.150:0/1670022696 shutdown_connections 2024-03-17T23:47:02.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.235+0000 7ff1aacd1640 1 -- 172.21.15.150:0/1670022696 wait complete. 2024-03-17T23:47:02.247 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:02.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.348+0000 7f2ca963d640 1 Processor -- start 2024-03-17T23:47:02.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.348+0000 7f2ca963d640 1 -- start start 2024-03-17T23:47:02.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.348+0000 7f2ca963d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2ca40cf120 0x7f2ca41619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:02.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.348+0000 7f2ca963d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2ca40b9860 0x7f2ca40d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:02.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.348+0000 7f2ca963d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ca41620a0 con 0x7f2ca415de30 2024-03-17T23:47:02.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.348+0000 7f2ca963d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f2ca4162940 con 0x7f2ca40cf120 2024-03-17T23:47:02.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.348+0000 7f2ca963d640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f2ca4162380 con 0x7f2ca40b9860 2024-03-17T23:47:02.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.348+0000 7f2ca27fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2ca40cf120 0x7f2ca41619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:02.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.349+0000 7f2ca2ffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f2ca415de30 0x7f2ca40cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39914/0 (socket says 172.21.15.150:39914) 2024-03-17T23:47:02.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.349+0000 7f2ca2ffd640 1 -- 172.21.15.150:0/3844356551 learned_addr learned my addr 172.21.15.150:0/3844356551 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:02.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.349+0000 7f2ca37fe640 1 --2- 172.21.15.150:0/3844356551 >> v2:172.21.15.157:3300/0 conn(0x7f2ca40b9860 0x7f2ca40d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:02.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.349+0000 7f2ca27fc640 1 -- 172.21.15.150:0/3844356551 >> v2:172.21.15.157:3300/0 conn(0x7f2ca40b9860 msgr2=0x7f2ca40d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:02.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.349+0000 7f2ca27fc640 1 --2- 172.21.15.150:0/3844356551 >> v2:172.21.15.157:3300/0 conn(0x7f2ca40b9860 0x7f2ca40d2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:02.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.349+0000 7f2ca27fc640 1 -- 172.21.15.150:0/3844356551 >> v1:172.21.15.150:6789/0 conn(0x7f2ca415de30 legacy=0x7f2ca40cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:02.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.349+0000 7f2ca1ffb640 1 -- 172.21.15.150:0/3844356551 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2417921355 0 0) 0x7f2ca41620a0 con 0x7f2ca415de30 2024-03-17T23:47:02.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.349+0000 7f2ca27fc640 1 -- 172.21.15.150:0/3844356551 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ca4162660 con 0x7f2ca40cf120 2024-03-17T23:47:02.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.349+0000 7f2ca27fc640 1 --2- 172.21.15.150:0/3844356551 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ca40cf120 0x7f2ca41619a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2ca40d0400 tx=0x7f2c9802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40bbc552cc4355f server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:02.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.350+0000 7f2ca1ffb640 1 -- 172.21.15.150:0/3844356551 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c980315a0 con 0x7f2ca40cf120 2024-03-17T23:47:02.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.350+0000 7f2ca1ffb640 1 -- 172.21.15.150:0/3844356551 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c98037070 con 0x7f2ca40cf120 2024-03-17T23:47:02.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.350+0000 7f2ca1ffb640 1 -- 172.21.15.150:0/3844356551 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c98031960 con 0x7f2ca40cf120 2024-03-17T23:47:02.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.350+0000 7f2ca963d640 1 -- 172.21.15.150:0/3844356551 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ca40cf120 msgr2=0x7f2ca41619a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:02.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.350+0000 7f2ca963d640 1 --2- 172.21.15.150:0/3844356551 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ca40cf120 0x7f2ca41619a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2ca40d0400 tx=0x7f2c9802b840 comp rx=0 tx=0).stop 2024-03-17T23:47:02.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.350+0000 7f2ca963d640 1 -- 172.21.15.150:0/3844356551 shutdown_connections 2024-03-17T23:47:02.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.350+0000 7f2ca963d640 1 --2- 172.21.15.150:0/3844356551 >> v2:172.21.15.157:3300/0 conn(0x7f2ca40b9860 0x7f2ca40d2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:02.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.350+0000 7f2ca963d640 1 --2- 172.21.15.150:0/3844356551 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ca40cf120 0x7f2ca41619a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:02.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.350+0000 7f2ca963d640 1 -- 172.21.15.150:0/3844356551 >> 172.21.15.150:0/3844356551 conn(0x7f2ca40730b0 msgr2=0x7f2ca40d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:02.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.350+0000 7f2ca963d640 1 -- 172.21.15.150:0/3844356551 shutdown_connections 2024-03-17T23:47:02.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.350+0000 7f2ca963d640 1 -- 172.21.15.150:0/3844356551 wait complete. 2024-03-17T23:47:02.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.351+0000 7f2ca963d640 1 Processor -- start 2024-03-17T23:47:02.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.352+0000 7f2ca963d640 1 -- start start 2024-03-17T23:47:02.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.352+0000 7f2ca963d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ca40b9860 0x7f2ca40ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:02.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.352+0000 7f2ca963d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ca40cf120 0x7f2ca40b5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:02.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.352+0000 7f2ca963d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ca415de30 0x7f2ca40b56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:02.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.352+0000 7f2ca963d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca415e210 con 0x7f2ca40b9860 2024-03-17T23:47:02.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.352+0000 7f2ca963d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca40b9c90 con 0x7f2ca40cf120 2024-03-17T23:47:02.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.352+0000 7f2ca963d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2ca40d2bd0 con 0x7f2ca415de30 2024-03-17T23:47:02.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.352+0000 7f2ca2ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ca40b9860 0x7f2ca40ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:02.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.352+0000 7f2ca2ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ca40b9860 0x7f2ca40ccc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32838/0 (socket says 172.21.15.150:32838) 2024-03-17T23:47:02.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.352+0000 7f2ca2ffd640 1 -- 172.21.15.150:0/1707009986 learned_addr learned my addr 172.21.15.150:0/1707009986 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:02.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.352+0000 7f2ca37fe640 1 --2- 172.21.15.150:0/1707009986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ca415de30 0x7f2ca40b56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:02.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.352+0000 7f2ca27fc640 1 --2- 172.21.15.150:0/1707009986 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ca40cf120 0x7f2ca40b5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:02.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.353+0000 7f2ca2ffd640 1 -- 172.21.15.150:0/1707009986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ca415de30 msgr2=0x7f2ca40b56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:02.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.353+0000 7f2ca2ffd640 1 --2- 172.21.15.150:0/1707009986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ca415de30 0x7f2ca40b56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:02.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.353+0000 7f2ca2ffd640 1 -- 172.21.15.150:0/1707009986 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ca40cf120 msgr2=0x7f2ca40b5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:02.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.353+0000 7f2ca2ffd640 1 --2- 172.21.15.150:0/1707009986 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ca40cf120 0x7f2ca40b5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:02.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.353+0000 7f2ca2ffd640 1 -- 172.21.15.150:0/1707009986 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ca40cd2a0 con 0x7f2ca40b9860 2024-03-17T23:47:02.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.353+0000 7f2ca2ffd640 1 --2- 172.21.15.150:0/1707009986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ca40b9860 0x7f2ca40ccc00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2c9400c7c0 tx=0x7f2c9400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:02.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.353+0000 7f2c8b7fe640 1 -- 172.21.15.150:0/1707009986 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c94018070 con 0x7f2ca40b9860 2024-03-17T23:47:02.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.353+0000 7f2c8b7fe640 1 -- 172.21.15.150:0/1707009986 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c9400a640 con 0x7f2ca40b9860 2024-03-17T23:47:02.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.353+0000 7f2ca963d640 1 -- 172.21.15.150:0/1707009986 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ca40cd580 con 0x7f2ca40b9860 2024-03-17T23:47:02.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.353+0000 7f2c8b7fe640 1 -- 172.21.15.150:0/1707009986 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c9400aa30 con 0x7f2ca40b9860 2024-03-17T23:47:02.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.353+0000 7f2ca963d640 1 -- 172.21.15.150:0/1707009986 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ca40cdb40 con 0x7f2ca40b9860 2024-03-17T23:47:02.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.354+0000 7f2ca963d640 1 -- 172.21.15.150:0/1707009986 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ca40681a0 con 0x7f2ca40b9860 2024-03-17T23:47:02.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.354+0000 7f2c8b7fe640 1 -- 172.21.15.150:0/1707009986 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2c9401b070 con 0x7f2ca40b9860 2024-03-17T23:47:02.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.355+0000 7f2c8b7fe640 1 --2- 172.21.15.150:0/1707009986 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2c9006bbd0 0x7f2c9006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:02.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.355+0000 7f2c8b7fe640 1 -- 172.21.15.150:0/1707009986 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f2c9408a2d0 con 0x7f2ca40b9860 2024-03-17T23:47:02.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.355+0000 7f2ca27fc640 1 --2- 172.21.15.150:0/1707009986 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2c9006bbd0 0x7f2c9006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:02.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.355+0000 7f2ca27fc640 1 --2- 172.21.15.150:0/1707009986 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2c9006bbd0 0x7f2c9006e090 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2c98036000 tx=0x7f2c9803d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:02.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.359+0000 7f2c8b7fe640 1 -- 172.21.15.150:0/1707009986 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2c940584c0 con 0x7f2ca40b9860 2024-03-17T23:47:02.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.532+0000 7f2ca963d640 1 -- 172.21.15.150:0/1707009986 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2ca415d700 con 0x7f2ca40b9860 2024-03-17T23:47:02.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.533+0000 7f2c8b7fe640 1 -- 172.21.15.150:0/1707009986 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18550 (secure 0 0 0) 0x7f2c9405c560 con 0x7f2ca40b9860 2024-03-17T23:47:02.535 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:02.535 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:01.940019+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:02.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.536+0000 7f2ca963d640 1 -- 172.21.15.150:0/1707009986 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2c9006bbd0 msgr2=0x7f2c9006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:02.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.536+0000 7f2ca963d640 1 --2- 172.21.15.150:0/1707009986 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2c9006bbd0 0x7f2c9006e090 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2c98036000 tx=0x7f2c9803d040 comp rx=0 tx=0).stop 2024-03-17T23:47:02.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.536+0000 7f2ca963d640 1 -- 172.21.15.150:0/1707009986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ca40b9860 msgr2=0x7f2ca40ccc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:02.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.536+0000 7f2ca963d640 1 --2- 172.21.15.150:0/1707009986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ca40b9860 0x7f2ca40ccc00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2c9400c7c0 tx=0x7f2c9400cc80 comp rx=0 tx=0).stop 2024-03-17T23:47:02.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.536+0000 7f2ca963d640 1 -- 172.21.15.150:0/1707009986 shutdown_connections 2024-03-17T23:47:02.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.536+0000 7f2ca963d640 1 --2- 172.21.15.150:0/1707009986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2ca415de30 0x7f2ca40b56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:02.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.536+0000 7f2ca963d640 1 --2- 172.21.15.150:0/1707009986 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2c9006bbd0 0x7f2c9006e090 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:02.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.536+0000 7f2ca963d640 1 --2- 172.21.15.150:0/1707009986 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2ca40cf120 0x7f2ca40b5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:02.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.536+0000 7f2ca963d640 1 --2- 172.21.15.150:0/1707009986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2ca40b9860 0x7f2ca40ccc00 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:02.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.536+0000 7f2ca963d640 1 -- 172.21.15.150:0/1707009986 >> 172.21.15.150:0/1707009986 conn(0x7f2ca40730b0 msgr2=0x7f2ca40d2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:02.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.536+0000 7f2ca963d640 1 -- 172.21.15.150:0/1707009986 shutdown_connections 2024-03-17T23:47:02.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:02.536+0000 7f2ca963d640 1 -- 172.21.15.150:0/1707009986 wait complete. 2024-03-17T23:47:02.951 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-17T23:46:55.253739+0000', 'modified': '2024-03-17T23:47:00.292677+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4297}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4297': {'gid': 4297, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.157:6834/1625058792', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6832', 'nonce': 1625058792}, {'type': 'v1', 'addr': '172.21.15.157:6834', 'nonce': 1625058792}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4297, 'qdb_cluster': [4297]} 2024-03-17T23:47:02.951 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-17T23:47:02.951 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-17T23:47:03.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.052+0000 7f1d58c9a640 1 Processor -- start 2024-03-17T23:47:03.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.052+0000 7f1d58c9a640 1 -- start start 2024-03-17T23:47:03.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.053+0000 7f1d58c9a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1d5415e1a0 0x7f1d540afe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.053+0000 7f1d58c9a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1d54099a70 0x7f1d540b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.053+0000 7f1d58c9a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d5415e580 con 0x7f1d540af980 2024-03-17T23:47:03.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.053+0000 7f1d58c9a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f1d5406bcd0 con 0x7f1d5415e1a0 2024-03-17T23:47:03.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.053+0000 7f1d58c9a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1d5406bfb0 con 0x7f1d54099a70 2024-03-17T23:47:03.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.053+0000 7f1d5259c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1d5415e1a0 0x7f1d540afe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.053+0000 7f1d5259c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1d5415e1a0 0x7f1d540afe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32842/0 (socket says 172.21.15.150:32842) 2024-03-17T23:47:03.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.053+0000 7f1d5259c640 1 -- 172.21.15.150:0/2544217020 learned_addr learned my addr 172.21.15.150:0/2544217020 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:03.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.053+0000 7f1d51d9b640 1 --2- 172.21.15.150:0/2544217020 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1d54099a70 0x7f1d540b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.054+0000 7f1d5259c640 1 -- 172.21.15.150:0/2544217020 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1d54099a70 msgr2=0x7f1d540b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:03.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.054+0000 7f1d5259c640 1 --2- 172.21.15.150:0/2544217020 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1d54099a70 0x7f1d540b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.054+0000 7f1d5259c640 1 -- 172.21.15.150:0/2544217020 >> v1:172.21.15.157:6789/0 conn(0x7f1d540af980 legacy=0x7f1d540d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:03.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.054+0000 7f1d5259c640 1 -- 172.21.15.150:0/2544217020 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d5406c290 con 0x7f1d5415e1a0 2024-03-17T23:47:03.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.054+0000 7f1d5259c640 1 --2- 172.21.15.150:0/2544217020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d5415e1a0 0x7f1d540afe80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1d44009e50 tx=0x7f1d4402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a774a3c11797c53 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:03.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.054+0000 7f1d5159a640 1 -- 172.21.15.150:0/2544217020 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d4402ca70 con 0x7f1d5415e1a0 2024-03-17T23:47:03.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.054+0000 7f1d5159a640 1 -- 172.21.15.150:0/2544217020 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d4402cc10 con 0x7f1d5415e1a0 2024-03-17T23:47:03.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.054+0000 7f1d5159a640 1 -- 172.21.15.150:0/2544217020 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d440343f0 con 0x7f1d5415e1a0 2024-03-17T23:47:03.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.055+0000 7f1d58c9a640 1 -- 172.21.15.150:0/2544217020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d5415e1a0 msgr2=0x7f1d540afe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:03.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.055+0000 7f1d58c9a640 1 --2- 172.21.15.150:0/2544217020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d5415e1a0 0x7f1d540afe80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1d44009e50 tx=0x7f1d4402bed0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.055+0000 7f1d58c9a640 1 -- 172.21.15.150:0/2544217020 shutdown_connections 2024-03-17T23:47:03.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.055+0000 7f1d58c9a640 1 --2- 172.21.15.150:0/2544217020 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1d54099a70 0x7f1d540b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.055+0000 7f1d58c9a640 1 --2- 172.21.15.150:0/2544217020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d5415e1a0 0x7f1d540afe80 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.055+0000 7f1d58c9a640 1 -- 172.21.15.150:0/2544217020 >> 172.21.15.150:0/2544217020 conn(0x7f1d54072a60 msgr2=0x7f1d540b5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:03.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.055+0000 7f1d58c9a640 1 -- 172.21.15.150:0/2544217020 shutdown_connections 2024-03-17T23:47:03.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.055+0000 7f1d58c9a640 1 -- 172.21.15.150:0/2544217020 wait complete. 2024-03-17T23:47:03.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.056+0000 7f1d58c9a640 1 Processor -- start 2024-03-17T23:47:03.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.056+0000 7f1d58c9a640 1 -- start start 2024-03-17T23:47:03.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.057+0000 7f1d58c9a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1d54099a70 0x7f1d541f3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.057+0000 7f1d58c9a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d540af980 0x7f1d541e70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.057+0000 7f1d58c9a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1d5415e1a0 0x7f1d540aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.057+0000 7f1d58c9a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1d540b4110 con 0x7f1d540af980 2024-03-17T23:47:03.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.057+0000 7f1d58c9a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1d5406c850 con 0x7f1d5415e1a0 2024-03-17T23:47:03.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.057+0000 7f1d58c9a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1d540ae290 con 0x7f1d54099a70 2024-03-17T23:47:03.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.057+0000 7f1d51d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d540af980 0x7f1d541e70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.057+0000 7f1d51d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d540af980 0x7f1d541e70e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32846/0 (socket says 172.21.15.150:32846) 2024-03-17T23:47:03.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.057+0000 7f1d51d9b640 1 -- 172.21.15.150:0/1808073947 learned_addr learned my addr 172.21.15.150:0/1808073947 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:03.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.057+0000 7f1d5259c640 1 --2- 172.21.15.150:0/1808073947 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1d54099a70 0x7f1d541f3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.057+0000 7f1d52d9d640 1 --2- 172.21.15.150:0/1808073947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1d5415e1a0 0x7f1d540aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.057+0000 7f1d51d9b640 1 -- 172.21.15.150:0/1808073947 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1d54099a70 msgr2=0x7f1d541f3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:03.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.057+0000 7f1d51d9b640 1 --2- 172.21.15.150:0/1808073947 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1d54099a70 0x7f1d541f3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.057+0000 7f1d51d9b640 1 -- 172.21.15.150:0/1808073947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1d5415e1a0 msgr2=0x7f1d540aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:03.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.057+0000 7f1d51d9b640 1 --2- 172.21.15.150:0/1808073947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1d5415e1a0 0x7f1d540aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.057+0000 7f1d51d9b640 1 -- 172.21.15.150:0/1808073947 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d5409ca10 con 0x7f1d540af980 2024-03-17T23:47:03.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.058+0000 7f1d51d9b640 1 --2- 172.21.15.150:0/1808073947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d540af980 0x7f1d541e70e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1d4800e9b0 tx=0x7f1d4800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:03.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.058+0000 7f1d42ffd640 1 -- 172.21.15.150:0/1808073947 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d48017070 con 0x7f1d540af980 2024-03-17T23:47:03.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.058+0000 7f1d42ffd640 1 -- 172.21.15.150:0/1808073947 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d4800c9a0 con 0x7f1d540af980 2024-03-17T23:47:03.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.058+0000 7f1d58c9a640 1 -- 172.21.15.150:0/1808073947 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d5409aff0 con 0x7f1d540af980 2024-03-17T23:47:03.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.058+0000 7f1d42ffd640 1 -- 172.21.15.150:0/1808073947 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d4800cd90 con 0x7f1d540af980 2024-03-17T23:47:03.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.058+0000 7f1d58c9a640 1 -- 172.21.15.150:0/1808073947 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d5409cf90 con 0x7f1d540af980 2024-03-17T23:47:03.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.059+0000 7f1d42ffd640 1 -- 172.21.15.150:0/1808073947 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1d4801a070 con 0x7f1d540af980 2024-03-17T23:47:03.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.060+0000 7f1d42ffd640 1 --2- 172.21.15.150:0/1808073947 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1d3406bbd0 0x7f1d3406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.060+0000 7f1d42ffd640 1 -- 172.21.15.150:0/1808073947 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f1d4808ad70 con 0x7f1d540af980 2024-03-17T23:47:03.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.060+0000 7f1d5259c640 1 --2- 172.21.15.150:0/1808073947 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1d3406bbd0 0x7f1d3406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.060+0000 7f1d5259c640 1 --2- 172.21.15.150:0/1808073947 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1d3406bbd0 0x7f1d3406e090 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1d44035000 tx=0x7f1d44030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:03.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.060+0000 7f1d5b965740 1 Processor -- start 2024-03-17T23:47:03.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.061+0000 7f1d5b965740 1 -- start start 2024-03-17T23:47:03.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.061+0000 7f1d5b965740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b055fc80 0x55c8b0532c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.061+0000 7f1d5b965740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c8b0560060 0x55c8b0562a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.061+0000 7f1d5b965740 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c8b0533160 0x55c8b0562f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.061+0000 7f1d5b965740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55c8b049bc80 con 0x55c8b055fc80 2024-03-17T23:47:03.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.061+0000 7f1d5b965740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c8b0444aa0 con 0x55c8b0560060 2024-03-17T23:47:03.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.061+0000 7f1d5b965740 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x55c8b0497d20 con 0x55c8b0533160 2024-03-17T23:47:03.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.061+0000 7f1d5259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b055fc80 0x55c8b0532c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.061+0000 7f1d5259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b055fc80 0x55c8b0532c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32848/0 (socket says 172.21.15.150:32848) 2024-03-17T23:47:03.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.061+0000 7f1d5259c640 1 -- 172.21.15.150:0/535432842 learned_addr learned my addr 172.21.15.150:0/535432842 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:03.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.062+0000 7f1d52d9d640 1 --2- 172.21.15.150:0/535432842 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c8b0533160 0x55c8b0562f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.062+0000 7f1d51d9b640 1 --2- 172.21.15.150:0/535432842 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c8b0560060 0x55c8b0562a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.062+0000 7f1d5259c640 1 -- 172.21.15.150:0/535432842 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c8b0533160 msgr2=0x55c8b0562f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:03.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.062+0000 7f1d5259c640 1 --2- 172.21.15.150:0/535432842 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c8b0533160 0x55c8b0562f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.062+0000 7f1d5259c640 1 -- 172.21.15.150:0/535432842 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c8b0560060 msgr2=0x55c8b0562a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:03.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.062+0000 7f1d5259c640 1 --2- 172.21.15.150:0/535432842 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c8b0560060 0x55c8b0562a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.062+0000 7f1d5259c640 1 -- 172.21.15.150:0/535432842 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8b0466ff0 con 0x55c8b055fc80 2024-03-17T23:47:03.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.062+0000 7f1d5259c640 1 --2- 172.21.15.150:0/535432842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b055fc80 0x55c8b0532c20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1d44036040 tx=0x7f1d4402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23497010c7937dc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:03.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.062+0000 7f1d33fff640 1 -- 172.21.15.150:0/535432842 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d44050070 con 0x55c8b055fc80 2024-03-17T23:47:03.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.062+0000 7f1d33fff640 1 -- 172.21.15.150:0/535432842 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d4404b440 con 0x55c8b055fc80 2024-03-17T23:47:03.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.062+0000 7f1d33fff640 1 -- 172.21.15.150:0/535432842 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d4404b830 con 0x55c8b055fc80 2024-03-17T23:47:03.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.063+0000 7f1d5b965740 1 -- 172.21.15.150:0/535432842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b055fc80 msgr2=0x55c8b0532c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:03.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.063+0000 7f1d5b965740 1 --2- 172.21.15.150:0/535432842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b055fc80 0x55c8b0532c20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1d44036040 tx=0x7f1d4402e000 comp rx=0 tx=0).stop 2024-03-17T23:47:03.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.063+0000 7f1d5b965740 1 -- 172.21.15.150:0/535432842 shutdown_connections 2024-03-17T23:47:03.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.063+0000 7f1d5b965740 1 --2- 172.21.15.150:0/535432842 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c8b0533160 0x55c8b0562f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.063+0000 7f1d5b965740 1 --2- 172.21.15.150:0/535432842 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c8b0560060 0x55c8b0562a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.063+0000 7f1d5b965740 1 --2- 172.21.15.150:0/535432842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b055fc80 0x55c8b0532c20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.063+0000 7f1d5b965740 1 -- 172.21.15.150:0/535432842 >> 172.21.15.150:0/535432842 conn(0x55c8b0534e90 msgr2=0x55c8b057da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:03.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.063+0000 7f1d5b965740 1 -- 172.21.15.150:0/535432842 shutdown_connections 2024-03-17T23:47:03.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.063+0000 7f1d5b965740 1 -- 172.21.15.150:0/535432842 wait complete. 2024-03-17T23:47:03.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.063+0000 7f1d5b965740 1 -- start start 2024-03-17T23:47:03.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.063+0000 7f1d5b965740 1 Processor -- start 2024-03-17T23:47:03.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.064+0000 7f1d5b965740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c8b055fc80 0x55c8b057ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.064+0000 7f1d5b965740 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c8b0560060 0x55c8b05910d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.064+0000 7f1d5b965740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b05918b0 0x55c8b0553b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.064+0000 7f1d5b965740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55c8b0537530 con 0x55c8b05918b0 2024-03-17T23:47:03.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.064+0000 7f1d5b965740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c8b0457ef0 con 0x55c8b055fc80 2024-03-17T23:47:03.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.064+0000 7f1d5b965740 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x55c8b0539d70 con 0x55c8b0560060 2024-03-17T23:47:03.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.064+0000 7f1d5b965740 10 client.0 _ll_register_callbacks cb 0x55c8b0452dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:47:03.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.064+0000 7f1d52d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b05918b0 0x55c8b0553b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.064+0000 7f1d52d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b05918b0 0x55c8b0553b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32856/0 (socket says 172.21.15.150:32856) 2024-03-17T23:47:03.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.064+0000 7f1d52d9d640 1 -- 172.21.15.150:0/1428553764 learned_addr learned my addr 172.21.15.150:0/1428553764 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:03.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.064+0000 7f1d51d9b640 1 --2- 172.21.15.150:0/1428553764 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c8b0560060 0x55c8b05910d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.064+0000 7f1d5259c640 1 --2- 172.21.15.150:0/1428553764 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c8b055fc80 0x55c8b057ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.065+0000 7f1d52d9d640 1 -- 172.21.15.150:0/1428553764 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c8b0560060 msgr2=0x55c8b05910d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:03.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.065+0000 7f1d52d9d640 1 --2- 172.21.15.150:0/1428553764 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c8b0560060 0x55c8b05910d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.065+0000 7f1d52d9d640 1 -- 172.21.15.150:0/1428553764 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c8b055fc80 msgr2=0x55c8b057ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:03.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.065+0000 7f1d52d9d640 1 --2- 172.21.15.150:0/1428553764 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c8b055fc80 0x55c8b057ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.065+0000 7f1d52d9d640 1 -- 172.21.15.150:0/1428553764 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8b0464a30 con 0x55c8b05918b0 2024-03-17T23:47:03.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.065+0000 7f1d52d9d640 1 --2- 172.21.15.150:0/1428553764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b05918b0 0x55c8b0553b40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c001ea0 tx=0x7f1d4c012750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e2ee4c298a0051c server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:03.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.065+0000 7f1d317fa640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-17T23:47:03.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.065+0000 7f1d317fa640 1 -- 172.21.15.150:0/1428553764 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d4c019020 con 0x55c8b05918b0 2024-03-17T23:47:03.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.065+0000 7f1d5b965740 1 -- 172.21.15.150:0/1428553764 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d4404b440 con 0x55c8b05918b0 2024-03-17T23:47:03.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.065+0000 7f1d317fa640 1 -- 172.21.15.150:0/1428553764 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f1d4c0052e0 con 0x55c8b05918b0 2024-03-17T23:47:03.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.065+0000 7f1d317fa640 1 -- 172.21.15.150:0/1428553764 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) ==== 24+0+0 (secure 0 0 0) 0x7f1d4c023050 con 0x55c8b05918b0 2024-03-17T23:47:03.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.065+0000 7f1d5b965740 10 client.4413 fetch_fsmap learned FSMap version 6 2024-03-17T23:47:03.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.065+0000 7f1d5b965740 1 -- 172.21.15.150:0/1428553764 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) -- 0x55c8b01b02d0 con 0x55c8b05918b0 2024-03-17T23:47:03.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.065+0000 7f1d317fa640 1 -- 172.21.15.150:0/1428553764 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d4404b440 con 0x55c8b05918b0 2024-03-17T23:47:03.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.065+0000 7f1d317fa640 1 -- 172.21.15.150:0/1428553764 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 6) ==== 3502+0+0 (secure 0 0 0) 0x7f1d4c026020 con 0x55c8b05918b0 2024-03-17T23:47:03.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.065+0000 7f1d5b965740 10 client.4413 fetch_fsmap finished waiting for FSMap version 6 2024-03-17T23:47:03.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.065+0000 7f1d5b965740 10 client.4413 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-17T23:47:03.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.065+0000 7f1d5b965740 20 client.4413 populate_metadata read hostname 'smithi150' 2024-03-17T23:47:03.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.065+0000 7f1d5b965740 1 --2- 172.21.15.150:0/1428553764 >> [v2:172.21.15.157:6832/1625058792,v1:172.21.15.157:6834/1625058792] conn(0x55c8b0567590 0x55c8b0565330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.065+0000 7f1d5b965740 4 client.4413 mds_command: new command op to 4297 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-17T23:47:03.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.065+0000 7f1d5b965740 1 -- 172.21.15.150:0/1428553764 --> [v2:172.21.15.157:6832/1625058792,v1:172.21.15.157:6834/1625058792] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8b04a91c0 con 0x55c8b0567590 2024-03-17T23:47:03.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.066+0000 7f1d5259c640 1 --2- 172.21.15.150:0/1428553764 >> [v2:172.21.15.157:6832/1625058792,v1:172.21.15.157:6834/1625058792] conn(0x55c8b0567590 0x55c8b0565330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.066+0000 7f1d5259c640 1 --2- 172.21.15.150:0/1428553764 >> [v2:172.21.15.157:6832/1625058792,v1:172.21.15.157:6834/1625058792] conn(0x55c8b0567590 0x55c8b0565330 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cf168ee067bd6ca server_cookie=bc2a4c57094d6459 in_seq=0 out_seq=0 2024-03-17T23:47:03.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.066+0000 7f1d317fa640 10 client.4413 ms_handle_connect on v2:172.21.15.157:6832/1625058792 2024-03-17T23:47:03.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.066+0000 7f1d317fa640 1 -- 172.21.15.150:0/1428553764 <== mds.0 v2:172.21.15.157:6832/1625058792 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c8b04a91c0 con 0x55c8b0567590 2024-03-17T23:47:03.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.066+0000 7f1d317fa640 10 client.4413 handle_command_reply: tid=0 2024-03-17T23:47:03.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.066+0000 7f1d5b965740 1 client.4413 shutdown 2024-03-17T23:47:03.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.067+0000 7f1d5b965740 1 -- 172.21.15.150:0/1428553764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b05918b0 msgr2=0x55c8b0553b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:03.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.067+0000 7f1d5b965740 1 --2- 172.21.15.150:0/1428553764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b05918b0 0x55c8b0553b40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c001ea0 tx=0x7f1d4c012750 comp rx=0 tx=0).stop 2024-03-17T23:47:03.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.067+0000 7f1d5b965740 1 -- 172.21.15.150:0/1428553764 shutdown_connections 2024-03-17T23:47:03.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.067+0000 7f1d5b965740 1 --2- 172.21.15.150:0/1428553764 >> [v2:172.21.15.157:6832/1625058792,v1:172.21.15.157:6834/1625058792] conn(0x55c8b0567590 0x55c8b0565330 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.067+0000 7f1d5b965740 1 --2- 172.21.15.150:0/1428553764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b05918b0 0x55c8b0553b40 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.067+0000 7f1d317fa640 0 client.4413 ms_handle_reset on v2:172.21.15.157:6832/1625058792 2024-03-17T23:47:03.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.067+0000 7f1d5b965740 1 --2- 172.21.15.150:0/1428553764 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c8b0560060 0x55c8b05910d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.067+0000 7f1d5b965740 1 --2- 172.21.15.150:0/1428553764 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c8b055fc80 0x55c8b057ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.067+0000 7f1d5b965740 1 -- 172.21.15.150:0/1428553764 >> 172.21.15.150:0/1428553764 conn(0x55c8b0533160 msgr2=0x55c8b0536b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:03.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.067+0000 7f1d5b965740 1 -- 172.21.15.150:0/1428553764 shutdown_connections 2024-03-17T23:47:03.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.067+0000 7f1d5b965740 1 -- 172.21.15.150:0/1428553764 wait complete. 2024-03-17T23:47:03.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.067+0000 7f1d5b965740 20 client.4413 trim_cache size 0 max 16384 2024-03-17T23:47:03.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.085+0000 7f1d5b965740 1 Processor -- start 2024-03-17T23:47:03.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.085+0000 7f1d5b965740 1 -- start start 2024-03-17T23:47:03.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.085+0000 7f1d5b965740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b055fc80 0x55c8b05624e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.085+0000 7f1d5b965740 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c8b0560060 0x55c8b0580020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.085+0000 7f1d5b965740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c8b05918b0 0x55c8b0562a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.086+0000 7f1d5b965740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1d542008f0 con 0x55c8b055fc80 2024-03-17T23:47:03.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.086+0000 7f1d5b965740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1d541ffd80 con 0x55c8b05918b0 2024-03-17T23:47:03.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.086+0000 7f1d5b965740 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1d542002a0 con 0x55c8b0560060 2024-03-17T23:47:03.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.086+0000 7f1d5259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b055fc80 0x55c8b05624e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.086+0000 7f1d5259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b055fc80 0x55c8b05624e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32858/0 (socket says 172.21.15.150:32858) 2024-03-17T23:47:03.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.086+0000 7f1d5259c640 1 -- 172.21.15.150:0/3358866799 learned_addr learned my addr 172.21.15.150:0/3358866799 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:03.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.086+0000 7f1d51d9b640 1 --2- 172.21.15.150:0/3358866799 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c8b0560060 0x55c8b0580020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.086+0000 7f1d52d9d640 1 --2- 172.21.15.150:0/3358866799 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c8b05918b0 0x55c8b0562a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.086+0000 7f1d5259c640 1 -- 172.21.15.150:0/3358866799 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c8b0560060 msgr2=0x55c8b0580020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:03.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.086+0000 7f1d5259c640 1 --2- 172.21.15.150:0/3358866799 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c8b0560060 0x55c8b0580020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.086+0000 7f1d5259c640 1 -- 172.21.15.150:0/3358866799 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c8b05918b0 msgr2=0x55c8b0562a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:03.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.086+0000 7f1d5259c640 1 --2- 172.21.15.150:0/3358866799 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c8b05918b0 0x55c8b0562a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.086+0000 7f1d5259c640 1 -- 172.21.15.150:0/3358866799 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8b027a6b0 con 0x55c8b055fc80 2024-03-17T23:47:03.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.086+0000 7f1d5259c640 1 --2- 172.21.15.150:0/3358866799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b055fc80 0x55c8b05624e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1d44009c40 tx=0x7f1d4403f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b746d5c639f463c server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:03.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.086+0000 7f1d337fe640 1 -- 172.21.15.150:0/3358866799 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d44060020 con 0x55c8b055fc80 2024-03-17T23:47:03.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.086+0000 7f1d337fe640 1 -- 172.21.15.150:0/3358866799 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d4404bb30 con 0x55c8b055fc80 2024-03-17T23:47:03.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.086+0000 7f1d337fe640 1 -- 172.21.15.150:0/3358866799 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d44050040 con 0x55c8b055fc80 2024-03-17T23:47:03.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.087+0000 7f1d5b965740 1 -- 172.21.15.150:0/3358866799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b055fc80 msgr2=0x55c8b05624e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:03.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.087+0000 7f1d5b965740 1 --2- 172.21.15.150:0/3358866799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b055fc80 0x55c8b05624e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1d44009c40 tx=0x7f1d4403f000 comp rx=0 tx=0).stop 2024-03-17T23:47:03.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.087+0000 7f1d5b965740 1 -- 172.21.15.150:0/3358866799 shutdown_connections 2024-03-17T23:47:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.087+0000 7f1d5b965740 1 --2- 172.21.15.150:0/3358866799 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c8b05918b0 0x55c8b0562a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.087+0000 7f1d5b965740 1 --2- 172.21.15.150:0/3358866799 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c8b0560060 0x55c8b0580020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.087+0000 7f1d5b965740 1 --2- 172.21.15.150:0/3358866799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b055fc80 0x55c8b05624e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.087+0000 7f1d5b965740 1 -- 172.21.15.150:0/3358866799 >> 172.21.15.150:0/3358866799 conn(0x55c8b0533160 msgr2=0x55c8b0538d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.087+0000 7f1d5b965740 1 -- 172.21.15.150:0/3358866799 shutdown_connections 2024-03-17T23:47:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.087+0000 7f1d5b965740 1 -- 172.21.15.150:0/3358866799 wait complete. 2024-03-17T23:47:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.087+0000 7f1d5b965740 1 -- start start 2024-03-17T23:47:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.087+0000 7f1d5b965740 1 Processor -- start 2024-03-17T23:47:03.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.088+0000 7f1d5b965740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c8b055fc80 0x55c8b05f6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.088+0000 7f1d5b965740 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c8b0560060 0x55c8b05f7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.088+0000 7f1d5b965740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b05918b0 0x55c8b05655a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.088+0000 7f1d5b965740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55c8b0567410 con 0x55c8b05918b0 2024-03-17T23:47:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.088+0000 7f1d5b965740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x55c8b0564820 con 0x55c8b055fc80 2024-03-17T23:47:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.088+0000 7f1d5b965740 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x55c8b0567a60 con 0x55c8b0560060 2024-03-17T23:47:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.088+0000 7f1d5b965740 10 client.0 _ll_register_callbacks cb 0x7f1d4c0012b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:47:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.088+0000 7f1d52d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b05918b0 0x55c8b05655a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.088+0000 7f1d52d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b05918b0 0x55c8b05655a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32870/0 (socket says 172.21.15.150:32870) 2024-03-17T23:47:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.088+0000 7f1d5259c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c8b055fc80 0x55c8b05f6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.088+0000 7f1d52d9d640 1 -- 172.21.15.150:0/3439794093 learned_addr learned my addr 172.21.15.150:0/3439794093 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.088+0000 7f1d51d9b640 1 --2- 172.21.15.150:0/3439794093 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c8b0560060 0x55c8b05f7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.089+0000 7f1d52d9d640 1 -- 172.21.15.150:0/3439794093 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c8b0560060 msgr2=0x55c8b05f7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.089+0000 7f1d52d9d640 1 --2- 172.21.15.150:0/3439794093 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c8b0560060 0x55c8b05f7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.089+0000 7f1d52d9d640 1 -- 172.21.15.150:0/3439794093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c8b055fc80 msgr2=0x55c8b05f6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.089+0000 7f1d52d9d640 1 --2- 172.21.15.150:0/3439794093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c8b055fc80 0x55c8b05f6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.089+0000 7f1d52d9d640 1 -- 172.21.15.150:0/3439794093 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8b0562c30 con 0x55c8b05918b0 2024-03-17T23:47:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.089+0000 7f1d52d9d640 1 --2- 172.21.15.150:0/3439794093 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b05918b0 0x55c8b05655a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c001e70 tx=0x7f1d4c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ecddd0cbf9fee02 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.089+0000 7f1d317fa640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-17T23:47:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.089+0000 7f1d317fa640 1 -- 172.21.15.150:0/3439794093 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d4c019070 con 0x55c8b05918b0 2024-03-17T23:47:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.089+0000 7f1d317fa640 1 -- 172.21.15.150:0/3439794093 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f1d4c0067d0 con 0x55c8b05918b0 2024-03-17T23:47:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.089+0000 7f1d5b965740 1 -- 172.21.15.150:0/3439794093 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d4404bb30 con 0x55c8b05918b0 2024-03-17T23:47:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.089+0000 7f1d317fa640 1 -- 172.21.15.150:0/3439794093 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) ==== 24+0+0 (secure 0 0 0) 0x7f1d4c033050 con 0x55c8b05918b0 2024-03-17T23:47:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.089+0000 7f1d5b965740 10 client.4419 fetch_fsmap learned FSMap version 6 2024-03-17T23:47:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.089+0000 7f1d5b965740 1 -- 172.21.15.150:0/3439794093 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) -- 0x55c8b05f7a30 con 0x55c8b05918b0 2024-03-17T23:47:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.089+0000 7f1d317fa640 1 -- 172.21.15.150:0/3439794093 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d4404bb30 con 0x55c8b05918b0 2024-03-17T23:47:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.089+0000 7f1d317fa640 1 -- 172.21.15.150:0/3439794093 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 6) ==== 3502+0+0 (secure 0 0 0) 0x7f1d4c036020 con 0x55c8b05918b0 2024-03-17T23:47:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.089+0000 7f1d5b965740 10 client.4419 fetch_fsmap finished waiting for FSMap version 6 2024-03-17T23:47:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.089+0000 7f1d5b965740 10 client.4419 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-17T23:47:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.089+0000 7f1d5b965740 20 client.4419 populate_metadata read hostname 'smithi150' 2024-03-17T23:47:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.089+0000 7f1d5b965740 1 --2- 172.21.15.150:0/3439794093 >> [v2:172.21.15.157:6832/1625058792,v1:172.21.15.157:6834/1625058792] conn(0x55c8b0565bb0 0x55c8b057b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.089+0000 7f1d5b965740 4 client.4419 mds_command: new command op to 4297 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-17T23:47:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.089+0000 7f1d5b965740 1 -- 172.21.15.150:0/3439794093 --> [v2:172.21.15.157:6832/1625058792,v1:172.21.15.157:6834/1625058792] -- command(tid 0: {"prefix": "status"}) -- 0x55c8b04a3510 con 0x55c8b0565bb0 2024-03-17T23:47:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.089+0000 7f1d5259c640 1 --2- 172.21.15.150:0/3439794093 >> [v2:172.21.15.157:6832/1625058792,v1:172.21.15.157:6834/1625058792] conn(0x55c8b0565bb0 0x55c8b057b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.090+0000 7f1d317fa640 10 client.4419 ms_handle_connect on v2:172.21.15.157:6832/1625058792 2024-03-17T23:47:03.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.090+0000 7f1d5259c640 1 --2- 172.21.15.150:0/3439794093 >> [v2:172.21.15.157:6832/1625058792,v1:172.21.15.157:6834/1625058792] conn(0x55c8b0565bb0 0x55c8b057b8a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd1e93e8d089cb88 server_cookie=6e6298f5dcc7d295 in_seq=0 out_seq=0 2024-03-17T23:47:03.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.090+0000 7f1d317fa640 1 -- 172.21.15.150:0/3439794093 <== mds.0 v2:172.21.15.157:6832/1625058792 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x55c8b04a3510 con 0x55c8b0565bb0 2024-03-17T23:47:03.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.090+0000 7f1d317fa640 10 client.4419 handle_command_reply: tid=0 2024-03-17T23:47:03.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.090+0000 7f1d5b965740 1 client.4419 shutdown 2024-03-17T23:47:03.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.090+0000 7f1d5b965740 1 -- 172.21.15.150:0/3439794093 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b05918b0 msgr2=0x55c8b05655a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:03.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.090+0000 7f1d5b965740 1 --2- 172.21.15.150:0/3439794093 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b05918b0 0x55c8b05655a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c001e70 tx=0x7f1d4c015000 comp rx=0 tx=0).stop 2024-03-17T23:47:03.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.090+0000 7f1d5b965740 1 -- 172.21.15.150:0/3439794093 shutdown_connections 2024-03-17T23:47:03.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.090+0000 7f1d5b965740 1 --2- 172.21.15.150:0/3439794093 >> [v2:172.21.15.157:6832/1625058792,v1:172.21.15.157:6834/1625058792] conn(0x55c8b0565bb0 0x55c8b057b8a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.090+0000 7f1d5b965740 1 --2- 172.21.15.150:0/3439794093 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c8b05918b0 0x55c8b05655a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.090+0000 7f1d5b965740 1 --2- 172.21.15.150:0/3439794093 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x55c8b0560060 0x55c8b05f7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.090+0000 7f1d317fa640 0 client.4419 ms_handle_reset on v2:172.21.15.157:6832/1625058792 2024-03-17T23:47:03.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.090+0000 7f1d5b965740 1 --2- 172.21.15.150:0/3439794093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c8b055fc80 0x55c8b05f6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.090+0000 7f1d5b965740 1 -- 172.21.15.150:0/3439794093 >> 172.21.15.150:0/3439794093 conn(0x55c8b0533160 msgr2=0x55c8b055e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:03.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.090+0000 7f1d5b965740 1 -- 172.21.15.150:0/3439794093 shutdown_connections 2024-03-17T23:47:03.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.090+0000 7f1d5b965740 1 -- 172.21.15.150:0/3439794093 wait complete. 2024-03-17T23:47:03.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.090+0000 7f1d5b965740 20 client.4419 trim_cache size 0 max 16384 2024-03-17T23:47:03.101 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-17T23:47:03.101 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_fsid": "a1c5acd3-e97f-470b-8226-335736a7795b", 2024-03-17T23:47:03.101 INFO:teuthology.orchestra.run.smithi150.stdout: "whoami": 0, 2024-03-17T23:47:03.101 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 4297, 2024-03-17T23:47:03.101 INFO:teuthology.orchestra.run.smithi150.stdout: "want_state": "up:active", 2024-03-17T23:47:03.102 INFO:teuthology.orchestra.run.smithi150.stdout: "state": "up:active", 2024-03-17T23:47:03.102 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "cephfs", 2024-03-17T23:47:03.102 INFO:teuthology.orchestra.run.smithi150.stdout: "rank_uptime": 7.1689462820000003, 2024-03-17T23:47:03.102 INFO:teuthology.orchestra.run.smithi150.stdout: "mdsmap_epoch": 6, 2024-03-17T23:47:03.102 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch": 26, 2024-03-17T23:47:03.102 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch_barrier": 25, 2024-03-17T23:47:03.102 INFO:teuthology.orchestra.run.smithi150.stdout: "uptime": 13.898868863000001 2024-03-17T23:47:03.102 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-17T23:47:03.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.091+0000 7f1d58c9a640 1 -- 172.21.15.150:0/1808073947 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1d3406bbd0 msgr2=0x7f1d3406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:03.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.091+0000 7f1d58c9a640 1 --2- 172.21.15.150:0/1808073947 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1d3406bbd0 0x7f1d3406e090 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1d44035000 tx=0x7f1d44030000 comp rx=0 tx=0).stop 2024-03-17T23:47:03.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.091+0000 7f1d58c9a640 1 -- 172.21.15.150:0/1808073947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d540af980 msgr2=0x7f1d541e70e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:03.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.091+0000 7f1d58c9a640 1 --2- 172.21.15.150:0/1808073947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d540af980 0x7f1d541e70e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1d4800e9b0 tx=0x7f1d4800ee70 comp rx=0 tx=0).stop 2024-03-17T23:47:03.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.091+0000 7f1d58c9a640 1 -- 172.21.15.150:0/1808073947 shutdown_connections 2024-03-17T23:47:03.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.091+0000 7f1d58c9a640 1 --2- 172.21.15.150:0/1808073947 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1d3406bbd0 0x7f1d3406e090 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.091+0000 7f1d58c9a640 1 --2- 172.21.15.150:0/1808073947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1d5415e1a0 0x7f1d540aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.091+0000 7f1d58c9a640 1 --2- 172.21.15.150:0/1808073947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d540af980 0x7f1d541e70e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.091+0000 7f1d58c9a640 1 --2- 172.21.15.150:0/1808073947 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1d54099a70 0x7f1d541f3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.091+0000 7f1d58c9a640 1 -- 172.21.15.150:0/1808073947 >> 172.21.15.150:0/1808073947 conn(0x7f1d54072a60 msgr2=0x7f1d5406f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:03.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.091+0000 7f1d58c9a640 1 -- 172.21.15.150:0/1808073947 shutdown_connections 2024-03-17T23:47:03.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.091+0000 7f1d58c9a640 1 -- 172.21.15.150:0/1808073947 wait complete. 2024-03-17T23:47:03.105 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-17T23:47:03.106 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-17T23:47:03.106 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-17T23:47:03.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.204+0000 7f66d32e8640 1 Processor -- start 2024-03-17T23:47:03.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.204+0000 7f66d32e8640 1 -- start start 2024-03-17T23:47:03.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.205+0000 7f66d32e8640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f66cc09a470 0x7f66cc09a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.205+0000 7f66d32e8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f66cc09ad90 0x7f66cc15f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.205+0000 7f66d32e8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66cc15fcd0 0x7f66cc15c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.205+0000 7f66d32e8640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f66cc160100 con 0x7f66cc09ad90 2024-03-17T23:47:03.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.205+0000 7f66d32e8640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f66cc0cfad0 con 0x7f66cc09a470 2024-03-17T23:47:03.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.205+0000 7f66d32e8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f66cc0cfdb0 con 0x7f66cc15fcd0 2024-03-17T23:47:03.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.205+0000 7f66d0883640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f66cc09ad90 0x7f66cc15f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.205+0000 7f66d0883640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f66cc09ad90 0x7f66cc15f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32876/0 (socket says 172.21.15.150:32876) 2024-03-17T23:47:03.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.205+0000 7f66d0883640 1 -- 172.21.15.150:0/3239546964 learned_addr learned my addr 172.21.15.150:0/3239546964 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:03.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.205+0000 7f66d1084640 1 --2- 172.21.15.150:0/3239546964 >> v2:172.21.15.157:3300/0 conn(0x7f66cc09a470 0x7f66cc09a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.205+0000 7f66d1885640 1 --2- 172.21.15.150:0/3239546964 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66cc15fcd0 0x7f66cc15c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.205+0000 7f66d0883640 1 -- 172.21.15.150:0/3239546964 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66cc15fcd0 msgr2=0x7f66cc15c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:03.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.205+0000 7f66d0883640 1 --2- 172.21.15.150:0/3239546964 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66cc15fcd0 0x7f66cc15c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.205+0000 7f66d0883640 1 -- 172.21.15.150:0/3239546964 >> v2:172.21.15.157:3300/0 conn(0x7f66cc09a470 msgr2=0x7f66cc09a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:03.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.205+0000 7f66d0883640 1 --2- 172.21.15.150:0/3239546964 >> v2:172.21.15.157:3300/0 conn(0x7f66cc09a470 0x7f66cc09a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.205+0000 7f66d0883640 1 -- 172.21.15.150:0/3239546964 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66cc0d0090 con 0x7f66cc09ad90 2024-03-17T23:47:03.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.205+0000 7f66d1885640 1 --2- 172.21.15.150:0/3239546964 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66cc15fcd0 0x7f66cc15c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:03.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.206+0000 7f66d0883640 1 --2- 172.21.15.150:0/3239546964 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66cc09ad90 0x7f66cc15f790 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f66cc09b740 tx=0x7f66bc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60d3b4b3d2abc1f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:03.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.206+0000 7f66c3fff640 1 -- 172.21.15.150:0/3239546964 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66bc035020 con 0x7f66cc09ad90 2024-03-17T23:47:03.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.206+0000 7f66c3fff640 1 -- 172.21.15.150:0/3239546964 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66bc031870 con 0x7f66cc09ad90 2024-03-17T23:47:03.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.206+0000 7f66c3fff640 1 -- 172.21.15.150:0/3239546964 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66bc031c40 con 0x7f66cc09ad90 2024-03-17T23:47:03.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.206+0000 7f66d32e8640 1 -- 172.21.15.150:0/3239546964 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66cc09ad90 msgr2=0x7f66cc15f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:03.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.206+0000 7f66d32e8640 1 --2- 172.21.15.150:0/3239546964 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66cc09ad90 0x7f66cc15f790 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f66cc09b740 tx=0x7f66bc02b910 comp rx=0 tx=0).stop 2024-03-17T23:47:03.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.207+0000 7f66d32e8640 1 -- 172.21.15.150:0/3239546964 shutdown_connections 2024-03-17T23:47:03.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.207+0000 7f66d32e8640 1 --2- 172.21.15.150:0/3239546964 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66cc09ad90 0x7f66cc15f790 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.207+0000 7f66d32e8640 1 --2- 172.21.15.150:0/3239546964 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66cc15fcd0 0x7f66cc15c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.207+0000 7f66d32e8640 1 --2- 172.21.15.150:0/3239546964 >> v2:172.21.15.157:3300/0 conn(0x7f66cc09a470 0x7f66cc09a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.207+0000 7f66d32e8640 1 -- 172.21.15.150:0/3239546964 >> 172.21.15.150:0/3239546964 conn(0x7f66cc0b3630 msgr2=0x7f66cc0b3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:03.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.207+0000 7f66d32e8640 1 -- 172.21.15.150:0/3239546964 shutdown_connections 2024-03-17T23:47:03.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.207+0000 7f66d32e8640 1 -- 172.21.15.150:0/3239546964 wait complete. 2024-03-17T23:47:03.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.208+0000 7f66d32e8640 1 Processor -- start 2024-03-17T23:47:03.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.208+0000 7f66d32e8640 1 -- start start 2024-03-17T23:47:03.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.208+0000 7f66d32e8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f66cc09a470 0x7f66cc0cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.208+0000 7f66d32e8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66cc09ad90 0x7f66cc14e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.208+0000 7f66d32e8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66cc15fcd0 0x7f66cc0cea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.208+0000 7f66d32e8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f66cc0cf710 con 0x7f66cc09ad90 2024-03-17T23:47:03.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.208+0000 7f66d32e8640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f66cc0d0650 con 0x7f66cc09a470 2024-03-17T23:47:03.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.208+0000 7f66d32e8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f66cc0cf890 con 0x7f66cc15fcd0 2024-03-17T23:47:03.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.209+0000 7f66d0883640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66cc09ad90 0x7f66cc14e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.209+0000 7f66d0883640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66cc09ad90 0x7f66cc14e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:32886/0 (socket says 172.21.15.150:32886) 2024-03-17T23:47:03.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.209+0000 7f66d0883640 1 -- 172.21.15.150:0/498958683 learned_addr learned my addr 172.21.15.150:0/498958683 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:03.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.209+0000 7f66d1084640 1 --2- 172.21.15.150:0/498958683 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f66cc09a470 0x7f66cc0cb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.209+0000 7f66d1885640 1 --2- 172.21.15.150:0/498958683 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66cc15fcd0 0x7f66cc0cea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.209+0000 7f66d0883640 1 -- 172.21.15.150:0/498958683 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66cc15fcd0 msgr2=0x7f66cc0cea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:03.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.209+0000 7f66d0883640 1 --2- 172.21.15.150:0/498958683 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66cc15fcd0 0x7f66cc0cea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.209+0000 7f66d0883640 1 -- 172.21.15.150:0/498958683 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f66cc09a470 msgr2=0x7f66cc0cb610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:03.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.209+0000 7f66d0883640 1 --2- 172.21.15.150:0/498958683 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f66cc09a470 0x7f66cc0cb610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.209+0000 7f66d0883640 1 -- 172.21.15.150:0/498958683 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66cc0ca570 con 0x7f66cc09ad90 2024-03-17T23:47:03.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.209+0000 7f66d0883640 1 --2- 172.21.15.150:0/498958683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66cc09ad90 0x7f66cc14e1c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f66bc000c00 tx=0x7f66bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:03.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.209+0000 7f66c1ffb640 1 -- 172.21.15.150:0/498958683 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66bc035040 con 0x7f66cc09ad90 2024-03-17T23:47:03.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.209+0000 7f66c1ffb640 1 -- 172.21.15.150:0/498958683 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66bc030d00 con 0x7f66cc09ad90 2024-03-17T23:47:03.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.209+0000 7f66d32e8640 1 -- 172.21.15.150:0/498958683 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66cc0ca850 con 0x7f66cc09ad90 2024-03-17T23:47:03.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.209+0000 7f66c1ffb640 1 -- 172.21.15.150:0/498958683 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66bc0456c0 con 0x7f66cc09ad90 2024-03-17T23:47:03.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.209+0000 7f66d32e8640 1 -- 172.21.15.150:0/498958683 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66cc0cae10 con 0x7f66cc09ad90 2024-03-17T23:47:03.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.210+0000 7f66d32e8640 1 -- 172.21.15.150:0/498958683 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66cc0681a0 con 0x7f66cc09ad90 2024-03-17T23:47:03.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.211+0000 7f66c1ffb640 1 -- 172.21.15.150:0/498958683 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f66bc038020 con 0x7f66cc09ad90 2024-03-17T23:47:03.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.211+0000 7f66c1ffb640 1 --2- 172.21.15.150:0/498958683 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f66b006bbd0 0x7f66b006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:03.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.211+0000 7f66c1ffb640 1 -- 172.21.15.150:0/498958683 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f66bc0ad000 con 0x7f66cc09ad90 2024-03-17T23:47:03.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.211+0000 7f66d1084640 1 --2- 172.21.15.150:0/498958683 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f66b006bbd0 0x7f66b006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:03.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.212+0000 7f66d1084640 1 --2- 172.21.15.150:0/498958683 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f66b006bbd0 0x7f66b006e090 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f66c8009a20 tx=0x7f66c8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:03.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.216+0000 7f66c1ffb640 1 -- 172.21.15.150:0/498958683 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f66bc07b7b0 con 0x7f66cc09ad90 2024-03-17T23:47:03.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.425+0000 7f66d32e8640 1 -- 172.21.15.150:0/498958683 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f66cc0b0ac0 con 0x7f66cc09ad90 2024-03-17T23:47:03.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.425+0000 7f66c1ffb640 1 -- 172.21.15.150:0/498958683 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) ==== 53+0+88 (secure 0 0 0) 0x7f66bc07b0b0 con 0x7f66cc09ad90 2024-03-17T23:47:03.427 INFO:teuthology.orchestra.run.smithi150.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-17T23:47:03.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.427+0000 7f66d32e8640 1 -- 172.21.15.150:0/498958683 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f66b006bbd0 msgr2=0x7f66b006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:03.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.427+0000 7f66d32e8640 1 --2- 172.21.15.150:0/498958683 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f66b006bbd0 0x7f66b006e090 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f66c8009a20 tx=0x7f66c8009190 comp rx=0 tx=0).stop 2024-03-17T23:47:03.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.427+0000 7f66d32e8640 1 -- 172.21.15.150:0/498958683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66cc09ad90 msgr2=0x7f66cc14e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:03.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.427+0000 7f66d32e8640 1 --2- 172.21.15.150:0/498958683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66cc09ad90 0x7f66cc14e1c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f66bc000c00 tx=0x7f66bc0046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.428+0000 7f66d32e8640 1 -- 172.21.15.150:0/498958683 shutdown_connections 2024-03-17T23:47:03.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.428+0000 7f66d32e8640 1 --2- 172.21.15.150:0/498958683 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f66cc15fcd0 0x7f66cc0cea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.428+0000 7f66d32e8640 1 --2- 172.21.15.150:0/498958683 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f66cc09ad90 0x7f66cc14e1c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.428+0000 7f66d32e8640 1 --2- 172.21.15.150:0/498958683 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f66b006bbd0 0x7f66b006e090 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.428+0000 7f66d32e8640 1 --2- 172.21.15.150:0/498958683 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f66cc09a470 0x7f66cc0cb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:03.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.428+0000 7f66d32e8640 1 -- 172.21.15.150:0/498958683 >> 172.21.15.150:0/498958683 conn(0x7f66cc0b3630 msgr2=0x7f66cc06ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:03.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.428+0000 7f66d32e8640 1 -- 172.21.15.150:0/498958683 shutdown_connections 2024-03-17T23:47:03.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03.428+0000 7f66d32e8640 1 -- 172.21.15.150:0/498958683 wait complete. 2024-03-17T23:47:03.440 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-17T23:47:03.440 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-17T23:47:03.440 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi150.front.sepia.ceph.com 2024-03-17T23:47:03.440 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-17T23:47:03.440 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-17T23:47:03.440 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-17T23:47:03.441 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-17T23:47:03.441 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-17T23:47:03.441 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-17T23:47:03.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:03.441 DEBUG:teuthology.orchestra.run.smithi150:> ip addr 2024-03-17T23:47:03.456 INFO:teuthology.orchestra.run.smithi150.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-17T23:47:03.456 INFO:teuthology.orchestra.run.smithi150.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-17T23:47:03.456 INFO:teuthology.orchestra.run.smithi150.stdout: inet 127.0.0.1/8 scope host lo 2024-03-17T23:47:03.456 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:03.456 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 ::1/128 scope host 2024-03-17T23:47:03.456 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:03.456 INFO:teuthology.orchestra.run.smithi150.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:47:03.456 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bc brd ff:ff:ff:ff:ff:ff 2024-03-17T23:47:03.456 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f0 2024-03-17T23:47:03.456 INFO:teuthology.orchestra.run.smithi150.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:47:03.456 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bd brd ff:ff:ff:ff:ff:ff 2024-03-17T23:47:03.457 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f1 2024-03-17T23:47:03.457 INFO:teuthology.orchestra.run.smithi150.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:47:03.457 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:34 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:47:03.457 INFO:teuthology.orchestra.run.smithi150.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-17T23:47:03.457 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:35 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:47:03.457 INFO:teuthology.orchestra.run.smithi150.stdout: inet 172.21.15.150/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-17T23:47:03.457 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft 42431sec preferred_lft 42431sec 2024-03-17T23:47:03.457 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::ec4:7aff:fe88:6e35/64 scope link 2024-03-17T23:47:03.457 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:03.458 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-17T23:47:03.458 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:47:03.458 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-03-17T23:47:03.458 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link add name ceph-brx type bridge 2024-03-17T23:47:03.458 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip addr flush dev ceph-brx 2024-03-17T23:47:03.458 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set ceph-brx up 2024-03-17T23:47:03.458 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-17T23:47:03.458 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-03-17T23:47:03.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:47:03.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:03.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:03.653 DEBUG:teuthology.orchestra.run.smithi150:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-17T23:47:03.681 INFO:teuthology.orchestra.run.smithi150.stdout:1 2024-03-17T23:47:03.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:03.683 DEBUG:teuthology.orchestra.run.smithi150:> route 2024-03-17T23:47:03.753 INFO:teuthology.orchestra.run.smithi150.stdout:Kernel IP routing table 2024-03-17T23:47:03.753 INFO:teuthology.orchestra.run.smithi150.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-17T23:47:03.753 INFO:teuthology.orchestra.run.smithi150.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-17T23:47:03.753 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-17T23:47:03.753 INFO:teuthology.orchestra.run.smithi150.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-17T23:47:03.754 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:47:03.754 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-03-17T23:47:03.754 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-17T23:47:03.754 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-17T23:47:03.754 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-17T23:47:03.754 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-03-17T23:47:03.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:47:04.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:04.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:04.061 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-03-17T23:47:04.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:04.076 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list-id 2024-03-17T23:47:04.132 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:47:04.132 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-03-17T23:47:04.132 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-17T23:47:04.132 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-17T23:47:04.132 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-03-17T23:47:04.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:47:04.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:04.257 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-17T23:47:04.258 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:47:04.258 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-03-17T23:47:04.258 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-17T23:47:04.258 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-17T23:47:04.258 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-17T23:47:04.258 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-17T23:47:04.258 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-17T23:47:04.258 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-03-17T23:47:04.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:47:04.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:04.401 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:47:04.401 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-03-17T23:47:04.401 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set brx.0 up 2024-03-17T23:47:04.402 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set dev brx.0 master ceph-brx 2024-03-17T23:47:04.402 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-03-17T23:47:04.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:47:04.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:04.484 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-17T23:47:04.484 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-17T23:47:04.484 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-17T23:47:04.498 INFO:teuthology.orchestra.run.smithi150.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-17T23:47:04.498 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-17T23:47:04.498 DEBUG:teuthology.orchestra.run.smithi150:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-17T23:47:04.553 DEBUG:teuthology.orchestra.run.smithi150:> sudo modprobe fuse 2024-03-17T23:47:04.620 DEBUG:teuthology.orchestra.run.smithi150:> cat /proc/self/mounts | awk '{print $2}' 2024-03-17T23:47:04.677 INFO:teuthology.orchestra.run.smithi150.stdout:/proc 2024-03-17T23:47:04.677 INFO:teuthology.orchestra.run.smithi150.stdout:/sys 2024-03-17T23:47:04.677 INFO:teuthology.orchestra.run.smithi150.stdout:/dev 2024-03-17T23:47:04.677 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/security 2024-03-17T23:47:04.677 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/shm 2024-03-17T23:47:04.677 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/pts 2024-03-17T23:47:04.677 INFO:teuthology.orchestra.run.smithi150.stdout:/run 2024-03-17T23:47:04.677 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/cgroup 2024-03-17T23:47:04.677 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/pstore 2024-03-17T23:47:04.677 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/bpf 2024-03-17T23:47:04.678 INFO:teuthology.orchestra.run.smithi150.stdout:/ 2024-03-17T23:47:04.678 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/selinux 2024-03-17T23:47:04.678 INFO:teuthology.orchestra.run.smithi150.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:47:04.678 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/hugepages 2024-03-17T23:47:04.678 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/mqueue 2024-03-17T23:47:04.678 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/debug 2024-03-17T23:47:04.678 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/tracing 2024-03-17T23:47:04.678 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-17T23:47:04.678 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/fuse/connections 2024-03-17T23:47:04.678 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/config 2024-03-17T23:47:04.678 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-sysctl.service 2024-03-17T23:47:04.678 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-17T23:47:04.678 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/nfs/rpc_pipefs 2024-03-17T23:47:04.679 INFO:teuthology.orchestra.run.smithi150.stdout:/run/user/1000 2024-03-17T23:47:04.679 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph 2024-03-17T23:47:04.679 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-0 2024-03-17T23:47:04.679 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-1 2024-03-17T23:47:04.679 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-2 2024-03-17T23:47:04.679 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-3 2024-03-17T23:47:04.679 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns 2024-03-17T23:47:04.679 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-17T23:47:04.679 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-17T23:47:04.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:04.679 DEBUG:teuthology.orchestra.run.smithi150:> ls /sys/fs/fuse/connections 2024-03-17T23:47:04.731 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-17T23:47:04.732 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-17T23:47:04.775 DEBUG:teuthology.orchestra.run.smithi150:> sudo modprobe fuse 2024-03-17T23:47:04.801 DEBUG:teuthology.orchestra.run.smithi150:> cat /proc/self/mounts | awk '{print $2}' 2024-03-17T23:47:04.858 INFO:teuthology.orchestra.run.smithi150.stdout:/proc 2024-03-17T23:47:04.858 INFO:teuthology.orchestra.run.smithi150.stdout:/sys 2024-03-17T23:47:04.859 INFO:teuthology.orchestra.run.smithi150.stdout:/dev 2024-03-17T23:47:04.859 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/security 2024-03-17T23:47:04.859 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/shm 2024-03-17T23:47:04.859 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/pts 2024-03-17T23:47:04.859 INFO:teuthology.orchestra.run.smithi150.stdout:/run 2024-03-17T23:47:04.859 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/cgroup 2024-03-17T23:47:04.859 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/pstore 2024-03-17T23:47:04.859 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/bpf 2024-03-17T23:47:04.859 INFO:teuthology.orchestra.run.smithi150.stdout:/ 2024-03-17T23:47:04.859 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/selinux 2024-03-17T23:47:04.859 INFO:teuthology.orchestra.run.smithi150.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:47:04.859 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/hugepages 2024-03-17T23:47:04.859 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/mqueue 2024-03-17T23:47:04.859 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/debug 2024-03-17T23:47:04.860 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/tracing 2024-03-17T23:47:04.860 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-17T23:47:04.860 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/fuse/connections 2024-03-17T23:47:04.860 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/config 2024-03-17T23:47:04.860 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-sysctl.service 2024-03-17T23:47:04.860 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-17T23:47:04.860 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/nfs/rpc_pipefs 2024-03-17T23:47:04.860 INFO:teuthology.orchestra.run.smithi150.stdout:/run/user/1000 2024-03-17T23:47:04.860 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph 2024-03-17T23:47:04.860 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-0 2024-03-17T23:47:04.860 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-1 2024-03-17T23:47:04.860 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-2 2024-03-17T23:47:04.860 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-3 2024-03-17T23:47:04.860 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns 2024-03-17T23:47:04.860 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-17T23:47:04.861 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-17T23:47:04.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:04.861 DEBUG:teuthology.orchestra.run.smithi150:> ls /sys/fs/fuse/connections 2024-03-17T23:47:04.863 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi150.stderr:2024-03-17T23:47:04.862+0000 7f071ea52480 -1 init, newargv = 0x55bebedae240 newargc=15 2024-03-17T23:47:04.863 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi150.stderr:ceph-fuse[44808]: starting ceph client 2024-03-17T23:47:04.869 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi150.stderr:ceph-fuse[44808]: starting fuse 2024-03-17T23:47:04.878 INFO:teuthology.orchestra.run.smithi150.stdout:43 2024-03-17T23:47:04.878 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-17T23:47:04.879 DEBUG:teuthology.orchestra.run.smithi150:> sudo stdin-killer -- python3 -c ' 2024-03-17T23:47:04.879 DEBUG:teuthology.orchestra.run.smithi150:> import glob 2024-03-17T23:47:04.879 DEBUG:teuthology.orchestra.run.smithi150:> import re 2024-03-17T23:47:04.879 DEBUG:teuthology.orchestra.run.smithi150:> import os 2024-03-17T23:47:04.879 DEBUG:teuthology.orchestra.run.smithi150:> import subprocess 2024-03-17T23:47:04.879 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:47:04.879 DEBUG:teuthology.orchestra.run.smithi150:> def _find_admin_socket(client_name): 2024-03-17T23:47:04.879 DEBUG:teuthology.orchestra.run.smithi150:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-17T23:47:04.879 DEBUG:teuthology.orchestra.run.smithi150:> files = glob.glob(asok_path) 2024-03-17T23:47:04.879 DEBUG:teuthology.orchestra.run.smithi150:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-17T23:47:04.879 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:47:04.879 DEBUG:teuthology.orchestra.run.smithi150:> # Given a non-glob path, it better be there 2024-03-17T23:47:04.879 DEBUG:teuthology.orchestra.run.smithi150:> if "*" not in asok_path: 2024-03-17T23:47:04.879 DEBUG:teuthology.orchestra.run.smithi150:> assert(len(files) == 1) 2024-03-17T23:47:04.880 DEBUG:teuthology.orchestra.run.smithi150:> return files[0] 2024-03-17T23:47:04.880 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:47:04.880 DEBUG:teuthology.orchestra.run.smithi150:> for f in files: 2024-03-17T23:47:04.880 DEBUG:teuthology.orchestra.run.smithi150:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-17T23:47:04.880 DEBUG:teuthology.orchestra.run.smithi150:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-17T23:47:04.880 DEBUG:teuthology.orchestra.run.smithi150:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-17T23:47:04.880 DEBUG:teuthology.orchestra.run.smithi150:> contents = proc_f.read() 2024-03-17T23:47:04.880 DEBUG:teuthology.orchestra.run.smithi150:> if mountpoint in contents: 2024-03-17T23:47:04.880 DEBUG:teuthology.orchestra.run.smithi150:> return f 2024-03-17T23:47:04.880 DEBUG:teuthology.orchestra.run.smithi150:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-17T23:47:04.880 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:47:04.880 DEBUG:teuthology.orchestra.run.smithi150:> print(_find_admin_socket("client.admin")) 2024-03-17T23:47:04.880 DEBUG:teuthology.orchestra.run.smithi150:> ' 2024-03-17T23:47:05.000 INFO:teuthology.orchestra.run.smithi150.stdout:/var/run/ceph/ceph-client.admin.44808.asok 2024-03-17T23:47:05.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:05.011 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.44808.asok 2024-03-17T23:47:05.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:05.011 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.44808.asok status 2024-03-17T23:47:05.147 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-17T23:47:05.147 INFO:teuthology.orchestra.run.smithi150.stdout: "metadata": { 2024-03-17T23:47:05.147 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-17T23:47:05.147 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-17T23:47:05.147 INFO:teuthology.orchestra.run.smithi150.stdout: "entity_id": "admin", 2024-03-17T23:47:05.147 INFO:teuthology.orchestra.run.smithi150.stdout: "hostname": "smithi150", 2024-03-17T23:47:05.147 INFO:teuthology.orchestra.run.smithi150.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-17T23:47:05.147 INFO:teuthology.orchestra.run.smithi150.stdout: "pid": "44808", 2024-03-17T23:47:05.147 INFO:teuthology.orchestra.run.smithi150.stdout: "root": "/" 2024-03-17T23:47:05.147 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:47:05.148 INFO:teuthology.orchestra.run.smithi150.stdout: "dentry_count": 0, 2024-03-17T23:47:05.148 INFO:teuthology.orchestra.run.smithi150.stdout: "dentry_pinned_count": 0, 2024-03-17T23:47:05.148 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 4431, 2024-03-17T23:47:05.148 INFO:teuthology.orchestra.run.smithi150.stdout: "inst": { 2024-03-17T23:47:05.148 INFO:teuthology.orchestra.run.smithi150.stdout: "name": { 2024-03-17T23:47:05.148 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "client", 2024-03-17T23:47:05.148 INFO:teuthology.orchestra.run.smithi150.stdout: "num": 4431 2024-03-17T23:47:05.148 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:47:05.148 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": { 2024-03-17T23:47:05.148 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v1", 2024-03-17T23:47:05.148 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "192.168.0.1:0", 2024-03-17T23:47:05.148 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3251121707 2024-03-17T23:47:05.148 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:47:05.148 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:47:05.149 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": { 2024-03-17T23:47:05.149 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v1", 2024-03-17T23:47:05.149 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "192.168.0.1:0", 2024-03-17T23:47:05.149 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3251121707 2024-03-17T23:47:05.149 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:47:05.149 INFO:teuthology.orchestra.run.smithi150.stdout: "inst_str": "client.4431 192.168.0.1:0/3251121707", 2024-03-17T23:47:05.149 INFO:teuthology.orchestra.run.smithi150.stdout: "addr_str": "192.168.0.1:0/3251121707", 2024-03-17T23:47:05.149 INFO:teuthology.orchestra.run.smithi150.stdout: "inode_count": 1, 2024-03-17T23:47:05.149 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_epoch": 6, 2024-03-17T23:47:05.149 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_epoch": 26, 2024-03-17T23:47:05.149 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_epoch_barrier": 0, 2024-03-17T23:47:05.149 INFO:teuthology.orchestra.run.smithi150.stdout: "blocklisted": false, 2024-03-17T23:47:05.149 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "cephfs" 2024-03-17T23:47:05.150 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-17T23:47:05.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:05.155 DEBUG:teuthology.orchestra.run.smithi150:> stat --file-system '--printf=%T 2024-03-17T23:47:05.155 DEBUG:teuthology.orchestra.run.smithi150:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-17T23:47:05.171 INFO:teuthology.orchestra.run.smithi150.stdout:fuseblk 2024-03-17T23:47:05.171 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-17T23:47:05.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:05.171 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-17T23:47:05.239 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-17T23:47:05.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-17T23:47:05.349 INFO:teuthology.orchestra.run.smithi150.stdout:# file: . 2024-03-17T23:47:05.350 INFO:teuthology.orchestra.run.smithi150.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-17T23:47:05.350 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:05.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:05.355 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:47:05.356 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:47:05.371 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:05.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.510+0000 7f50809a9640 1 Processor -- start 2024-03-17T23:47:05.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.510+0000 7f50809a9640 1 -- start start 2024-03-17T23:47:05.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.511+0000 7f50809a9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f507c15e060 0x7f507c0d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:05.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.511+0000 7f50809a9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f507c0b57d0 0x7f507c0b5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:05.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.511+0000 7f50809a9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f507c0d2890 con 0x7f507c06c6e0 2024-03-17T23:47:05.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.511+0000 7f50809a9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f507c0d2b70 con 0x7f507c15e060 2024-03-17T23:47:05.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.511+0000 7f50809a9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f507c0d2e50 con 0x7f507c0b57d0 2024-03-17T23:47:05.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.511+0000 7f507a59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f507c15e060 0x7f507c0d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:05.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.511+0000 7f507a59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f507c15e060 0x7f507c0d1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36012/0 (socket says 172.21.15.150:36012) 2024-03-17T23:47:05.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.511+0000 7f507a59c640 1 -- 172.21.15.150:0/1489559988 learned_addr learned my addr 172.21.15.150:0/1489559988 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:05.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.511+0000 7f5079d9b640 1 --2- 172.21.15.150:0/1489559988 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f507c0b57d0 0x7f507c0b5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:05.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.511+0000 7f507a59c640 1 -- 172.21.15.150:0/1489559988 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f507c0b57d0 msgr2=0x7f507c0b5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:05.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.511+0000 7f507a59c640 1 --2- 172.21.15.150:0/1489559988 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f507c0b57d0 0x7f507c0b5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:05.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.511+0000 7f507a59c640 1 -- 172.21.15.150:0/1489559988 >> v1:172.21.15.157:6789/0 conn(0x7f507c06c6e0 legacy=0x7f507c06caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:05.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.511+0000 7f507a59c640 1 -- 172.21.15.150:0/1489559988 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f507c0d3130 con 0x7f507c15e060 2024-03-17T23:47:05.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.511+0000 7f507959a640 1 -- 172.21.15.150:0/1489559988 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2761252310 0 0) 0x7f507c0d2890 con 0x7f507c06c6e0 2024-03-17T23:47:05.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.512+0000 7f507a59c640 1 --2- 172.21.15.150:0/1489559988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f507c15e060 0x7f507c0d1f20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f506c009870 tx=0x7f506c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a855ab00ff47ab26 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:05.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.512+0000 7f507959a640 1 -- 172.21.15.150:0/1489559988 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f506c035020 con 0x7f507c15e060 2024-03-17T23:47:05.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.512+0000 7f507959a640 1 -- 172.21.15.150:0/1489559988 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f506c031640 con 0x7f507c15e060 2024-03-17T23:47:05.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.512+0000 7f507959a640 1 -- 172.21.15.150:0/1489559988 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f506c02d040 con 0x7f507c15e060 2024-03-17T23:47:05.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.513+0000 7f50809a9640 1 -- 172.21.15.150:0/1489559988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f507c15e060 msgr2=0x7f507c0d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:05.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.513+0000 7f50809a9640 1 --2- 172.21.15.150:0/1489559988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f507c15e060 0x7f507c0d1f20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f506c009870 tx=0x7f506c02b8f0 comp rx=0 tx=0).stop 2024-03-17T23:47:05.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.513+0000 7f50809a9640 1 -- 172.21.15.150:0/1489559988 shutdown_connections 2024-03-17T23:47:05.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.513+0000 7f50809a9640 1 --2- 172.21.15.150:0/1489559988 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f507c0b57d0 0x7f507c0b5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:05.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.513+0000 7f50809a9640 1 --2- 172.21.15.150:0/1489559988 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f507c15e060 0x7f507c0d1f20 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:05.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.513+0000 7f50809a9640 1 -- 172.21.15.150:0/1489559988 >> 172.21.15.150:0/1489559988 conn(0x7f507c073200 msgr2=0x7f507c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:05.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.513+0000 7f50809a9640 1 -- 172.21.15.150:0/1489559988 shutdown_connections 2024-03-17T23:47:05.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.513+0000 7f50809a9640 1 -- 172.21.15.150:0/1489559988 wait complete. 2024-03-17T23:47:05.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.514+0000 7f50809a9640 1 Processor -- start 2024-03-17T23:47:05.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.514+0000 7f50809a9640 1 -- start start 2024-03-17T23:47:05.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.515+0000 7f50809a9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f507c06c6e0 0x7f507c153900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:05.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.515+0000 7f50809a9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f507c0b57d0 0x7f507c158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:05.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.515+0000 7f50809a9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f507c15e060 0x7f507c0d1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:05.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.515+0000 7f50809a9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f507c15e440 con 0x7f507c06c6e0 2024-03-17T23:47:05.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.515+0000 7f50809a9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f507c0b7f00 con 0x7f507c0b57d0 2024-03-17T23:47:05.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.515+0000 7f50809a9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f507c0d2460 con 0x7f507c15e060 2024-03-17T23:47:05.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.515+0000 7f507a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f507c06c6e0 0x7f507c153900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:05.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.515+0000 7f507a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f507c06c6e0 0x7f507c153900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36024/0 (socket says 172.21.15.150:36024) 2024-03-17T23:47:05.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.515+0000 7f507a59c640 1 -- 172.21.15.150:0/2470377536 learned_addr learned my addr 172.21.15.150:0/2470377536 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:05.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.515+0000 7f5079d9b640 1 --2- 172.21.15.150:0/2470377536 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f507c0b57d0 0x7f507c158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:05.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.515+0000 7f507ad9d640 1 --2- 172.21.15.150:0/2470377536 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f507c15e060 0x7f507c0d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:05.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.515+0000 7f507a59c640 1 -- 172.21.15.150:0/2470377536 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f507c15e060 msgr2=0x7f507c0d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:05.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.515+0000 7f507a59c640 1 --2- 172.21.15.150:0/2470377536 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f507c15e060 0x7f507c0d1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:05.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.515+0000 7f507a59c640 1 -- 172.21.15.150:0/2470377536 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f507c0b57d0 msgr2=0x7f507c158d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:05.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.515+0000 7f507a59c640 1 --2- 172.21.15.150:0/2470377536 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f507c0b57d0 0x7f507c158d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:05.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.515+0000 7f507a59c640 1 -- 172.21.15.150:0/2470377536 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f507c1595e0 con 0x7f507c06c6e0 2024-03-17T23:47:05.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.515+0000 7f507ad9d640 1 --2- 172.21.15.150:0/2470377536 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f507c15e060 0x7f507c0d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:47:05.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.515+0000 7f507a59c640 1 --2- 172.21.15.150:0/2470377536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f507c06c6e0 0x7f507c153900 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f506c000c00 tx=0x7f506c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:05.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.516+0000 7f506affd640 1 -- 172.21.15.150:0/2470377536 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f506c040070 con 0x7f507c06c6e0 2024-03-17T23:47:05.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.516+0000 7f506affd640 1 -- 172.21.15.150:0/2470377536 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f506c0399f0 con 0x7f507c06c6e0 2024-03-17T23:47:05.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.516+0000 7f50809a9640 1 -- 172.21.15.150:0/2470377536 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f507c1f3b10 con 0x7f507c06c6e0 2024-03-17T23:47:05.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.516+0000 7f506affd640 1 -- 172.21.15.150:0/2470377536 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f506c035040 con 0x7f507c06c6e0 2024-03-17T23:47:05.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.516+0000 7f50809a9640 1 -- 172.21.15.150:0/2470377536 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f507c153e40 con 0x7f507c06c6e0 2024-03-17T23:47:05.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.517+0000 7f50809a9640 1 -- 172.21.15.150:0/2470377536 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f507c0688b0 con 0x7f507c06c6e0 2024-03-17T23:47:05.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.517+0000 7f506affd640 1 -- 172.21.15.150:0/2470377536 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f506c04b060 con 0x7f507c06c6e0 2024-03-17T23:47:05.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.517+0000 7f506affd640 1 --2- 172.21.15.150:0/2470377536 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f505c06bbd0 0x7f505c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:05.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.517+0000 7f506affd640 1 -- 172.21.15.150:0/2470377536 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f506c0ac5d0 con 0x7f507c06c6e0 2024-03-17T23:47:05.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.518+0000 7f5079d9b640 1 --2- 172.21.15.150:0/2470377536 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f505c06bbd0 0x7f505c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:05.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.518+0000 7f5079d9b640 1 --2- 172.21.15.150:0/2470377536 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f505c06bbd0 0x7f505c06e090 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f507c0b6300 tx=0x7f5070009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:05.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.522+0000 7f506affd640 1 -- 172.21.15.150:0/2470377536 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f506c07ad80 con 0x7f507c06c6e0 2024-03-17T23:47:05.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.695+0000 7f50809a9640 1 -- 172.21.15.150:0/2470377536 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f507c1f38d0 con 0x7f507c06c6e0 2024-03-17T23:47:05.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.696+0000 7f506affd640 1 -- 172.21.15.150:0/2470377536 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18550 (secure 0 0 0) 0x7f506c07a680 con 0x7f507c06c6e0 2024-03-17T23:47:05.696 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:05.697 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:01.940019+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:05.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.698+0000 7f50809a9640 1 -- 172.21.15.150:0/2470377536 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f505c06bbd0 msgr2=0x7f505c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:05.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.698+0000 7f50809a9640 1 --2- 172.21.15.150:0/2470377536 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f505c06bbd0 0x7f505c06e090 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f507c0b6300 tx=0x7f5070009190 comp rx=0 tx=0).stop 2024-03-17T23:47:05.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.698+0000 7f50809a9640 1 -- 172.21.15.150:0/2470377536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f507c06c6e0 msgr2=0x7f507c153900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:05.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.698+0000 7f50809a9640 1 --2- 172.21.15.150:0/2470377536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f507c06c6e0 0x7f507c153900 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f506c000c00 tx=0x7f506c0046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:05.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.698+0000 7f50809a9640 1 -- 172.21.15.150:0/2470377536 shutdown_connections 2024-03-17T23:47:05.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.698+0000 7f50809a9640 1 --2- 172.21.15.150:0/2470377536 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f507c15e060 0x7f507c0d1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:05.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.698+0000 7f50809a9640 1 --2- 172.21.15.150:0/2470377536 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f505c06bbd0 0x7f505c06e090 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:05.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.698+0000 7f50809a9640 1 --2- 172.21.15.150:0/2470377536 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f507c0b57d0 0x7f507c158d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:05.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.698+0000 7f50809a9640 1 --2- 172.21.15.150:0/2470377536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f507c06c6e0 0x7f507c153900 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:05.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.698+0000 7f50809a9640 1 -- 172.21.15.150:0/2470377536 >> 172.21.15.150:0/2470377536 conn(0x7f507c073200 msgr2=0x7f507c0b8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:05.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.698+0000 7f50809a9640 1 -- 172.21.15.150:0/2470377536 shutdown_connections 2024-03-17T23:47:05.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.698+0000 7f50809a9640 1 -- 172.21.15.150:0/2470377536 wait complete. 2024-03-17T23:47:05.709 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:46:53.610678+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:46:54.227335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-17T23:46:57.949235+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-17T23:47:05.710 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:47:05.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.813+0000 7ff3dfaa1640 1 Processor -- start 2024-03-17T23:47:05.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.813+0000 7ff3dfaa1640 1 -- start start 2024-03-17T23:47:05.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.814+0000 7ff3dfaa1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3d80d0850 0x7ff3d8067690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:05.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.814+0000 7ff3dfaa1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3d8161970 con 0x7ff3d8161590 2024-03-17T23:47:05.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.814+0000 7ff3dfaa1640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3d806e7d0 con 0x7ff3d8072a20 2024-03-17T23:47:05.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.814+0000 7ff3dfaa1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff3d806e580 con 0x7ff3d80d0850 2024-03-17T23:47:05.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.814+0000 7ff3dd83d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7ff3d8161590 0x7ff3d80d0140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59282/0 (socket says 172.21.15.150:59282) 2024-03-17T23:47:05.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.814+0000 7ff3dd83d640 1 -- 172.21.15.150:0/2519931655 learned_addr learned my addr 172.21.15.150:0/2519931655 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:05.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.814+0000 7ff3dd03c640 1 --2- 172.21.15.150:0/2519931655 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3d80d0850 0x7ff3d8067690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:05.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.814+0000 7ff3dc83b640 1 -- 172.21.15.150:0/2519931655 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1394007684 0 0) 0x7ff3d8161970 con 0x7ff3d8161590 2024-03-17T23:47:05.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.814+0000 7ff3dc83b640 1 -- 172.21.15.150:0/2519931655 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3c4003520 con 0x7ff3d8161590 2024-03-17T23:47:05.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.814+0000 7ff3dc83b640 1 -- 172.21.15.150:0/2519931655 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 765572490 0 0) 0x7ff3d806e7d0 con 0x7ff3d8072a20 2024-03-17T23:47:05.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.814+0000 7ff3dc83b640 1 -- 172.21.15.150:0/2519931655 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3d8161970 con 0x7ff3d8072a20 2024-03-17T23:47:05.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.814+0000 7ff3dd03c640 1 -- 172.21.15.150:0/2519931655 >> v1:172.21.15.157:6789/0 conn(0x7ff3d8072a20 legacy=0x7ff3d806b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:05.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.814+0000 7ff3dd03c640 1 -- 172.21.15.150:0/2519931655 >> v1:172.21.15.150:6789/0 conn(0x7ff3d8161590 legacy=0x7ff3d80d0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:05.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.814+0000 7ff3dd03c640 1 -- 172.21.15.150:0/2519931655 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3d806eab0 con 0x7ff3d80d0850 2024-03-17T23:47:05.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.815+0000 7ff3dd03c640 1 --2- 172.21.15.150:0/2519931655 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3d80d0850 0x7ff3d8067690 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff3c80097d0 tx=0x7ff3c802b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb822d0850af2b7c server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:05.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.815+0000 7ff3dc83b640 1 -- 172.21.15.150:0/2519931655 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8031770 con 0x7ff3d80d0850 2024-03-17T23:47:05.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.815+0000 7ff3dc83b640 1 -- 172.21.15.150:0/2519931655 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3c8037070 con 0x7ff3d80d0850 2024-03-17T23:47:05.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.815+0000 7ff3dc83b640 1 -- 172.21.15.150:0/2519931655 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8031b30 con 0x7ff3d80d0850 2024-03-17T23:47:05.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.816+0000 7ff3dfaa1640 1 -- 172.21.15.150:0/2519931655 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3d80d0850 msgr2=0x7ff3d8067690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:05.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.816+0000 7ff3dfaa1640 1 --2- 172.21.15.150:0/2519931655 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3d80d0850 0x7ff3d8067690 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff3c80097d0 tx=0x7ff3c802b810 comp rx=0 tx=0).stop 2024-03-17T23:47:05.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.816+0000 7ff3dfaa1640 1 -- 172.21.15.150:0/2519931655 shutdown_connections 2024-03-17T23:47:05.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.816+0000 7ff3dfaa1640 1 --2- 172.21.15.150:0/2519931655 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3d80d0850 0x7ff3d8067690 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:05.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.816+0000 7ff3dfaa1640 1 -- 172.21.15.150:0/2519931655 >> 172.21.15.150:0/2519931655 conn(0x7ff3d80b3da0 msgr2=0x7ff3d80b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:05.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.816+0000 7ff3dfaa1640 1 -- 172.21.15.150:0/2519931655 shutdown_connections 2024-03-17T23:47:05.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.816+0000 7ff3dfaa1640 1 -- 172.21.15.150:0/2519931655 wait complete. 2024-03-17T23:47:05.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.817+0000 7ff3dfaa1640 1 Processor -- start 2024-03-17T23:47:05.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.817+0000 7ff3dfaa1640 1 -- start start 2024-03-17T23:47:05.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.818+0000 7ff3dfaa1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3d8072a20 0x7ff3d8150c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:05.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.818+0000 7ff3dfaa1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3d80d0850 0x7ff3d8153410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:05.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.818+0000 7ff3dfaa1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3d8161590 0x7ff3d8153950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:05.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.818+0000 7ff3dfaa1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d8099c20 con 0x7ff3d8161590 2024-03-17T23:47:05.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.818+0000 7ff3dfaa1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d80ad180 con 0x7ff3d8072a20 2024-03-17T23:47:05.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.818+0000 7ff3dfaa1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff3d8158e40 con 0x7ff3d80d0850 2024-03-17T23:47:05.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.818+0000 7ff3dd83d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3d8072a20 0x7ff3d8150c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:05.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.818+0000 7ff3de03e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3d8161590 0x7ff3d8153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:05.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.818+0000 7ff3dd03c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3d80d0850 0x7ff3d8153410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:05.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.818+0000 7ff3dd83d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3d8072a20 0x7ff3d8150c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:33572/0 (socket says 172.21.15.150:33572) 2024-03-17T23:47:05.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.818+0000 7ff3dd83d640 1 -- 172.21.15.150:0/3761657949 learned_addr learned my addr 172.21.15.150:0/3761657949 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:05.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.818+0000 7ff3dd83d640 1 -- 172.21.15.150:0/3761657949 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3d80d0850 msgr2=0x7ff3d8153410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:05.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.818+0000 7ff3dd83d640 1 --2- 172.21.15.150:0/3761657949 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3d80d0850 0x7ff3d8153410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:05.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.818+0000 7ff3dd83d640 1 -- 172.21.15.150:0/3761657949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3d8161590 msgr2=0x7ff3d8153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:05.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.818+0000 7ff3dd83d640 1 --2- 172.21.15.150:0/3761657949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3d8161590 0x7ff3d8153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:05.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.818+0000 7ff3dd83d640 1 -- 172.21.15.150:0/3761657949 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3d809d2c0 con 0x7ff3d8072a20 2024-03-17T23:47:05.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.818+0000 7ff3de03e640 1 --2- 172.21.15.150:0/3761657949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3d8161590 0x7ff3d8153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:05.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.818+0000 7ff3dd83d640 1 --2- 172.21.15.150:0/3761657949 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3d8072a20 0x7ff3d8150c40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff3d400c410 tx=0x7ff3d400c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:05.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.819+0000 7ff3ce7fc640 1 -- 172.21.15.150:0/3761657949 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d4017070 con 0x7ff3d8072a20 2024-03-17T23:47:05.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.819+0000 7ff3ce7fc640 1 -- 172.21.15.150:0/3761657949 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3d4005ce0 con 0x7ff3d8072a20 2024-03-17T23:47:05.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.819+0000 7ff3dfaa1640 1 -- 172.21.15.150:0/3761657949 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3d809d5a0 con 0x7ff3d8072a20 2024-03-17T23:47:05.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.819+0000 7ff3ce7fc640 1 -- 172.21.15.150:0/3761657949 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3d40072f0 con 0x7ff3d8072a20 2024-03-17T23:47:05.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.819+0000 7ff3dfaa1640 1 -- 172.21.15.150:0/3761657949 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3d809db60 con 0x7ff3d8072a20 2024-03-17T23:47:05.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.820+0000 7ff3dfaa1640 1 -- 172.21.15.150:0/3761657949 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3d80681a0 con 0x7ff3d8072a20 2024-03-17T23:47:05.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.820+0000 7ff3ce7fc640 1 -- 172.21.15.150:0/3761657949 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff3d401a070 con 0x7ff3d8072a20 2024-03-17T23:47:05.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.820+0000 7ff3ce7fc640 1 --2- 172.21.15.150:0/3761657949 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff3bc06bbd0 0x7ff3bc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:05.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.821+0000 7ff3ce7fc640 1 -- 172.21.15.150:0/3761657949 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7ff3d408a200 con 0x7ff3d8072a20 2024-03-17T23:47:05.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.821+0000 7ff3dd03c640 1 --2- 172.21.15.150:0/3761657949 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff3bc06bbd0 0x7ff3bc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:05.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.821+0000 7ff3dd03c640 1 --2- 172.21.15.150:0/3761657949 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff3bc06bbd0 0x7ff3bc06e090 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8036000 tx=0x7ff3c803d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:05.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.825+0000 7ff3ce7fc640 1 -- 172.21.15.150:0/3761657949 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff3d4058380 con 0x7ff3d8072a20 2024-03-17T23:47:05.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.999+0000 7ff3dfaa1640 1 -- 172.21.15.150:0/3761657949 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff3d806cb70 con 0x7ff3d8072a20 2024-03-17T23:47:05.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:05.999+0000 7ff3ce7fc640 1 -- 172.21.15.150:0/3761657949 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7ff3d405c420 con 0x7ff3d8072a20 2024-03-17T23:47:06.000 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:47:06.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.001+0000 7ff3dfaa1640 1 -- 172.21.15.150:0/3761657949 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff3bc06bbd0 msgr2=0x7ff3bc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:06.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.001+0000 7ff3dfaa1640 1 --2- 172.21.15.150:0/3761657949 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff3bc06bbd0 0x7ff3bc06e090 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8036000 tx=0x7ff3c803d040 comp rx=0 tx=0).stop 2024-03-17T23:47:06.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.001+0000 7ff3dfaa1640 1 -- 172.21.15.150:0/3761657949 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3d8072a20 msgr2=0x7ff3d8150c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:06.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.001+0000 7ff3dfaa1640 1 --2- 172.21.15.150:0/3761657949 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3d8072a20 0x7ff3d8150c40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff3d400c410 tx=0x7ff3d400c8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.001+0000 7ff3dfaa1640 1 -- 172.21.15.150:0/3761657949 shutdown_connections 2024-03-17T23:47:06.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.001+0000 7ff3dfaa1640 1 --2- 172.21.15.150:0/3761657949 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3d8161590 0x7ff3d8153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.001+0000 7ff3dfaa1640 1 --2- 172.21.15.150:0/3761657949 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3d80d0850 0x7ff3d8153410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.001+0000 7ff3dfaa1640 1 --2- 172.21.15.150:0/3761657949 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff3bc06bbd0 0x7ff3bc06e090 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.001+0000 7ff3dfaa1640 1 --2- 172.21.15.150:0/3761657949 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3d8072a20 0x7ff3d8150c40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.001+0000 7ff3dfaa1640 1 -- 172.21.15.150:0/3761657949 >> 172.21.15.150:0/3761657949 conn(0x7ff3d80b3da0 msgr2=0x7ff3d815cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:06.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.002+0000 7ff3dfaa1640 1 -- 172.21.15.150:0/3761657949 shutdown_connections 2024-03-17T23:47:06.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.002+0000 7ff3dfaa1640 1 -- 172.21.15.150:0/3761657949 wait complete. 2024-03-17T23:47:06.013 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:47:06.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.116+0000 7f81d279d640 1 Processor -- start 2024-03-17T23:47:06.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.116+0000 7f81d279d640 1 -- start start 2024-03-17T23:47:06.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.117+0000 7f81d279d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f81cc0b95f0 0x7f81cc0ce810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:06.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.117+0000 7f81d279d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81cc0d25f0 con 0x7f81cc0d3690 2024-03-17T23:47:06.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.117+0000 7f81d279d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81cc0d28d0 con 0x7f81cc15dee0 2024-03-17T23:47:06.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.117+0000 7f81d279d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f81cc0d3200 con 0x7f81cc0b95f0 2024-03-17T23:47:06.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.117+0000 7f81d0f9a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f81cc0b95f0 0x7f81cc0ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:06.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.117+0000 7f81d1f9c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f81cc0d3690 0x7f81cc0b6ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59294/0 (socket says 172.21.15.150:59294) 2024-03-17T23:47:06.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.117+0000 7f81d1f9c640 1 -- 172.21.15.150:0/1130958310 learned_addr learned my addr 172.21.15.150:0/1130958310 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:06.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.117+0000 7f81d0f9a640 1 --2- 172.21.15.150:0/1130958310 >> v2:172.21.15.150:3300/0 conn(0x7f81cc0b95f0 0x7f81cc0ce810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36046/0 (socket says 172.21.15.150:36046) 2024-03-17T23:47:06.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.118+0000 7f81d0f9a640 1 -- 172.21.15.150:0/1130958310 >> v1:172.21.15.157:6789/0 conn(0x7f81cc15dee0 legacy=0x7f81cc0d1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:06.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.118+0000 7f81d0f9a640 1 -- 172.21.15.150:0/1130958310 >> v1:172.21.15.150:6789/0 conn(0x7f81cc0d3690 legacy=0x7f81cc0b6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:06.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.118+0000 7f81c3fff640 1 -- 172.21.15.150:0/1130958310 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2939744279 0 0) 0x7f81cc0d28d0 con 0x7f81cc15dee0 2024-03-17T23:47:06.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.118+0000 7f81d0f9a640 1 -- 172.21.15.150:0/1130958310 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81cc0d2bb0 con 0x7f81cc0b95f0 2024-03-17T23:47:06.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.118+0000 7f81d0f9a640 1 --2- 172.21.15.150:0/1130958310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81cc0b95f0 0x7f81cc0ce810 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f81bc0097a0 tx=0x7f81bc02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a2234110e8a298f server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:06.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.118+0000 7f81c3fff640 1 -- 172.21.15.150:0/1130958310 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81bc037070 con 0x7f81cc0b95f0 2024-03-17T23:47:06.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.118+0000 7f81c3fff640 1 -- 172.21.15.150:0/1130958310 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81bc030710 con 0x7f81cc0b95f0 2024-03-17T23:47:06.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.118+0000 7f81c3fff640 1 -- 172.21.15.150:0/1130958310 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81bc030ae0 con 0x7f81cc0b95f0 2024-03-17T23:47:06.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.119+0000 7f81d279d640 1 -- 172.21.15.150:0/1130958310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81cc0b95f0 msgr2=0x7f81cc0ce810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:06.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.119+0000 7f81d279d640 1 --2- 172.21.15.150:0/1130958310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81cc0b95f0 0x7f81cc0ce810 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f81bc0097a0 tx=0x7f81bc02b790 comp rx=0 tx=0).stop 2024-03-17T23:47:06.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.119+0000 7f81d279d640 1 -- 172.21.15.150:0/1130958310 shutdown_connections 2024-03-17T23:47:06.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.119+0000 7f81d279d640 1 --2- 172.21.15.150:0/1130958310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81cc0b95f0 0x7f81cc0ce810 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.119+0000 7f81d279d640 1 -- 172.21.15.150:0/1130958310 >> 172.21.15.150:0/1130958310 conn(0x7f81cc06c820 msgr2=0x7f81cc06cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:06.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.119+0000 7f81d279d640 1 -- 172.21.15.150:0/1130958310 shutdown_connections 2024-03-17T23:47:06.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.119+0000 7f81d279d640 1 -- 172.21.15.150:0/1130958310 wait complete. 2024-03-17T23:47:06.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.120+0000 7f81d279d640 1 Processor -- start 2024-03-17T23:47:06.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.120+0000 7f81d279d640 1 -- start start 2024-03-17T23:47:06.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.121+0000 7f81d279d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81cc0d3690 0x7f81cc159110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:06.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.121+0000 7f81d279d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81cc15dee0 0x7f81cc1f6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:06.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.121+0000 7f81d279d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81cc1f7300 0x7f81cc0cc570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:06.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.121+0000 7f81d279d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f81cc0ced50 con 0x7f81cc15dee0 2024-03-17T23:47:06.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.121+0000 7f81d279d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f81cc0d2390 con 0x7f81cc1f7300 2024-03-17T23:47:06.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.121+0000 7f81d279d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f81cc0ceed0 con 0x7f81cc0d3690 2024-03-17T23:47:06.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.121+0000 7f81d0f9a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81cc15dee0 0x7f81cc1f6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:06.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.121+0000 7f81d0f9a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81cc15dee0 0x7f81cc1f6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36060/0 (socket says 172.21.15.150:36060) 2024-03-17T23:47:06.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.121+0000 7f81d0f9a640 1 -- 172.21.15.150:0/3566212380 learned_addr learned my addr 172.21.15.150:0/3566212380 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:06.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.121+0000 7f81d179b640 1 --2- 172.21.15.150:0/3566212380 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81cc0d3690 0x7f81cc159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:06.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.121+0000 7f81d1f9c640 1 --2- 172.21.15.150:0/3566212380 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81cc1f7300 0x7f81cc0cc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:06.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.121+0000 7f81d0f9a640 1 -- 172.21.15.150:0/3566212380 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81cc0d3690 msgr2=0x7f81cc159110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:06.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.121+0000 7f81d0f9a640 1 --2- 172.21.15.150:0/3566212380 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81cc0d3690 0x7f81cc159110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.121+0000 7f81d0f9a640 1 -- 172.21.15.150:0/3566212380 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81cc1f7300 msgr2=0x7f81cc0cc570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:06.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.121+0000 7f81d0f9a640 1 --2- 172.21.15.150:0/3566212380 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81cc1f7300 0x7f81cc0cc570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.121+0000 7f81d0f9a640 1 -- 172.21.15.150:0/3566212380 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81cc1f7bb0 con 0x7f81cc15dee0 2024-03-17T23:47:06.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.122+0000 7f81d0f9a640 1 --2- 172.21.15.150:0/3566212380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81cc15dee0 0x7f81cc1f6dc0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f81bc000c00 tx=0x7f81bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:06.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.122+0000 7f81c1ffb640 1 -- 172.21.15.150:0/3566212380 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81bc041020 con 0x7f81cc15dee0 2024-03-17T23:47:06.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.122+0000 7f81c1ffb640 1 -- 172.21.15.150:0/3566212380 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81bc032a40 con 0x7f81cc15dee0 2024-03-17T23:47:06.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.122+0000 7f81c1ffb640 1 -- 172.21.15.150:0/3566212380 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81bc037040 con 0x7f81cc15dee0 2024-03-17T23:47:06.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.122+0000 7f81d279d640 1 -- 172.21.15.150:0/3566212380 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81cc159650 con 0x7f81cc15dee0 2024-03-17T23:47:06.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.122+0000 7f81d279d640 1 -- 172.21.15.150:0/3566212380 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81cc0cd4c0 con 0x7f81cc15dee0 2024-03-17T23:47:06.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.123+0000 7f81d279d640 1 -- 172.21.15.150:0/3566212380 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81cc074c40 con 0x7f81cc15dee0 2024-03-17T23:47:06.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.123+0000 7f81c1ffb640 1 -- 172.21.15.150:0/3566212380 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f81bc044020 con 0x7f81cc15dee0 2024-03-17T23:47:06.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.124+0000 7f81c1ffb640 1 --2- 172.21.15.150:0/3566212380 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f81b006bbd0 0x7f81b006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:06.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.124+0000 7f81d179b640 1 --2- 172.21.15.150:0/3566212380 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f81b006bbd0 0x7f81b006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:06.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.124+0000 7f81c1ffb640 1 -- 172.21.15.150:0/3566212380 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f81bc0ac7e0 con 0x7f81cc15dee0 2024-03-17T23:47:06.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.124+0000 7f81d179b640 1 --2- 172.21.15.150:0/3566212380 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f81b006bbd0 0x7f81b006e090 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f81c80012e0 tx=0x7f81c8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:06.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.128+0000 7f81c1ffb640 1 -- 172.21.15.150:0/3566212380 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f81bc04b030 con 0x7f81cc15dee0 2024-03-17T23:47:06.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.301+0000 7f81d279d640 1 -- 172.21.15.150:0/3566212380 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f81cc0b90c0 con 0x7f81cc15dee0 2024-03-17T23:47:06.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.301+0000 7f81c1ffb640 1 -- 172.21.15.150:0/3566212380 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f81bc07a940 con 0x7f81cc15dee0 2024-03-17T23:47:06.302 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:06.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.303+0000 7f81d279d640 1 -- 172.21.15.150:0/3566212380 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f81b006bbd0 msgr2=0x7f81b006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:06.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.303+0000 7f81d279d640 1 --2- 172.21.15.150:0/3566212380 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f81b006bbd0 0x7f81b006e090 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f81c80012e0 tx=0x7f81c8009040 comp rx=0 tx=0).stop 2024-03-17T23:47:06.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.303+0000 7f81d279d640 1 -- 172.21.15.150:0/3566212380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81cc15dee0 msgr2=0x7f81cc1f6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:06.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.303+0000 7f81d279d640 1 --2- 172.21.15.150:0/3566212380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81cc15dee0 0x7f81cc1f6dc0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f81bc000c00 tx=0x7f81bc0046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.303+0000 7f81d279d640 1 -- 172.21.15.150:0/3566212380 shutdown_connections 2024-03-17T23:47:06.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.303+0000 7f81d279d640 1 --2- 172.21.15.150:0/3566212380 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f81b006bbd0 0x7f81b006e090 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.303+0000 7f81d279d640 1 --2- 172.21.15.150:0/3566212380 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81cc1f7300 0x7f81cc0cc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.303+0000 7f81d279d640 1 --2- 172.21.15.150:0/3566212380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81cc15dee0 0x7f81cc1f6dc0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.303+0000 7f81d279d640 1 --2- 172.21.15.150:0/3566212380 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81cc0d3690 0x7f81cc159110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.303+0000 7f81d279d640 1 -- 172.21.15.150:0/3566212380 >> 172.21.15.150:0/3566212380 conn(0x7f81cc06c820 msgr2=0x7f81cc0b6ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:06.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.303+0000 7f81d279d640 1 -- 172.21.15.150:0/3566212380 shutdown_connections 2024-03-17T23:47:06.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.303+0000 7f81d279d640 1 -- 172.21.15.150:0/3566212380 wait complete. 2024-03-17T23:47:06.314 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:47:06.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.418+0000 7f00c1de6640 1 Processor -- start 2024-03-17T23:47:06.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.418+0000 7f00c1de6640 1 -- start start 2024-03-17T23:47:06.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.419+0000 7f00c1de6640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f00bc06b190 0x7f00bc06b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:06.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.419+0000 7f00c1de6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00bc0ce620 0x7f00bc0ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:06.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.419+0000 7f00c1de6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00bc0d0850 con 0x7f00bc15e170 2024-03-17T23:47:06.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.419+0000 7f00c1de6640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f00bc06bb30 con 0x7f00bc06b190 2024-03-17T23:47:06.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.419+0000 7f00c1de6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f00bc0d0b30 con 0x7f00bc0ce620 2024-03-17T23:47:06.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.419+0000 7f00b2ffd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f00bc06b190 0x7f00bc06b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:06.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.419+0000 7f00bbfff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00bc0ce620 0x7f00bc0ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:06.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.419+0000 7f00bb7fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f00bc15e170 0x7f00bc0cd9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:37726/0 (socket says 172.21.15.150:37726) 2024-03-17T23:47:06.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.419+0000 7f00bbfff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00bc0ce620 0x7f00bc0ceab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:50824/0 (socket says 172.21.15.150:50824) 2024-03-17T23:47:06.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.419+0000 7f00bb7fe640 1 -- 172.21.15.150:0/992333286 learned_addr learned my addr 172.21.15.150:0/992333286 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:06.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.420+0000 7f00b2ffd640 1 -- 172.21.15.150:0/992333286 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00bc0ce620 msgr2=0x7f00bc0ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:06.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.420+0000 7f00b2ffd640 1 --2- 172.21.15.150:0/992333286 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00bc0ce620 0x7f00bc0ceab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.420+0000 7f00b2ffd640 1 -- 172.21.15.150:0/992333286 >> v1:172.21.15.157:6789/0 conn(0x7f00bc15e170 legacy=0x7f00bc0cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:06.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.420+0000 7f00baffd640 1 -- 172.21.15.150:0/992333286 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2922815163 0 0) 0x7f00bc0d0850 con 0x7f00bc15e170 2024-03-17T23:47:06.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.420+0000 7f00b2ffd640 1 -- 172.21.15.150:0/992333286 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00bc0d0e10 con 0x7f00bc06b190 2024-03-17T23:47:06.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.420+0000 7f00b2ffd640 1 --2- 172.21.15.150:0/992333286 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00bc06b190 0x7f00bc06b5f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f00a40097a0 tx=0x7f00a402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3eec053cf71ac1a server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:06.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.421+0000 7f00baffd640 1 -- 172.21.15.150:0/992333286 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a40315a0 con 0x7f00bc06b190 2024-03-17T23:47:06.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.421+0000 7f00baffd640 1 -- 172.21.15.150:0/992333286 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00a4037070 con 0x7f00bc06b190 2024-03-17T23:47:06.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.421+0000 7f00baffd640 1 -- 172.21.15.150:0/992333286 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a4031960 con 0x7f00bc06b190 2024-03-17T23:47:06.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.421+0000 7f00c1de6640 1 -- 172.21.15.150:0/992333286 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00bc06b190 msgr2=0x7f00bc06b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:06.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.421+0000 7f00c1de6640 1 --2- 172.21.15.150:0/992333286 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00bc06b190 0x7f00bc06b5f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f00a40097a0 tx=0x7f00a402b840 comp rx=0 tx=0).stop 2024-03-17T23:47:06.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.421+0000 7f00c1de6640 1 -- 172.21.15.150:0/992333286 shutdown_connections 2024-03-17T23:47:06.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.421+0000 7f00c1de6640 1 --2- 172.21.15.150:0/992333286 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00bc0ce620 0x7f00bc0ceab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.421+0000 7f00c1de6640 1 --2- 172.21.15.150:0/992333286 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00bc06b190 0x7f00bc06b5f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.421+0000 7f00c1de6640 1 -- 172.21.15.150:0/992333286 >> 172.21.15.150:0/992333286 conn(0x7f00bc099a70 msgr2=0x7f00bc099e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:06.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.421+0000 7f00c1de6640 1 -- 172.21.15.150:0/992333286 shutdown_connections 2024-03-17T23:47:06.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.421+0000 7f00c1de6640 1 -- 172.21.15.150:0/992333286 wait complete. 2024-03-17T23:47:06.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.422+0000 7f00c1de6640 1 Processor -- start 2024-03-17T23:47:06.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.423+0000 7f00c1de6640 1 -- start start 2024-03-17T23:47:06.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.423+0000 7f00c1de6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00bc06b190 0x7f00bc0b60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:06.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.423+0000 7f00c1de6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00bc0ce620 0x7f00bc149790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:06.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.423+0000 7f00c1de6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00bc15e170 0x7f00bc0b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:06.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.423+0000 7f00c1de6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f00bc0afcf0 con 0x7f00bc15e170 2024-03-17T23:47:06.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.423+0000 7f00c1de6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f00bc0d13d0 con 0x7f00bc0ce620 2024-03-17T23:47:06.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.423+0000 7f00c1de6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f00bc0afe70 con 0x7f00bc06b190 2024-03-17T23:47:06.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.423+0000 7f00bbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00bc15e170 0x7f00bc0b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:06.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.423+0000 7f00b2ffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00bc0ce620 0x7f00bc149790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:06.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.423+0000 7f00bbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00bc15e170 0x7f00bc0b04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36070/0 (socket says 172.21.15.150:36070) 2024-03-17T23:47:06.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.423+0000 7f00bb7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00bc06b190 0x7f00bc0b60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:06.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.423+0000 7f00bbfff640 1 -- 172.21.15.150:0/3204986536 learned_addr learned my addr 172.21.15.150:0/3204986536 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:06.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.424+0000 7f00b2ffd640 1 -- 172.21.15.150:0/3204986536 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00bc06b190 msgr2=0x7f00bc0b60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:06.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.424+0000 7f00b2ffd640 1 --2- 172.21.15.150:0/3204986536 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00bc06b190 0x7f00bc0b60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.424+0000 7f00b2ffd640 1 -- 172.21.15.150:0/3204986536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00bc15e170 msgr2=0x7f00bc0b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:06.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.424+0000 7f00b2ffd640 1 --2- 172.21.15.150:0/3204986536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00bc15e170 0x7f00bc0b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.424+0000 7f00b2ffd640 1 -- 172.21.15.150:0/3204986536 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00bc15fad0 con 0x7f00bc0ce620 2024-03-17T23:47:06.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.424+0000 7f00bbfff640 1 --2- 172.21.15.150:0/3204986536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00bc15e170 0x7f00bc0b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:06.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.424+0000 7f00b2ffd640 1 --2- 172.21.15.150:0/3204986536 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00bc0ce620 0x7f00bc149790 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f00a4033000 tx=0x7f00a40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:06.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.424+0000 7f00b8ff9640 1 -- 172.21.15.150:0/3204986536 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a4004230 con 0x7f00bc0ce620 2024-03-17T23:47:06.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.424+0000 7f00b8ff9640 1 -- 172.21.15.150:0/3204986536 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00a40043f0 con 0x7f00bc0ce620 2024-03-17T23:47:06.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.424+0000 7f00c1de6640 1 -- 172.21.15.150:0/3204986536 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00bc15fdb0 con 0x7f00bc0ce620 2024-03-17T23:47:06.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.424+0000 7f00b8ff9640 1 -- 172.21.15.150:0/3204986536 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a4003850 con 0x7f00bc0ce620 2024-03-17T23:47:06.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.424+0000 7f00c1de6640 1 -- 172.21.15.150:0/3204986536 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00bc160370 con 0x7f00bc0ce620 2024-03-17T23:47:06.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.425+0000 7f00c1de6640 1 -- 172.21.15.150:0/3204986536 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00bc0688b0 con 0x7f00bc0ce620 2024-03-17T23:47:06.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.426+0000 7f00b8ff9640 1 -- 172.21.15.150:0/3204986536 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f00a40039f0 con 0x7f00bc0ce620 2024-03-17T23:47:06.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.426+0000 7f00b8ff9640 1 --2- 172.21.15.150:0/3204986536 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f009c06bbd0 0x7f009c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:06.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.426+0000 7f00b8ff9640 1 -- 172.21.15.150:0/3204986536 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f00a402f030 con 0x7f00bc0ce620 2024-03-17T23:47:06.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.426+0000 7f00bb7fe640 1 --2- 172.21.15.150:0/3204986536 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f009c06bbd0 0x7f009c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:06.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.426+0000 7f00bb7fe640 1 --2- 172.21.15.150:0/3204986536 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f009c06bbd0 0x7f009c06e090 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f00ac007fd0 tx=0x7f00ac007de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:06.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.435+0000 7f00b8ff9640 1 -- 172.21.15.150:0/3204986536 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f00a4041070 con 0x7f00bc0ce620 2024-03-17T23:47:06.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.606+0000 7f00c1de6640 1 -- 172.21.15.150:0/3204986536 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f00bc0cd560 con 0x7f00bc0ce620 2024-03-17T23:47:06.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.607+0000 7f00b8ff9640 1 -- 172.21.15.150:0/3204986536 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f00a407ae80 con 0x7f00bc0ce620 2024-03-17T23:47:06.608 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:47:06.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.609+0000 7f00c1de6640 1 -- 172.21.15.150:0/3204986536 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f009c06bbd0 msgr2=0x7f009c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:06.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.609+0000 7f00c1de6640 1 --2- 172.21.15.150:0/3204986536 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f009c06bbd0 0x7f009c06e090 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f00ac007fd0 tx=0x7f00ac007de0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.609+0000 7f00c1de6640 1 -- 172.21.15.150:0/3204986536 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00bc0ce620 msgr2=0x7f00bc149790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:06.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.609+0000 7f00c1de6640 1 --2- 172.21.15.150:0/3204986536 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00bc0ce620 0x7f00bc149790 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f00a4033000 tx=0x7f00a40046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.609+0000 7f00c1de6640 1 -- 172.21.15.150:0/3204986536 shutdown_connections 2024-03-17T23:47:06.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.609+0000 7f00c1de6640 1 --2- 172.21.15.150:0/3204986536 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00bc15e170 0x7f00bc0b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.609+0000 7f00c1de6640 1 --2- 172.21.15.150:0/3204986536 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f009c06bbd0 0x7f009c06e090 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.609+0000 7f00c1de6640 1 --2- 172.21.15.150:0/3204986536 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00bc0ce620 0x7f00bc149790 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.609+0000 7f00c1de6640 1 --2- 172.21.15.150:0/3204986536 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00bc06b190 0x7f00bc0b60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.609+0000 7f00c1de6640 1 -- 172.21.15.150:0/3204986536 >> 172.21.15.150:0/3204986536 conn(0x7f00bc099a70 msgr2=0x7f00bc099e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:06.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.609+0000 7f00c1de6640 1 -- 172.21.15.150:0/3204986536 shutdown_connections 2024-03-17T23:47:06.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.609+0000 7f00c1de6640 1 -- 172.21.15.150:0/3204986536 wait complete. 2024-03-17T23:47:06.620 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-17T23:47:06.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.723+0000 7f0690f59640 1 Processor -- start 2024-03-17T23:47:06.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.723+0000 7f0690f59640 1 -- start start 2024-03-17T23:47:06.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.724+0000 7f0690f59640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f068c0cd420 0x7f068c165d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:06.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.724+0000 7f0690f59640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f068c0cf3b0 con 0x7f068c15bfd0 2024-03-17T23:47:06.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.724+0000 7f0690f59640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f068c0cf690 con 0x7f068c155320 2024-03-17T23:47:06.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.724+0000 7f0690f59640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f068c15c550 con 0x7f068c0cd420 2024-03-17T23:47:06.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.724+0000 7f0689d9b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f068c15bfd0 0x7f068c0ccd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59298/0 (socket says 172.21.15.150:59298) 2024-03-17T23:47:06.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.724+0000 7f0689d9b640 1 -- 172.21.15.150:0/3999734774 learned_addr learned my addr 172.21.15.150:0/3999734774 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:06.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.724+0000 7f068ad9d640 1 --2- 172.21.15.150:0/3999734774 >> v2:172.21.15.157:3300/0 conn(0x7f068c0cd420 0x7f068c165d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:06.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.725+0000 7f068959a640 1 -- 172.21.15.150:0/3999734774 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3025442568 0 0) 0x7f068c0cf3b0 con 0x7f068c15bfd0 2024-03-17T23:47:06.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.725+0000 7f068959a640 1 -- 172.21.15.150:0/3999734774 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0674003520 con 0x7f068c15bfd0 2024-03-17T23:47:06.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.725+0000 7f068ad9d640 1 -- 172.21.15.150:0/3999734774 >> v1:172.21.15.157:6789/0 conn(0x7f068c155320 legacy=0x7f068c06dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:06.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.725+0000 7f068ad9d640 1 -- 172.21.15.150:0/3999734774 >> v1:172.21.15.150:6789/0 conn(0x7f068c15bfd0 legacy=0x7f068c0ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:06.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.725+0000 7f068ad9d640 1 -- 172.21.15.150:0/3999734774 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f068c15c7a0 con 0x7f068c0cd420 2024-03-17T23:47:06.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.725+0000 7f068ad9d640 1 --2- 172.21.15.150:0/3999734774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f068c0cd420 0x7f068c165d10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f06840097a0 tx=0x7f068402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ce05506a45b884c server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:06.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.726+0000 7f068959a640 1 -- 172.21.15.150:0/3999734774 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0684034070 con 0x7f068c0cd420 2024-03-17T23:47:06.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.726+0000 7f068959a640 1 -- 172.21.15.150:0/3999734774 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0684030610 con 0x7f068c0cd420 2024-03-17T23:47:06.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.726+0000 7f068959a640 1 -- 172.21.15.150:0/3999734774 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06840309e0 con 0x7f068c0cd420 2024-03-17T23:47:06.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.726+0000 7f0690f59640 1 -- 172.21.15.150:0/3999734774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f068c0cd420 msgr2=0x7f068c165d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:06.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.726+0000 7f0690f59640 1 --2- 172.21.15.150:0/3999734774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f068c0cd420 0x7f068c165d10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f06840097a0 tx=0x7f068402b960 comp rx=0 tx=0).stop 2024-03-17T23:47:06.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.726+0000 7f0690f59640 1 -- 172.21.15.150:0/3999734774 shutdown_connections 2024-03-17T23:47:06.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.726+0000 7f0690f59640 1 --2- 172.21.15.150:0/3999734774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f068c0cd420 0x7f068c165d10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.726+0000 7f0690f59640 1 -- 172.21.15.150:0/3999734774 >> 172.21.15.150:0/3999734774 conn(0x7f068c06b190 msgr2=0x7f068c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:06.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.726+0000 7f0690f59640 1 -- 172.21.15.150:0/3999734774 shutdown_connections 2024-03-17T23:47:06.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.726+0000 7f0690f59640 1 -- 172.21.15.150:0/3999734774 wait complete. 2024-03-17T23:47:06.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.728+0000 7f0690f59640 1 Processor -- start 2024-03-17T23:47:06.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.728+0000 7f0690f59640 1 -- start start 2024-03-17T23:47:06.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.728+0000 7f0690f59640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f068c0cd420 0x7f068c13c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:06.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.728+0000 7f0690f59640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f068c155320 0x7f068c148690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:06.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.728+0000 7f0690f59640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f068c15bfd0 0x7f068c148bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:06.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.728+0000 7f0690f59640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f068c155880 con 0x7f068c15bfd0 2024-03-17T23:47:06.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.728+0000 7f0690f59640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f068c155700 con 0x7f068c155320 2024-03-17T23:47:06.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.728+0000 7f0690f59640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f068c15dcd0 con 0x7f068c0cd420 2024-03-17T23:47:06.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.728+0000 7f068ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f068c15bfd0 0x7f068c148bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:06.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.728+0000 7f068ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f068c15bfd0 0x7f068c148bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36084/0 (socket says 172.21.15.150:36084) 2024-03-17T23:47:06.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.728+0000 7f068ad9d640 1 -- 172.21.15.150:0/1246672498 learned_addr learned my addr 172.21.15.150:0/1246672498 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:06.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.729+0000 7f0689d9b640 1 --2- 172.21.15.150:0/1246672498 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f068c155320 0x7f068c148690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:06.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.729+0000 7f068a59c640 1 --2- 172.21.15.150:0/1246672498 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f068c0cd420 0x7f068c13c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:06.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.729+0000 7f068ad9d640 1 -- 172.21.15.150:0/1246672498 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f068c0cd420 msgr2=0x7f068c13c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:06.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.729+0000 7f068ad9d640 1 --2- 172.21.15.150:0/1246672498 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f068c0cd420 0x7f068c13c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.729+0000 7f068ad9d640 1 -- 172.21.15.150:0/1246672498 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f068c155320 msgr2=0x7f068c148690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:06.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.729+0000 7f068ad9d640 1 --2- 172.21.15.150:0/1246672498 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f068c155320 0x7f068c148690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.729+0000 7f068ad9d640 1 -- 172.21.15.150:0/1246672498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f068c13d1b0 con 0x7f068c15bfd0 2024-03-17T23:47:06.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.729+0000 7f068ad9d640 1 --2- 172.21.15.150:0/1246672498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f068c15bfd0 0x7f068c148bd0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f06840097a0 tx=0x7f06840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:06.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.729+0000 7f067affd640 1 -- 172.21.15.150:0/1246672498 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0684034040 con 0x7f068c15bfd0 2024-03-17T23:47:06.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.729+0000 7f067affd640 1 -- 172.21.15.150:0/1246672498 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0684030410 con 0x7f068c15bfd0 2024-03-17T23:47:06.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.729+0000 7f0690f59640 1 -- 172.21.15.150:0/1246672498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f068c149200 con 0x7f068c15bfd0 2024-03-17T23:47:06.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.729+0000 7f067affd640 1 -- 172.21.15.150:0/1246672498 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0684039bb0 con 0x7f068c15bfd0 2024-03-17T23:47:06.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.729+0000 7f0690f59640 1 -- 172.21.15.150:0/1246672498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f068c1473c0 con 0x7f068c15bfd0 2024-03-17T23:47:06.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.730+0000 7f0690f59640 1 -- 172.21.15.150:0/1246672498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f068c0681a0 con 0x7f068c15bfd0 2024-03-17T23:47:06.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.730+0000 7f067affd640 1 -- 172.21.15.150:0/1246672498 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0684037070 con 0x7f068c15bfd0 2024-03-17T23:47:06.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.731+0000 7f067affd640 1 --2- 172.21.15.150:0/1246672498 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f066c06bbd0 0x7f066c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:06.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.731+0000 7f067affd640 1 -- 172.21.15.150:0/1246672498 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f06840acba0 con 0x7f068c15bfd0 2024-03-17T23:47:06.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.731+0000 7f068a59c640 1 --2- 172.21.15.150:0/1246672498 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f066c06bbd0 0x7f066c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:06.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.731+0000 7f068a59c640 1 --2- 172.21.15.150:0/1246672498 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f066c06bbd0 0x7f066c06e090 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f067c001f10 tx=0x7f067c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:06.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.736+0000 7f067affd640 1 -- 172.21.15.150:0/1246672498 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f068407ad00 con 0x7f068c15bfd0 2024-03-17T23:47:06.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.909+0000 7f0690f59640 1 -- 172.21.15.150:0/1246672498 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f068c06ccb0 con 0x7f068c15bfd0 2024-03-17T23:47:06.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.910+0000 7f067affd640 1 -- 172.21.15.150:0/1246672498 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f06840ac0a0 con 0x7f068c15bfd0 2024-03-17T23:47:06.910 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:06.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.912+0000 7f0690f59640 1 -- 172.21.15.150:0/1246672498 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f066c06bbd0 msgr2=0x7f066c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:06.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.912+0000 7f0690f59640 1 --2- 172.21.15.150:0/1246672498 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f066c06bbd0 0x7f066c06e090 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f067c001f10 tx=0x7f067c009040 comp rx=0 tx=0).stop 2024-03-17T23:47:06.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.912+0000 7f0690f59640 1 -- 172.21.15.150:0/1246672498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f068c15bfd0 msgr2=0x7f068c148bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:06.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.912+0000 7f0690f59640 1 --2- 172.21.15.150:0/1246672498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f068c15bfd0 0x7f068c148bd0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f06840097a0 tx=0x7f06840046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.912+0000 7f0690f59640 1 -- 172.21.15.150:0/1246672498 shutdown_connections 2024-03-17T23:47:06.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.912+0000 7f0690f59640 1 --2- 172.21.15.150:0/1246672498 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f068c15bfd0 0x7f068c148bd0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.912+0000 7f0690f59640 1 --2- 172.21.15.150:0/1246672498 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f066c06bbd0 0x7f066c06e090 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.912+0000 7f0690f59640 1 --2- 172.21.15.150:0/1246672498 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f068c155320 0x7f068c148690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.912+0000 7f0690f59640 1 --2- 172.21.15.150:0/1246672498 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f068c0cd420 0x7f068c13c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:06.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.912+0000 7f0690f59640 1 -- 172.21.15.150:0/1246672498 >> 172.21.15.150:0/1246672498 conn(0x7f068c06b190 msgr2=0x7f068c159450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:06.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.912+0000 7f0690f59640 1 -- 172.21.15.150:0/1246672498 shutdown_connections 2024-03-17T23:47:06.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:06.912+0000 7f0690f59640 1 -- 172.21.15.150:0/1246672498 wait complete. 2024-03-17T23:47:06.923 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-17T23:47:07.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.025+0000 7fd162b3b640 1 Processor -- start 2024-03-17T23:47:07.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.025+0000 7fd162b3b640 1 -- start start 2024-03-17T23:47:07.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.025+0000 7fd162b3b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd15c15e2a0 0x7fd15c09a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:07.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.025+0000 7fd162b3b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd15c0cfdf0 0x7fd15c0d1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:07.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.025+0000 7fd162b3b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd15c06ccb0 0x7fd15c06d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:07.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.025+0000 7fd162b3b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fd15c0d1990 con 0x7fd15c0cfdf0 2024-03-17T23:47:07.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.025+0000 7fd162b3b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd15c0d03b0 con 0x7fd15c06ccb0 2024-03-17T23:47:07.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.025+0000 7fd162b3b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd15c0d0690 con 0x7fd15c15e2a0 2024-03-17T23:47:07.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.025+0000 7fd15bfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd15c0cfdf0 0x7fd15c0d1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:07.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.026+0000 7fd15bfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd15c0cfdf0 0x7fd15c0d1450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36088/0 (socket says 172.21.15.150:36088) 2024-03-17T23:47:07.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.026+0000 7fd15bfff640 1 -- 172.21.15.150:0/1701627502 learned_addr learned my addr 172.21.15.150:0/1701627502 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:07.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.026+0000 7fd1610d8640 1 --2- 172.21.15.150:0/1701627502 >> v2:172.21.15.157:3300/0 conn(0x7fd15c06ccb0 0x7fd15c06d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:07.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.026+0000 7fd1608d7640 1 --2- 172.21.15.150:0/1701627502 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd15c15e2a0 0x7fd15c09a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:07.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.026+0000 7fd15bfff640 1 -- 172.21.15.150:0/1701627502 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd15c15e2a0 msgr2=0x7fd15c09a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:07.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.026+0000 7fd15bfff640 1 --2- 172.21.15.150:0/1701627502 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd15c15e2a0 0x7fd15c09a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.026+0000 7fd15bfff640 1 -- 172.21.15.150:0/1701627502 >> v2:172.21.15.157:3300/0 conn(0x7fd15c06ccb0 msgr2=0x7fd15c06d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:07.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.026+0000 7fd15bfff640 1 --2- 172.21.15.150:0/1701627502 >> v2:172.21.15.157:3300/0 conn(0x7fd15c06ccb0 0x7fd15c06d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.026+0000 7fd15bfff640 1 -- 172.21.15.150:0/1701627502 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd15c0d0970 con 0x7fd15c0cfdf0 2024-03-17T23:47:07.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.026+0000 7fd15bfff640 1 --2- 172.21.15.150:0/1701627502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd15c0cfdf0 0x7fd15c0d1450 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd148009870 tx=0x7fd14802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edcfcfe69578d233 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:07.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.027+0000 7fd15b7fe640 1 -- 172.21.15.150:0/1701627502 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd148035020 con 0x7fd15c0cfdf0 2024-03-17T23:47:07.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.027+0000 7fd15b7fe640 1 -- 172.21.15.150:0/1701627502 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd148031640 con 0x7fd15c0cfdf0 2024-03-17T23:47:07.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.027+0000 7fd15b7fe640 1 -- 172.21.15.150:0/1701627502 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14802d040 con 0x7fd15c0cfdf0 2024-03-17T23:47:07.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.027+0000 7fd162b3b640 1 -- 172.21.15.150:0/1701627502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd15c0cfdf0 msgr2=0x7fd15c0d1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:07.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.027+0000 7fd162b3b640 1 --2- 172.21.15.150:0/1701627502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd15c0cfdf0 0x7fd15c0d1450 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd148009870 tx=0x7fd14802b8f0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.027+0000 7fd162b3b640 1 -- 172.21.15.150:0/1701627502 shutdown_connections 2024-03-17T23:47:07.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.027+0000 7fd162b3b640 1 --2- 172.21.15.150:0/1701627502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd15c0cfdf0 0x7fd15c0d1450 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.027+0000 7fd162b3b640 1 --2- 172.21.15.150:0/1701627502 >> v2:172.21.15.157:3300/0 conn(0x7fd15c06ccb0 0x7fd15c06d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.027+0000 7fd162b3b640 1 --2- 172.21.15.150:0/1701627502 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd15c15e2a0 0x7fd15c09a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.027+0000 7fd162b3b640 1 -- 172.21.15.150:0/1701627502 >> 172.21.15.150:0/1701627502 conn(0x7fd15c0b36a0 msgr2=0x7fd15c0b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:07.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.027+0000 7fd162b3b640 1 -- 172.21.15.150:0/1701627502 shutdown_connections 2024-03-17T23:47:07.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.027+0000 7fd162b3b640 1 -- 172.21.15.150:0/1701627502 wait complete. 2024-03-17T23:47:07.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.028+0000 7fd162b3b640 1 Processor -- start 2024-03-17T23:47:07.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.028+0000 7fd162b3b640 1 -- start start 2024-03-17T23:47:07.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.029+0000 7fd162b3b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd15c06ccb0 0x7fd15c0b4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:07.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.029+0000 7fd162b3b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd15c0cfdf0 0x7fd15c0ca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:07.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.029+0000 7fd162b3b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd15c15e2a0 0x7fd15c0caf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:07.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.029+0000 7fd162b3b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd15c06c740 con 0x7fd15c0cfdf0 2024-03-17T23:47:07.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.029+0000 7fd162b3b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd15c0d0f30 con 0x7fd15c15e2a0 2024-03-17T23:47:07.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.029+0000 7fd162b3b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd15c0cc200 con 0x7fd15c06ccb0 2024-03-17T23:47:07.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.029+0000 7fd15bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd15c0cfdf0 0x7fd15c0ca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:07.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.029+0000 7fd15bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd15c0cfdf0 0x7fd15c0ca9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36098/0 (socket says 172.21.15.150:36098) 2024-03-17T23:47:07.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.029+0000 7fd15bfff640 1 -- 172.21.15.150:0/1273417017 learned_addr learned my addr 172.21.15.150:0/1273417017 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:07.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.029+0000 7fd1608d7640 1 --2- 172.21.15.150:0/1273417017 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd15c06ccb0 0x7fd15c0b4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:07.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.029+0000 7fd1610d8640 1 --2- 172.21.15.150:0/1273417017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd15c15e2a0 0x7fd15c0caf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:07.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.030+0000 7fd15bfff640 1 -- 172.21.15.150:0/1273417017 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd15c06ccb0 msgr2=0x7fd15c0b4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:07.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.030+0000 7fd15bfff640 1 --2- 172.21.15.150:0/1273417017 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd15c06ccb0 0x7fd15c0b4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.030+0000 7fd15bfff640 1 -- 172.21.15.150:0/1273417017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd15c15e2a0 msgr2=0x7fd15c0caf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:07.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.030+0000 7fd15bfff640 1 --2- 172.21.15.150:0/1273417017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd15c15e2a0 0x7fd15c0caf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.030+0000 7fd15bfff640 1 -- 172.21.15.150:0/1273417017 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd15c1485f0 con 0x7fd15c0cfdf0 2024-03-17T23:47:07.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.030+0000 7fd15bfff640 1 --2- 172.21.15.150:0/1273417017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd15c0cfdf0 0x7fd15c0ca9d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd148000c00 tx=0x7fd1480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:07.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.030+0000 7fd1597fa640 1 -- 172.21.15.150:0/1273417017 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd148040070 con 0x7fd15c0cfdf0 2024-03-17T23:47:07.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.030+0000 7fd1597fa640 1 -- 172.21.15.150:0/1273417017 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1480399f0 con 0x7fd15c0cfdf0 2024-03-17T23:47:07.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.030+0000 7fd162b3b640 1 -- 172.21.15.150:0/1273417017 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd15c1488d0 con 0x7fd15c0cfdf0 2024-03-17T23:47:07.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.030+0000 7fd1597fa640 1 -- 172.21.15.150:0/1273417017 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd148035040 con 0x7fd15c0cfdf0 2024-03-17T23:47:07.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.030+0000 7fd162b3b640 1 -- 172.21.15.150:0/1273417017 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd15c148e90 con 0x7fd15c0cfdf0 2024-03-17T23:47:07.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.031+0000 7fd162b3b640 1 -- 172.21.15.150:0/1273417017 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd15c0742e0 con 0x7fd15c0cfdf0 2024-03-17T23:47:07.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.032+0000 7fd1597fa640 1 -- 172.21.15.150:0/1273417017 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd14804b060 con 0x7fd15c0cfdf0 2024-03-17T23:47:07.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.032+0000 7fd1597fa640 1 --2- 172.21.15.150:0/1273417017 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd14006bbd0 0x7fd14006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:07.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.032+0000 7fd1597fa640 1 -- 172.21.15.150:0/1273417017 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fd1480ac490 con 0x7fd15c0cfdf0 2024-03-17T23:47:07.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.032+0000 7fd1608d7640 1 --2- 172.21.15.150:0/1273417017 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd14006bbd0 0x7fd14006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:07.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.032+0000 7fd1608d7640 1 --2- 172.21.15.150:0/1273417017 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd14006bbd0 0x7fd14006e090 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd150009aa0 tx=0x7fd150009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:07.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.036+0000 7fd1597fa640 1 -- 172.21.15.150:0/1273417017 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd15c0742e0 con 0x7fd15c0cfdf0 2024-03-17T23:47:07.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.221+0000 7fd162b3b640 1 -- 172.21.15.150:0/1273417017 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd15c0688b0 con 0x7fd15c0cfdf0 2024-03-17T23:47:07.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.221+0000 7fd1597fa640 1 -- 172.21.15.150:0/1273417017 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7fd14807a5f0 con 0x7fd15c0cfdf0 2024-03-17T23:47:07.222 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-03-17T23:47:07.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.223+0000 7fd162b3b640 1 -- 172.21.15.150:0/1273417017 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd14006bbd0 msgr2=0x7fd14006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:07.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.223+0000 7fd162b3b640 1 --2- 172.21.15.150:0/1273417017 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd14006bbd0 0x7fd14006e090 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd150009aa0 tx=0x7fd150009190 comp rx=0 tx=0).stop 2024-03-17T23:47:07.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.223+0000 7fd162b3b640 1 -- 172.21.15.150:0/1273417017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd15c0cfdf0 msgr2=0x7fd15c0ca9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:07.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.223+0000 7fd162b3b640 1 --2- 172.21.15.150:0/1273417017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd15c0cfdf0 0x7fd15c0ca9d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd148000c00 tx=0x7fd1480046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.224+0000 7fd162b3b640 1 -- 172.21.15.150:0/1273417017 shutdown_connections 2024-03-17T23:47:07.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.224+0000 7fd162b3b640 1 --2- 172.21.15.150:0/1273417017 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd14006bbd0 0x7fd14006e090 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.224+0000 7fd162b3b640 1 --2- 172.21.15.150:0/1273417017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd15c15e2a0 0x7fd15c0caf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.224+0000 7fd162b3b640 1 --2- 172.21.15.150:0/1273417017 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd15c0cfdf0 0x7fd15c0ca9d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.224+0000 7fd162b3b640 1 --2- 172.21.15.150:0/1273417017 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd15c06ccb0 0x7fd15c0b4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.224+0000 7fd162b3b640 1 -- 172.21.15.150:0/1273417017 >> 172.21.15.150:0/1273417017 conn(0x7fd15c0b36a0 msgr2=0x7fd15c099db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:07.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.224+0000 7fd162b3b640 1 -- 172.21.15.150:0/1273417017 shutdown_connections 2024-03-17T23:47:07.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.224+0000 7fd162b3b640 1 -- 172.21.15.150:0/1273417017 wait complete. 2024-03-17T23:47:07.235 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:47:07.235 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:47:07.251 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:07.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.390+0000 7fc819959640 1 Processor -- start 2024-03-17T23:47:07.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.390+0000 7fc819959640 1 -- start start 2024-03-17T23:47:07.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.391+0000 7fc819959640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc81415e060 0x7fc8140d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:07.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.391+0000 7fc819959640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc8140b57d0 0x7fc8140b5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:07.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.391+0000 7fc819959640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8140d2890 con 0x7fc81406c6e0 2024-03-17T23:47:07.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.391+0000 7fc819959640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc8140b6170 con 0x7fc8140b57d0 2024-03-17T23:47:07.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.391+0000 7fc819959640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc8140d2b70 con 0x7fc81415e060 2024-03-17T23:47:07.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.391+0000 7fc8137fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fc81406c6e0 0x7fc81406cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59300/0 (socket says 172.21.15.150:59300) 2024-03-17T23:47:07.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.391+0000 7fc8137fe640 1 -- 172.21.15.150:0/1593871346 learned_addr learned my addr 172.21.15.150:0/1593871346 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:07.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.391+0000 7fc8127fc640 1 --2- 172.21.15.150:0/1593871346 >> v2:172.21.15.157:3300/0 conn(0x7fc8140b57d0 0x7fc8140b5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:07.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.391+0000 7fc812ffd640 1 --2- 172.21.15.150:0/1593871346 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc81415e060 0x7fc8140d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:07.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.392+0000 7fc811ffb640 1 -- 172.21.15.150:0/1593871346 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 388403664 0 0) 0x7fc8140d2890 con 0x7fc81406c6e0 2024-03-17T23:47:07.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.392+0000 7fc811ffb640 1 -- 172.21.15.150:0/1593871346 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7fc003520 con 0x7fc81406c6e0 2024-03-17T23:47:07.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.392+0000 7fc8127fc640 1 -- 172.21.15.150:0/1593871346 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc81415e060 msgr2=0x7fc8140d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:07.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.392+0000 7fc8127fc640 1 --2- 172.21.15.150:0/1593871346 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc81415e060 0x7fc8140d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.392+0000 7fc8127fc640 1 -- 172.21.15.150:0/1593871346 >> v1:172.21.15.150:6789/0 conn(0x7fc81406c6e0 legacy=0x7fc81406cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:07.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.392+0000 7fc8127fc640 1 -- 172.21.15.150:0/1593871346 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8140d2e50 con 0x7fc8140b57d0 2024-03-17T23:47:07.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.392+0000 7fc8127fc640 1 --2- 172.21.15.150:0/1593871346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8140b57d0 0x7fc8140b5c30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc8080097a0 tx=0x7fc80802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9627175fe8454750 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:07.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.393+0000 7fc811ffb640 1 -- 172.21.15.150:0/1593871346 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8080315a0 con 0x7fc8140b57d0 2024-03-17T23:47:07.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.393+0000 7fc811ffb640 1 -- 172.21.15.150:0/1593871346 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc808037070 con 0x7fc8140b57d0 2024-03-17T23:47:07.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.393+0000 7fc811ffb640 1 -- 172.21.15.150:0/1593871346 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc808031940 con 0x7fc8140b57d0 2024-03-17T23:47:07.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.393+0000 7fc819959640 1 -- 172.21.15.150:0/1593871346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8140b57d0 msgr2=0x7fc8140b5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:07.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.393+0000 7fc819959640 1 --2- 172.21.15.150:0/1593871346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8140b57d0 0x7fc8140b5c30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc8080097a0 tx=0x7fc80802b840 comp rx=0 tx=0).stop 2024-03-17T23:47:07.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.393+0000 7fc819959640 1 -- 172.21.15.150:0/1593871346 shutdown_connections 2024-03-17T23:47:07.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.393+0000 7fc819959640 1 --2- 172.21.15.150:0/1593871346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8140b57d0 0x7fc8140b5c30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.393+0000 7fc819959640 1 --2- 172.21.15.150:0/1593871346 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc81415e060 0x7fc8140d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.393+0000 7fc819959640 1 -- 172.21.15.150:0/1593871346 >> 172.21.15.150:0/1593871346 conn(0x7fc814073200 msgr2=0x7fc8140d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:07.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.393+0000 7fc819959640 1 -- 172.21.15.150:0/1593871346 shutdown_connections 2024-03-17T23:47:07.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.393+0000 7fc819959640 1 -- 172.21.15.150:0/1593871346 wait complete. 2024-03-17T23:47:07.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.394+0000 7fc819959640 1 Processor -- start 2024-03-17T23:47:07.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.395+0000 7fc819959640 1 -- start start 2024-03-17T23:47:07.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.395+0000 7fc819959640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc81406c6e0 0x7fc8141f77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:07.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.395+0000 7fc819959640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8140b57d0 0x7fc8140cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:07.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.395+0000 7fc819959640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc81415e060 0x7fc8140b3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:07.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.395+0000 7fc819959640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc8140b7f00 con 0x7fc81406c6e0 2024-03-17T23:47:07.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.395+0000 7fc819959640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc8140d3410 con 0x7fc8140b57d0 2024-03-17T23:47:07.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.395+0000 7fc819959640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc81415e440 con 0x7fc81415e060 2024-03-17T23:47:07.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.395+0000 7fc812ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc81406c6e0 0x7fc8141f77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:07.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.395+0000 7fc812ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc81406c6e0 0x7fc8141f77f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36104/0 (socket says 172.21.15.150:36104) 2024-03-17T23:47:07.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.395+0000 7fc812ffd640 1 -- 172.21.15.150:0/2954310582 learned_addr learned my addr 172.21.15.150:0/2954310582 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:07.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.395+0000 7fc8137fe640 1 --2- 172.21.15.150:0/2954310582 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc81415e060 0x7fc8140b3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:07.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.396+0000 7fc8127fc640 1 --2- 172.21.15.150:0/2954310582 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8140b57d0 0x7fc8140cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:07.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.396+0000 7fc812ffd640 1 -- 172.21.15.150:0/2954310582 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc81415e060 msgr2=0x7fc8140b3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:07.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.396+0000 7fc812ffd640 1 --2- 172.21.15.150:0/2954310582 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc81415e060 0x7fc8140b3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.396+0000 7fc812ffd640 1 -- 172.21.15.150:0/2954310582 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8140b57d0 msgr2=0x7fc8140cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:07.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.396+0000 7fc812ffd640 1 --2- 172.21.15.150:0/2954310582 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8140b57d0 0x7fc8140cd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.396+0000 7fc812ffd640 1 -- 172.21.15.150:0/2954310582 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc81415edb0 con 0x7fc81406c6e0 2024-03-17T23:47:07.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.396+0000 7fc812ffd640 1 --2- 172.21.15.150:0/2954310582 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc81406c6e0 0x7fc8141f77f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc80400efc0 tx=0x7fc80400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:07.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.396+0000 7fc7fbfff640 1 -- 172.21.15.150:0/2954310582 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc804018020 con 0x7fc81406c6e0 2024-03-17T23:47:07.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.396+0000 7fc7fbfff640 1 -- 172.21.15.150:0/2954310582 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc804014480 con 0x7fc81406c6e0 2024-03-17T23:47:07.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.396+0000 7fc819959640 1 -- 172.21.15.150:0/2954310582 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc81415f090 con 0x7fc81406c6e0 2024-03-17T23:47:07.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.396+0000 7fc7fbfff640 1 -- 172.21.15.150:0/2954310582 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc804014850 con 0x7fc81406c6e0 2024-03-17T23:47:07.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.396+0000 7fc819959640 1 -- 172.21.15.150:0/2954310582 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc81415f650 con 0x7fc81406c6e0 2024-03-17T23:47:07.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.397+0000 7fc819959640 1 -- 172.21.15.150:0/2954310582 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8140688b0 con 0x7fc81406c6e0 2024-03-17T23:47:07.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.397+0000 7fc7fbfff640 1 -- 172.21.15.150:0/2954310582 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc804005d10 con 0x7fc81406c6e0 2024-03-17T23:47:07.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.398+0000 7fc7fbfff640 1 --2- 172.21.15.150:0/2954310582 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc80006bbd0 0x7fc80006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:07.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.398+0000 7fc7fbfff640 1 -- 172.21.15.150:0/2954310582 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fc80408b6d0 con 0x7fc81406c6e0 2024-03-17T23:47:07.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.398+0000 7fc8127fc640 1 --2- 172.21.15.150:0/2954310582 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc80006bbd0 0x7fc80006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:07.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.398+0000 7fc8127fc640 1 --2- 172.21.15.150:0/2954310582 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc80006bbd0 0x7fc80006e090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc808036000 tx=0x7fc80803d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:07.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.402+0000 7fc7fbfff640 1 -- 172.21.15.150:0/2954310582 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc804022070 con 0x7fc81406c6e0 2024-03-17T23:47:07.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.575+0000 7fc819959640 1 -- 172.21.15.150:0/2954310582 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc81415fa20 con 0x7fc81406c6e0 2024-03-17T23:47:07.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.576+0000 7fc7fbfff640 1 -- 172.21.15.150:0/2954310582 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18550 (secure 0 0 0) 0x7fc804010040 con 0x7fc81406c6e0 2024-03-17T23:47:07.577 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:07.577 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:01.940019+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:07.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.578+0000 7fc819959640 1 -- 172.21.15.150:0/2954310582 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc80006bbd0 msgr2=0x7fc80006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:07.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.578+0000 7fc819959640 1 --2- 172.21.15.150:0/2954310582 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc80006bbd0 0x7fc80006e090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc808036000 tx=0x7fc80803d040 comp rx=0 tx=0).stop 2024-03-17T23:47:07.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.578+0000 7fc819959640 1 -- 172.21.15.150:0/2954310582 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc81406c6e0 msgr2=0x7fc8141f77f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:07.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.578+0000 7fc819959640 1 --2- 172.21.15.150:0/2954310582 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc81406c6e0 0x7fc8141f77f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc80400efc0 tx=0x7fc80400c490 comp rx=0 tx=0).stop 2024-03-17T23:47:07.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.578+0000 7fc819959640 1 -- 172.21.15.150:0/2954310582 shutdown_connections 2024-03-17T23:47:07.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.578+0000 7fc819959640 1 --2- 172.21.15.150:0/2954310582 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc81415e060 0x7fc8140b3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.578+0000 7fc819959640 1 --2- 172.21.15.150:0/2954310582 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc80006bbd0 0x7fc80006e090 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.578+0000 7fc819959640 1 --2- 172.21.15.150:0/2954310582 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8140b57d0 0x7fc8140cd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.578+0000 7fc819959640 1 --2- 172.21.15.150:0/2954310582 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc81406c6e0 0x7fc8141f77f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.578+0000 7fc819959640 1 -- 172.21.15.150:0/2954310582 >> 172.21.15.150:0/2954310582 conn(0x7fc814073200 msgr2=0x7fc8141e8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:07.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.579+0000 7fc819959640 1 -- 172.21.15.150:0/2954310582 shutdown_connections 2024-03-17T23:47:07.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.579+0000 7fc819959640 1 -- 172.21.15.150:0/2954310582 wait complete. 2024-03-17T23:47:07.590 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:46:53.610678+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:46:54.227335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-17T23:46:57.949235+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-17T23:47:07.591 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:47:07.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.694+0000 7f6991959640 1 Processor -- start 2024-03-17T23:47:07.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.694+0000 7f6991959640 1 -- start start 2024-03-17T23:47:07.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.695+0000 7f6991959640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f698c161520 0x7f698c0b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:07.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.695+0000 7f6991959640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f698c161d30 con 0x7f698c0d0ce0 2024-03-17T23:47:07.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.695+0000 7f6991959640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f698c162010 con 0x7f698c0b2bf0 2024-03-17T23:47:07.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.695+0000 7f6991959640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f698c0b9bf0 con 0x7f698c161520 2024-03-17T23:47:07.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.695+0000 7f698b7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f698c0d0ce0 0x7f698c0d1fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59306/0 (socket says 172.21.15.150:59306) 2024-03-17T23:47:07.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.695+0000 7f698b7fe640 1 -- 172.21.15.150:0/3228348769 learned_addr learned my addr 172.21.15.150:0/3228348769 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:07.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.695+0000 7f698affd640 1 --2- 172.21.15.150:0/3228348769 >> v2:172.21.15.157:3300/0 conn(0x7f698c161520 0x7f698c0b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:07.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.695+0000 7f698affd640 1 -- 172.21.15.150:0/3228348769 >> v1:172.21.15.157:6789/0 conn(0x7f698c0b2bf0 legacy=0x7f698c0d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:07.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.695+0000 7f698affd640 1 -- 172.21.15.150:0/3228348769 >> v1:172.21.15.150:6789/0 conn(0x7f698c0d0ce0 legacy=0x7f698c0d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:07.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.695+0000 7f6989ffb640 1 -- 172.21.15.150:0/3228348769 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 538614397 0 0) 0x7f698c161d30 con 0x7f698c0d0ce0 2024-03-17T23:47:07.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.695+0000 7f698affd640 1 -- 172.21.15.150:0/3228348769 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698c1622f0 con 0x7f698c161520 2024-03-17T23:47:07.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.696+0000 7f698affd640 1 --2- 172.21.15.150:0/3228348769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f698c161520 0x7f698c0b96b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f69840097a0 tx=0x7f698402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e875dbf081a1659a server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:07.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.696+0000 7f6989ffb640 1 -- 172.21.15.150:0/3228348769 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69840315a0 con 0x7f698c161520 2024-03-17T23:47:07.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.696+0000 7f6989ffb640 1 -- 172.21.15.150:0/3228348769 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6984037070 con 0x7f698c161520 2024-03-17T23:47:07.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.696+0000 7f6989ffb640 1 -- 172.21.15.150:0/3228348769 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6984031960 con 0x7f698c161520 2024-03-17T23:47:07.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.697+0000 7f6991959640 1 -- 172.21.15.150:0/3228348769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f698c161520 msgr2=0x7f698c0b96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:07.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.697+0000 7f6991959640 1 --2- 172.21.15.150:0/3228348769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f698c161520 0x7f698c0b96b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f69840097a0 tx=0x7f698402b840 comp rx=0 tx=0).stop 2024-03-17T23:47:07.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.697+0000 7f6991959640 1 -- 172.21.15.150:0/3228348769 shutdown_connections 2024-03-17T23:47:07.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.697+0000 7f6991959640 1 --2- 172.21.15.150:0/3228348769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f698c161520 0x7f698c0b96b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.697+0000 7f6991959640 1 -- 172.21.15.150:0/3228348769 >> 172.21.15.150:0/3228348769 conn(0x7f698c06ca00 msgr2=0x7f698c06ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:07.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.697+0000 7f6991959640 1 -- 172.21.15.150:0/3228348769 shutdown_connections 2024-03-17T23:47:07.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.697+0000 7f6991959640 1 -- 172.21.15.150:0/3228348769 wait complete. 2024-03-17T23:47:07.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.698+0000 7f6991959640 1 Processor -- start 2024-03-17T23:47:07.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.698+0000 7f6991959640 1 -- start start 2024-03-17T23:47:07.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.699+0000 7f6991959640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f698c0b2bf0 0x7f698c15b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:07.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.699+0000 7f6991959640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f698c0d0ce0 0x7f698c0b4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:07.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.699+0000 7f6991959640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f698c161520 0x7f698c15f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:07.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.699+0000 7f6991959640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f698c161900 con 0x7f698c161520 2024-03-17T23:47:07.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.699+0000 7f6991959640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f698c162940 con 0x7f698c0b2bf0 2024-03-17T23:47:07.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.699+0000 7f6991959640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f698c0d26b0 con 0x7f698c0d0ce0 2024-03-17T23:47:07.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.699+0000 7f698b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f698c161520 0x7f698c15f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:07.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.699+0000 7f698b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f698c161520 0x7f698c15f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36108/0 (socket says 172.21.15.150:36108) 2024-03-17T23:47:07.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.699+0000 7f698b7fe640 1 -- 172.21.15.150:0/1055204604 learned_addr learned my addr 172.21.15.150:0/1055204604 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:07.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.699+0000 7f698affd640 1 --2- 172.21.15.150:0/1055204604 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f698c0b2bf0 0x7f698c15b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:07.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.699+0000 7f698a7fc640 1 --2- 172.21.15.150:0/1055204604 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f698c0d0ce0 0x7f698c0b4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:07.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.699+0000 7f698affd640 1 -- 172.21.15.150:0/1055204604 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f698c0d0ce0 msgr2=0x7f698c0b4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:07.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.699+0000 7f698affd640 1 --2- 172.21.15.150:0/1055204604 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f698c0d0ce0 0x7f698c0b4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.699+0000 7f698affd640 1 -- 172.21.15.150:0/1055204604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f698c161520 msgr2=0x7f698c15f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:07.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.699+0000 7f698affd640 1 --2- 172.21.15.150:0/1055204604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f698c161520 0x7f698c15f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.699+0000 7f698affd640 1 -- 172.21.15.150:0/1055204604 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698c0b4580 con 0x7f698c0b2bf0 2024-03-17T23:47:07.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.699+0000 7f698b7fe640 1 --2- 172.21.15.150:0/1055204604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f698c161520 0x7f698c15f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:07.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.699+0000 7f698affd640 1 --2- 172.21.15.150:0/1055204604 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f698c0b2bf0 0x7f698c15b330 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6984033000 tx=0x7f69840046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:07.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.699+0000 7f6973fff640 1 -- 172.21.15.150:0/1055204604 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6984032840 con 0x7f698c0b2bf0 2024-03-17T23:47:07.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.699+0000 7f6973fff640 1 -- 172.21.15.150:0/1055204604 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6984032a20 con 0x7f698c0b2bf0 2024-03-17T23:47:07.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.699+0000 7f6973fff640 1 -- 172.21.15.150:0/1055204604 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6984032e10 con 0x7f698c0b2bf0 2024-03-17T23:47:07.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.699+0000 7f6991959640 1 -- 172.21.15.150:0/1055204604 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f698c0cd5b0 con 0x7f698c0b2bf0 2024-03-17T23:47:07.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.699+0000 7f6991959640 1 -- 172.21.15.150:0/1055204604 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f698c0cdb70 con 0x7f698c0b2bf0 2024-03-17T23:47:07.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.700+0000 7f6991959640 1 -- 172.21.15.150:0/1055204604 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f698c0681a0 con 0x7f698c0b2bf0 2024-03-17T23:47:07.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.701+0000 7f6973fff640 1 -- 172.21.15.150:0/1055204604 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f698404d3f0 con 0x7f698c0b2bf0 2024-03-17T23:47:07.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.701+0000 7f6973fff640 1 --2- 172.21.15.150:0/1055204604 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f697806bbd0 0x7f697806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:07.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.701+0000 7f6973fff640 1 -- 172.21.15.150:0/1055204604 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f698402f030 con 0x7f698c0b2bf0 2024-03-17T23:47:07.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.701+0000 7f698a7fc640 1 --2- 172.21.15.150:0/1055204604 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f697806bbd0 0x7f697806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:07.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.702+0000 7f698a7fc640 1 --2- 172.21.15.150:0/1055204604 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f697806bbd0 0x7f697806e090 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f698c15d090 tx=0x7f6980009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:07.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.710+0000 7f6973fff640 1 -- 172.21.15.150:0/1055204604 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6984040070 con 0x7f698c0b2bf0 2024-03-17T23:47:07.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.882+0000 7f6991959640 1 -- 172.21.15.150:0/1055204604 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f698c0d19c0 con 0x7f698c0b2bf0 2024-03-17T23:47:07.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.882+0000 7f6973fff640 1 -- 172.21.15.150:0/1055204604 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f698407af20 con 0x7f698c0b2bf0 2024-03-17T23:47:07.883 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:47:07.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.884+0000 7f6991959640 1 -- 172.21.15.150:0/1055204604 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f697806bbd0 msgr2=0x7f697806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:07.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.884+0000 7f6991959640 1 --2- 172.21.15.150:0/1055204604 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f697806bbd0 0x7f697806e090 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f698c15d090 tx=0x7f6980009040 comp rx=0 tx=0).stop 2024-03-17T23:47:07.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.884+0000 7f6991959640 1 -- 172.21.15.150:0/1055204604 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f698c0b2bf0 msgr2=0x7f698c15b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:07.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.884+0000 7f6991959640 1 --2- 172.21.15.150:0/1055204604 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f698c0b2bf0 0x7f698c15b330 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6984033000 tx=0x7f69840046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.884+0000 7f6991959640 1 -- 172.21.15.150:0/1055204604 shutdown_connections 2024-03-17T23:47:07.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.884+0000 7f6991959640 1 --2- 172.21.15.150:0/1055204604 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f698c161520 0x7f698c15f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.884+0000 7f6991959640 1 --2- 172.21.15.150:0/1055204604 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f698c0d0ce0 0x7f698c0b4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.884+0000 7f6991959640 1 --2- 172.21.15.150:0/1055204604 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f697806bbd0 0x7f697806e090 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.884+0000 7f6991959640 1 --2- 172.21.15.150:0/1055204604 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f698c0b2bf0 0x7f698c15b330 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:07.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.884+0000 7f6991959640 1 -- 172.21.15.150:0/1055204604 >> 172.21.15.150:0/1055204604 conn(0x7f698c06ca00 msgr2=0x7f698c0cf3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:07.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.885+0000 7f6991959640 1 -- 172.21.15.150:0/1055204604 shutdown_connections 2024-03-17T23:47:07.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:07.885+0000 7f6991959640 1 -- 172.21.15.150:0/1055204604 wait complete. 2024-03-17T23:47:07.896 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:47:08.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.002+0000 7fca388bf640 1 Processor -- start 2024-03-17T23:47:08.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.002+0000 7fca388bf640 1 -- start start 2024-03-17T23:47:08.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.002+0000 7fca388bf640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fca341479c0 0x7fca3406f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:08.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.003+0000 7fca388bf640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fca34072980 0x7fca340ae380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:08.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.003+0000 7fca388bf640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca3415e8e0 con 0x7fca3406fab0 2024-03-17T23:47:08.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.003+0000 7fca388bf640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fca3415ebc0 con 0x7fca341479c0 2024-03-17T23:47:08.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.003+0000 7fca388bf640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fca3415eea0 con 0x7fca34072980 2024-03-17T23:47:08.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.003+0000 7fca3259c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fca341479c0 0x7fca3406f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:08.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.003+0000 7fca2bfff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fca3406fab0 0x7fca34072270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59308/0 (socket says 172.21.15.150:59308) 2024-03-17T23:47:08.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.003+0000 7fca3259c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fca341479c0 0x7fca3406f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36110/0 (socket says 172.21.15.150:36110) 2024-03-17T23:47:08.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.003+0000 7fca2bfff640 1 -- 172.21.15.150:0/3360359893 learned_addr learned my addr 172.21.15.150:0/3360359893 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:08.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.003+0000 7fca32d9d640 1 --2- 172.21.15.150:0/3360359893 >> v2:172.21.15.157:3300/0 conn(0x7fca34072980 0x7fca340ae380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:08.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.003+0000 7fca3259c640 1 -- 172.21.15.150:0/3360359893 >> v2:172.21.15.157:3300/0 conn(0x7fca34072980 msgr2=0x7fca340ae380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:08.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.003+0000 7fca3259c640 1 --2- 172.21.15.150:0/3360359893 >> v2:172.21.15.157:3300/0 conn(0x7fca34072980 0x7fca340ae380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.003+0000 7fca3259c640 1 -- 172.21.15.150:0/3360359893 >> v1:172.21.15.150:6789/0 conn(0x7fca3406fab0 legacy=0x7fca34072270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:08.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.003+0000 7fca3259c640 1 -- 172.21.15.150:0/3360359893 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca3415f180 con 0x7fca341479c0 2024-03-17T23:47:08.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.003+0000 7fca31d9b640 1 -- 172.21.15.150:0/3360359893 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 775924922 0 0) 0x7fca3415e8e0 con 0x7fca3406fab0 2024-03-17T23:47:08.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.004+0000 7fca3259c640 1 --2- 172.21.15.150:0/3360359893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca341479c0 0x7fca3406f570 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fca340ad140 tx=0x7fca2402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d645972d0c7d2649 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:08.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.004+0000 7fca31d9b640 1 -- 172.21.15.150:0/3360359893 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca24038070 con 0x7fca341479c0 2024-03-17T23:47:08.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.004+0000 7fca31d9b640 1 -- 172.21.15.150:0/3360359893 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca24031710 con 0x7fca341479c0 2024-03-17T23:47:08.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.004+0000 7fca31d9b640 1 -- 172.21.15.150:0/3360359893 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca24031ae0 con 0x7fca341479c0 2024-03-17T23:47:08.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.004+0000 7fca388bf640 1 -- 172.21.15.150:0/3360359893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca341479c0 msgr2=0x7fca3406f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:08.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.004+0000 7fca388bf640 1 --2- 172.21.15.150:0/3360359893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca341479c0 0x7fca3406f570 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fca340ad140 tx=0x7fca2402b7b0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.004+0000 7fca388bf640 1 -- 172.21.15.150:0/3360359893 shutdown_connections 2024-03-17T23:47:08.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.004+0000 7fca388bf640 1 --2- 172.21.15.150:0/3360359893 >> v2:172.21.15.157:3300/0 conn(0x7fca34072980 0x7fca340ae380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.004+0000 7fca388bf640 1 --2- 172.21.15.150:0/3360359893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca341479c0 0x7fca3406f570 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.004+0000 7fca388bf640 1 -- 172.21.15.150:0/3360359893 >> 172.21.15.150:0/3360359893 conn(0x7fca3406b370 msgr2=0x7fca3406b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:08.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.005+0000 7fca388bf640 1 -- 172.21.15.150:0/3360359893 shutdown_connections 2024-03-17T23:47:08.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.005+0000 7fca388bf640 1 -- 172.21.15.150:0/3360359893 wait complete. 2024-03-17T23:47:08.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.006+0000 7fca388bf640 1 Processor -- start 2024-03-17T23:47:08.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.006+0000 7fca388bf640 1 -- start start 2024-03-17T23:47:08.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.007+0000 7fca388bf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca3406fab0 0x7fca3415d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:08.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.007+0000 7fca388bf640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca34072980 0x7fca341395a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:08.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.007+0000 7fca388bf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca341479c0 0x7fca34139ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:08.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.007+0000 7fca388bf640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fca340aea40 con 0x7fca3406fab0 2024-03-17T23:47:08.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.007+0000 7fca388bf640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fca340ae8c0 con 0x7fca34072980 2024-03-17T23:47:08.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.007+0000 7fca388bf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fca340aebc0 con 0x7fca341479c0 2024-03-17T23:47:08.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.007+0000 7fca3259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca3406fab0 0x7fca3415d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:08.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.007+0000 7fca3259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca3406fab0 0x7fca3415d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36122/0 (socket says 172.21.15.150:36122) 2024-03-17T23:47:08.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.007+0000 7fca3259c640 1 -- 172.21.15.150:0/657045926 learned_addr learned my addr 172.21.15.150:0/657045926 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:08.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.007+0000 7fca2bfff640 1 --2- 172.21.15.150:0/657045926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca34072980 0x7fca341395a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:08.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.007+0000 7fca32d9d640 1 --2- 172.21.15.150:0/657045926 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca341479c0 0x7fca34139ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:08.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.007+0000 7fca3259c640 1 -- 172.21.15.150:0/657045926 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca341479c0 msgr2=0x7fca34139ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:08.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.007+0000 7fca3259c640 1 --2- 172.21.15.150:0/657045926 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca341479c0 0x7fca34139ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.007+0000 7fca3259c640 1 -- 172.21.15.150:0/657045926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca34072980 msgr2=0x7fca341395a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:08.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.007+0000 7fca3259c640 1 --2- 172.21.15.150:0/657045926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca34072980 0x7fca341395a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.007+0000 7fca3259c640 1 -- 172.21.15.150:0/657045926 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca340b2a20 con 0x7fca3406fab0 2024-03-17T23:47:08.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.007+0000 7fca3259c640 1 --2- 172.21.15.150:0/657045926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca3406fab0 0x7fca3415d900 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fca24000c00 tx=0x7fca240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:08.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.007+0000 7fca2affd640 1 -- 172.21.15.150:0/657045926 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca24042070 con 0x7fca3406fab0 2024-03-17T23:47:08.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.007+0000 7fca2affd640 1 -- 172.21.15.150:0/657045926 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca2403d870 con 0x7fca3406fab0 2024-03-17T23:47:08.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.007+0000 7fca388bf640 1 -- 172.21.15.150:0/657045926 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca340c50e0 con 0x7fca3406fab0 2024-03-17T23:47:08.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.007+0000 7fca2affd640 1 -- 172.21.15.150:0/657045926 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca24038040 con 0x7fca3406fab0 2024-03-17T23:47:08.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.007+0000 7fca388bf640 1 -- 172.21.15.150:0/657045926 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca340b0780 con 0x7fca3406fab0 2024-03-17T23:47:08.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.009+0000 7fca388bf640 1 -- 172.21.15.150:0/657045926 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca340681a0 con 0x7fca3406fab0 2024-03-17T23:47:08.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.009+0000 7fca2affd640 1 -- 172.21.15.150:0/657045926 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fca2403b070 con 0x7fca3406fab0 2024-03-17T23:47:08.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.009+0000 7fca2affd640 1 --2- 172.21.15.150:0/657045926 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fca1406bbd0 0x7fca1406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:08.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.009+0000 7fca2affd640 1 -- 172.21.15.150:0/657045926 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fca240ac510 con 0x7fca3406fab0 2024-03-17T23:47:08.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.009+0000 7fca2bfff640 1 --2- 172.21.15.150:0/657045926 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fca1406bbd0 0x7fca1406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:08.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.010+0000 7fca2bfff640 1 --2- 172.21.15.150:0/657045926 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fca1406bbd0 0x7fca1406e090 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fca3413f3b0 tx=0x7fca1c007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:08.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.014+0000 7fca2affd640 1 -- 172.21.15.150:0/657045926 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fca2407a700 con 0x7fca3406fab0 2024-03-17T23:47:08.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.185+0000 7fca388bf640 1 -- 172.21.15.150:0/657045926 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fca34147170 con 0x7fca3406fab0 2024-03-17T23:47:08.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.186+0000 7fca2affd640 1 -- 172.21.15.150:0/657045926 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7fca2407e7a0 con 0x7fca3406fab0 2024-03-17T23:47:08.187 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:08.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.188+0000 7fca388bf640 1 -- 172.21.15.150:0/657045926 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fca1406bbd0 msgr2=0x7fca1406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:08.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.188+0000 7fca388bf640 1 --2- 172.21.15.150:0/657045926 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fca1406bbd0 0x7fca1406e090 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fca3413f3b0 tx=0x7fca1c007d50 comp rx=0 tx=0).stop 2024-03-17T23:47:08.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.188+0000 7fca388bf640 1 -- 172.21.15.150:0/657045926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca3406fab0 msgr2=0x7fca3415d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:08.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.188+0000 7fca388bf640 1 --2- 172.21.15.150:0/657045926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca3406fab0 0x7fca3415d900 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fca24000c00 tx=0x7fca240046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.188+0000 7fca388bf640 1 -- 172.21.15.150:0/657045926 shutdown_connections 2024-03-17T23:47:08.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.188+0000 7fca388bf640 1 --2- 172.21.15.150:0/657045926 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fca341479c0 0x7fca34139ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.188+0000 7fca388bf640 1 --2- 172.21.15.150:0/657045926 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fca1406bbd0 0x7fca1406e090 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.188+0000 7fca388bf640 1 --2- 172.21.15.150:0/657045926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fca34072980 0x7fca341395a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.188+0000 7fca388bf640 1 --2- 172.21.15.150:0/657045926 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fca3406fab0 0x7fca3415d900 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.188+0000 7fca388bf640 1 -- 172.21.15.150:0/657045926 >> 172.21.15.150:0/657045926 conn(0x7fca3406b370 msgr2=0x7fca340ccb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:08.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.188+0000 7fca388bf640 1 -- 172.21.15.150:0/657045926 shutdown_connections 2024-03-17T23:47:08.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.188+0000 7fca388bf640 1 -- 172.21.15.150:0/657045926 wait complete. 2024-03-17T23:47:08.199 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:47:08.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.302+0000 7f2e18a81640 1 Processor -- start 2024-03-17T23:47:08.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.302+0000 7f2e18a81640 1 -- start start 2024-03-17T23:47:08.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.303+0000 7f2e18a81640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2e14155320 0x7f2e1415bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:08.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.303+0000 7f2e18a81640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2e1406dd30 0x7f2e140ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:08.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.303+0000 7f2e18a81640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e140cf3b0 con 0x7f2e140cd250 2024-03-17T23:47:08.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.303+0000 7f2e18a81640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f2e140cd6c0 con 0x7f2e1406dd30 2024-03-17T23:47:08.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.303+0000 7f2e18a81640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f2e1415c550 con 0x7f2e14155320 2024-03-17T23:47:08.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.303+0000 7f2e0bfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2e1406dd30 0x7f2e140ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:08.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.303+0000 7f2e0bfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2e1406dd30 0x7f2e140ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36128/0 (socket says 172.21.15.150:36128) 2024-03-17T23:47:08.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.303+0000 7f2e1259c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2e14155320 0x7f2e1415bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:08.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.303+0000 7f2e0bfff640 1 -- 172.21.15.150:0/3755525428 learned_addr learned my addr 172.21.15.150:0/3755525428 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:08.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.303+0000 7f2e1259c640 1 -- 172.21.15.150:0/3755525428 >> v2:172.21.15.150:3300/0 conn(0x7f2e1406dd30 msgr2=0x7f2e140ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:08.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.303+0000 7f2e1259c640 1 --2- 172.21.15.150:0/3755525428 >> v2:172.21.15.150:3300/0 conn(0x7f2e1406dd30 0x7f2e140ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.303+0000 7f2e1259c640 1 -- 172.21.15.150:0/3755525428 >> v1:172.21.15.157:6789/0 conn(0x7f2e140cd250 legacy=0x7f2e1415e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:08.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.303+0000 7f2e1259c640 1 -- 172.21.15.150:0/3755525428 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e140cf690 con 0x7f2e14155320 2024-03-17T23:47:08.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.303+0000 7f2e11d9b640 1 -- 172.21.15.150:0/3755525428 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 104075478 0 0) 0x7f2e140cf3b0 con 0x7f2e140cd250 2024-03-17T23:47:08.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.303+0000 7f2e0bfff640 1 --2- 172.21.15.150:0/3755525428 >> v2:172.21.15.150:3300/0 conn(0x7f2e1406dd30 0x7f2e140ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:08.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.304+0000 7f2e1259c640 1 --2- 172.21.15.150:0/3755525428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e14155320 0x7f2e1415bfd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2e1406d5a0 tx=0x7f2e0402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=966a097e3d84dfe6 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:08.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.304+0000 7f2e11d9b640 1 -- 172.21.15.150:0/3755525428 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e04035020 con 0x7f2e14155320 2024-03-17T23:47:08.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.304+0000 7f2e11d9b640 1 -- 172.21.15.150:0/3755525428 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e04031830 con 0x7f2e14155320 2024-03-17T23:47:08.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.304+0000 7f2e11d9b640 1 -- 172.21.15.150:0/3755525428 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e04031c00 con 0x7f2e14155320 2024-03-17T23:47:08.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.305+0000 7f2e18a81640 1 -- 172.21.15.150:0/3755525428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e14155320 msgr2=0x7f2e1415bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:08.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.305+0000 7f2e18a81640 1 --2- 172.21.15.150:0/3755525428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e14155320 0x7f2e1415bfd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2e1406d5a0 tx=0x7f2e0402b8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.305+0000 7f2e18a81640 1 -- 172.21.15.150:0/3755525428 shutdown_connections 2024-03-17T23:47:08.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.305+0000 7f2e18a81640 1 --2- 172.21.15.150:0/3755525428 >> v2:172.21.15.150:3300/0 conn(0x7f2e1406dd30 0x7f2e140ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.305+0000 7f2e18a81640 1 --2- 172.21.15.150:0/3755525428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e14155320 0x7f2e1415bfd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.305+0000 7f2e18a81640 1 -- 172.21.15.150:0/3755525428 >> 172.21.15.150:0/3755525428 conn(0x7f2e1406b190 msgr2=0x7f2e1406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:08.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.305+0000 7f2e18a81640 1 -- 172.21.15.150:0/3755525428 shutdown_connections 2024-03-17T23:47:08.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.305+0000 7f2e18a81640 1 -- 172.21.15.150:0/3755525428 wait complete. 2024-03-17T23:47:08.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.306+0000 7f2e18a81640 1 Processor -- start 2024-03-17T23:47:08.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.306+0000 7f2e18a81640 1 -- start start 2024-03-17T23:47:08.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.307+0000 7f2e18a81640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e1406dd30 0x7f2e141f1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:08.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.307+0000 7f2e18a81640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e140cd250 0x7f2e1414f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:08.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.307+0000 7f2e18a81640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e14155320 0x7f2e1406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:08.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.307+0000 7f2e18a81640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1406e190 con 0x7f2e14155320 2024-03-17T23:47:08.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.307+0000 7f2e18a81640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2e140cf060 con 0x7f2e1406dd30 2024-03-17T23:47:08.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.307+0000 7f2e18a81640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2e14155700 con 0x7f2e140cd250 2024-03-17T23:47:08.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.307+0000 7f2e12d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e14155320 0x7f2e1406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:08.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.307+0000 7f2e1259c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e1406dd30 0x7f2e141f1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:08.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.307+0000 7f2e12d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e14155320 0x7f2e1406ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36136/0 (socket says 172.21.15.150:36136) 2024-03-17T23:47:08.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.307+0000 7f2e12d9d640 1 -- 172.21.15.150:0/3177259694 learned_addr learned my addr 172.21.15.150:0/3177259694 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:08.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.307+0000 7f2e0bfff640 1 --2- 172.21.15.150:0/3177259694 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e140cd250 0x7f2e1414f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:08.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.307+0000 7f2e12d9d640 1 -- 172.21.15.150:0/3177259694 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e140cd250 msgr2=0x7f2e1414f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:08.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.307+0000 7f2e12d9d640 1 --2- 172.21.15.150:0/3177259694 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e140cd250 0x7f2e1414f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.307+0000 7f2e12d9d640 1 -- 172.21.15.150:0/3177259694 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e1406dd30 msgr2=0x7f2e141f1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:08.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.307+0000 7f2e12d9d640 1 --2- 172.21.15.150:0/3177259694 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e1406dd30 0x7f2e141f1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.307+0000 7f2e12d9d640 1 -- 172.21.15.150:0/3177259694 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e141f5160 con 0x7f2e14155320 2024-03-17T23:47:08.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.307+0000 7f2e12d9d640 1 --2- 172.21.15.150:0/3177259694 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e14155320 0x7f2e1406ed40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c00c790 tx=0x7f2e0c00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:08.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.307+0000 7f2e0affd640 1 -- 172.21.15.150:0/3177259694 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c003da0 con 0x7f2e14155320 2024-03-17T23:47:08.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.308+0000 7f2e0affd640 1 -- 172.21.15.150:0/3177259694 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e0c005070 con 0x7f2e14155320 2024-03-17T23:47:08.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.308+0000 7f2e18a81640 1 -- 172.21.15.150:0/3177259694 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e141f5440 con 0x7f2e14155320 2024-03-17T23:47:08.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.308+0000 7f2e0affd640 1 -- 172.21.15.150:0/3177259694 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c005460 con 0x7f2e14155320 2024-03-17T23:47:08.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.308+0000 7f2e18a81640 1 -- 172.21.15.150:0/3177259694 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e141f5a00 con 0x7f2e14155320 2024-03-17T23:47:08.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.309+0000 7f2e18a81640 1 -- 172.21.15.150:0/3177259694 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e140681a0 con 0x7f2e14155320 2024-03-17T23:47:08.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.309+0000 7f2e0affd640 1 -- 172.21.15.150:0/3177259694 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2e0c005600 con 0x7f2e14155320 2024-03-17T23:47:08.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.310+0000 7f2e0affd640 1 --2- 172.21.15.150:0/3177259694 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2df406bbd0 0x7f2df406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:08.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.310+0000 7f2e1259c640 1 --2- 172.21.15.150:0/3177259694 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2df406bbd0 0x7f2df406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:08.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.310+0000 7f2e0affd640 1 -- 172.21.15.150:0/3177259694 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f2e0c08ab20 con 0x7f2e14155320 2024-03-17T23:47:08.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.310+0000 7f2e1259c640 1 --2- 172.21.15.150:0/3177259694 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2df406bbd0 0x7f2df406e090 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2e04036000 tx=0x7f2e04008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:08.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.314+0000 7f2e0affd640 1 -- 172.21.15.150:0/3177259694 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2e0c058d10 con 0x7f2e14155320 2024-03-17T23:47:08.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.487+0000 7f2e18a81640 1 -- 172.21.15.150:0/3177259694 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2e1415bfb0 con 0x7f2e14155320 2024-03-17T23:47:08.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.487+0000 7f2e0affd640 1 -- 172.21.15.150:0/3177259694 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f2e0c05cdb0 con 0x7f2e14155320 2024-03-17T23:47:08.519 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:47:08.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.491+0000 7f2e18a81640 1 -- 172.21.15.150:0/3177259694 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2df406bbd0 msgr2=0x7f2df406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:08.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.491+0000 7f2e18a81640 1 --2- 172.21.15.150:0/3177259694 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2df406bbd0 0x7f2df406e090 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2e04036000 tx=0x7f2e04008030 comp rx=0 tx=0).stop 2024-03-17T23:47:08.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.491+0000 7f2e18a81640 1 -- 172.21.15.150:0/3177259694 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e14155320 msgr2=0x7f2e1406ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:08.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.491+0000 7f2e18a81640 1 --2- 172.21.15.150:0/3177259694 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e14155320 0x7f2e1406ed40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c00c790 tx=0x7f2e0c00cc50 comp rx=0 tx=0).stop 2024-03-17T23:47:08.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.491+0000 7f2e18a81640 1 -- 172.21.15.150:0/3177259694 shutdown_connections 2024-03-17T23:47:08.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.491+0000 7f2e18a81640 1 --2- 172.21.15.150:0/3177259694 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e14155320 0x7f2e1406ed40 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.491+0000 7f2e18a81640 1 --2- 172.21.15.150:0/3177259694 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e140cd250 0x7f2e1414f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.491+0000 7f2e18a81640 1 --2- 172.21.15.150:0/3177259694 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2df406bbd0 0x7f2df406e090 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.491+0000 7f2e18a81640 1 --2- 172.21.15.150:0/3177259694 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e1406dd30 0x7f2e141f1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.491+0000 7f2e18a81640 1 -- 172.21.15.150:0/3177259694 >> 172.21.15.150:0/3177259694 conn(0x7f2e1406b190 msgr2=0x7f2e141542c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:08.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.491+0000 7f2e18a81640 1 -- 172.21.15.150:0/3177259694 shutdown_connections 2024-03-17T23:47:08.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.491+0000 7f2e18a81640 1 -- 172.21.15.150:0/3177259694 wait complete. 2024-03-17T23:47:08.521 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-17T23:47:08.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.625+0000 7f96d6f9d640 1 Processor -- start 2024-03-17T23:47:08.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.625+0000 7f96d6f9d640 1 -- start start 2024-03-17T23:47:08.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.626+0000 7f96d6f9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96d015dee0 0x7f96d00d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:08.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.626+0000 7f96d6f9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f96d00cef20 0x7f96d00b6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:08.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.626+0000 7f96d6f9d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96d00d25f0 con 0x7f96d00b95f0 2024-03-17T23:47:08.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.626+0000 7f96d6f9d640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f96d00d3200 con 0x7f96d00cef20 2024-03-17T23:47:08.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.626+0000 7f96d6f9d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f96d00d28d0 con 0x7f96d015dee0 2024-03-17T23:47:08.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.626+0000 7f96d679c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f96d00cef20 0x7f96d00b6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:08.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.626+0000 7f96d5f9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96d015dee0 0x7f96d00d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:08.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.626+0000 7f96d579a640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f96d00b95f0 0x7f96d00ce810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:37762/0 (socket says 172.21.15.150:37762) 2024-03-17T23:47:08.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.626+0000 7f96d579a640 1 -- 172.21.15.150:0/1694638215 learned_addr learned my addr 172.21.15.150:0/1694638215 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:08.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.627+0000 7f96d5f9b640 1 -- 172.21.15.150:0/1694638215 >> v2:172.21.15.157:3300/0 conn(0x7f96d00cef20 msgr2=0x7f96d00b6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:08.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.627+0000 7f96d5f9b640 1 --2- 172.21.15.150:0/1694638215 >> v2:172.21.15.157:3300/0 conn(0x7f96d00cef20 0x7f96d00b6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.627+0000 7f96d5f9b640 1 -- 172.21.15.150:0/1694638215 >> v1:172.21.15.157:6789/0 conn(0x7f96d00b95f0 legacy=0x7f96d00ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:08.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.627+0000 7f96d5f9b640 1 -- 172.21.15.150:0/1694638215 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96d00d2bb0 con 0x7f96d015dee0 2024-03-17T23:47:08.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.627+0000 7f96d679c640 1 --2- 172.21.15.150:0/1694638215 >> v2:172.21.15.157:3300/0 conn(0x7f96d00cef20 0x7f96d00b6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:08.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.627+0000 7f96d5f9b640 1 --2- 172.21.15.150:0/1694638215 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96d015dee0 0x7f96d00d1c80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009d90 tx=0x7f96cc02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a79f7bd7108e845 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:08.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.627+0000 7f96d4f99640 1 -- 172.21.15.150:0/1694638215 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96cc039070 con 0x7f96d015dee0 2024-03-17T23:47:08.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.628+0000 7f96d4f99640 1 -- 172.21.15.150:0/1694638215 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96cc02cc50 con 0x7f96d015dee0 2024-03-17T23:47:08.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.628+0000 7f96d4f99640 1 -- 172.21.15.150:0/1694638215 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96cc034440 con 0x7f96d015dee0 2024-03-17T23:47:08.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.628+0000 7f96d6f9d640 1 -- 172.21.15.150:0/1694638215 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96d015dee0 msgr2=0x7f96d00d1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:08.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.628+0000 7f96d6f9d640 1 --2- 172.21.15.150:0/1694638215 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96d015dee0 0x7f96d00d1c80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009d90 tx=0x7f96cc02bea0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.628+0000 7f96d6f9d640 1 -- 172.21.15.150:0/1694638215 shutdown_connections 2024-03-17T23:47:08.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.628+0000 7f96d6f9d640 1 --2- 172.21.15.150:0/1694638215 >> v2:172.21.15.157:3300/0 conn(0x7f96d00cef20 0x7f96d00b6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.628+0000 7f96d6f9d640 1 --2- 172.21.15.150:0/1694638215 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96d015dee0 0x7f96d00d1c80 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.628+0000 7f96d6f9d640 1 -- 172.21.15.150:0/1694638215 >> 172.21.15.150:0/1694638215 conn(0x7f96d006c820 msgr2=0x7f96d006cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:08.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.628+0000 7f96d6f9d640 1 -- 172.21.15.150:0/1694638215 shutdown_connections 2024-03-17T23:47:08.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.628+0000 7f96d6f9d640 1 -- 172.21.15.150:0/1694638215 wait complete. 2024-03-17T23:47:08.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.629+0000 7f96d6f9d640 1 Processor -- start 2024-03-17T23:47:08.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.629+0000 7f96d6f9d640 1 -- start start 2024-03-17T23:47:08.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.630+0000 7f96d6f9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96d00cef20 0x7f96d00b5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:08.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.630+0000 7f96d6f9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96d015dee0 0x7f96d015f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:08.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.630+0000 7f96d6f9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96d015adc0 0x7f96d00cce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:08.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.630+0000 7f96d6f9d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f96d00d2340 con 0x7f96d015dee0 2024-03-17T23:47:08.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.630+0000 7f96d6f9d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f96d00d21c0 con 0x7f96d00cef20 2024-03-17T23:47:08.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.630+0000 7f96d6f9d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f96d00d3690 con 0x7f96d015adc0 2024-03-17T23:47:08.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.630+0000 7f96d579a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96d015dee0 0x7f96d015f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:08.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.630+0000 7f96d579a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96d015dee0 0x7f96d015f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36142/0 (socket says 172.21.15.150:36142) 2024-03-17T23:47:08.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.630+0000 7f96d579a640 1 -- 172.21.15.150:0/1965933365 learned_addr learned my addr 172.21.15.150:0/1965933365 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:08.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.630+0000 7f96d679c640 1 --2- 172.21.15.150:0/1965933365 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96d015adc0 0x7f96d00cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:08.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.630+0000 7f96d5f9b640 1 --2- 172.21.15.150:0/1965933365 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96d00cef20 0x7f96d00b5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:08.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.630+0000 7f96d579a640 1 -- 172.21.15.150:0/1965933365 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96d015adc0 msgr2=0x7f96d00cce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:08.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.630+0000 7f96d579a640 1 --2- 172.21.15.150:0/1965933365 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96d015adc0 0x7f96d00cce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.630+0000 7f96d579a640 1 -- 172.21.15.150:0/1965933365 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96d00cef20 msgr2=0x7f96d00b5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:08.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.630+0000 7f96d579a640 1 --2- 172.21.15.150:0/1965933365 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96d00cef20 0x7f96d00b5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.630+0000 7f96d579a640 1 -- 172.21.15.150:0/1965933365 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96d00b6330 con 0x7f96d015dee0 2024-03-17T23:47:08.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.630+0000 7f96d679c640 1 --2- 172.21.15.150:0/1965933365 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96d015adc0 0x7f96d00cce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:08.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.631+0000 7f96d579a640 1 --2- 172.21.15.150:0/1965933365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96d015dee0 0x7f96d015f2d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f96c000c430 tx=0x7f96c000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:08.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.631+0000 7f96c67fc640 1 -- 172.21.15.150:0/1965933365 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c0017070 con 0x7f96d015dee0 2024-03-17T23:47:08.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.631+0000 7f96c67fc640 1 -- 172.21.15.150:0/1965933365 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96c0005ce0 con 0x7f96d015dee0 2024-03-17T23:47:08.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.631+0000 7f96c67fc640 1 -- 172.21.15.150:0/1965933365 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c00072f0 con 0x7f96d015dee0 2024-03-17T23:47:08.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.631+0000 7f96d6f9d640 1 -- 172.21.15.150:0/1965933365 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96d015f810 con 0x7f96d015dee0 2024-03-17T23:47:08.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.631+0000 7f96d6f9d640 1 -- 172.21.15.150:0/1965933365 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96d015e7f0 con 0x7f96d015dee0 2024-03-17T23:47:08.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.632+0000 7f96d6f9d640 1 -- 172.21.15.150:0/1965933365 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96d006cec0 con 0x7f96d015dee0 2024-03-17T23:47:08.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.632+0000 7f96c67fc640 1 -- 172.21.15.150:0/1965933365 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f96c001a070 con 0x7f96d015dee0 2024-03-17T23:47:08.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.633+0000 7f96c67fc640 1 --2- 172.21.15.150:0/1965933365 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f96b406bbd0 0x7f96b406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:08.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.633+0000 7f96c67fc640 1 -- 172.21.15.150:0/1965933365 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f96c0089f80 con 0x7f96d015dee0 2024-03-17T23:47:08.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.633+0000 7f96d5f9b640 1 --2- 172.21.15.150:0/1965933365 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f96b406bbd0 0x7f96b406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:08.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.633+0000 7f96d5f9b640 1 --2- 172.21.15.150:0/1965933365 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f96b406bbd0 0x7f96b406e090 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f96cc036000 tx=0x7f96cc030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:08.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.637+0000 7f96c67fc640 1 -- 172.21.15.150:0/1965933365 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f96d006cec0 con 0x7f96d015dee0 2024-03-17T23:47:08.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.807+0000 7f96d6f9d640 1 -- 172.21.15.150:0/1965933365 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f96d0074c40 con 0x7f96d015dee0 2024-03-17T23:47:08.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.807+0000 7f96c67fc640 1 -- 172.21.15.150:0/1965933365 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f96d00b6330 con 0x7f96d015dee0 2024-03-17T23:47:08.808 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:08.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.809+0000 7f96d6f9d640 1 -- 172.21.15.150:0/1965933365 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f96b406bbd0 msgr2=0x7f96b406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:08.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.809+0000 7f96d6f9d640 1 --2- 172.21.15.150:0/1965933365 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f96b406bbd0 0x7f96b406e090 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f96cc036000 tx=0x7f96cc030000 comp rx=0 tx=0).stop 2024-03-17T23:47:08.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.809+0000 7f96d6f9d640 1 -- 172.21.15.150:0/1965933365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96d015dee0 msgr2=0x7f96d015f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:08.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.809+0000 7f96d6f9d640 1 --2- 172.21.15.150:0/1965933365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96d015dee0 0x7f96d015f2d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f96c000c430 tx=0x7f96c000c830 comp rx=0 tx=0).stop 2024-03-17T23:47:08.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.809+0000 7f96d6f9d640 1 -- 172.21.15.150:0/1965933365 shutdown_connections 2024-03-17T23:47:08.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.809+0000 7f96d6f9d640 1 --2- 172.21.15.150:0/1965933365 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96d015adc0 0x7f96d00cce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.809+0000 7f96d6f9d640 1 --2- 172.21.15.150:0/1965933365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f96d015dee0 0x7f96d015f2d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.809+0000 7f96d6f9d640 1 --2- 172.21.15.150:0/1965933365 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f96b406bbd0 0x7f96b406e090 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.809+0000 7f96d6f9d640 1 --2- 172.21.15.150:0/1965933365 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96d00cef20 0x7f96d00b5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.809+0000 7f96d6f9d640 1 -- 172.21.15.150:0/1965933365 >> 172.21.15.150:0/1965933365 conn(0x7f96d006c820 msgr2=0x7f96d00c7bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:08.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.809+0000 7f96d6f9d640 1 -- 172.21.15.150:0/1965933365 shutdown_connections 2024-03-17T23:47:08.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.809+0000 7f96d6f9d640 1 -- 172.21.15.150:0/1965933365 wait complete. 2024-03-17T23:47:08.822 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-17T23:47:08.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.923+0000 7f89da17e640 1 Processor -- start 2024-03-17T23:47:08.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.923+0000 7f89da17e640 1 -- start start 2024-03-17T23:47:08.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.924+0000 7f89da17e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f89d4147830 0x7f89d4099a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:08.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.924+0000 7f89da17e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89d415e440 con 0x7f89d40d1750 2024-03-17T23:47:08.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.924+0000 7f89da17e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89d409a310 con 0x7f89d40b59e0 2024-03-17T23:47:08.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.924+0000 7f89da17e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f89d415e700 con 0x7f89d4147830 2024-03-17T23:47:08.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.924+0000 7f89d3fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f89d40d1750 0x7f89d406ee80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59320/0 (socket says 172.21.15.150:59320) 2024-03-17T23:47:08.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.924+0000 7f89d3fff640 1 -- 172.21.15.150:0/3587456346 learned_addr learned my addr 172.21.15.150:0/3587456346 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:08.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.924+0000 7f89d37fe640 1 --2- 172.21.15.150:0/3587456346 >> v2:172.21.15.157:3300/0 conn(0x7f89d4147830 0x7f89d4099a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:08.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.924+0000 7f89d27fc640 1 -- 172.21.15.150:0/3587456346 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2664497916 0 0) 0x7f89d415e440 con 0x7f89d40d1750 2024-03-17T23:47:08.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.924+0000 7f89d27fc640 1 -- 172.21.15.150:0/3587456346 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89bc003520 con 0x7f89d40d1750 2024-03-17T23:47:08.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.924+0000 7f89d27fc640 1 -- 172.21.15.150:0/3587456346 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4162903463 0 0) 0x7f89d409a310 con 0x7f89d40b59e0 2024-03-17T23:47:08.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.924+0000 7f89d27fc640 1 -- 172.21.15.150:0/3587456346 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89d415e440 con 0x7f89d40b59e0 2024-03-17T23:47:08.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.924+0000 7f89d37fe640 1 -- 172.21.15.150:0/3587456346 >> v1:172.21.15.157:6789/0 conn(0x7f89d40b59e0 legacy=0x7f89d40d1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:08.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.924+0000 7f89d37fe640 1 -- 172.21.15.150:0/3587456346 >> v1:172.21.15.150:6789/0 conn(0x7f89d40d1750 legacy=0x7f89d406ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:08.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.924+0000 7f89d37fe640 1 -- 172.21.15.150:0/3587456346 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d415e9e0 con 0x7f89d4147830 2024-03-17T23:47:08.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.925+0000 7f89d37fe640 1 --2- 172.21.15.150:0/3587456346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89d4147830 0x7f89d4099a70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f89c40097a0 tx=0x7f89c402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb044039bc1c1262 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:08.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.925+0000 7f89d27fc640 1 -- 172.21.15.150:0/3587456346 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c40315a0 con 0x7f89d4147830 2024-03-17T23:47:08.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.925+0000 7f89d27fc640 1 -- 172.21.15.150:0/3587456346 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89c4037070 con 0x7f89d4147830 2024-03-17T23:47:08.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.925+0000 7f89d27fc640 1 -- 172.21.15.150:0/3587456346 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c4031960 con 0x7f89d4147830 2024-03-17T23:47:08.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.926+0000 7f89da17e640 1 -- 172.21.15.150:0/3587456346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89d4147830 msgr2=0x7f89d4099a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:08.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.926+0000 7f89da17e640 1 --2- 172.21.15.150:0/3587456346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89d4147830 0x7f89d4099a70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f89c40097a0 tx=0x7f89c402b840 comp rx=0 tx=0).stop 2024-03-17T23:47:08.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.926+0000 7f89da17e640 1 -- 172.21.15.150:0/3587456346 shutdown_connections 2024-03-17T23:47:08.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.926+0000 7f89da17e640 1 --2- 172.21.15.150:0/3587456346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89d4147830 0x7f89d4099a70 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.926+0000 7f89da17e640 1 -- 172.21.15.150:0/3587456346 >> 172.21.15.150:0/3587456346 conn(0x7f89d40b5ec0 msgr2=0x7f89d40b62d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:08.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.926+0000 7f89da17e640 1 -- 172.21.15.150:0/3587456346 shutdown_connections 2024-03-17T23:47:08.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.926+0000 7f89da17e640 1 -- 172.21.15.150:0/3587456346 wait complete. 2024-03-17T23:47:08.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.927+0000 7f89da17e640 1 Processor -- start 2024-03-17T23:47:08.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.927+0000 7f89da17e640 1 -- start start 2024-03-17T23:47:08.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.928+0000 7f89da17e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f89d40b59e0 0x7f89d4160960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:08.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.928+0000 7f89da17e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89d40d1750 0x7f89d41f2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:08.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.928+0000 7f89da17e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89d4147830 0x7f89d41f2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:08.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.928+0000 7f89da17e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f89d415f1e0 con 0x7f89d40d1750 2024-03-17T23:47:08.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.928+0000 7f89da17e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f89d4099fb0 con 0x7f89d4147830 2024-03-17T23:47:08.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.928+0000 7f89da17e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f89d4136960 con 0x7f89d40b59e0 2024-03-17T23:47:08.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.928+0000 7f89d2ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89d40d1750 0x7f89d41f2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:08.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.928+0000 7f89d2ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89d40d1750 0x7f89d41f2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36146/0 (socket says 172.21.15.150:36146) 2024-03-17T23:47:08.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.928+0000 7f89d2ffd640 1 -- 172.21.15.150:0/1387950185 learned_addr learned my addr 172.21.15.150:0/1387950185 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:08.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.928+0000 7f89d37fe640 1 --2- 172.21.15.150:0/1387950185 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f89d40b59e0 0x7f89d4160960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:08.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.928+0000 7f89d3fff640 1 --2- 172.21.15.150:0/1387950185 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89d4147830 0x7f89d41f2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:08.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.928+0000 7f89d2ffd640 1 -- 172.21.15.150:0/1387950185 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f89d40b59e0 msgr2=0x7f89d4160960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:08.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.928+0000 7f89d2ffd640 1 --2- 172.21.15.150:0/1387950185 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f89d40b59e0 0x7f89d4160960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.928+0000 7f89d2ffd640 1 -- 172.21.15.150:0/1387950185 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89d4147830 msgr2=0x7f89d41f2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:08.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.928+0000 7f89d2ffd640 1 --2- 172.21.15.150:0/1387950185 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89d4147830 0x7f89d41f2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:08.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.928+0000 7f89d2ffd640 1 -- 172.21.15.150:0/1387950185 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d41f6bd0 con 0x7f89d40d1750 2024-03-17T23:47:08.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.929+0000 7f89d2ffd640 1 --2- 172.21.15.150:0/1387950185 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89d40d1750 0x7f89d41f2950 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f89c800c460 tx=0x7f89c800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:08.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.929+0000 7f89bbfff640 1 -- 172.21.15.150:0/1387950185 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c8017070 con 0x7f89d40d1750 2024-03-17T23:47:08.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.929+0000 7f89bbfff640 1 -- 172.21.15.150:0/1387950185 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89c80052f0 con 0x7f89d40d1750 2024-03-17T23:47:08.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.929+0000 7f89da17e640 1 -- 172.21.15.150:0/1387950185 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89d41f6eb0 con 0x7f89d40d1750 2024-03-17T23:47:08.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.929+0000 7f89bbfff640 1 -- 172.21.15.150:0/1387950185 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c8003830 con 0x7f89d40d1750 2024-03-17T23:47:08.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.929+0000 7f89da17e640 1 -- 172.21.15.150:0/1387950185 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89d4160ea0 con 0x7f89d40d1750 2024-03-17T23:47:08.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.930+0000 7f89da17e640 1 -- 172.21.15.150:0/1387950185 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89d40681a0 con 0x7f89d40d1750 2024-03-17T23:47:08.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.930+0000 7f89bbfff640 1 -- 172.21.15.150:0/1387950185 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f89c801a070 con 0x7f89d40d1750 2024-03-17T23:47:08.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.930+0000 7f89bbfff640 1 --2- 172.21.15.150:0/1387950185 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f89c006bbd0 0x7f89c006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:08.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.931+0000 7f89bbfff640 1 -- 172.21.15.150:0/1387950185 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f89c808a050 con 0x7f89d40d1750 2024-03-17T23:47:08.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.931+0000 7f89d37fe640 1 --2- 172.21.15.150:0/1387950185 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f89c006bbd0 0x7f89c006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:08.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.931+0000 7f89d37fe640 1 --2- 172.21.15.150:0/1387950185 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f89c006bbd0 0x7f89c006e090 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f89c4036000 tx=0x7f89c403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:08.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:08.935+0000 7f89bbfff640 1 -- 172.21.15.150:0/1387950185 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f89c8058240 con 0x7f89d40d1750 2024-03-17T23:47:09.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.122+0000 7f89da17e640 1 -- 172.21.15.150:0/1387950185 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f89d41604e0 con 0x7f89d40d1750 2024-03-17T23:47:09.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.122+0000 7f89bbfff640 1 -- 172.21.15.150:0/1387950185 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f89c805c2e0 con 0x7f89d40d1750 2024-03-17T23:47:09.123 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-03-17T23:47:09.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.124+0000 7f89da17e640 1 -- 172.21.15.150:0/1387950185 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f89c006bbd0 msgr2=0x7f89c006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:09.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.124+0000 7f89da17e640 1 --2- 172.21.15.150:0/1387950185 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f89c006bbd0 0x7f89c006e090 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f89c4036000 tx=0x7f89c403d040 comp rx=0 tx=0).stop 2024-03-17T23:47:09.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.124+0000 7f89da17e640 1 -- 172.21.15.150:0/1387950185 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89d40d1750 msgr2=0x7f89d41f2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:09.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.124+0000 7f89da17e640 1 --2- 172.21.15.150:0/1387950185 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89d40d1750 0x7f89d41f2950 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f89c800c460 tx=0x7f89c800c920 comp rx=0 tx=0).stop 2024-03-17T23:47:09.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.124+0000 7f89da17e640 1 -- 172.21.15.150:0/1387950185 shutdown_connections 2024-03-17T23:47:09.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.124+0000 7f89da17e640 1 --2- 172.21.15.150:0/1387950185 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f89c006bbd0 0x7f89c006e090 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:09.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.124+0000 7f89da17e640 1 --2- 172.21.15.150:0/1387950185 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89d4147830 0x7f89d41f2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:09.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.124+0000 7f89da17e640 1 --2- 172.21.15.150:0/1387950185 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89d40d1750 0x7f89d41f2950 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:09.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.124+0000 7f89da17e640 1 --2- 172.21.15.150:0/1387950185 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f89d40b59e0 0x7f89d4160960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:09.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.124+0000 7f89da17e640 1 -- 172.21.15.150:0/1387950185 >> 172.21.15.150:0/1387950185 conn(0x7f89d40b5ec0 msgr2=0x7f89d40b1570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:09.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.125+0000 7f89da17e640 1 -- 172.21.15.150:0/1387950185 shutdown_connections 2024-03-17T23:47:09.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.125+0000 7f89da17e640 1 -- 172.21.15.150:0/1387950185 wait complete. 2024-03-17T23:47:09.136 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi150.front.sepia.ceph.com... 2024-03-17T23:47:09.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:09.136 DEBUG:teuthology.orchestra.run.smithi150:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-17T23:47:09.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi150.stderr:ceph-fuse[44808]: fuse finished with error 0 and tester_r 0 2024-03-17T23:47:09.163 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:47:09.254 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi150.front.sepia.ceph.com 2024-03-17T23:47:09.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:09.255 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-17T23:47:09.279 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:09.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.420+0000 7f278a169640 1 Processor -- start 2024-03-17T23:47:09.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.420+0000 7f278a169640 1 -- start start 2024-03-17T23:47:09.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.421+0000 7f278a169640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2784155320 0x7f278415bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:09.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.421+0000 7f278a169640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f278406dd30 0x7f27840ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:09.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.421+0000 7f278a169640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27840cd250 0x7f278415e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:09.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.421+0000 7f278a169640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f278415c550 con 0x7f278406dd30 2024-03-17T23:47:09.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.421+0000 7f278a169640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f27840cf3b0 con 0x7f2784155320 2024-03-17T23:47:09.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.421+0000 7f278a169640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f27840cf690 con 0x7f27840cd250 2024-03-17T23:47:09.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.421+0000 7f2782ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f278406dd30 0x7f27840ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:09.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.421+0000 7f27837fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2784155320 0x7f278415bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:09.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.421+0000 7f2782ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f278406dd30 0x7f27840ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36154/0 (socket says 172.21.15.150:36154) 2024-03-17T23:47:09.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.421+0000 7f2782ffd640 1 -- 172.21.15.150:0/1722243141 learned_addr learned my addr 172.21.15.150:0/1722243141 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:09.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.421+0000 7f2783fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27840cd250 0x7f278415e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:09.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.421+0000 7f2782ffd640 1 -- 172.21.15.150:0/1722243141 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27840cd250 msgr2=0x7f278415e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:09.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.421+0000 7f2782ffd640 1 --2- 172.21.15.150:0/1722243141 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27840cd250 0x7f278415e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:09.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.421+0000 7f2782ffd640 1 -- 172.21.15.150:0/1722243141 >> v2:172.21.15.157:3300/0 conn(0x7f2784155320 msgr2=0x7f278415bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:09.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.421+0000 7f2782ffd640 1 --2- 172.21.15.150:0/1722243141 >> v2:172.21.15.157:3300/0 conn(0x7f2784155320 0x7f278415bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:09.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.421+0000 7f2782ffd640 1 -- 172.21.15.150:0/1722243141 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f278415c7a0 con 0x7f278406dd30 2024-03-17T23:47:09.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.421+0000 7f27837fe640 1 --2- 172.21.15.150:0/1722243141 >> v2:172.21.15.157:3300/0 conn(0x7f2784155320 0x7f278415bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:09.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.421+0000 7f2783fff640 1 --2- 172.21.15.150:0/1722243141 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27840cd250 0x7f278415e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:09.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.422+0000 7f2782ffd640 1 --2- 172.21.15.150:0/1722243141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f278406dd30 0x7f27840ccd10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f278406d5a0 tx=0x7f277802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aba15785ac6f5a54 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:09.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.422+0000 7f27827fc640 1 -- 172.21.15.150:0/1722243141 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2778035020 con 0x7f278406dd30 2024-03-17T23:47:09.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.422+0000 7f27827fc640 1 -- 172.21.15.150:0/1722243141 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2778031870 con 0x7f278406dd30 2024-03-17T23:47:09.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.422+0000 7f27827fc640 1 -- 172.21.15.150:0/1722243141 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2778031c40 con 0x7f278406dd30 2024-03-17T23:47:09.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.422+0000 7f278a169640 1 -- 172.21.15.150:0/1722243141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f278406dd30 msgr2=0x7f27840ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:09.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.422+0000 7f278a169640 1 --2- 172.21.15.150:0/1722243141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f278406dd30 0x7f27840ccd10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f278406d5a0 tx=0x7f277802b910 comp rx=0 tx=0).stop 2024-03-17T23:47:09.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.422+0000 7f278a169640 1 -- 172.21.15.150:0/1722243141 shutdown_connections 2024-03-17T23:47:09.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.422+0000 7f278a169640 1 --2- 172.21.15.150:0/1722243141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f278406dd30 0x7f27840ccd10 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:09.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.422+0000 7f278a169640 1 --2- 172.21.15.150:0/1722243141 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27840cd250 0x7f278415e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:09.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.422+0000 7f278a169640 1 --2- 172.21.15.150:0/1722243141 >> v2:172.21.15.157:3300/0 conn(0x7f2784155320 0x7f278415bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:09.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.422+0000 7f278a169640 1 -- 172.21.15.150:0/1722243141 >> 172.21.15.150:0/1722243141 conn(0x7f278406b190 msgr2=0x7f278406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:09.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.423+0000 7f278a169640 1 -- 172.21.15.150:0/1722243141 shutdown_connections 2024-03-17T23:47:09.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.423+0000 7f278a169640 1 -- 172.21.15.150:0/1722243141 wait complete. 2024-03-17T23:47:09.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.424+0000 7f278a169640 1 Processor -- start 2024-03-17T23:47:09.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.424+0000 7f278a169640 1 -- start start 2024-03-17T23:47:09.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.424+0000 7f278a169640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f278406dd30 0x7f2784141110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:09.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.424+0000 7f278a169640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27840cd250 0x7f2784141650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:09.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.424+0000 7f278a169640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2784155320 0x7f278414b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:09.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.424+0000 7f278a169640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2784155700 con 0x7f278406dd30 2024-03-17T23:47:09.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.424+0000 7f278a169640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f278406e190 con 0x7f2784155320 2024-03-17T23:47:09.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.424+0000 7f278a169640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2784155880 con 0x7f27840cd250 2024-03-17T23:47:09.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.425+0000 7f27837fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f278406dd30 0x7f2784141110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:09.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.425+0000 7f27837fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f278406dd30 0x7f2784141110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36162/0 (socket says 172.21.15.150:36162) 2024-03-17T23:47:09.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.425+0000 7f27837fe640 1 -- 172.21.15.150:0/4171015395 learned_addr learned my addr 172.21.15.150:0/4171015395 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:09.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.425+0000 7f2783fff640 1 --2- 172.21.15.150:0/4171015395 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2784155320 0x7f278414b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:09.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.425+0000 7f2782ffd640 1 --2- 172.21.15.150:0/4171015395 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27840cd250 0x7f2784141650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:09.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.425+0000 7f27837fe640 1 -- 172.21.15.150:0/4171015395 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27840cd250 msgr2=0x7f2784141650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:09.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.425+0000 7f27837fe640 1 --2- 172.21.15.150:0/4171015395 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27840cd250 0x7f2784141650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:09.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.425+0000 7f27837fe640 1 -- 172.21.15.150:0/4171015395 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2784155320 msgr2=0x7f278414b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:09.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.425+0000 7f27837fe640 1 --2- 172.21.15.150:0/4171015395 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2784155320 0x7f278414b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:09.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.425+0000 7f27837fe640 1 -- 172.21.15.150:0/4171015395 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2784151610 con 0x7f278406dd30 2024-03-17T23:47:09.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.425+0000 7f27837fe640 1 --2- 172.21.15.150:0/4171015395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f278406dd30 0x7f2784141110 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f277400ebe0 tx=0x7f2774009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:09.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.425+0000 7f276bfff640 1 -- 172.21.15.150:0/4171015395 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2774017070 con 0x7f278406dd30 2024-03-17T23:47:09.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.425+0000 7f276bfff640 1 -- 172.21.15.150:0/4171015395 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f277400ca00 con 0x7f278406dd30 2024-03-17T23:47:09.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.425+0000 7f276bfff640 1 -- 172.21.15.150:0/4171015395 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277400cdd0 con 0x7f278406dd30 2024-03-17T23:47:09.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.425+0000 7f278a169640 1 -- 172.21.15.150:0/4171015395 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2784155de0 con 0x7f278406dd30 2024-03-17T23:47:09.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.425+0000 7f278a169640 1 -- 172.21.15.150:0/4171015395 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27841563a0 con 0x7f278406dd30 2024-03-17T23:47:09.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.426+0000 7f278a169640 1 -- 172.21.15.150:0/4171015395 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27840681a0 con 0x7f278406dd30 2024-03-17T23:47:09.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.427+0000 7f276bfff640 1 -- 172.21.15.150:0/4171015395 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f277401a070 con 0x7f278406dd30 2024-03-17T23:47:09.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.427+0000 7f276bfff640 1 --2- 172.21.15.150:0/4171015395 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f277006bbd0 0x7f277006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:09.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.427+0000 7f276bfff640 1 -- 172.21.15.150:0/4171015395 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f277408af70 con 0x7f278406dd30 2024-03-17T23:47:09.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.427+0000 7f2782ffd640 1 --2- 172.21.15.150:0/4171015395 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f277006bbd0 0x7f277006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:09.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.428+0000 7f2782ffd640 1 --2- 172.21.15.150:0/4171015395 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f277006bbd0 0x7f277006e090 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2778036000 tx=0x7f277803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:09.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.431+0000 7f276bfff640 1 -- 172.21.15.150:0/4171015395 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f27740590d0 con 0x7f278406dd30 2024-03-17T23:47:09.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.605+0000 7f278a169640 1 -- 172.21.15.150:0/4171015395 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f278415b7e0 con 0x7f278406dd30 2024-03-17T23:47:09.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.606+0000 7f276bfff640 1 -- 172.21.15.150:0/4171015395 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18550 (secure 0 0 0) 0x7f277405d170 con 0x7f278406dd30 2024-03-17T23:47:09.607 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:09.607 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:01.940019+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:09.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.608+0000 7f278a169640 1 -- 172.21.15.150:0/4171015395 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f277006bbd0 msgr2=0x7f277006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:09.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.608+0000 7f278a169640 1 --2- 172.21.15.150:0/4171015395 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f277006bbd0 0x7f277006e090 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2778036000 tx=0x7f277803c040 comp rx=0 tx=0).stop 2024-03-17T23:47:09.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.608+0000 7f278a169640 1 -- 172.21.15.150:0/4171015395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f278406dd30 msgr2=0x7f2784141110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:09.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.608+0000 7f278a169640 1 --2- 172.21.15.150:0/4171015395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f278406dd30 0x7f2784141110 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f277400ebe0 tx=0x7f2774009e20 comp rx=0 tx=0).stop 2024-03-17T23:47:09.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.608+0000 7f278a169640 1 -- 172.21.15.150:0/4171015395 shutdown_connections 2024-03-17T23:47:09.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.608+0000 7f278a169640 1 --2- 172.21.15.150:0/4171015395 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f277006bbd0 0x7f277006e090 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:09.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.608+0000 7f278a169640 1 --2- 172.21.15.150:0/4171015395 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2784155320 0x7f278414b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:09.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.608+0000 7f278a169640 1 --2- 172.21.15.150:0/4171015395 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27840cd250 0x7f2784141650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:09.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.608+0000 7f278a169640 1 --2- 172.21.15.150:0/4171015395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f278406dd30 0x7f2784141110 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:09.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.608+0000 7f278a169640 1 -- 172.21.15.150:0/4171015395 >> 172.21.15.150:0/4171015395 conn(0x7f278406b190 msgr2=0x7f2784073c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:09.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.608+0000 7f278a169640 1 -- 172.21.15.150:0/4171015395 shutdown_connections 2024-03-17T23:47:09.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.608+0000 7f278a169640 1 -- 172.21.15.150:0/4171015395 wait complete. 2024-03-17T23:47:09.620 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:09.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.720+0000 7f328628a640 1 Processor -- start 2024-03-17T23:47:09.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.720+0000 7f328628a640 1 -- start start 2024-03-17T23:47:09.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.720+0000 7f328628a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f328015e060 0x7f32800d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:09.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.720+0000 7f328628a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f32800b57d0 0x7f32800b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:09.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.720+0000 7f328628a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32800d2890 con 0x7f328006c6e0 2024-03-17T23:47:09.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.720+0000 7f328628a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f32800b6180 con 0x7f328015e060 2024-03-17T23:47:09.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.720+0000 7f328628a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f32800d2b70 con 0x7f32800b57d0 2024-03-17T23:47:09.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.721+0000 7f327ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f328015e060 0x7f32800d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:09.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.721+0000 7f327f7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f32800b57d0 0x7f32800b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:09.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.721+0000 7f327ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f328015e060 0x7f32800d1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36176/0 (socket says 172.21.15.150:36176) 2024-03-17T23:47:09.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.721+0000 7f3284827640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f328006c6e0 0x7f328006cb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:37784/0 (socket says 172.21.15.150:37784) 2024-03-17T23:47:09.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.721+0000 7f327ffff640 1 -- 172.21.15.150:0/520679139 learned_addr learned my addr 172.21.15.150:0/520679139 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:09.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.721+0000 7f327ffff640 1 -- 172.21.15.150:0/520679139 >> v2:172.21.15.157:3300/0 conn(0x7f32800b57d0 msgr2=0x7f32800b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:09.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.721+0000 7f327ffff640 1 --2- 172.21.15.150:0/520679139 >> v2:172.21.15.157:3300/0 conn(0x7f32800b57d0 0x7f32800b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:09.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.721+0000 7f327ffff640 1 -- 172.21.15.150:0/520679139 >> v1:172.21.15.157:6789/0 conn(0x7f328006c6e0 legacy=0x7f328006cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:09.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.721+0000 7f327ffff640 1 -- 172.21.15.150:0/520679139 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32800d2e50 con 0x7f328015e060 2024-03-17T23:47:09.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.721+0000 7f327f7fe640 1 --2- 172.21.15.150:0/520679139 >> v2:172.21.15.157:3300/0 conn(0x7f32800b57d0 0x7f32800b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:09.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.722+0000 7f327ffff640 1 --2- 172.21.15.150:0/520679139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f328015e060 0x7f32800d1f20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f32800b9480 tx=0x7f327002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f51cd3509a82428 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:09.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.722+0000 7f327effd640 1 -- 172.21.15.150:0/520679139 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3270035020 con 0x7f328015e060 2024-03-17T23:47:09.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.722+0000 7f327effd640 1 -- 172.21.15.150:0/520679139 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3270031870 con 0x7f328015e060 2024-03-17T23:47:09.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.722+0000 7f327effd640 1 -- 172.21.15.150:0/520679139 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3270031c40 con 0x7f328015e060 2024-03-17T23:47:09.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.722+0000 7f328628a640 1 -- 172.21.15.150:0/520679139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f328015e060 msgr2=0x7f32800d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:09.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.722+0000 7f328628a640 1 --2- 172.21.15.150:0/520679139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f328015e060 0x7f32800d1f20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f32800b9480 tx=0x7f327002b910 comp rx=0 tx=0).stop 2024-03-17T23:47:09.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.722+0000 7f328628a640 1 -- 172.21.15.150:0/520679139 shutdown_connections 2024-03-17T23:47:09.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.722+0000 7f328628a640 1 --2- 172.21.15.150:0/520679139 >> v2:172.21.15.157:3300/0 conn(0x7f32800b57d0 0x7f32800b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:09.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.722+0000 7f328628a640 1 --2- 172.21.15.150:0/520679139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f328015e060 0x7f32800d1f20 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:09.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.722+0000 7f328628a640 1 -- 172.21.15.150:0/520679139 >> 172.21.15.150:0/520679139 conn(0x7f3280073200 msgr2=0x7f32800d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:09.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.722+0000 7f328628a640 1 -- 172.21.15.150:0/520679139 shutdown_connections 2024-03-17T23:47:09.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.722+0000 7f328628a640 1 -- 172.21.15.150:0/520679139 wait complete. 2024-03-17T23:47:09.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.723+0000 7f328628a640 1 Processor -- start 2024-03-17T23:47:09.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.724+0000 7f328628a640 1 -- start start 2024-03-17T23:47:09.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.724+0000 7f328628a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f328006c6e0 0x7f328014bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:09.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.724+0000 7f328628a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32800b57d0 0x7f3280150ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:09.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.724+0000 7f328628a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f328015e060 0x7f3280160d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:09.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.724+0000 7f328628a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f32800b7f00 con 0x7f32800b57d0 2024-03-17T23:47:09.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.724+0000 7f328628a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f32800d3410 con 0x7f328006c6e0 2024-03-17T23:47:09.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.724+0000 7f328628a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f328015e440 con 0x7f328015e060 2024-03-17T23:47:09.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.724+0000 7f327f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32800b57d0 0x7f3280150ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:09.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.724+0000 7f327f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32800b57d0 0x7f3280150ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36186/0 (socket says 172.21.15.150:36186) 2024-03-17T23:47:09.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.725+0000 7f327f7fe640 1 -- 172.21.15.150:0/4145523108 learned_addr learned my addr 172.21.15.150:0/4145523108 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:09.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.725+0000 7f327ffff640 1 --2- 172.21.15.150:0/4145523108 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f328006c6e0 0x7f328014bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:09.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.725+0000 7f3284827640 1 --2- 172.21.15.150:0/4145523108 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f328015e060 0x7f3280160d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:09.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.725+0000 7f327f7fe640 1 -- 172.21.15.150:0/4145523108 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f328015e060 msgr2=0x7f3280160d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:09.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.725+0000 7f327f7fe640 1 --2- 172.21.15.150:0/4145523108 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f328015e060 0x7f3280160d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:09.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.725+0000 7f327f7fe640 1 -- 172.21.15.150:0/4145523108 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f328006c6e0 msgr2=0x7f328014bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:09.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.725+0000 7f327f7fe640 1 --2- 172.21.15.150:0/4145523108 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f328006c6e0 0x7f328014bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:09.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.725+0000 7f327f7fe640 1 -- 172.21.15.150:0/4145523108 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3280160210 con 0x7f32800b57d0 2024-03-17T23:47:09.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.725+0000 7f327f7fe640 1 --2- 172.21.15.150:0/4145523108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32800b57d0 0x7f3280150ce0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f327400ec10 tx=0x7f327400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:09.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.725+0000 7f327cff9640 1 -- 172.21.15.150:0/4145523108 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3274017070 con 0x7f32800b57d0 2024-03-17T23:47:09.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.725+0000 7f327cff9640 1 -- 172.21.15.150:0/4145523108 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f327400cc30 con 0x7f32800b57d0 2024-03-17T23:47:09.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.725+0000 7f328628a640 1 -- 172.21.15.150:0/4145523108 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f328015f690 con 0x7f32800b57d0 2024-03-17T23:47:09.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.725+0000 7f327cff9640 1 -- 172.21.15.150:0/4145523108 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3274013470 con 0x7f32800b57d0 2024-03-17T23:47:09.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.725+0000 7f328628a640 1 -- 172.21.15.150:0/4145523108 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f328015fc50 con 0x7f32800b57d0 2024-03-17T23:47:09.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.726+0000 7f328628a640 1 -- 172.21.15.150:0/4145523108 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32800688b0 con 0x7f32800b57d0 2024-03-17T23:47:09.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.726+0000 7f327cff9640 1 -- 172.21.15.150:0/4145523108 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f327401a070 con 0x7f32800b57d0 2024-03-17T23:47:09.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.727+0000 7f327cff9640 1 --2- 172.21.15.150:0/4145523108 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f326c06bc00 0x7f326c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:09.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.727+0000 7f327ffff640 1 --2- 172.21.15.150:0/4145523108 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f326c06bc00 0x7f326c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:09.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.727+0000 7f327cff9640 1 -- 172.21.15.150:0/4145523108 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f327408b070 con 0x7f32800b57d0 2024-03-17T23:47:09.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.727+0000 7f327ffff640 1 --2- 172.21.15.150:0/4145523108 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f326c06bc00 0x7f326c06e0c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3270036000 tx=0x7f327003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:09.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.731+0000 7f327cff9640 1 -- 172.21.15.150:0/4145523108 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3274059140 con 0x7f32800b57d0 2024-03-17T23:47:09.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.905+0000 7f328628a640 1 -- 172.21.15.150:0/4145523108 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f32800d0b50 con 0x7f32800b57d0 2024-03-17T23:47:09.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.906+0000 7f327cff9640 1 -- 172.21.15.150:0/4145523108 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18550 (secure 0 0 0) 0x7f327405d1e0 con 0x7f32800b57d0 2024-03-17T23:47:09.907 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:09.907 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:01.940019+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:09.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.909+0000 7f328628a640 1 -- 172.21.15.150:0/4145523108 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f326c06bc00 msgr2=0x7f326c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:09.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.909+0000 7f328628a640 1 --2- 172.21.15.150:0/4145523108 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f326c06bc00 0x7f326c06e0c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3270036000 tx=0x7f327003c040 comp rx=0 tx=0).stop 2024-03-17T23:47:09.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.909+0000 7f328628a640 1 -- 172.21.15.150:0/4145523108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32800b57d0 msgr2=0x7f3280150ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:09.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.909+0000 7f328628a640 1 --2- 172.21.15.150:0/4145523108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32800b57d0 0x7f3280150ce0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f327400ec10 tx=0x7f327400c520 comp rx=0 tx=0).stop 2024-03-17T23:47:09.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.909+0000 7f328628a640 1 -- 172.21.15.150:0/4145523108 shutdown_connections 2024-03-17T23:47:09.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.909+0000 7f328628a640 1 --2- 172.21.15.150:0/4145523108 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f328015e060 0x7f3280160d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:09.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.909+0000 7f328628a640 1 --2- 172.21.15.150:0/4145523108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32800b57d0 0x7f3280150ce0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:09.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.909+0000 7f328628a640 1 --2- 172.21.15.150:0/4145523108 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f326c06bc00 0x7f326c06e0c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:09.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.909+0000 7f328628a640 1 --2- 172.21.15.150:0/4145523108 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f328006c6e0 0x7f328014bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:09.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.909+0000 7f328628a640 1 -- 172.21.15.150:0/4145523108 >> 172.21.15.150:0/4145523108 conn(0x7f3280073200 msgr2=0x7f32800d4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:09.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.909+0000 7f328628a640 1 -- 172.21.15.150:0/4145523108 shutdown_connections 2024-03-17T23:47:09.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:09.909+0000 7f328628a640 1 -- 172.21.15.150:0/4145523108 wait complete. 2024-03-17T23:47:09.921 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-17T23:47:10.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.024+0000 7f6a4b962640 1 Processor -- start 2024-03-17T23:47:10.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.024+0000 7f6a4b962640 1 -- start start 2024-03-17T23:47:10.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.025+0000 7f6a4b962640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6a4415f3d0 0x7f6a440cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:10.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.025+0000 7f6a4b962640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6a440d0880 0x7f6a441603c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:10.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.025+0000 7f6a4b962640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a440ccc50 con 0x7f6a440cc770 2024-03-17T23:47:10.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.025+0000 7f6a4b962640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f6a4415f7b0 con 0x7f6a440d0880 2024-03-17T23:47:10.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.025+0000 7f6a4b962640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f6a44160900 con 0x7f6a4415f3d0 2024-03-17T23:47:10.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.025+0000 7f6a49eff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6a440d0880 0x7f6a441603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:10.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.025+0000 7f6a496fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6a4415f3d0 0x7f6a440cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:10.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.025+0000 7f6a49eff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6a440d0880 0x7f6a441603c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36188/0 (socket says 172.21.15.150:36188) 2024-03-17T23:47:10.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.025+0000 7f6a49eff640 1 -- 172.21.15.150:0/2307624803 learned_addr learned my addr 172.21.15.150:0/2307624803 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:10.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.026+0000 7f6a49eff640 1 -- 172.21.15.150:0/2307624803 >> v2:172.21.15.157:3300/0 conn(0x7f6a4415f3d0 msgr2=0x7f6a440cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:10.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.026+0000 7f6a49eff640 1 --2- 172.21.15.150:0/2307624803 >> v2:172.21.15.157:3300/0 conn(0x7f6a4415f3d0 0x7f6a440cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:10.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.026+0000 7f6a49eff640 1 -- 172.21.15.150:0/2307624803 >> v1:172.21.15.157:6789/0 conn(0x7f6a440cc770 legacy=0x7f6a440d0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:10.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.026+0000 7f6a49eff640 1 -- 172.21.15.150:0/2307624803 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a440ccf30 con 0x7f6a440d0880 2024-03-17T23:47:10.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.026+0000 7f6a49eff640 1 --2- 172.21.15.150:0/2307624803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a440d0880 0x7f6a441603c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6a440728e0 tx=0x7f6a3c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=664aadcdc8881712 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:10.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.027+0000 7f6a3bfff640 1 -- 172.21.15.150:0/2307624803 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c034070 con 0x7f6a440d0880 2024-03-17T23:47:10.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.027+0000 7f6a3bfff640 1 -- 172.21.15.150:0/2307624803 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a3c030860 con 0x7f6a440d0880 2024-03-17T23:47:10.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.027+0000 7f6a3bfff640 1 -- 172.21.15.150:0/2307624803 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c030c30 con 0x7f6a440d0880 2024-03-17T23:47:10.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.027+0000 7f6a4b962640 1 -- 172.21.15.150:0/2307624803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a440d0880 msgr2=0x7f6a441603c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:10.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.027+0000 7f6a4b962640 1 --2- 172.21.15.150:0/2307624803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a440d0880 0x7f6a441603c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6a440728e0 tx=0x7f6a3c02b9a0 comp rx=0 tx=0).stop 2024-03-17T23:47:10.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.027+0000 7f6a4b962640 1 -- 172.21.15.150:0/2307624803 shutdown_connections 2024-03-17T23:47:10.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.027+0000 7f6a4b962640 1 --2- 172.21.15.150:0/2307624803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a440d0880 0x7f6a441603c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:10.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.027+0000 7f6a4b962640 1 --2- 172.21.15.150:0/2307624803 >> v2:172.21.15.157:3300/0 conn(0x7f6a4415f3d0 0x7f6a440cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:10.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.027+0000 7f6a4b962640 1 -- 172.21.15.150:0/2307624803 >> 172.21.15.150:0/2307624803 conn(0x7f6a440b36c0 msgr2=0x7f6a440b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:10.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.027+0000 7f6a4b962640 1 -- 172.21.15.150:0/2307624803 shutdown_connections 2024-03-17T23:47:10.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.027+0000 7f6a4b962640 1 -- 172.21.15.150:0/2307624803 wait complete. 2024-03-17T23:47:10.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.028+0000 7f6a4b962640 1 Processor -- start 2024-03-17T23:47:10.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.028+0000 7f6a4b962640 1 -- start start 2024-03-17T23:47:10.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.029+0000 7f6a4b962640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a440cc770 0x7f6a4415b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:10.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.029+0000 7f6a4b962640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a440d0880 0x7f6a440b7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:10.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.029+0000 7f6a4b962640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a4415f3d0 0x7f6a440cb450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:10.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.029+0000 7f6a4b962640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6a440cd7d0 con 0x7f6a440d0880 2024-03-17T23:47:10.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.029+0000 7f6a4b962640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6a440cd4f0 con 0x7f6a440cc770 2024-03-17T23:47:10.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.029+0000 7f6a4b962640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6a440cfa30 con 0x7f6a4415f3d0 2024-03-17T23:47:10.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.029+0000 7f6a48efd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a440d0880 0x7f6a440b7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:10.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.029+0000 7f6a48efd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a440d0880 0x7f6a440b7340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36204/0 (socket says 172.21.15.150:36204) 2024-03-17T23:47:10.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.029+0000 7f6a496fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a440cc770 0x7f6a4415b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:10.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.029+0000 7f6a48efd640 1 -- 172.21.15.150:0/1618437725 learned_addr learned my addr 172.21.15.150:0/1618437725 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:10.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.029+0000 7f6a49eff640 1 --2- 172.21.15.150:0/1618437725 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a4415f3d0 0x7f6a440cb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:10.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.030+0000 7f6a48efd640 1 -- 172.21.15.150:0/1618437725 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a4415f3d0 msgr2=0x7f6a440cb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:10.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.030+0000 7f6a48efd640 1 --2- 172.21.15.150:0/1618437725 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a4415f3d0 0x7f6a440cb450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:10.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.030+0000 7f6a48efd640 1 -- 172.21.15.150:0/1618437725 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a440cc770 msgr2=0x7f6a4415b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:10.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.030+0000 7f6a48efd640 1 --2- 172.21.15.150:0/1618437725 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a440cc770 0x7f6a4415b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:10.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.030+0000 7f6a48efd640 1 -- 172.21.15.150:0/1618437725 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a440ca3b0 con 0x7f6a440d0880 2024-03-17T23:47:10.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.030+0000 7f6a48efd640 1 --2- 172.21.15.150:0/1618437725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a440d0880 0x7f6a440b7340 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6a3400c430 tx=0x7f6a3400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:10.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.030+0000 7f6a39ffb640 1 -- 172.21.15.150:0/1618437725 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a340052d0 con 0x7f6a440d0880 2024-03-17T23:47:10.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.030+0000 7f6a39ffb640 1 -- 172.21.15.150:0/1618437725 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a34005490 con 0x7f6a440d0880 2024-03-17T23:47:10.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.030+0000 7f6a39ffb640 1 -- 172.21.15.150:0/1618437725 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a34007260 con 0x7f6a440d0880 2024-03-17T23:47:10.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.030+0000 7f6a4b962640 1 -- 172.21.15.150:0/1618437725 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a440ca690 con 0x7f6a440d0880 2024-03-17T23:47:10.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.030+0000 7f6a4b962640 1 -- 172.21.15.150:0/1618437725 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a440cac50 con 0x7f6a440d0880 2024-03-17T23:47:10.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.031+0000 7f6a4b962640 1 -- 172.21.15.150:0/1618437725 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a4415c060 con 0x7f6a440d0880 2024-03-17T23:47:10.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.031+0000 7f6a39ffb640 1 -- 172.21.15.150:0/1618437725 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6a34007400 con 0x7f6a440d0880 2024-03-17T23:47:10.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.032+0000 7f6a39ffb640 1 --2- 172.21.15.150:0/1618437725 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6a2806bc00 0x7f6a2806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:10.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.032+0000 7f6a39ffb640 1 -- 172.21.15.150:0/1618437725 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f6a3408a2a0 con 0x7f6a440d0880 2024-03-17T23:47:10.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.032+0000 7f6a496fe640 1 --2- 172.21.15.150:0/1618437725 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6a2806bc00 0x7f6a2806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:10.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.032+0000 7f6a496fe640 1 --2- 172.21.15.150:0/1618437725 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6a2806bc00 0x7f6a2806e0c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6a40011340 tx=0x7f6a4000b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:10.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.036+0000 7f6a39ffb640 1 -- 172.21.15.150:0/1618437725 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6a34058370 con 0x7f6a440d0880 2024-03-17T23:47:10.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.246+0000 7f6a4b962640 1 -- 172.21.15.150:0/1618437725 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f6a440688b0 con 0x7f6a440d0880 2024-03-17T23:47:10.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.967+0000 7f6a39ffb640 1 -- 172.21.15.150:0/1618437725 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 118+0+0 (secure 0 0 0) 0x7f6a3405c410 con 0x7f6a440d0880 2024-03-17T23:47:10.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.970+0000 7f6a4b962640 1 -- 172.21.15.150:0/1618437725 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6a2806bc00 msgr2=0x7f6a2806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:10.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.970+0000 7f6a4b962640 1 --2- 172.21.15.150:0/1618437725 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6a2806bc00 0x7f6a2806e0c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6a40011340 tx=0x7f6a4000b040 comp rx=0 tx=0).stop 2024-03-17T23:47:10.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.970+0000 7f6a4b962640 1 -- 172.21.15.150:0/1618437725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a440d0880 msgr2=0x7f6a440b7340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:10.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.970+0000 7f6a4b962640 1 --2- 172.21.15.150:0/1618437725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a440d0880 0x7f6a440b7340 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6a3400c430 tx=0x7f6a3400c8f0 comp rx=0 tx=0).stop 2024-03-17T23:47:10.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.970+0000 7f6a4b962640 1 -- 172.21.15.150:0/1618437725 shutdown_connections 2024-03-17T23:47:10.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.970+0000 7f6a4b962640 1 --2- 172.21.15.150:0/1618437725 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a4415f3d0 0x7f6a440cb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:10.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.970+0000 7f6a4b962640 1 --2- 172.21.15.150:0/1618437725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a440d0880 0x7f6a440b7340 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:10.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.970+0000 7f6a4b962640 1 --2- 172.21.15.150:0/1618437725 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6a2806bc00 0x7f6a2806e0c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:10.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.970+0000 7f6a4b962640 1 --2- 172.21.15.150:0/1618437725 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a440cc770 0x7f6a4415b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:10.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.970+0000 7f6a4b962640 1 -- 172.21.15.150:0/1618437725 >> 172.21.15.150:0/1618437725 conn(0x7f6a440b36c0 msgr2=0x7f6a440c5700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:10.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.970+0000 7f6a4b962640 1 -- 172.21.15.150:0/1618437725 shutdown_connections 2024-03-17T23:47:10.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:10.970+0000 7f6a4b962640 1 -- 172.21.15.150:0/1618437725 wait complete. 2024-03-17T23:47:10.983 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-03-17T23:47:10.984 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-17T23:47:11.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.093+0000 7f9bf27d4640 1 Processor -- start 2024-03-17T23:47:11.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.093+0000 7f9bf27d4640 1 -- start start 2024-03-17T23:47:11.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.094+0000 7f9bf27d4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9bec155320 0x7f9bec15bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:11.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.094+0000 7f9bf27d4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9bec0cd420 0x7f9bec15ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:11.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.094+0000 7f9bf27d4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bec0cf3b0 con 0x7f9bec06dd30 2024-03-17T23:47:11.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.094+0000 7f9bf27d4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f9bec15c550 con 0x7f9bec155320 2024-03-17T23:47:11.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.094+0000 7f9bf27d4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9bec0cf690 con 0x7f9bec0cd420 2024-03-17T23:47:11.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.094+0000 7f9beb7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f9bec06dd30 0x7f9bec0ccd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59330/0 (socket says 172.21.15.150:59330) 2024-03-17T23:47:11.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.094+0000 7f9beb7fe640 1 -- 172.21.15.150:0/99826410 learned_addr learned my addr 172.21.15.150:0/99826410 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:11.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.094+0000 7f9bf0d71640 1 --2- 172.21.15.150:0/99826410 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9bec0cd420 0x7f9bec15ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:11.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.094+0000 7f9bebfff640 1 --2- 172.21.15.150:0/99826410 >> v2:172.21.15.157:3300/0 conn(0x7f9bec155320 0x7f9bec15bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:11.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.094+0000 7f9beaffd640 1 -- 172.21.15.150:0/99826410 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2913559972 0 0) 0x7f9bec0cf3b0 con 0x7f9bec06dd30 2024-03-17T23:47:11.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.094+0000 7f9beaffd640 1 -- 172.21.15.150:0/99826410 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9bd4003520 con 0x7f9bec06dd30 2024-03-17T23:47:11.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.094+0000 7f9bf0d71640 1 -- 172.21.15.150:0/99826410 >> v2:172.21.15.157:3300/0 conn(0x7f9bec155320 msgr2=0x7f9bec15bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:11.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.094+0000 7f9bf0d71640 1 --2- 172.21.15.150:0/99826410 >> v2:172.21.15.157:3300/0 conn(0x7f9bec155320 0x7f9bec15bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:11.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.094+0000 7f9bf0d71640 1 -- 172.21.15.150:0/99826410 >> v1:172.21.15.150:6789/0 conn(0x7f9bec06dd30 legacy=0x7f9bec0ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:11.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.094+0000 7f9bf0d71640 1 -- 172.21.15.150:0/99826410 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bec15c7a0 con 0x7f9bec0cd420 2024-03-17T23:47:11.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.094+0000 7f9bebfff640 1 --2- 172.21.15.150:0/99826410 >> v2:172.21.15.157:3300/0 conn(0x7f9bec155320 0x7f9bec15bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:11.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.095+0000 7f9bf0d71640 1 --2- 172.21.15.150:0/99826410 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9bec0cd420 0x7f9bec15ccc0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009d90 tx=0x7f9be402e0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48a85120222c2460 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:11.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.095+0000 7f9beaffd640 1 -- 172.21.15.150:0/99826410 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be403b070 con 0x7f9bec0cd420 2024-03-17T23:47:11.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.095+0000 7f9beaffd640 1 -- 172.21.15.150:0/99826410 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9be4036440 con 0x7f9bec0cd420 2024-03-17T23:47:11.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.095+0000 7f9beaffd640 1 -- 172.21.15.150:0/99826410 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be402ee60 con 0x7f9bec0cd420 2024-03-17T23:47:11.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.096+0000 7f9bf27d4640 1 -- 172.21.15.150:0/99826410 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9bec0cd420 msgr2=0x7f9bec15ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:11.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.096+0000 7f9bf27d4640 1 --2- 172.21.15.150:0/99826410 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9bec0cd420 0x7f9bec15ccc0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009d90 tx=0x7f9be402e0b0 comp rx=0 tx=0).stop 2024-03-17T23:47:11.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.096+0000 7f9bf27d4640 1 -- 172.21.15.150:0/99826410 shutdown_connections 2024-03-17T23:47:11.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.096+0000 7f9bf27d4640 1 --2- 172.21.15.150:0/99826410 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9bec0cd420 0x7f9bec15ccc0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:11.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.096+0000 7f9bf27d4640 1 --2- 172.21.15.150:0/99826410 >> v2:172.21.15.157:3300/0 conn(0x7f9bec155320 0x7f9bec15bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:11.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.096+0000 7f9bf27d4640 1 -- 172.21.15.150:0/99826410 >> 172.21.15.150:0/99826410 conn(0x7f9bec06b190 msgr2=0x7f9bec06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:11.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.096+0000 7f9bf27d4640 1 -- 172.21.15.150:0/99826410 shutdown_connections 2024-03-17T23:47:11.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.096+0000 7f9bf27d4640 1 -- 172.21.15.150:0/99826410 wait complete. 2024-03-17T23:47:11.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.097+0000 7f9bf27d4640 1 Processor -- start 2024-03-17T23:47:11.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.097+0000 7f9bf27d4640 1 -- start start 2024-03-17T23:47:11.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.098+0000 7f9bf27d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bec06dd30 0x7f9bec1484b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:11.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.098+0000 7f9bf27d4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bec0cd420 0x7f9bec0b3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:11.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.098+0000 7f9bf27d4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9bec155320 0x7f9bec0b3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:11.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.098+0000 7f9bf27d4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9bec155700 con 0x7f9bec06dd30 2024-03-17T23:47:11.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.098+0000 7f9bf27d4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9bec06e190 con 0x7f9bec0cd420 2024-03-17T23:47:11.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.098+0000 7f9bf27d4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9bec155880 con 0x7f9bec155320 2024-03-17T23:47:11.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.098+0000 7f9bebfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bec06dd30 0x7f9bec1484b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:11.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.098+0000 7f9bebfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bec06dd30 0x7f9bec1484b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36218/0 (socket says 172.21.15.150:36218) 2024-03-17T23:47:11.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.098+0000 7f9bebfff640 1 -- 172.21.15.150:0/291134937 learned_addr learned my addr 172.21.15.150:0/291134937 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:11.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.098+0000 7f9bf0d71640 1 --2- 172.21.15.150:0/291134937 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9bec155320 0x7f9bec0b3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:11.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.098+0000 7f9beb7fe640 1 --2- 172.21.15.150:0/291134937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bec0cd420 0x7f9bec0b3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:11.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.098+0000 7f9bebfff640 1 -- 172.21.15.150:0/291134937 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9bec155320 msgr2=0x7f9bec0b3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:11.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.098+0000 7f9bebfff640 1 --2- 172.21.15.150:0/291134937 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9bec155320 0x7f9bec0b3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:11.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.098+0000 7f9bebfff640 1 -- 172.21.15.150:0/291134937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bec0cd420 msgr2=0x7f9bec0b3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:11.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.098+0000 7f9bebfff640 1 --2- 172.21.15.150:0/291134937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bec0cd420 0x7f9bec0b3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:11.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.098+0000 7f9bebfff640 1 -- 172.21.15.150:0/291134937 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bec15ccf0 con 0x7f9bec06dd30 2024-03-17T23:47:11.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.098+0000 7f9bebfff640 1 --2- 172.21.15.150:0/291134937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bec06dd30 0x7f9bec1484b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc00e990 tx=0x7f9bdc00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:11.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.098+0000 7f9be8ff9640 1 -- 172.21.15.150:0/291134937 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bdc017070 con 0x7f9bec06dd30 2024-03-17T23:47:11.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.098+0000 7f9be8ff9640 1 -- 172.21.15.150:0/291134937 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bdc00c9e0 con 0x7f9bec06dd30 2024-03-17T23:47:11.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.098+0000 7f9bf27d4640 1 -- 172.21.15.150:0/291134937 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bec15cfd0 con 0x7f9bec06dd30 2024-03-17T23:47:11.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.098+0000 7f9be8ff9640 1 -- 172.21.15.150:0/291134937 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bdc00cdb0 con 0x7f9bec06dd30 2024-03-17T23:47:11.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.098+0000 7f9bf27d4640 1 -- 172.21.15.150:0/291134937 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bec15d590 con 0x7f9bec06dd30 2024-03-17T23:47:11.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.100+0000 7f9bf27d4640 1 -- 172.21.15.150:0/291134937 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bec0681a0 con 0x7f9bec06dd30 2024-03-17T23:47:11.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.100+0000 7f9be8ff9640 1 -- 172.21.15.150:0/291134937 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9bdc01a070 con 0x7f9bec06dd30 2024-03-17T23:47:11.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.100+0000 7f9be8ff9640 1 --2- 172.21.15.150:0/291134937 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f9bd806bc00 0x7f9bd806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:11.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.100+0000 7f9be8ff9640 1 -- 172.21.15.150:0/291134937 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f9bdc08ae90 con 0x7f9bec06dd30 2024-03-17T23:47:11.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.100+0000 7f9beb7fe640 1 --2- 172.21.15.150:0/291134937 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f9bd806bc00 0x7f9bd806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:11.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.101+0000 7f9beb7fe640 1 --2- 172.21.15.150:0/291134937 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f9bd806bc00 0x7f9bd806e0c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9bec0b4c50 tx=0x7f9be0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:11.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.105+0000 7f9be8ff9640 1 -- 172.21.15.150:0/291134937 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9bdc059080 con 0x7f9bec06dd30 2024-03-17T23:47:11.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.315+0000 7f9bf27d4640 1 -- 172.21.15.150:0/291134937 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f9bec159aa0 con 0x7f9bec06dd30 2024-03-17T23:47:11.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.971+0000 7f9be8ff9640 1 -- 172.21.15.150:0/291134937 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) ==== 122+0+0 (secure 0 0 0) 0x7f9bdc05d120 con 0x7f9bec06dd30 2024-03-17T23:47:11.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.973+0000 7f9bf27d4640 1 -- 172.21.15.150:0/291134937 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f9bd806bc00 msgr2=0x7f9bd806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:11.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.973+0000 7f9bf27d4640 1 --2- 172.21.15.150:0/291134937 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f9bd806bc00 0x7f9bd806e0c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9bec0b4c50 tx=0x7f9be0009040 comp rx=0 tx=0).stop 2024-03-17T23:47:11.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.973+0000 7f9bf27d4640 1 -- 172.21.15.150:0/291134937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bec06dd30 msgr2=0x7f9bec1484b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:11.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.973+0000 7f9bf27d4640 1 --2- 172.21.15.150:0/291134937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bec06dd30 0x7f9bec1484b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc00e990 tx=0x7f9bdc00ee50 comp rx=0 tx=0).stop 2024-03-17T23:47:11.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.973+0000 7f9bf27d4640 1 -- 172.21.15.150:0/291134937 shutdown_connections 2024-03-17T23:47:11.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.973+0000 7f9bf27d4640 1 --2- 172.21.15.150:0/291134937 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9bec155320 0x7f9bec0b3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:11.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.973+0000 7f9bf27d4640 1 --2- 172.21.15.150:0/291134937 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f9bd806bc00 0x7f9bd806e0c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:11.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.973+0000 7f9bf27d4640 1 --2- 172.21.15.150:0/291134937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bec0cd420 0x7f9bec0b3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:11.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.973+0000 7f9bf27d4640 1 --2- 172.21.15.150:0/291134937 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9bec06dd30 0x7f9bec1484b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:11.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.973+0000 7f9bf27d4640 1 -- 172.21.15.150:0/291134937 >> 172.21.15.150:0/291134937 conn(0x7f9bec06b190 msgr2=0x7f9bec0b2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:11.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.973+0000 7f9bf27d4640 1 -- 172.21.15.150:0/291134937 shutdown_connections 2024-03-17T23:47:11.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:11.973+0000 7f9bf27d4640 1 -- 172.21.15.150:0/291134937 wait complete. 2024-03-17T23:47:11.986 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-17T23:47:11.986 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:47:12.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.090+0000 7f959e06a640 1 Processor -- start 2024-03-17T23:47:12.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.090+0000 7f959e06a640 1 -- start start 2024-03-17T23:47:12.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.091+0000 7f959e06a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f95981514f0 0x7f959815bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:12.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.091+0000 7f959e06a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f959806ecd0 0x7f959806f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:12.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.091+0000 7f959e06a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95980ce790 con 0x7f959815ab90 2024-03-17T23:47:12.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.091+0000 7f959e06a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f959806fba0 con 0x7f959806ecd0 2024-03-17T23:47:12.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.091+0000 7f959e06a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f959815b050 con 0x7f95981514f0 2024-03-17T23:47:12.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.091+0000 7f9597fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f959806ecd0 0x7f959806f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:12.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.091+0000 7f9597fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f959806ecd0 0x7f959806f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36234/0 (socket says 172.21.15.150:36234) 2024-03-17T23:47:12.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.091+0000 7f9597fff640 1 -- 172.21.15.150:0/580702375 learned_addr learned my addr 172.21.15.150:0/580702375 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:12.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.091+0000 7f95977fe640 1 --2- 172.21.15.150:0/580702375 >> v2:172.21.15.157:3300/0 conn(0x7f95981514f0 0x7f959815bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:12.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.092+0000 7f9597fff640 1 -- 172.21.15.150:0/580702375 >> v2:172.21.15.157:3300/0 conn(0x7f95981514f0 msgr2=0x7f959815bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:12.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.092+0000 7f9597fff640 1 --2- 172.21.15.150:0/580702375 >> v2:172.21.15.157:3300/0 conn(0x7f95981514f0 0x7f959815bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.092+0000 7f9597fff640 1 -- 172.21.15.150:0/580702375 >> v1:172.21.15.157:6789/0 conn(0x7f959815ab90 legacy=0x7f959806e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:12.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.092+0000 7f9597fff640 1 -- 172.21.15.150:0/580702375 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f959815b330 con 0x7f959806ecd0 2024-03-17T23:47:12.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.092+0000 7f95977fe640 1 --2- 172.21.15.150:0/580702375 >> v2:172.21.15.157:3300/0 conn(0x7f95981514f0 0x7f959815bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:12.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.092+0000 7f9597fff640 1 --2- 172.21.15.150:0/580702375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f959806ecd0 0x7f959806f660 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9598072d90 tx=0x7f959002d090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbc8a1a7f2bb86f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:12.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.092+0000 7f95967fc640 1 -- 172.21.15.150:0/580702375 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f959002da30 con 0x7f959806ecd0 2024-03-17T23:47:12.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.092+0000 7f95967fc640 1 -- 172.21.15.150:0/580702375 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f959002dbd0 con 0x7f959806ecd0 2024-03-17T23:47:12.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.092+0000 7f95967fc640 1 -- 172.21.15.150:0/580702375 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9590008500 con 0x7f959806ecd0 2024-03-17T23:47:12.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.093+0000 7f959e06a640 1 -- 172.21.15.150:0/580702375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f959806ecd0 msgr2=0x7f959806f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:12.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.093+0000 7f959e06a640 1 --2- 172.21.15.150:0/580702375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f959806ecd0 0x7f959806f660 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9598072d90 tx=0x7f959002d090 comp rx=0 tx=0).stop 2024-03-17T23:47:12.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.093+0000 7f959e06a640 1 -- 172.21.15.150:0/580702375 shutdown_connections 2024-03-17T23:47:12.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.093+0000 7f959e06a640 1 --2- 172.21.15.150:0/580702375 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f959806ecd0 0x7f959806f660 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.093+0000 7f959e06a640 1 --2- 172.21.15.150:0/580702375 >> v2:172.21.15.157:3300/0 conn(0x7f95981514f0 0x7f959815bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.093+0000 7f959e06a640 1 -- 172.21.15.150:0/580702375 >> 172.21.15.150:0/580702375 conn(0x7f959806b190 msgr2=0x7f959806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:12.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.093+0000 7f959e06a640 1 -- 172.21.15.150:0/580702375 shutdown_connections 2024-03-17T23:47:12.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.093+0000 7f959e06a640 1 -- 172.21.15.150:0/580702375 wait complete. 2024-03-17T23:47:12.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.094+0000 7f959e06a640 1 Processor -- start 2024-03-17T23:47:12.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.094+0000 7f959e06a640 1 -- start start 2024-03-17T23:47:12.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.095+0000 7f959e06a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f959806ecd0 0x7f9598155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:12.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.095+0000 7f959e06a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95981514f0 0x7f9598155750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:12.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.095+0000 7f959e06a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f959815ab90 0x7f95981f4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:12.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.095+0000 7f959e06a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f95980af510 con 0x7f959806ecd0 2024-03-17T23:47:12.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.095+0000 7f959e06a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f959815b8f0 con 0x7f959815ab90 2024-03-17T23:47:12.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.095+0000 7f959e06a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f95981518d0 con 0x7f95981514f0 2024-03-17T23:47:12.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.095+0000 7f95977fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f959806ecd0 0x7f9598155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:12.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.095+0000 7f95977fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f959806ecd0 0x7f9598155210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36246/0 (socket says 172.21.15.150:36246) 2024-03-17T23:47:12.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.095+0000 7f95977fe640 1 -- 172.21.15.150:0/2922399073 learned_addr learned my addr 172.21.15.150:0/2922399073 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:12.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.095+0000 7f9597fff640 1 --2- 172.21.15.150:0/2922399073 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f959815ab90 0x7f95981f4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:12.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.095+0000 7f9596ffd640 1 --2- 172.21.15.150:0/2922399073 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95981514f0 0x7f9598155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:12.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.095+0000 7f95977fe640 1 -- 172.21.15.150:0/2922399073 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95981514f0 msgr2=0x7f9598155750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:12.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.095+0000 7f95977fe640 1 --2- 172.21.15.150:0/2922399073 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95981514f0 0x7f9598155750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.095+0000 7f95977fe640 1 -- 172.21.15.150:0/2922399073 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f959815ab90 msgr2=0x7f95981f4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:12.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.095+0000 7f95977fe640 1 --2- 172.21.15.150:0/2922399073 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f959815ab90 0x7f95981f4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.095+0000 7f95977fe640 1 -- 172.21.15.150:0/2922399073 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9598152760 con 0x7f959806ecd0 2024-03-17T23:47:12.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.096+0000 7f95977fe640 1 --2- 172.21.15.150:0/2922399073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f959806ecd0 0x7f9598155210 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f958800ec10 tx=0x7f958800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:12.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.096+0000 7f957ffff640 1 -- 172.21.15.150:0/2922399073 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9588017070 con 0x7f959806ecd0 2024-03-17T23:47:12.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.096+0000 7f957ffff640 1 -- 172.21.15.150:0/2922399073 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f958800cc30 con 0x7f959806ecd0 2024-03-17T23:47:12.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.096+0000 7f959e06a640 1 -- 172.21.15.150:0/2922399073 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9598152a40 con 0x7f959806ecd0 2024-03-17T23:47:12.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.096+0000 7f957ffff640 1 -- 172.21.15.150:0/2922399073 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9588013470 con 0x7f959806ecd0 2024-03-17T23:47:12.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.096+0000 7f959e06a640 1 -- 172.21.15.150:0/2922399073 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9598153000 con 0x7f959806ecd0 2024-03-17T23:47:12.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.097+0000 7f959e06a640 1 -- 172.21.15.150:0/2922399073 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95980681a0 con 0x7f959806ecd0 2024-03-17T23:47:12.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.097+0000 7f957ffff640 1 -- 172.21.15.150:0/2922399073 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f958801a070 con 0x7f959806ecd0 2024-03-17T23:47:12.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.098+0000 7f957ffff640 1 --2- 172.21.15.150:0/2922399073 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f958406bbd0 0x7f958406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:12.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.098+0000 7f957ffff640 1 -- 172.21.15.150:0/2922399073 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f958808af60 con 0x7f959806ecd0 2024-03-17T23:47:12.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.098+0000 7f9596ffd640 1 --2- 172.21.15.150:0/2922399073 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f958406bbd0 0x7f958406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:12.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.098+0000 7f9596ffd640 1 --2- 172.21.15.150:0/2922399073 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f958406bbd0 0x7f958406e090 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f959814e400 tx=0x7f958c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:12.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.102+0000 7f957ffff640 1 -- 172.21.15.150:0/2922399073 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9588059150 con 0x7f959806ecd0 2024-03-17T23:47:12.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.311+0000 7f959e06a640 1 -- 172.21.15.150:0/2922399073 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95980affb0 con 0x7f959806ecd0 2024-03-17T23:47:12.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.312+0000 7f957ffff640 1 -- 172.21.15.150:0/2922399073 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5186 (secure 0 0 0) 0x7f958805d1f0 con 0x7f959806ecd0 2024-03-17T23:47:12.312 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:12.313 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/1949609985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6833","nonce":1949609985},{"type":"v1","addr":"172.21.15.157:6835","nonce":1949609985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/758264728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":758264728},{"type":"v1","addr":"172.21.15.150:6835","nonce":758264728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3655589010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3655589010},{"type":"v1","addr":"172.21.15.150:6837","nonce":3655589010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-17T23:46:55.253739+0000","modified":"2024-03-17T23:47:11.316249+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6834/1625058792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1625058792},{"type":"v1","addr":"172.21.15.157:6834","nonce":1625058792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4297,"qdb_cluster":[4297]},"id":1}]} 2024-03-17T23:47:12.313 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 8 2024-03-17T23:47:12.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.314+0000 7f959e06a640 1 -- 172.21.15.150:0/2922399073 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f958406bbd0 msgr2=0x7f958406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:12.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.314+0000 7f959e06a640 1 --2- 172.21.15.150:0/2922399073 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f958406bbd0 0x7f958406e090 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f959814e400 tx=0x7f958c009040 comp rx=0 tx=0).stop 2024-03-17T23:47:12.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.314+0000 7f959e06a640 1 -- 172.21.15.150:0/2922399073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f959806ecd0 msgr2=0x7f9598155210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:12.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.314+0000 7f959e06a640 1 --2- 172.21.15.150:0/2922399073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f959806ecd0 0x7f9598155210 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f958800ec10 tx=0x7f958800c520 comp rx=0 tx=0).stop 2024-03-17T23:47:12.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.314+0000 7f959e06a640 1 -- 172.21.15.150:0/2922399073 shutdown_connections 2024-03-17T23:47:12.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.314+0000 7f959e06a640 1 --2- 172.21.15.150:0/2922399073 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f958406bbd0 0x7f958406e090 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.314+0000 7f959e06a640 1 --2- 172.21.15.150:0/2922399073 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f959815ab90 0x7f95981f4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.314+0000 7f959e06a640 1 --2- 172.21.15.150:0/2922399073 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95981514f0 0x7f9598155750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.314+0000 7f959e06a640 1 --2- 172.21.15.150:0/2922399073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f959806ecd0 0x7f9598155210 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.314+0000 7f959e06a640 1 -- 172.21.15.150:0/2922399073 >> 172.21.15.150:0/2922399073 conn(0x7f959806b190 msgr2=0x7f95980b07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:12.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.314+0000 7f959e06a640 1 -- 172.21.15.150:0/2922399073 shutdown_connections 2024-03-17T23:47:12.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.314+0000 7f959e06a640 1 -- 172.21.15.150:0/2922399073 wait complete. 2024-03-17T23:47:12.325 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:12.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.425+0000 7fb7dcc12640 1 Processor -- start 2024-03-17T23:47:12.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.425+0000 7fb7dcc12640 1 -- start start 2024-03-17T23:47:12.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.426+0000 7fb7dcc12640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb7d81586d0 0x7fb7d815f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:12.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.426+0000 7fb7dcc12640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7d8072a20 0x7fb7d8099a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:12.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.426+0000 7fb7dcc12640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7d815bdb0 con 0x7fb7d806f630 2024-03-17T23:47:12.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.426+0000 7fb7dcc12640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fb7d815c090 con 0x7fb7d81586d0 2024-03-17T23:47:12.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.426+0000 7fb7dcc12640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb7d815e8b0 con 0x7fb7d8072a20 2024-03-17T23:47:12.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.426+0000 7fb7d659c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb7d81586d0 0x7fb7d815f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:12.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.426+0000 7fb7d659c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb7d81586d0 0x7fb7d815f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36254/0 (socket says 172.21.15.150:36254) 2024-03-17T23:47:12.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.426+0000 7fb7d659c640 1 -- 172.21.15.150:0/3985373554 learned_addr learned my addr 172.21.15.150:0/3985373554 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:12.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.426+0000 7fb7cffff640 1 --2- 172.21.15.150:0/3985373554 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7d8072a20 0x7fb7d8099a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:12.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.426+0000 7fb7d659c640 1 -- 172.21.15.150:0/3985373554 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7d8072a20 msgr2=0x7fb7d8099a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:12.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.426+0000 7fb7d659c640 1 --2- 172.21.15.150:0/3985373554 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7d8072a20 0x7fb7d8099a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.426+0000 7fb7d659c640 1 -- 172.21.15.150:0/3985373554 >> v1:172.21.15.157:6789/0 conn(0x7fb7d806f630 legacy=0x7fb7d806b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:12.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.426+0000 7fb7d5d9b640 1 -- 172.21.15.150:0/3985373554 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2972467808 0 0) 0x7fb7d815bdb0 con 0x7fb7d806f630 2024-03-17T23:47:12.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.426+0000 7fb7d659c640 1 -- 172.21.15.150:0/3985373554 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7d815eb90 con 0x7fb7d81586d0 2024-03-17T23:47:12.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.426+0000 7fb7cffff640 1 --2- 172.21.15.150:0/3985373554 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7d8072a20 0x7fb7d8099a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:12.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.427+0000 7fb7d659c640 1 --2- 172.21.15.150:0/3985373554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7d81586d0 0x7fb7d815f390 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8009870 tx=0x7fb7c802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aaef1df1c585aaa server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:12.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.427+0000 7fb7d5d9b640 1 -- 172.21.15.150:0/3985373554 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8034070 con 0x7fb7d81586d0 2024-03-17T23:47:12.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.427+0000 7fb7d5d9b640 1 -- 172.21.15.150:0/3985373554 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7c8030840 con 0x7fb7d81586d0 2024-03-17T23:47:12.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.427+0000 7fb7d5d9b640 1 -- 172.21.15.150:0/3985373554 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8030c10 con 0x7fb7d81586d0 2024-03-17T23:47:12.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.427+0000 7fb7dcc12640 1 -- 172.21.15.150:0/3985373554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7d81586d0 msgr2=0x7fb7d815f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:12.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.427+0000 7fb7dcc12640 1 --2- 172.21.15.150:0/3985373554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7d81586d0 0x7fb7d815f390 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8009870 tx=0x7fb7c802b980 comp rx=0 tx=0).stop 2024-03-17T23:47:12.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.428+0000 7fb7dcc12640 1 -- 172.21.15.150:0/3985373554 shutdown_connections 2024-03-17T23:47:12.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.428+0000 7fb7dcc12640 1 --2- 172.21.15.150:0/3985373554 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7d8072a20 0x7fb7d8099a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.428+0000 7fb7dcc12640 1 --2- 172.21.15.150:0/3985373554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7d81586d0 0x7fb7d815f390 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.428+0000 7fb7dcc12640 1 -- 172.21.15.150:0/3985373554 >> 172.21.15.150:0/3985373554 conn(0x7fb7d80b3da0 msgr2=0x7fb7d80b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:12.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.428+0000 7fb7dcc12640 1 -- 172.21.15.150:0/3985373554 shutdown_connections 2024-03-17T23:47:12.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.428+0000 7fb7dcc12640 1 -- 172.21.15.150:0/3985373554 wait complete. 2024-03-17T23:47:12.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.429+0000 7fb7dcc12640 1 Processor -- start 2024-03-17T23:47:12.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.429+0000 7fb7dcc12640 1 -- start start 2024-03-17T23:47:12.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.430+0000 7fb7dcc12640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7d806f630 0x7fb7d809bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:12.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.430+0000 7fb7dcc12640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7d8072a20 0x7fb7d80ce380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:12.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.430+0000 7fb7dcc12640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7d81586d0 0x7fb7d80b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:12.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.430+0000 7fb7dcc12640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d8158c30 con 0x7fb7d8072a20 2024-03-17T23:47:12.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.430+0000 7fb7dcc12640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb7d8158ab0 con 0x7fb7d806f630 2024-03-17T23:47:12.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.430+0000 7fb7dcc12640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb7d806fac0 con 0x7fb7d81586d0 2024-03-17T23:47:12.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.430+0000 7fb7cffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7d8072a20 0x7fb7d80ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:12.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.430+0000 7fb7cffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7d8072a20 0x7fb7d80ce380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36268/0 (socket says 172.21.15.150:36268) 2024-03-17T23:47:12.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.430+0000 7fb7cffff640 1 -- 172.21.15.150:0/580831598 learned_addr learned my addr 172.21.15.150:0/580831598 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:12.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.430+0000 7fb7d6d9d640 1 --2- 172.21.15.150:0/580831598 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7d81586d0 0x7fb7d80b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:12.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.430+0000 7fb7d659c640 1 --2- 172.21.15.150:0/580831598 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7d806f630 0x7fb7d809bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:12.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.430+0000 7fb7cffff640 1 -- 172.21.15.150:0/580831598 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7d81586d0 msgr2=0x7fb7d80b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:12.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.430+0000 7fb7cffff640 1 --2- 172.21.15.150:0/580831598 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7d81586d0 0x7fb7d80b2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.430+0000 7fb7cffff640 1 -- 172.21.15.150:0/580831598 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7d806f630 msgr2=0x7fb7d809bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:12.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.430+0000 7fb7cffff640 1 --2- 172.21.15.150:0/580831598 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7d806f630 0x7fb7d809bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.430+0000 7fb7cffff640 1 -- 172.21.15.150:0/580831598 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7d809c140 con 0x7fb7d8072a20 2024-03-17T23:47:12.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.430+0000 7fb7cffff640 1 --2- 172.21.15.150:0/580831598 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7d8072a20 0x7fb7d80ce380 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb7c000ebe0 tx=0x7fb7c000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:12.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.430+0000 7fb7ceffd640 1 -- 172.21.15.150:0/580831598 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c0017070 con 0x7fb7d8072a20 2024-03-17T23:47:12.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.430+0000 7fb7ceffd640 1 -- 172.21.15.150:0/580831598 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7c000cc30 con 0x7fb7d8072a20 2024-03-17T23:47:12.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.430+0000 7fb7dcc12640 1 -- 172.21.15.150:0/580831598 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7d814ccb0 con 0x7fb7d8072a20 2024-03-17T23:47:12.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.430+0000 7fb7ceffd640 1 -- 172.21.15.150:0/580831598 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c0013470 con 0x7fb7d8072a20 2024-03-17T23:47:12.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.430+0000 7fb7dcc12640 1 -- 172.21.15.150:0/580831598 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7d814d270 con 0x7fb7d8072a20 2024-03-17T23:47:12.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.431+0000 7fb7dcc12640 1 -- 172.21.15.150:0/580831598 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7d80681a0 con 0x7fb7d8072a20 2024-03-17T23:47:12.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.432+0000 7fb7ceffd640 1 -- 172.21.15.150:0/580831598 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb7c001a070 con 0x7fb7d8072a20 2024-03-17T23:47:12.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.432+0000 7fb7ceffd640 1 --2- 172.21.15.150:0/580831598 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb7b806bbd0 0x7fb7b806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:12.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.432+0000 7fb7d659c640 1 --2- 172.21.15.150:0/580831598 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb7b806bbd0 0x7fb7b806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:12.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.432+0000 7fb7ceffd640 1 -- 172.21.15.150:0/580831598 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fb7c008ac30 con 0x7fb7d8072a20 2024-03-17T23:47:12.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.433+0000 7fb7d659c640 1 --2- 172.21.15.150:0/580831598 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb7b806bbd0 0x7fb7b806e090 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8036000 tx=0x7fb7c803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:12.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.437+0000 7fb7ceffd640 1 -- 172.21.15.150:0/580831598 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb7d809c140 con 0x7fb7d8072a20 2024-03-17T23:47:12.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.610+0000 7fb7dcc12640 1 -- 172.21.15.150:0/580831598 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb7d815e000 con 0x7fb7d8072a20 2024-03-17T23:47:12.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.611+0000 7fb7ceffd640 1 -- 172.21.15.150:0/580831598 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18550 (secure 0 0 0) 0x7fb7c0058d90 con 0x7fb7d8072a20 2024-03-17T23:47:12.612 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:12.612 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:01.940019+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:12.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.613+0000 7fb7dcc12640 1 -- 172.21.15.150:0/580831598 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb7b806bbd0 msgr2=0x7fb7b806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:12.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.613+0000 7fb7dcc12640 1 --2- 172.21.15.150:0/580831598 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb7b806bbd0 0x7fb7b806e090 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb7c8036000 tx=0x7fb7c803b040 comp rx=0 tx=0).stop 2024-03-17T23:47:12.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.613+0000 7fb7dcc12640 1 -- 172.21.15.150:0/580831598 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7d8072a20 msgr2=0x7fb7d80ce380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:12.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.613+0000 7fb7dcc12640 1 --2- 172.21.15.150:0/580831598 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7d8072a20 0x7fb7d80ce380 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb7c000ebe0 tx=0x7fb7c000c520 comp rx=0 tx=0).stop 2024-03-17T23:47:12.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.613+0000 7fb7dcc12640 1 -- 172.21.15.150:0/580831598 shutdown_connections 2024-03-17T23:47:12.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.613+0000 7fb7dcc12640 1 --2- 172.21.15.150:0/580831598 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7d81586d0 0x7fb7d80b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.613+0000 7fb7dcc12640 1 --2- 172.21.15.150:0/580831598 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7d8072a20 0x7fb7d80ce380 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.613+0000 7fb7dcc12640 1 --2- 172.21.15.150:0/580831598 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb7b806bbd0 0x7fb7b806e090 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.613+0000 7fb7dcc12640 1 --2- 172.21.15.150:0/580831598 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7d806f630 0x7fb7d809bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.613+0000 7fb7dcc12640 1 -- 172.21.15.150:0/580831598 >> 172.21.15.150:0/580831598 conn(0x7fb7d80b3da0 msgr2=0x7fb7d80ae0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:12.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.613+0000 7fb7dcc12640 1 -- 172.21.15.150:0/580831598 shutdown_connections 2024-03-17T23:47:12.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.613+0000 7fb7dcc12640 1 -- 172.21.15.150:0/580831598 wait complete. 2024-03-17T23:47:12.625 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-17T23:47:12.626 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-17T23:47:12.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.725+0000 7f552c390640 1 Processor -- start 2024-03-17T23:47:12.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.725+0000 7f552c390640 1 -- start start 2024-03-17T23:47:12.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.726+0000 7f552c390640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f552415bfd0 0x7f55240ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:12.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.726+0000 7f552c390640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f55240cd250 0x7f552415ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:12.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.726+0000 7f552c390640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55240cf3b0 con 0x7f5524155320 2024-03-17T23:47:12.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.726+0000 7f552c390640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f552415c550 con 0x7f55240cd250 2024-03-17T23:47:12.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.726+0000 7f552c390640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f55240cf690 con 0x7f552415bfd0 2024-03-17T23:47:12.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.727+0000 7f552a12c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f5524155320 0x7f552406dd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59338/0 (socket says 172.21.15.150:59338) 2024-03-17T23:47:12.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.727+0000 7f552a12c640 1 -- 172.21.15.150:0/1535297937 learned_addr learned my addr 172.21.15.150:0/1535297937 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:12.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.727+0000 7f552a92d640 1 --2- 172.21.15.150:0/1535297937 >> v2:172.21.15.157:3300/0 conn(0x7f55240cd250 0x7f552415ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:12.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.727+0000 7f552992b640 1 --2- 172.21.15.150:0/1535297937 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f552415bfd0 0x7f55240ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:12.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.727+0000 7f552912a640 1 -- 172.21.15.150:0/1535297937 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3546779781 0 0) 0x7f55240cf3b0 con 0x7f5524155320 2024-03-17T23:47:12.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.727+0000 7f552912a640 1 -- 172.21.15.150:0/1535297937 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5510003520 con 0x7f5524155320 2024-03-17T23:47:12.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.727+0000 7f552a92d640 1 -- 172.21.15.150:0/1535297937 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f552415bfd0 msgr2=0x7f55240ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:12.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.727+0000 7f552a92d640 1 --2- 172.21.15.150:0/1535297937 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f552415bfd0 0x7f55240ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.727+0000 7f552a92d640 1 -- 172.21.15.150:0/1535297937 >> v1:172.21.15.150:6789/0 conn(0x7f5524155320 legacy=0x7f552406dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:12.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.727+0000 7f552a92d640 1 -- 172.21.15.150:0/1535297937 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f552415c7a0 con 0x7f55240cd250 2024-03-17T23:47:12.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.727+0000 7f552992b640 1 --2- 172.21.15.150:0/1535297937 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f552415bfd0 0x7f55240ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:12.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.728+0000 7f552a92d640 1 --2- 172.21.15.150:0/1535297937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55240cd250 0x7f552415ccc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f551c00bfa0 tx=0x7f551c02e0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=430b38ec1f608396 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:12.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.728+0000 7f552912a640 1 -- 172.21.15.150:0/1535297937 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551c02ea40 con 0x7f55240cd250 2024-03-17T23:47:12.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.728+0000 7f552912a640 1 -- 172.21.15.150:0/1535297937 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f551c02ebe0 con 0x7f55240cd250 2024-03-17T23:47:12.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.728+0000 7f552912a640 1 -- 172.21.15.150:0/1535297937 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551c009540 con 0x7f55240cd250 2024-03-17T23:47:12.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.728+0000 7f552c390640 1 -- 172.21.15.150:0/1535297937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55240cd250 msgr2=0x7f552415ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:12.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.728+0000 7f552c390640 1 --2- 172.21.15.150:0/1535297937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55240cd250 0x7f552415ccc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f551c00bfa0 tx=0x7f551c02e0a0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.728+0000 7f552c390640 1 -- 172.21.15.150:0/1535297937 shutdown_connections 2024-03-17T23:47:12.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.728+0000 7f552c390640 1 --2- 172.21.15.150:0/1535297937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55240cd250 0x7f552415ccc0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.728+0000 7f552c390640 1 --2- 172.21.15.150:0/1535297937 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f552415bfd0 0x7f55240ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.728+0000 7f552c390640 1 -- 172.21.15.150:0/1535297937 >> 172.21.15.150:0/1535297937 conn(0x7f552406b190 msgr2=0x7f552406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:12.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.729+0000 7f552c390640 1 -- 172.21.15.150:0/1535297937 shutdown_connections 2024-03-17T23:47:12.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.729+0000 7f552c390640 1 -- 172.21.15.150:0/1535297937 wait complete. 2024-03-17T23:47:12.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.730+0000 7f552c390640 1 Processor -- start 2024-03-17T23:47:12.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.730+0000 7f552c390640 1 -- start start 2024-03-17T23:47:12.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.730+0000 7f552c390640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55240cd250 0x7f552414cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:12.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.730+0000 7f552c390640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5524155320 0x7f552414bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:12.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.730+0000 7f552c390640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f552415bfd0 0x7f5524158120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:12.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.730+0000 7f552c390640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5524155880 con 0x7f5524155320 2024-03-17T23:47:12.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.730+0000 7f552c390640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5524155700 con 0x7f552415bfd0 2024-03-17T23:47:12.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.730+0000 7f552c390640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f552415d200 con 0x7f55240cd250 2024-03-17T23:47:12.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.730+0000 7f552992b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5524155320 0x7f552414bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:12.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.730+0000 7f552992b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5524155320 0x7f552414bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36274/0 (socket says 172.21.15.150:36274) 2024-03-17T23:47:12.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.730+0000 7f552992b640 1 -- 172.21.15.150:0/1661680578 learned_addr learned my addr 172.21.15.150:0/1661680578 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:12.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.730+0000 7f552a12c640 1 --2- 172.21.15.150:0/1661680578 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55240cd250 0x7f552414cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:12.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.731+0000 7f552a92d640 1 --2- 172.21.15.150:0/1661680578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f552415bfd0 0x7f5524158120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:12.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.731+0000 7f552992b640 1 -- 172.21.15.150:0/1661680578 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55240cd250 msgr2=0x7f552414cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:12.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.731+0000 7f552992b640 1 --2- 172.21.15.150:0/1661680578 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55240cd250 0x7f552414cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.731+0000 7f552992b640 1 -- 172.21.15.150:0/1661680578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f552415bfd0 msgr2=0x7f5524158120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:12.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.731+0000 7f552992b640 1 --2- 172.21.15.150:0/1661680578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f552415bfd0 0x7f5524158120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.731+0000 7f552992b640 1 -- 172.21.15.150:0/1661680578 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f552414c8d0 con 0x7f5524155320 2024-03-17T23:47:12.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.731+0000 7f552992b640 1 --2- 172.21.15.150:0/1661680578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5524155320 0x7f552414bd00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f552000e960 tx=0x7f552000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:12.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.731+0000 7f5516ffd640 1 -- 172.21.15.150:0/1661680578 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5520017070 con 0x7f5524155320 2024-03-17T23:47:12.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.731+0000 7f552c390640 1 -- 172.21.15.150:0/1661680578 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55241594d0 con 0x7f5524155320 2024-03-17T23:47:12.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.731+0000 7f5516ffd640 1 -- 172.21.15.150:0/1661680578 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f552000c9a0 con 0x7f5524155320 2024-03-17T23:47:12.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.731+0000 7f5516ffd640 1 -- 172.21.15.150:0/1661680578 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f552000cd70 con 0x7f5524155320 2024-03-17T23:47:12.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.731+0000 7f552c390640 1 -- 172.21.15.150:0/1661680578 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5524159d70 con 0x7f5524155320 2024-03-17T23:47:12.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.732+0000 7f552c390640 1 -- 172.21.15.150:0/1661680578 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55240681a0 con 0x7f5524155320 2024-03-17T23:47:12.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.733+0000 7f5516ffd640 1 -- 172.21.15.150:0/1661680578 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f552001a070 con 0x7f5524155320 2024-03-17T23:47:12.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.733+0000 7f5516ffd640 1 --2- 172.21.15.150:0/1661680578 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f550806bc00 0x7f550806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:12.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.733+0000 7f5516ffd640 1 -- 172.21.15.150:0/1661680578 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f552008ab70 con 0x7f5524155320 2024-03-17T23:47:12.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.733+0000 7f552a12c640 1 --2- 172.21.15.150:0/1661680578 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f550806bc00 0x7f550806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:12.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.733+0000 7f552a12c640 1 --2- 172.21.15.150:0/1661680578 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f550806bc00 0x7f550806e0c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5518005880 tx=0x7f5518009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:12.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.737+0000 7f5516ffd640 1 -- 172.21.15.150:0/1661680578 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f552414c8d0 con 0x7f5524155320 2024-03-17T23:47:12.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.906+0000 7f552c390640 1 -- 172.21.15.150:0/1661680578 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f55240af830 con 0x7f550806bc00 2024-03-17T23:47:12.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.914+0000 7f5516ffd640 1 -- 172.21.15.150:0/1661680578 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7f55240af830 con 0x7f550806bc00 2024-03-17T23:47:12.920 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:12.922 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-03-17T23:47:12.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.918+0000 7f552c390640 1 -- 172.21.15.150:0/1661680578 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f550806bc00 msgr2=0x7f550806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:12.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.918+0000 7f552c390640 1 --2- 172.21.15.150:0/1661680578 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f550806bc00 0x7f550806e0c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5518005880 tx=0x7f5518009040 comp rx=0 tx=0).stop 2024-03-17T23:47:12.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.918+0000 7f552c390640 1 -- 172.21.15.150:0/1661680578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5524155320 msgr2=0x7f552414bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:12.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.918+0000 7f552c390640 1 --2- 172.21.15.150:0/1661680578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5524155320 0x7f552414bd00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f552000e960 tx=0x7f552000ee20 comp rx=0 tx=0).stop 2024-03-17T23:47:12.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.918+0000 7f552c390640 1 -- 172.21.15.150:0/1661680578 shutdown_connections 2024-03-17T23:47:12.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.918+0000 7f552c390640 1 --2- 172.21.15.150:0/1661680578 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f550806bc00 0x7f550806e0c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.918+0000 7f552c390640 1 --2- 172.21.15.150:0/1661680578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f552415bfd0 0x7f5524158120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.918+0000 7f552c390640 1 --2- 172.21.15.150:0/1661680578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5524155320 0x7f552414bd00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.918+0000 7f552c390640 1 --2- 172.21.15.150:0/1661680578 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55240cd250 0x7f552414cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:12.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.918+0000 7f552c390640 1 -- 172.21.15.150:0/1661680578 >> 172.21.15.150:0/1661680578 conn(0x7f552406b190 msgr2=0x7f5524071cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:12.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.918+0000 7f552c390640 1 -- 172.21.15.150:0/1661680578 shutdown_connections 2024-03-17T23:47:12.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:12.918+0000 7f552c390640 1 -- 172.21.15.150:0/1661680578 wait complete. 2024-03-17T23:47:12.929 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-17T23:47:11.599293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594867,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":206,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":232008,"kb_used_data":3368,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754742712,"statfs":{"total":773094113280,"available":772856537088,"internally_reserved":0,"allocated":3448832,"data_stored":2079526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2089,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002137"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713953+0000","last_change":"2024-03-17T23:46:54.923706+0000","last_active":"2024-03-17T23:47:02.713953+0000","last_peered":"2024-03-17T23:47:02.713953+0000","last_clean":"2024-03-17T23:47:02.713953+0000","last_became_active":"2024-03-17T23:46:54.923462+0000","last_became_peered":"2024-03-17T23:46:54.923462+0000","last_unstale":"2024-03-17T23:47:02.713953+0000","last_undegraded":"2024-03-17T23:47:02.713953+0000","last_fullsized":"2024-03-17T23:47:02.713953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:34:48.054365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943580+0000","last_change":"2024-03-17T23:46:55.932962+0000","last_active":"2024-03-17T23:47:01.943580+0000","last_peered":"2024-03-17T23:47:01.943580+0000","last_clean":"2024-03-17T23:47:01.943580+0000","last_became_active":"2024-03-17T23:46:55.932316+0000","last_became_peered":"2024-03-17T23:46:55.932316+0000","last_unstale":"2024-03-17T23:47:01.943580+0000","last_undegraded":"2024-03-17T23:47:01.943580+0000","last_fullsized":"2024-03-17T23:47:01.943580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:45:20.327436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943046+0000","last_change":"2024-03-17T23:47:00.412319+0000","last_active":"2024-03-17T23:47:01.943046+0000","last_peered":"2024-03-17T23:47:01.943046+0000","last_clean":"2024-03-17T23:47:01.943046+0000","last_became_active":"2024-03-17T23:47:00.411544+0000","last_became_peered":"2024-03-17T23:47:00.411544+0000","last_unstale":"2024-03-17T23:47:01.943046+0000","last_undegraded":"2024-03-17T23:47:01.943046+0000","last_fullsized":"2024-03-17T23:47:01.943046+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:24:18.544213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943689+0000","last_change":"2024-03-17T23:46:54.921077+0000","last_active":"2024-03-17T23:47:01.943689+0000","last_peered":"2024-03-17T23:47:01.943689+0000","last_clean":"2024-03-17T23:47:01.943689+0000","last_became_active":"2024-03-17T23:46:54.920797+0000","last_became_peered":"2024-03-17T23:46:54.920797+0000","last_unstale":"2024-03-17T23:47:01.943689+0000","last_undegraded":"2024-03-17T23:47:01.943689+0000","last_fullsized":"2024-03-17T23:47:01.943689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:18.195935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713708+0000","last_change":"2024-03-17T23:46:55.943217+0000","last_active":"2024-03-17T23:47:02.713708+0000","last_peered":"2024-03-17T23:47:02.713708+0000","last_clean":"2024-03-17T23:47:02.713708+0000","last_became_active":"2024-03-17T23:46:55.942641+0000","last_became_peered":"2024-03-17T23:46:55.942641+0000","last_unstale":"2024-03-17T23:47:02.713708+0000","last_undegraded":"2024-03-17T23:47:02.713708+0000","last_fullsized":"2024-03-17T23:47:02.713708+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:45:20.327436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106123+0000","last_change":"2024-03-17T23:46:59.957747+0000","last_active":"2024-03-17T23:47:02.106123+0000","last_peered":"2024-03-17T23:47:02.106123+0000","last_clean":"2024-03-17T23:47:02.106123+0000","last_became_active":"2024-03-17T23:46:59.957329+0000","last_became_peered":"2024-03-17T23:46:59.957329+0000","last_unstale":"2024-03-17T23:47:02.106123+0000","last_undegraded":"2024-03-17T23:47:02.106123+0000","last_fullsized":"2024-03-17T23:47:02.106123+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:27:51.964285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713914+0000","last_change":"2024-03-17T23:46:54.926580+0000","last_active":"2024-03-17T23:47:02.713914+0000","last_peered":"2024-03-17T23:47:02.713914+0000","last_clean":"2024-03-17T23:47:02.713914+0000","last_became_active":"2024-03-17T23:46:54.926254+0000","last_became_peered":"2024-03-17T23:46:54.926254+0000","last_unstale":"2024-03-17T23:47:02.713914+0000","last_undegraded":"2024-03-17T23:47:02.713914+0000","last_fullsized":"2024-03-17T23:47:02.713914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:11:01.353944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943965+0000","last_change":"2024-03-17T23:46:55.937073+0000","last_active":"2024-03-17T23:47:01.943965+0000","last_peered":"2024-03-17T23:47:01.943965+0000","last_clean":"2024-03-17T23:47:01.943965+0000","last_became_active":"2024-03-17T23:46:55.936068+0000","last_became_peered":"2024-03-17T23:46:55.936068+0000","last_unstale":"2024-03-17T23:47:01.943965+0000","last_undegraded":"2024-03-17T23:47:01.943965+0000","last_fullsized":"2024-03-17T23:47:01.943965+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:45:29.447600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106472+0000","last_change":"2024-03-17T23:46:59.957945+0000","last_active":"2024-03-17T23:47:02.106472+0000","last_peered":"2024-03-17T23:47:02.106472+0000","last_clean":"2024-03-17T23:47:02.106472+0000","last_became_active":"2024-03-17T23:46:59.957324+0000","last_became_peered":"2024-03-17T23:46:59.957324+0000","last_unstale":"2024-03-17T23:47:02.106472+0000","last_undegraded":"2024-03-17T23:47:02.106472+0000","last_fullsized":"2024-03-17T23:47:02.106472+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:52:17.152097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713691+0000","last_change":"2024-03-17T23:46:54.924488+0000","last_active":"2024-03-17T23:47:02.713691+0000","last_peered":"2024-03-17T23:47:02.713691+0000","last_clean":"2024-03-17T23:47:02.713691+0000","last_became_active":"2024-03-17T23:46:54.924170+0000","last_became_peered":"2024-03-17T23:46:54.924170+0000","last_unstale":"2024-03-17T23:47:02.713691+0000","last_undegraded":"2024-03-17T23:47:02.713691+0000","last_fullsized":"2024-03-17T23:47:02.713691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:06:54.433520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944190+0000","last_change":"2024-03-17T23:46:55.938268+0000","last_active":"2024-03-17T23:47:01.944190+0000","last_peered":"2024-03-17T23:47:01.944190+0000","last_clean":"2024-03-17T23:47:01.944190+0000","last_became_active":"2024-03-17T23:46:55.937957+0000","last_became_peered":"2024-03-17T23:46:55.937957+0000","last_unstale":"2024-03-17T23:47:01.944190+0000","last_undegraded":"2024-03-17T23:47:01.944190+0000","last_fullsized":"2024-03-17T23:47:01.944190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T23:52:25.543746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106183+0000","last_change":"2024-03-17T23:46:59.959993+0000","last_active":"2024-03-17T23:47:02.106183+0000","last_peered":"2024-03-17T23:47:02.106183+0000","last_clean":"2024-03-17T23:47:02.106183+0000","last_became_active":"2024-03-17T23:46:59.959636+0000","last_became_peered":"2024-03-17T23:46:59.959636+0000","last_unstale":"2024-03-17T23:47:02.106183+0000","last_undegraded":"2024-03-17T23:47:02.106183+0000","last_fullsized":"2024-03-17T23:47:02.106183+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:39:36.583214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943413+0000","last_change":"2024-03-17T23:46:55.917068+0000","last_active":"2024-03-17T23:47:01.943413+0000","last_peered":"2024-03-17T23:47:01.943413+0000","last_clean":"2024-03-17T23:47:01.943413+0000","last_became_active":"2024-03-17T23:46:55.916759+0000","last_became_peered":"2024-03-17T23:46:55.916759+0000","last_unstale":"2024-03-17T23:47:01.943413+0000","last_undegraded":"2024-03-17T23:47:01.943413+0000","last_fullsized":"2024-03-17T23:47:01.943413+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:45:20.327436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713785+0000","last_change":"2024-03-17T23:46:54.925485+0000","last_active":"2024-03-17T23:47:02.713785+0000","last_peered":"2024-03-17T23:47:02.713785+0000","last_clean":"2024-03-17T23:47:02.713785+0000","last_became_active":"2024-03-17T23:46:54.925173+0000","last_became_peered":"2024-03-17T23:46:54.925173+0000","last_unstale":"2024-03-17T23:47:02.713785+0000","last_undegraded":"2024-03-17T23:47:02.713785+0000","last_fullsized":"2024-03-17T23:47:02.713785+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:51:18.545842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106704+0000","last_change":"2024-03-17T23:46:59.961930+0000","last_active":"2024-03-17T23:47:02.106704+0000","last_peered":"2024-03-17T23:47:02.106704+0000","last_clean":"2024-03-17T23:47:02.106704+0000","last_became_active":"2024-03-17T23:46:59.961495+0000","last_became_peered":"2024-03-17T23:46:59.961495+0000","last_unstale":"2024-03-17T23:47:02.106704+0000","last_undegraded":"2024-03-17T23:47:02.106704+0000","last_fullsized":"2024-03-17T23:47:02.106704+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:59:20.183441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943415+0000","last_change":"2024-03-17T23:46:55.927904+0000","last_active":"2024-03-17T23:47:01.943415+0000","last_peered":"2024-03-17T23:47:01.943415+0000","last_clean":"2024-03-17T23:47:01.943415+0000","last_became_active":"2024-03-17T23:46:55.927637+0000","last_became_peered":"2024-03-17T23:46:55.927637+0000","last_unstale":"2024-03-17T23:47:01.943415+0000","last_undegraded":"2024-03-17T23:47:01.943415+0000","last_fullsized":"2024-03-17T23:47:01.943415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:03.353861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714207+0000","last_change":"2024-03-17T23:46:54.920822+0000","last_active":"2024-03-17T23:47:02.714207+0000","last_peered":"2024-03-17T23:47:02.714207+0000","last_clean":"2024-03-17T23:47:02.714207+0000","last_became_active":"2024-03-17T23:46:54.920534+0000","last_became_peered":"2024-03-17T23:46:54.920534+0000","last_unstale":"2024-03-17T23:47:02.714207+0000","last_undegraded":"2024-03-17T23:47:02.714207+0000","last_fullsized":"2024-03-17T23:47:02.714207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:56.665566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106433+0000","last_change":"2024-03-17T23:46:59.961245+0000","last_active":"2024-03-17T23:47:02.106433+0000","last_peered":"2024-03-17T23:47:02.106433+0000","last_clean":"2024-03-17T23:47:02.106433+0000","last_became_active":"2024-03-17T23:46:59.960895+0000","last_became_peered":"2024-03-17T23:46:59.960895+0000","last_unstale":"2024-03-17T23:47:02.106433+0000","last_undegraded":"2024-03-17T23:47:02.106433+0000","last_fullsized":"2024-03-17T23:47:02.106433+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:57.296991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943288+0000","last_change":"2024-03-17T23:46:55.924013+0000","last_active":"2024-03-17T23:47:01.943288+0000","last_peered":"2024-03-17T23:47:01.943288+0000","last_clean":"2024-03-17T23:47:01.943288+0000","last_became_active":"2024-03-17T23:46:55.923741+0000","last_became_peered":"2024-03-17T23:46:55.923741+0000","last_unstale":"2024-03-17T23:47:01.943288+0000","last_undegraded":"2024-03-17T23:47:01.943288+0000","last_fullsized":"2024-03-17T23:47:01.943288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:00.571611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713487+0000","last_change":"2024-03-17T23:46:54.918712+0000","last_active":"2024-03-17T23:47:02.713487+0000","last_peered":"2024-03-17T23:47:02.713487+0000","last_clean":"2024-03-17T23:47:02.713487+0000","last_became_active":"2024-03-17T23:46:54.918470+0000","last_became_peered":"2024-03-17T23:46:54.918470+0000","last_unstale":"2024-03-17T23:47:02.713487+0000","last_undegraded":"2024-03-17T23:47:02.713487+0000","last_fullsized":"2024-03-17T23:47:02.713487+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:18.195935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.715105+0000","last_change":"2024-03-17T23:47:00.410049+0000","last_active":"2024-03-17T23:47:02.715105+0000","last_peered":"2024-03-17T23:47:02.715105+0000","last_clean":"2024-03-17T23:47:02.715105+0000","last_became_active":"2024-03-17T23:47:00.409688+0000","last_became_peered":"2024-03-17T23:47:00.409688+0000","last_unstale":"2024-03-17T23:47:02.715105+0000","last_undegraded":"2024-03-17T23:47:02.715105+0000","last_fullsized":"2024-03-17T23:47:02.715105+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:39:36.583214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106313+0000","last_change":"2024-03-17T23:46:55.925699+0000","last_active":"2024-03-17T23:47:02.106313+0000","last_peered":"2024-03-17T23:47:02.106313+0000","last_clean":"2024-03-17T23:47:02.106313+0000","last_became_active":"2024-03-17T23:46:55.925467+0000","last_became_peered":"2024-03-17T23:46:55.925467+0000","last_unstale":"2024-03-17T23:47:02.106313+0000","last_undegraded":"2024-03-17T23:47:02.106313+0000","last_fullsized":"2024-03-17T23:47:02.106313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:11:02.357691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713372+0000","last_change":"2024-03-17T23:46:54.923992+0000","last_active":"2024-03-17T23:47:02.713372+0000","last_peered":"2024-03-17T23:47:02.713372+0000","last_clean":"2024-03-17T23:47:02.713372+0000","last_became_active":"2024-03-17T23:46:54.923721+0000","last_became_peered":"2024-03-17T23:46:54.923721+0000","last_unstale":"2024-03-17T23:47:02.713372+0000","last_undegraded":"2024-03-17T23:47:02.713372+0000","last_fullsized":"2024-03-17T23:47:02.713372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:46:16.782215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106661+0000","last_change":"2024-03-17T23:46:59.960940+0000","last_active":"2024-03-17T23:47:02.106661+0000","last_peered":"2024-03-17T23:47:02.106661+0000","last_clean":"2024-03-17T23:47:02.106661+0000","last_became_active":"2024-03-17T23:46:59.960472+0000","last_became_peered":"2024-03-17T23:46:59.960472+0000","last_unstale":"2024-03-17T23:47:02.106661+0000","last_undegraded":"2024-03-17T23:47:02.106661+0000","last_fullsized":"2024-03-17T23:47:02.106661+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:21:41.104850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714557+0000","last_change":"2024-03-17T23:46:55.927164+0000","last_active":"2024-03-17T23:47:02.714557+0000","last_peered":"2024-03-17T23:47:02.714557+0000","last_clean":"2024-03-17T23:47:02.714557+0000","last_became_active":"2024-03-17T23:46:55.926883+0000","last_became_peered":"2024-03-17T23:46:55.926883+0000","last_unstale":"2024-03-17T23:47:02.714557+0000","last_undegraded":"2024-03-17T23:47:02.714557+0000","last_fullsized":"2024-03-17T23:47:02.714557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:03.353861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944736+0000","last_change":"2024-03-17T23:46:54.921997+0000","last_active":"2024-03-17T23:47:01.944736+0000","last_peered":"2024-03-17T23:47:01.944736+0000","last_clean":"2024-03-17T23:47:01.944736+0000","last_became_active":"2024-03-17T23:46:54.921701+0000","last_became_peered":"2024-03-17T23:46:54.921701+0000","last_unstale":"2024-03-17T23:47:01.944736+0000","last_undegraded":"2024-03-17T23:47:01.944736+0000","last_fullsized":"2024-03-17T23:47:01.944736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:11:01.353944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106094+0000","last_change":"2024-03-17T23:47:00.410238+0000","last_active":"2024-03-17T23:47:02.106094+0000","last_peered":"2024-03-17T23:47:02.106094+0000","last_clean":"2024-03-17T23:47:02.106094+0000","last_became_active":"2024-03-17T23:47:00.409845+0000","last_became_peered":"2024-03-17T23:47:00.409845+0000","last_unstale":"2024-03-17T23:47:02.106094+0000","last_undegraded":"2024-03-17T23:47:02.106094+0000","last_fullsized":"2024-03-17T23:47:02.106094+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:43:57.153828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943841+0000","last_change":"2024-03-17T23:46:55.929991+0000","last_active":"2024-03-17T23:47:01.943841+0000","last_peered":"2024-03-17T23:47:01.943841+0000","last_clean":"2024-03-17T23:47:01.943841+0000","last_became_active":"2024-03-17T23:46:55.929722+0000","last_became_peered":"2024-03-17T23:46:55.929722+0000","last_unstale":"2024-03-17T23:47:01.943841+0000","last_undegraded":"2024-03-17T23:47:01.943841+0000","last_fullsized":"2024-03-17T23:47:01.943841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:11:08.829506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713963+0000","last_change":"2024-03-17T23:46:54.913493+0000","last_active":"2024-03-17T23:47:02.713963+0000","last_peered":"2024-03-17T23:47:02.713963+0000","last_clean":"2024-03-17T23:47:02.713963+0000","last_became_active":"2024-03-17T23:46:54.913233+0000","last_became_peered":"2024-03-17T23:46:54.913233+0000","last_unstale":"2024-03-17T23:47:02.713963+0000","last_undegraded":"2024-03-17T23:47:02.713963+0000","last_fullsized":"2024-03-17T23:47:02.713963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:58.417421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944060+0000","last_change":"2024-03-17T23:47:00.408616+0000","last_active":"2024-03-17T23:47:01.944060+0000","last_peered":"2024-03-17T23:47:01.944060+0000","last_clean":"2024-03-17T23:47:01.944060+0000","last_became_active":"2024-03-17T23:47:00.408251+0000","last_became_peered":"2024-03-17T23:47:00.408251+0000","last_unstale":"2024-03-17T23:47:01.944060+0000","last_undegraded":"2024-03-17T23:47:01.944060+0000","last_fullsized":"2024-03-17T23:47:01.944060+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:16:52.426485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714810+0000","last_change":"2024-03-17T23:46:55.921416+0000","last_active":"2024-03-17T23:47:02.714810+0000","last_peered":"2024-03-17T23:47:02.714810+0000","last_clean":"2024-03-17T23:47:02.714810+0000","last_became_active":"2024-03-17T23:46:55.921128+0000","last_became_peered":"2024-03-17T23:46:55.921128+0000","last_unstale":"2024-03-17T23:47:02.714810+0000","last_undegraded":"2024-03-17T23:47:02.714810+0000","last_fullsized":"2024-03-17T23:47:02.714810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:49:47.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714025+0000","last_change":"2024-03-17T23:46:54.926748+0000","last_active":"2024-03-17T23:47:02.714025+0000","last_peered":"2024-03-17T23:47:02.714025+0000","last_clean":"2024-03-17T23:47:02.714025+0000","last_became_active":"2024-03-17T23:46:54.926467+0000","last_became_peered":"2024-03-17T23:46:54.926467+0000","last_unstale":"2024-03-17T23:47:02.714025+0000","last_undegraded":"2024-03-17T23:47:02.714025+0000","last_fullsized":"2024-03-17T23:47:02.714025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:58.417421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714085+0000","last_change":"2024-03-17T23:47:00.409306+0000","last_active":"2024-03-17T23:47:02.714085+0000","last_peered":"2024-03-17T23:47:02.714085+0000","last_clean":"2024-03-17T23:47:02.714085+0000","last_became_active":"2024-03-17T23:47:00.408934+0000","last_became_peered":"2024-03-17T23:47:00.408934+0000","last_unstale":"2024-03-17T23:47:02.714085+0000","last_undegraded":"2024-03-17T23:47:02.714085+0000","last_fullsized":"2024-03-17T23:47:02.714085+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:35:49.838832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944302+0000","last_change":"2024-03-17T23:46:55.933418+0000","last_active":"2024-03-17T23:47:01.944302+0000","last_peered":"2024-03-17T23:47:01.944302+0000","last_clean":"2024-03-17T23:47:01.944302+0000","last_became_active":"2024-03-17T23:46:55.931133+0000","last_became_peered":"2024-03-17T23:46:55.931133+0000","last_unstale":"2024-03-17T23:47:01.944302+0000","last_undegraded":"2024-03-17T23:47:01.944302+0000","last_fullsized":"2024-03-17T23:47:01.944302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:32:27.349125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714826+0000","last_change":"2024-03-17T23:46:54.914746+0000","last_active":"2024-03-17T23:47:02.714826+0000","last_peered":"2024-03-17T23:47:02.714826+0000","last_clean":"2024-03-17T23:47:02.714826+0000","last_became_active":"2024-03-17T23:46:54.914482+0000","last_became_peered":"2024-03-17T23:46:54.914482+0000","last_unstale":"2024-03-17T23:47:02.714826+0000","last_undegraded":"2024-03-17T23:47:02.714826+0000","last_fullsized":"2024-03-17T23:47:02.714826+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:42:33.325150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713844+0000","last_change":"2024-03-17T23:46:59.961836+0000","last_active":"2024-03-17T23:47:02.713844+0000","last_peered":"2024-03-17T23:47:02.713844+0000","last_clean":"2024-03-17T23:47:02.713844+0000","last_became_active":"2024-03-17T23:46:59.961458+0000","last_became_peered":"2024-03-17T23:46:59.961458+0000","last_unstale":"2024-03-17T23:47:02.713844+0000","last_undegraded":"2024-03-17T23:47:02.713844+0000","last_fullsized":"2024-03-17T23:47:02.713844+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:43:57.153828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106542+0000","last_change":"2024-03-17T23:46:55.926629+0000","last_active":"2024-03-17T23:47:02.106542+0000","last_peered":"2024-03-17T23:47:02.106542+0000","last_clean":"2024-03-17T23:47:02.106542+0000","last_became_active":"2024-03-17T23:46:55.926391+0000","last_became_peered":"2024-03-17T23:46:55.926391+0000","last_unstale":"2024-03-17T23:47:02.106542+0000","last_undegraded":"2024-03-17T23:47:02.106542+0000","last_fullsized":"2024-03-17T23:47:02.106542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:59.421169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943876+0000","last_change":"2024-03-17T23:46:54.921058+0000","last_active":"2024-03-17T23:47:01.943876+0000","last_peered":"2024-03-17T23:47:01.943876+0000","last_clean":"2024-03-17T23:47:01.943876+0000","last_became_active":"2024-03-17T23:46:54.919926+0000","last_became_peered":"2024-03-17T23:46:54.919926+0000","last_unstale":"2024-03-17T23:47:01.943876+0000","last_undegraded":"2024-03-17T23:47:01.943876+0000","last_fullsized":"2024-03-17T23:47:01.943876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:11:01.353944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714725+0000","last_change":"2024-03-17T23:46:59.957516+0000","last_active":"2024-03-17T23:47:02.714725+0000","last_peered":"2024-03-17T23:47:02.714725+0000","last_clean":"2024-03-17T23:47:02.714725+0000","last_became_active":"2024-03-17T23:46:59.957168+0000","last_became_peered":"2024-03-17T23:46:59.957168+0000","last_unstale":"2024-03-17T23:47:02.714725+0000","last_undegraded":"2024-03-17T23:47:02.714725+0000","last_fullsized":"2024-03-17T23:47:02.714725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:35:49.838832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713602+0000","last_change":"2024-03-17T23:46:55.921369+0000","last_active":"2024-03-17T23:47:02.713602+0000","last_peered":"2024-03-17T23:47:02.713602+0000","last_clean":"2024-03-17T23:47:02.713602+0000","last_became_active":"2024-03-17T23:46:55.921109+0000","last_became_peered":"2024-03-17T23:46:55.921109+0000","last_unstale":"2024-03-17T23:47:02.713602+0000","last_undegraded":"2024-03-17T23:47:02.713602+0000","last_fullsized":"2024-03-17T23:47:02.713602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:24.419077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713228+0000","last_change":"2024-03-17T23:46:54.918722+0000","last_active":"2024-03-17T23:47:02.713228+0000","last_peered":"2024-03-17T23:47:02.713228+0000","last_clean":"2024-03-17T23:47:02.713228+0000","last_became_active":"2024-03-17T23:46:54.918440+0000","last_became_peered":"2024-03-17T23:46:54.918440+0000","last_unstale":"2024-03-17T23:47:02.713228+0000","last_undegraded":"2024-03-17T23:47:02.713228+0000","last_fullsized":"2024-03-17T23:47:02.713228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:56.665566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.715106+0000","last_change":"2024-03-17T23:46:59.962185+0000","last_active":"2024-03-17T23:47:02.715106+0000","last_peered":"2024-03-17T23:47:02.715106+0000","last_clean":"2024-03-17T23:47:02.715106+0000","last_became_active":"2024-03-17T23:46:59.961682+0000","last_became_peered":"2024-03-17T23:46:59.961682+0000","last_unstale":"2024-03-17T23:47:02.715106+0000","last_undegraded":"2024-03-17T23:47:02.715106+0000","last_fullsized":"2024-03-17T23:47:02.715106+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:47:52.502020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713596+0000","last_change":"2024-03-17T23:46:55.924464+0000","last_active":"2024-03-17T23:47:02.713596+0000","last_peered":"2024-03-17T23:47:02.713596+0000","last_clean":"2024-03-17T23:47:02.713596+0000","last_became_active":"2024-03-17T23:46:55.924193+0000","last_became_peered":"2024-03-17T23:46:55.924193+0000","last_unstale":"2024-03-17T23:47:02.713596+0000","last_undegraded":"2024-03-17T23:47:02.713596+0000","last_fullsized":"2024-03-17T23:47:02.713596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:54.348184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713381+0000","last_change":"2024-03-17T23:46:54.921188+0000","last_active":"2024-03-17T23:47:02.713381+0000","last_peered":"2024-03-17T23:47:02.713381+0000","last_clean":"2024-03-17T23:47:02.713381+0000","last_became_active":"2024-03-17T23:46:54.920926+0000","last_became_peered":"2024-03-17T23:46:54.920926+0000","last_unstale":"2024-03-17T23:47:02.713381+0000","last_undegraded":"2024-03-17T23:47:02.713381+0000","last_fullsized":"2024-03-17T23:47:02.713381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:01:23.150557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714399+0000","last_change":"2024-03-17T23:47:00.410855+0000","last_active":"2024-03-17T23:47:02.714399+0000","last_peered":"2024-03-17T23:47:02.714399+0000","last_clean":"2024-03-17T23:47:02.714399+0000","last_became_active":"2024-03-17T23:47:00.410485+0000","last_became_peered":"2024-03-17T23:47:00.410485+0000","last_unstale":"2024-03-17T23:47:02.714399+0000","last_undegraded":"2024-03-17T23:47:02.714399+0000","last_fullsized":"2024-03-17T23:47:02.714399+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:21:41.104850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943282+0000","last_change":"2024-03-17T23:46:55.931906+0000","last_active":"2024-03-17T23:47:01.943282+0000","last_peered":"2024-03-17T23:47:01.943282+0000","last_clean":"2024-03-17T23:47:01.943282+0000","last_became_active":"2024-03-17T23:46:55.931643+0000","last_became_peered":"2024-03-17T23:46:55.931643+0000","last_unstale":"2024-03-17T23:47:01.943282+0000","last_undegraded":"2024-03-17T23:47:01.943282+0000","last_fullsized":"2024-03-17T23:47:01.943282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:24.419077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943618+0000","last_change":"2024-03-17T23:46:54.920477+0000","last_active":"2024-03-17T23:47:01.943618+0000","last_peered":"2024-03-17T23:47:01.943618+0000","last_clean":"2024-03-17T23:47:01.943618+0000","last_became_active":"2024-03-17T23:46:54.920116+0000","last_became_peered":"2024-03-17T23:46:54.920116+0000","last_unstale":"2024-03-17T23:47:01.943618+0000","last_undegraded":"2024-03-17T23:47:01.943618+0000","last_fullsized":"2024-03-17T23:47:01.943618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:56.665566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":180,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713494+0000","last_change":"2024-03-17T23:46:44.768668+0000","last_active":"2024-03-17T23:47:02.713494+0000","last_peered":"2024-03-17T23:47:02.713494+0000","last_clean":"2024-03-17T23:47:02.713494+0000","last_became_active":"2024-03-17T23:46:44.767826+0000","last_became_peered":"2024-03-17T23:46:44.767826+0000","last_unstale":"2024-03-17T23:47:02.713494+0000","last_undegraded":"2024-03-17T23:47:02.713494+0000","last_fullsized":"2024-03-17T23:47:02.713494+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_clean_scrub_stamp":"2024-03-17T23:46:43.744878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:54:22.286155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944382+0000","last_change":"2024-03-17T23:47:00.410345+0000","last_active":"2024-03-17T23:47:01.944382+0000","last_peered":"2024-03-17T23:47:01.944382+0000","last_clean":"2024-03-17T23:47:01.944382+0000","last_became_active":"2024-03-17T23:47:00.409948+0000","last_became_peered":"2024-03-17T23:47:00.409948+0000","last_unstale":"2024-03-17T23:47:01.944382+0000","last_undegraded":"2024-03-17T23:47:01.944382+0000","last_fullsized":"2024-03-17T23:47:01.944382+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:20:25.538662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713826+0000","last_change":"2024-03-17T23:46:55.943072+0000","last_active":"2024-03-17T23:47:02.713826+0000","last_peered":"2024-03-17T23:47:02.713826+0000","last_clean":"2024-03-17T23:47:02.713826+0000","last_became_active":"2024-03-17T23:46:55.942516+0000","last_became_peered":"2024-03-17T23:46:55.942516+0000","last_unstale":"2024-03-17T23:47:02.713826+0000","last_undegraded":"2024-03-17T23:47:02.713826+0000","last_fullsized":"2024-03-17T23:47:02.713826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:00.571611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713510+0000","last_change":"2024-03-17T23:46:54.916398+0000","last_active":"2024-03-17T23:47:02.713510+0000","last_peered":"2024-03-17T23:47:02.713510+0000","last_clean":"2024-03-17T23:47:02.713510+0000","last_became_active":"2024-03-17T23:46:54.916167+0000","last_became_peered":"2024-03-17T23:46:54.916167+0000","last_unstale":"2024-03-17T23:47:02.713510+0000","last_undegraded":"2024-03-17T23:47:02.713510+0000","last_fullsized":"2024-03-17T23:47:02.713510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:42:33.325150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943370+0000","last_change":"2024-03-17T23:47:00.408551+0000","last_active":"2024-03-17T23:47:01.943370+0000","last_peered":"2024-03-17T23:47:01.943370+0000","last_clean":"2024-03-17T23:47:01.943370+0000","last_became_active":"2024-03-17T23:47:00.408223+0000","last_became_peered":"2024-03-17T23:47:00.408223+0000","last_unstale":"2024-03-17T23:47:01.943370+0000","last_undegraded":"2024-03-17T23:47:01.943370+0000","last_fullsized":"2024-03-17T23:47:01.943370+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:52:49.194504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944055+0000","last_change":"2024-03-17T23:46:55.933223+0000","last_active":"2024-03-17T23:47:01.944055+0000","last_peered":"2024-03-17T23:47:01.944055+0000","last_clean":"2024-03-17T23:47:01.944055+0000","last_became_active":"2024-03-17T23:46:55.932957+0000","last_became_peered":"2024-03-17T23:46:55.932957+0000","last_unstale":"2024-03-17T23:47:01.944055+0000","last_undegraded":"2024-03-17T23:47:01.944055+0000","last_fullsized":"2024-03-17T23:47:01.944055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:52:45.163567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713856+0000","last_change":"2024-03-17T23:46:54.919441+0000","last_active":"2024-03-17T23:47:02.713856+0000","last_peered":"2024-03-17T23:47:02.713856+0000","last_clean":"2024-03-17T23:47:02.713856+0000","last_became_active":"2024-03-17T23:46:54.919151+0000","last_became_peered":"2024-03-17T23:46:54.919151+0000","last_unstale":"2024-03-17T23:47:02.713856+0000","last_undegraded":"2024-03-17T23:47:02.713856+0000","last_fullsized":"2024-03-17T23:47:02.713856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:11:01.353944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714221+0000","last_change":"2024-03-17T23:46:59.963445+0000","last_active":"2024-03-17T23:47:02.714221+0000","last_peered":"2024-03-17T23:47:02.714221+0000","last_clean":"2024-03-17T23:47:02.714221+0000","last_became_active":"2024-03-17T23:46:59.963078+0000","last_became_peered":"2024-03-17T23:46:59.963078+0000","last_unstale":"2024-03-17T23:47:02.714221+0000","last_undegraded":"2024-03-17T23:47:02.714221+0000","last_fullsized":"2024-03-17T23:47:02.714221+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:47:52.502020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713966+0000","last_change":"2024-03-17T23:46:55.930951+0000","last_active":"2024-03-17T23:47:02.713966+0000","last_peered":"2024-03-17T23:47:02.713966+0000","last_clean":"2024-03-17T23:47:02.713966+0000","last_became_active":"2024-03-17T23:46:55.930689+0000","last_became_peered":"2024-03-17T23:46:55.930689+0000","last_unstale":"2024-03-17T23:47:02.713966+0000","last_undegraded":"2024-03-17T23:47:02.713966+0000","last_fullsized":"2024-03-17T23:47:02.713966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:43:53.122891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713358+0000","last_change":"2024-03-17T23:46:54.921408+0000","last_active":"2024-03-17T23:47:02.713358+0000","last_peered":"2024-03-17T23:47:02.713358+0000","last_clean":"2024-03-17T23:47:02.713358+0000","last_became_active":"2024-03-17T23:46:54.920825+0000","last_became_peered":"2024-03-17T23:46:54.920825+0000","last_unstale":"2024-03-17T23:47:02.713358+0000","last_undegraded":"2024-03-17T23:47:02.713358+0000","last_fullsized":"2024-03-17T23:47:02.713358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:18.195935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.942947+0000","last_change":"2024-03-17T23:47:00.411371+0000","last_active":"2024-03-17T23:47:01.942947+0000","last_peered":"2024-03-17T23:47:01.942947+0000","last_clean":"2024-03-17T23:47:01.942947+0000","last_became_active":"2024-03-17T23:47:00.410726+0000","last_became_peered":"2024-03-17T23:47:00.410726+0000","last_unstale":"2024-03-17T23:47:01.942947+0000","last_undegraded":"2024-03-17T23:47:01.942947+0000","last_fullsized":"2024-03-17T23:47:01.942947+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:47:52.502020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713787+0000","last_change":"2024-03-17T23:46:55.933053+0000","last_active":"2024-03-17T23:47:02.713787+0000","last_peered":"2024-03-17T23:47:02.713787+0000","last_clean":"2024-03-17T23:47:02.713787+0000","last_became_active":"2024-03-17T23:46:55.932756+0000","last_became_peered":"2024-03-17T23:46:55.932756+0000","last_unstale":"2024-03-17T23:47:02.713787+0000","last_undegraded":"2024-03-17T23:47:02.713787+0000","last_fullsized":"2024-03-17T23:47:02.713787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:45:29.447600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106603+0000","last_change":"2024-03-17T23:46:54.916992+0000","last_active":"2024-03-17T23:47:02.106603+0000","last_peered":"2024-03-17T23:47:02.106603+0000","last_clean":"2024-03-17T23:47:02.106603+0000","last_became_active":"2024-03-17T23:46:54.916296+0000","last_became_peered":"2024-03-17T23:46:54.916296+0000","last_unstale":"2024-03-17T23:47:02.106603+0000","last_undegraded":"2024-03-17T23:47:02.106603+0000","last_fullsized":"2024-03-17T23:47:02.106603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:17:26.217921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106127+0000","last_change":"2024-03-17T23:46:59.957769+0000","last_active":"2024-03-17T23:47:02.106127+0000","last_peered":"2024-03-17T23:47:02.106127+0000","last_clean":"2024-03-17T23:47:02.106127+0000","last_became_active":"2024-03-17T23:46:59.957385+0000","last_became_peered":"2024-03-17T23:46:59.957385+0000","last_unstale":"2024-03-17T23:47:02.106127+0000","last_undegraded":"2024-03-17T23:47:02.106127+0000","last_fullsized":"2024-03-17T23:47:02.106127+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:35:49.838832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713581+0000","last_change":"2024-03-17T23:46:55.935175+0000","last_active":"2024-03-17T23:47:02.713581+0000","last_peered":"2024-03-17T23:47:02.713581+0000","last_clean":"2024-03-17T23:47:02.713581+0000","last_became_active":"2024-03-17T23:46:55.934338+0000","last_became_peered":"2024-03-17T23:46:55.934338+0000","last_unstale":"2024-03-17T23:47:02.713581+0000","last_undegraded":"2024-03-17T23:47:02.713581+0000","last_fullsized":"2024-03-17T23:47:02.713581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:59:32.027430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713064+0000","last_change":"2024-03-17T23:46:54.913663+0000","last_active":"2024-03-17T23:47:02.713064+0000","last_peered":"2024-03-17T23:47:02.713064+0000","last_clean":"2024-03-17T23:47:02.713064+0000","last_became_active":"2024-03-17T23:46:54.913410+0000","last_became_peered":"2024-03-17T23:46:54.913410+0000","last_unstale":"2024-03-17T23:47:02.713064+0000","last_undegraded":"2024-03-17T23:47:02.713064+0000","last_fullsized":"2024-03-17T23:47:02.713064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:45:17.571056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714251+0000","last_change":"2024-03-17T23:46:59.967164+0000","last_active":"2024-03-17T23:47:02.714251+0000","last_peered":"2024-03-17T23:47:02.714251+0000","last_clean":"2024-03-17T23:47:02.714251+0000","last_became_active":"2024-03-17T23:46:59.966653+0000","last_became_peered":"2024-03-17T23:46:59.966653+0000","last_unstale":"2024-03-17T23:47:02.714251+0000","last_undegraded":"2024-03-17T23:47:02.714251+0000","last_fullsized":"2024-03-17T23:47:02.714251+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:57.296991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943052+0000","last_change":"2024-03-17T23:46:55.925322+0000","last_active":"2024-03-17T23:47:01.943052+0000","last_peered":"2024-03-17T23:47:01.943052+0000","last_clean":"2024-03-17T23:47:01.943052+0000","last_became_active":"2024-03-17T23:46:55.925037+0000","last_became_peered":"2024-03-17T23:46:55.925037+0000","last_unstale":"2024-03-17T23:47:01.943052+0000","last_undegraded":"2024-03-17T23:47:01.943052+0000","last_fullsized":"2024-03-17T23:47:01.943052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:39.972607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943437+0000","last_change":"2024-03-17T23:46:54.918033+0000","last_active":"2024-03-17T23:47:01.943437+0000","last_peered":"2024-03-17T23:47:01.943437+0000","last_clean":"2024-03-17T23:47:01.943437+0000","last_became_active":"2024-03-17T23:46:54.917735+0000","last_became_peered":"2024-03-17T23:46:54.917735+0000","last_unstale":"2024-03-17T23:47:01.943437+0000","last_undegraded":"2024-03-17T23:47:01.943437+0000","last_fullsized":"2024-03-17T23:47:01.943437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:42:33.325150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943778+0000","last_change":"2024-03-17T23:47:00.412019+0000","last_active":"2024-03-17T23:47:01.943778+0000","last_peered":"2024-03-17T23:47:01.943778+0000","last_clean":"2024-03-17T23:47:01.943778+0000","last_became_active":"2024-03-17T23:47:00.411603+0000","last_became_peered":"2024-03-17T23:47:00.411603+0000","last_unstale":"2024-03-17T23:47:01.943778+0000","last_undegraded":"2024-03-17T23:47:01.943778+0000","last_fullsized":"2024-03-17T23:47:01.943778+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:18:26.998562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713747+0000","last_change":"2024-03-17T23:46:55.935249+0000","last_active":"2024-03-17T23:47:02.713747+0000","last_peered":"2024-03-17T23:47:02.713747+0000","last_clean":"2024-03-17T23:47:02.713747+0000","last_became_active":"2024-03-17T23:46:55.934833+0000","last_became_peered":"2024-03-17T23:46:55.934833+0000","last_unstale":"2024-03-17T23:47:02.713747+0000","last_undegraded":"2024-03-17T23:47:02.713747+0000","last_fullsized":"2024-03-17T23:47:02.713747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:32:27.349125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943797+0000","last_change":"2024-03-17T23:46:54.924340+0000","last_active":"2024-03-17T23:47:01.943797+0000","last_peered":"2024-03-17T23:47:01.943797+0000","last_clean":"2024-03-17T23:47:01.943797+0000","last_became_active":"2024-03-17T23:46:54.924080+0000","last_became_peered":"2024-03-17T23:46:54.924080+0000","last_unstale":"2024-03-17T23:47:01.943797+0000","last_undegraded":"2024-03-17T23:47:01.943797+0000","last_fullsized":"2024-03-17T23:47:01.943797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:06:54.433520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943804+0000","last_change":"2024-03-17T23:47:00.408517+0000","last_active":"2024-03-17T23:47:01.943804+0000","last_peered":"2024-03-17T23:47:01.943804+0000","last_clean":"2024-03-17T23:47:01.943804+0000","last_became_active":"2024-03-17T23:47:00.408125+0000","last_became_peered":"2024-03-17T23:47:00.408125+0000","last_unstale":"2024-03-17T23:47:01.943804+0000","last_undegraded":"2024-03-17T23:47:01.943804+0000","last_fullsized":"2024-03-17T23:47:01.943804+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:12:39.327858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714486+0000","last_change":"2024-03-17T23:46:55.928837+0000","last_active":"2024-03-17T23:47:02.714486+0000","last_peered":"2024-03-17T23:47:02.714486+0000","last_clean":"2024-03-17T23:47:02.714486+0000","last_became_active":"2024-03-17T23:46:55.928557+0000","last_became_peered":"2024-03-17T23:46:55.928557+0000","last_unstale":"2024-03-17T23:47:02.714486+0000","last_undegraded":"2024-03-17T23:47:02.714486+0000","last_fullsized":"2024-03-17T23:47:02.714486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:23:58.221114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714395+0000","last_change":"2024-03-17T23:46:54.912215+0000","last_active":"2024-03-17T23:47:02.714395+0000","last_peered":"2024-03-17T23:47:02.714395+0000","last_clean":"2024-03-17T23:47:02.714395+0000","last_became_active":"2024-03-17T23:46:54.911909+0000","last_became_peered":"2024-03-17T23:46:54.911909+0000","last_unstale":"2024-03-17T23:47:02.714395+0000","last_undegraded":"2024-03-17T23:47:02.714395+0000","last_fullsized":"2024-03-17T23:47:02.714395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:45:17.571056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944243+0000","last_change":"2024-03-17T23:47:00.408798+0000","last_active":"2024-03-17T23:47:01.944243+0000","last_peered":"2024-03-17T23:47:01.944243+0000","last_clean":"2024-03-17T23:47:01.944243+0000","last_became_active":"2024-03-17T23:47:00.408420+0000","last_became_peered":"2024-03-17T23:47:00.408420+0000","last_unstale":"2024-03-17T23:47:01.944243+0000","last_undegraded":"2024-03-17T23:47:01.944243+0000","last_fullsized":"2024-03-17T23:47:01.944243+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:57.296991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713721+0000","last_change":"2024-03-17T23:46:55.935456+0000","last_active":"2024-03-17T23:47:02.713721+0000","last_peered":"2024-03-17T23:47:02.713721+0000","last_clean":"2024-03-17T23:47:02.713721+0000","last_became_active":"2024-03-17T23:46:55.935187+0000","last_became_peered":"2024-03-17T23:46:55.935187+0000","last_unstale":"2024-03-17T23:47:02.713721+0000","last_undegraded":"2024-03-17T23:47:02.713721+0000","last_fullsized":"2024-03-17T23:47:02.713721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:11:08.829506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"26'8","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:04.869244+0000","last_change":"2024-03-17T23:46:54.918644+0000","last_active":"2024-03-17T23:47:04.869244+0000","last_peered":"2024-03-17T23:47:04.869244+0000","last_clean":"2024-03-17T23:47:04.869244+0000","last_became_active":"2024-03-17T23:46:54.918290+0000","last_became_peered":"2024-03-17T23:46:54.918290+0000","last_unstale":"2024-03-17T23:47:04.869244+0000","last_undegraded":"2024-03-17T23:47:04.869244+0000","last_fullsized":"2024-03-17T23:47:04.869244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:58.417421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3701,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714038+0000","last_change":"2024-03-17T23:46:59.961721+0000","last_active":"2024-03-17T23:47:02.714038+0000","last_peered":"2024-03-17T23:47:02.714038+0000","last_clean":"2024-03-17T23:47:02.714038+0000","last_became_active":"2024-03-17T23:46:59.960948+0000","last_became_peered":"2024-03-17T23:46:59.960948+0000","last_unstale":"2024-03-17T23:47:02.714038+0000","last_undegraded":"2024-03-17T23:47:02.714038+0000","last_fullsized":"2024-03-17T23:47:02.714038+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:27:51.964285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106592+0000","last_change":"2024-03-17T23:46:55.928225+0000","last_active":"2024-03-17T23:47:02.106592+0000","last_peered":"2024-03-17T23:47:02.106592+0000","last_clean":"2024-03-17T23:47:02.106592+0000","last_became_active":"2024-03-17T23:46:55.927359+0000","last_became_peered":"2024-03-17T23:46:55.927359+0000","last_unstale":"2024-03-17T23:47:02.106592+0000","last_undegraded":"2024-03-17T23:47:02.106592+0000","last_fullsized":"2024-03-17T23:47:02.106592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:46:17.785962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713797+0000","last_change":"2024-03-17T23:46:54.917416+0000","last_active":"2024-03-17T23:47:02.713797+0000","last_peered":"2024-03-17T23:47:02.713797+0000","last_clean":"2024-03-17T23:47:02.713797+0000","last_became_active":"2024-03-17T23:46:54.917136+0000","last_became_peered":"2024-03-17T23:46:54.917136+0000","last_unstale":"2024-03-17T23:47:02.713797+0000","last_undegraded":"2024-03-17T23:47:02.713797+0000","last_fullsized":"2024-03-17T23:47:02.713797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:06:54.433520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106414+0000","last_change":"2024-03-17T23:46:59.957651+0000","last_active":"2024-03-17T23:47:02.106414+0000","last_peered":"2024-03-17T23:47:02.106414+0000","last_clean":"2024-03-17T23:47:02.106414+0000","last_became_active":"2024-03-17T23:46:59.957261+0000","last_became_peered":"2024-03-17T23:46:59.957261+0000","last_unstale":"2024-03-17T23:47:02.106414+0000","last_undegraded":"2024-03-17T23:47:02.106414+0000","last_fullsized":"2024-03-17T23:47:02.106414+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:47:52.502020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713868+0000","last_change":"2024-03-17T23:46:55.937105+0000","last_active":"2024-03-17T23:47:02.713868+0000","last_peered":"2024-03-17T23:47:02.713868+0000","last_clean":"2024-03-17T23:47:02.713868+0000","last_became_active":"2024-03-17T23:46:55.936762+0000","last_became_peered":"2024-03-17T23:46:55.936762+0000","last_unstale":"2024-03-17T23:47:02.713868+0000","last_undegraded":"2024-03-17T23:47:02.713868+0000","last_fullsized":"2024-03-17T23:47:02.713868+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:39.972607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944371+0000","last_change":"2024-03-17T23:46:54.922118+0000","last_active":"2024-03-17T23:47:01.944371+0000","last_peered":"2024-03-17T23:47:01.944371+0000","last_clean":"2024-03-17T23:47:01.944371+0000","last_became_active":"2024-03-17T23:46:54.921834+0000","last_became_peered":"2024-03-17T23:46:54.921834+0000","last_unstale":"2024-03-17T23:47:01.944371+0000","last_undegraded":"2024-03-17T23:47:01.944371+0000","last_fullsized":"2024-03-17T23:47:01.944371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:42:33.325150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943156+0000","last_change":"2024-03-17T23:46:59.957652+0000","last_active":"2024-03-17T23:47:01.943156+0000","last_peered":"2024-03-17T23:47:01.943156+0000","last_clean":"2024-03-17T23:47:01.943156+0000","last_became_active":"2024-03-17T23:46:59.957242+0000","last_became_peered":"2024-03-17T23:46:59.957242+0000","last_unstale":"2024-03-17T23:47:01.943156+0000","last_undegraded":"2024-03-17T23:47:01.943156+0000","last_fullsized":"2024-03-17T23:47:01.943156+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:48:08.469403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944280+0000","last_change":"2024-03-17T23:46:55.930428+0000","last_active":"2024-03-17T23:47:01.944280+0000","last_peered":"2024-03-17T23:47:01.944280+0000","last_clean":"2024-03-17T23:47:01.944280+0000","last_became_active":"2024-03-17T23:46:55.930119+0000","last_became_peered":"2024-03-17T23:46:55.930119+0000","last_unstale":"2024-03-17T23:47:01.944280+0000","last_undegraded":"2024-03-17T23:47:01.944280+0000","last_fullsized":"2024-03-17T23:47:01.944280+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:00.571611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944550+0000","last_change":"2024-03-17T23:46:54.918186+0000","last_active":"2024-03-17T23:47:01.944550+0000","last_peered":"2024-03-17T23:47:01.944550+0000","last_clean":"2024-03-17T23:47:01.944550+0000","last_became_active":"2024-03-17T23:46:54.917839+0000","last_became_peered":"2024-03-17T23:46:54.917839+0000","last_unstale":"2024-03-17T23:47:01.944550+0000","last_undegraded":"2024-03-17T23:47:01.944550+0000","last_fullsized":"2024-03-17T23:47:01.944550+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:06:54.433520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943131+0000","last_change":"2024-03-17T23:47:00.411924+0000","last_active":"2024-03-17T23:47:01.943131+0000","last_peered":"2024-03-17T23:47:01.943131+0000","last_clean":"2024-03-17T23:47:01.943131+0000","last_became_active":"2024-03-17T23:47:00.411410+0000","last_became_peered":"2024-03-17T23:47:00.411410+0000","last_unstale":"2024-03-17T23:47:01.943131+0000","last_undegraded":"2024-03-17T23:47:01.943131+0000","last_fullsized":"2024-03-17T23:47:01.943131+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:56:09.808314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714107+0000","last_change":"2024-03-17T23:47:00.410837+0000","last_active":"2024-03-17T23:47:02.714107+0000","last_peered":"2024-03-17T23:47:02.714107+0000","last_clean":"2024-03-17T23:47:02.714107+0000","last_became_active":"2024-03-17T23:47:00.410459+0000","last_became_peered":"2024-03-17T23:47:00.410459+0000","last_unstale":"2024-03-17T23:47:02.714107+0000","last_undegraded":"2024-03-17T23:47:02.714107+0000","last_fullsized":"2024-03-17T23:47:02.714107+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:39:36.583214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713305+0000","last_change":"2024-03-17T23:46:55.928213+0000","last_active":"2024-03-17T23:47:02.713305+0000","last_peered":"2024-03-17T23:47:02.713305+0000","last_clean":"2024-03-17T23:47:02.713305+0000","last_became_active":"2024-03-17T23:46:55.927957+0000","last_became_peered":"2024-03-17T23:46:55.927957+0000","last_unstale":"2024-03-17T23:47:02.713305+0000","last_undegraded":"2024-03-17T23:47:02.713305+0000","last_fullsized":"2024-03-17T23:47:02.713305+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:03.353861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943171+0000","last_change":"2024-03-17T23:46:54.919392+0000","last_active":"2024-03-17T23:47:01.943171+0000","last_peered":"2024-03-17T23:47:01.943171+0000","last_clean":"2024-03-17T23:47:01.943171+0000","last_became_active":"2024-03-17T23:46:54.918594+0000","last_became_peered":"2024-03-17T23:46:54.918594+0000","last_unstale":"2024-03-17T23:47:01.943171+0000","last_undegraded":"2024-03-17T23:47:01.943171+0000","last_fullsized":"2024-03-17T23:47:01.943171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:18.195935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944720+0000","last_change":"2024-03-17T23:46:59.959748+0000","last_active":"2024-03-17T23:47:01.944720+0000","last_peered":"2024-03-17T23:47:01.944720+0000","last_clean":"2024-03-17T23:47:01.944720+0000","last_became_active":"2024-03-17T23:46:59.959383+0000","last_became_peered":"2024-03-17T23:46:59.959383+0000","last_unstale":"2024-03-17T23:47:01.944720+0000","last_undegraded":"2024-03-17T23:47:01.944720+0000","last_fullsized":"2024-03-17T23:47:01.944720+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:43:57.153828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943202+0000","last_change":"2024-03-17T23:46:55.929140+0000","last_active":"2024-03-17T23:47:01.943202+0000","last_peered":"2024-03-17T23:47:01.943202+0000","last_clean":"2024-03-17T23:47:01.943202+0000","last_became_active":"2024-03-17T23:46:55.928874+0000","last_became_peered":"2024-03-17T23:46:55.928874+0000","last_unstale":"2024-03-17T23:47:01.943202+0000","last_undegraded":"2024-03-17T23:47:01.943202+0000","last_fullsized":"2024-03-17T23:47:01.943202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:59:32.027430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943228+0000","last_change":"2024-03-17T23:46:54.923319+0000","last_active":"2024-03-17T23:47:01.943228+0000","last_peered":"2024-03-17T23:47:01.943228+0000","last_clean":"2024-03-17T23:47:01.943228+0000","last_became_active":"2024-03-17T23:46:54.923060+0000","last_became_peered":"2024-03-17T23:46:54.923060+0000","last_unstale":"2024-03-17T23:47:01.943228+0000","last_undegraded":"2024-03-17T23:47:01.943228+0000","last_fullsized":"2024-03-17T23:47:01.943228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:45:17.571056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943597+0000","last_change":"2024-03-17T23:46:54.917589+0000","last_active":"2024-03-17T23:47:01.943597+0000","last_peered":"2024-03-17T23:47:01.943597+0000","last_clean":"2024-03-17T23:47:01.943597+0000","last_became_active":"2024-03-17T23:46:54.917183+0000","last_became_peered":"2024-03-17T23:46:54.917183+0000","last_unstale":"2024-03-17T23:47:01.943597+0000","last_undegraded":"2024-03-17T23:47:01.943597+0000","last_fullsized":"2024-03-17T23:47:01.943597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:18.195935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714330+0000","last_change":"2024-03-17T23:46:55.924339+0000","last_active":"2024-03-17T23:47:02.714330+0000","last_peered":"2024-03-17T23:47:02.714330+0000","last_clean":"2024-03-17T23:47:02.714330+0000","last_became_active":"2024-03-17T23:46:55.923891+0000","last_became_peered":"2024-03-17T23:46:55.923891+0000","last_unstale":"2024-03-17T23:47:02.714330+0000","last_undegraded":"2024-03-17T23:47:02.714330+0000","last_fullsized":"2024-03-17T23:47:02.714330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:11:08.829506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713322+0000","last_change":"2024-03-17T23:46:59.959425+0000","last_active":"2024-03-17T23:47:02.713322+0000","last_peered":"2024-03-17T23:47:02.713322+0000","last_clean":"2024-03-17T23:47:02.713322+0000","last_became_active":"2024-03-17T23:46:59.958851+0000","last_became_peered":"2024-03-17T23:46:59.958851+0000","last_unstale":"2024-03-17T23:47:02.713322+0000","last_undegraded":"2024-03-17T23:47:02.713322+0000","last_fullsized":"2024-03-17T23:47:02.713322+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:58.379120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943490+0000","last_change":"2024-03-17T23:46:54.921325+0000","last_active":"2024-03-17T23:47:01.943490+0000","last_peered":"2024-03-17T23:47:01.943490+0000","last_clean":"2024-03-17T23:47:01.943490+0000","last_became_active":"2024-03-17T23:46:54.921093+0000","last_became_peered":"2024-03-17T23:46:54.921093+0000","last_unstale":"2024-03-17T23:47:01.943490+0000","last_undegraded":"2024-03-17T23:47:01.943490+0000","last_fullsized":"2024-03-17T23:47:01.943490+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:56.665566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106239+0000","last_change":"2024-03-17T23:46:55.922093+0000","last_active":"2024-03-17T23:47:02.106239+0000","last_peered":"2024-03-17T23:47:02.106239+0000","last_clean":"2024-03-17T23:47:02.106239+0000","last_became_active":"2024-03-17T23:46:55.919778+0000","last_became_peered":"2024-03-17T23:46:55.919778+0000","last_unstale":"2024-03-17T23:47:02.106239+0000","last_undegraded":"2024-03-17T23:47:02.106239+0000","last_fullsized":"2024-03-17T23:47:02.106239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:06:55.437268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943978+0000","last_change":"2024-03-17T23:46:59.964222+0000","last_active":"2024-03-17T23:47:01.943978+0000","last_peered":"2024-03-17T23:47:01.943978+0000","last_clean":"2024-03-17T23:47:01.943978+0000","last_became_active":"2024-03-17T23:46:59.963807+0000","last_became_peered":"2024-03-17T23:46:59.963807+0000","last_unstale":"2024-03-17T23:47:01.943978+0000","last_undegraded":"2024-03-17T23:47:01.943978+0000","last_fullsized":"2024-03-17T23:47:01.943978+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:25:11.791005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.942962+0000","last_change":"2024-03-17T23:46:55.926785+0000","last_active":"2024-03-17T23:47:01.942962+0000","last_peered":"2024-03-17T23:47:01.942962+0000","last_clean":"2024-03-17T23:47:01.942962+0000","last_became_active":"2024-03-17T23:46:55.926494+0000","last_became_peered":"2024-03-17T23:46:55.926494+0000","last_unstale":"2024-03-17T23:47:01.942962+0000","last_undegraded":"2024-03-17T23:47:01.942962+0000","last_fullsized":"2024-03-17T23:47:01.942962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:24.419077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713141+0000","last_change":"2024-03-17T23:46:55.930094+0000","last_active":"2024-03-17T23:47:02.713141+0000","last_peered":"2024-03-17T23:47:02.713141+0000","last_clean":"2024-03-17T23:47:02.713141+0000","last_became_active":"2024-03-17T23:46:55.929834+0000","last_became_peered":"2024-03-17T23:46:55.929834+0000","last_unstale":"2024-03-17T23:47:02.713141+0000","last_undegraded":"2024-03-17T23:47:02.713141+0000","last_fullsized":"2024-03-17T23:47:02.713141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:59.421169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943791+0000","last_change":"2024-03-17T23:46:55.934265+0000","last_active":"2024-03-17T23:47:01.943791+0000","last_peered":"2024-03-17T23:47:01.943791+0000","last_clean":"2024-03-17T23:47:01.943791+0000","last_became_active":"2024-03-17T23:46:55.933735+0000","last_became_peered":"2024-03-17T23:46:55.933735+0000","last_unstale":"2024-03-17T23:47:01.943791+0000","last_undegraded":"2024-03-17T23:47:01.943791+0000","last_fullsized":"2024-03-17T23:47:01.943791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:59:32.027430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714250+0000","last_change":"2024-03-17T23:46:55.930627+0000","last_active":"2024-03-17T23:47:02.714250+0000","last_peered":"2024-03-17T23:47:02.714250+0000","last_clean":"2024-03-17T23:47:02.714250+0000","last_became_active":"2024-03-17T23:46:55.930368+0000","last_became_peered":"2024-03-17T23:46:55.930368+0000","last_unstale":"2024-03-17T23:47:02.714250+0000","last_undegraded":"2024-03-17T23:47:02.714250+0000","last_fullsized":"2024-03-17T23:47:02.714250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:47:30.023063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943914+0000","last_change":"2024-03-17T23:46:55.930544+0000","last_active":"2024-03-17T23:47:01.943914+0000","last_peered":"2024-03-17T23:47:01.943914+0000","last_clean":"2024-03-17T23:47:01.943914+0000","last_became_active":"2024-03-17T23:46:55.930277+0000","last_became_peered":"2024-03-17T23:46:55.930277+0000","last_unstale":"2024-03-17T23:47:01.943914+0000","last_undegraded":"2024-03-17T23:47:01.943914+0000","last_fullsized":"2024-03-17T23:47:01.943914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:23:58.221114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713632+0000","last_change":"2024-03-17T23:46:55.927054+0000","last_active":"2024-03-17T23:47:02.713632+0000","last_peered":"2024-03-17T23:47:02.713632+0000","last_clean":"2024-03-17T23:47:02.713632+0000","last_became_active":"2024-03-17T23:46:55.926762+0000","last_became_peered":"2024-03-17T23:46:55.926762+0000","last_unstale":"2024-03-17T23:47:02.713632+0000","last_undegraded":"2024-03-17T23:47:02.713632+0000","last_fullsized":"2024-03-17T23:47:02.713632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:39.972607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944787+0000","last_change":"2024-03-17T23:46:55.929740+0000","last_active":"2024-03-17T23:47:01.944787+0000","last_peered":"2024-03-17T23:47:01.944787+0000","last_clean":"2024-03-17T23:47:01.944787+0000","last_became_active":"2024-03-17T23:46:55.929433+0000","last_became_peered":"2024-03-17T23:46:55.929433+0000","last_unstale":"2024-03-17T23:47:01.944787+0000","last_undegraded":"2024-03-17T23:47:01.944787+0000","last_fullsized":"2024-03-17T23:47:01.944787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:11:08.829506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943753+0000","last_change":"2024-03-17T23:46:55.930851+0000","last_active":"2024-03-17T23:47:01.943753+0000","last_peered":"2024-03-17T23:47:01.943753+0000","last_clean":"2024-03-17T23:47:01.943753+0000","last_became_active":"2024-03-17T23:46:55.930305+0000","last_became_peered":"2024-03-17T23:46:55.930305+0000","last_unstale":"2024-03-17T23:47:01.943753+0000","last_undegraded":"2024-03-17T23:47:01.943753+0000","last_fullsized":"2024-03-17T23:47:01.943753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:47:30.023063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943134+0000","last_change":"2024-03-17T23:46:55.933139+0000","last_active":"2024-03-17T23:47:01.943134+0000","last_peered":"2024-03-17T23:47:01.943134+0000","last_clean":"2024-03-17T23:47:01.943134+0000","last_became_active":"2024-03-17T23:46:55.932843+0000","last_became_peered":"2024-03-17T23:46:55.932843+0000","last_unstale":"2024-03-17T23:47:01.943134+0000","last_undegraded":"2024-03-17T23:47:01.943134+0000","last_fullsized":"2024-03-17T23:47:01.943134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:44:47.002489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943147+0000","last_change":"2024-03-17T23:46:55.930881+0000","last_active":"2024-03-17T23:47:01.943147+0000","last_peered":"2024-03-17T23:47:01.943147+0000","last_clean":"2024-03-17T23:47:01.943147+0000","last_became_active":"2024-03-17T23:46:55.930629+0000","last_became_peered":"2024-03-17T23:46:55.930629+0000","last_unstale":"2024-03-17T23:47:01.943147+0000","last_undegraded":"2024-03-17T23:47:01.943147+0000","last_fullsized":"2024-03-17T23:47:01.943147+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:49:47.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943696+0000","last_change":"2024-03-17T23:46:55.936236+0000","last_active":"2024-03-17T23:47:01.943696+0000","last_peered":"2024-03-17T23:47:01.943696+0000","last_clean":"2024-03-17T23:47:01.943696+0000","last_became_active":"2024-03-17T23:46:55.933485+0000","last_became_peered":"2024-03-17T23:46:55.933485+0000","last_unstale":"2024-03-17T23:47:01.943696+0000","last_undegraded":"2024-03-17T23:47:01.943696+0000","last_fullsized":"2024-03-17T23:47:01.943696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:11:08.829506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714118+0000","last_change":"2024-03-17T23:46:55.930067+0000","last_active":"2024-03-17T23:47:02.714118+0000","last_peered":"2024-03-17T23:47:02.714118+0000","last_clean":"2024-03-17T23:47:02.714118+0000","last_became_active":"2024-03-17T23:46:55.929808+0000","last_became_peered":"2024-03-17T23:46:55.929808+0000","last_unstale":"2024-03-17T23:47:02.714118+0000","last_undegraded":"2024-03-17T23:47:02.714118+0000","last_fullsized":"2024-03-17T23:47:02.714118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:49:47.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943993+0000","last_change":"2024-03-17T23:46:55.917074+0000","last_active":"2024-03-17T23:47:01.943993+0000","last_peered":"2024-03-17T23:47:01.943993+0000","last_clean":"2024-03-17T23:47:01.943993+0000","last_became_active":"2024-03-17T23:46:55.916784+0000","last_became_peered":"2024-03-17T23:46:55.916784+0000","last_unstale":"2024-03-17T23:47:01.943993+0000","last_undegraded":"2024-03-17T23:47:01.943993+0000","last_fullsized":"2024-03-17T23:47:01.943993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:49:47.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713939+0000","last_change":"2024-03-17T23:46:55.928539+0000","last_active":"2024-03-17T23:47:02.713939+0000","last_peered":"2024-03-17T23:47:02.713939+0000","last_clean":"2024-03-17T23:47:02.713939+0000","last_became_active":"2024-03-17T23:46:55.928275+0000","last_became_peered":"2024-03-17T23:46:55.928275+0000","last_unstale":"2024-03-17T23:47:02.713939+0000","last_undegraded":"2024-03-17T23:47:02.713939+0000","last_fullsized":"2024-03-17T23:47:02.713939+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T23:52:25.543746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713273+0000","last_change":"2024-03-17T23:46:55.924366+0000","last_active":"2024-03-17T23:47:02.713273+0000","last_peered":"2024-03-17T23:47:02.713273+0000","last_clean":"2024-03-17T23:47:02.713273+0000","last_became_active":"2024-03-17T23:46:55.924100+0000","last_became_peered":"2024-03-17T23:46:55.924100+0000","last_unstale":"2024-03-17T23:47:02.713273+0000","last_undegraded":"2024-03-17T23:47:02.713273+0000","last_fullsized":"2024-03-17T23:47:02.713273+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:51:19.549589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944530+0000","last_change":"2024-03-17T23:46:55.932514+0000","last_active":"2024-03-17T23:47:01.944530+0000","last_peered":"2024-03-17T23:47:01.944530+0000","last_clean":"2024-03-17T23:47:01.944530+0000","last_became_active":"2024-03-17T23:46:55.930899+0000","last_became_peered":"2024-03-17T23:46:55.930899+0000","last_unstale":"2024-03-17T23:47:01.944530+0000","last_undegraded":"2024-03-17T23:47:01.944530+0000","last_fullsized":"2024-03-17T23:47:01.944530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:47:30.023063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943088+0000","last_change":"2024-03-17T23:46:55.940176+0000","last_active":"2024-03-17T23:47:01.943088+0000","last_peered":"2024-03-17T23:47:01.943088+0000","last_clean":"2024-03-17T23:47:01.943088+0000","last_became_active":"2024-03-17T23:46:55.939853+0000","last_became_peered":"2024-03-17T23:46:55.939853+0000","last_unstale":"2024-03-17T23:47:01.943088+0000","last_undegraded":"2024-03-17T23:47:01.943088+0000","last_fullsized":"2024-03-17T23:47:01.943088+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:49:47.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713396+0000","last_change":"2024-03-17T23:46:55.917430+0000","last_active":"2024-03-17T23:47:02.713396+0000","last_peered":"2024-03-17T23:47:02.713396+0000","last_clean":"2024-03-17T23:47:02.713396+0000","last_became_active":"2024-03-17T23:46:55.917173+0000","last_became_peered":"2024-03-17T23:46:55.917173+0000","last_unstale":"2024-03-17T23:47:02.713396+0000","last_undegraded":"2024-03-17T23:47:02.713396+0000","last_fullsized":"2024-03-17T23:47:02.713396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:23:58.221114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714399+0000","last_change":"2024-03-17T23:46:55.927198+0000","last_active":"2024-03-17T23:47:02.714399+0000","last_peered":"2024-03-17T23:47:02.714399+0000","last_clean":"2024-03-17T23:47:02.714399+0000","last_became_active":"2024-03-17T23:46:55.926919+0000","last_became_peered":"2024-03-17T23:46:55.926919+0000","last_unstale":"2024-03-17T23:47:02.714399+0000","last_undegraded":"2024-03-17T23:47:02.714399+0000","last_fullsized":"2024-03-17T23:47:02.714399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:24.419077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943395+0000","last_change":"2024-03-17T23:46:55.932835+0000","last_active":"2024-03-17T23:47:01.943395+0000","last_peered":"2024-03-17T23:47:01.943395+0000","last_clean":"2024-03-17T23:47:01.943395+0000","last_became_active":"2024-03-17T23:46:55.932339+0000","last_became_peered":"2024-03-17T23:46:55.932339+0000","last_unstale":"2024-03-17T23:47:01.943395+0000","last_undegraded":"2024-03-17T23:47:01.943395+0000","last_fullsized":"2024-03-17T23:47:01.943395+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:39.972607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106535+0000","last_change":"2024-03-17T23:46:55.916730+0000","last_active":"2024-03-17T23:47:02.106535+0000","last_peered":"2024-03-17T23:47:02.106535+0000","last_clean":"2024-03-17T23:47:02.106535+0000","last_became_active":"2024-03-17T23:46:55.916438+0000","last_became_peered":"2024-03-17T23:46:55.916438+0000","last_unstale":"2024-03-17T23:47:02.106535+0000","last_undegraded":"2024-03-17T23:47:02.106535+0000","last_fullsized":"2024-03-17T23:47:02.106535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:51:19.549589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944384+0000","last_change":"2024-03-17T23:46:55.932468+0000","last_active":"2024-03-17T23:47:01.944384+0000","last_peered":"2024-03-17T23:47:01.944384+0000","last_clean":"2024-03-17T23:47:01.944384+0000","last_became_active":"2024-03-17T23:46:55.932210+0000","last_became_peered":"2024-03-17T23:46:55.932210+0000","last_unstale":"2024-03-17T23:47:01.944384+0000","last_undegraded":"2024-03-17T23:47:01.944384+0000","last_fullsized":"2024-03-17T23:47:01.944384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:23:58.221114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944784+0000","last_change":"2024-03-17T23:46:55.934093+0000","last_active":"2024-03-17T23:47:01.944784+0000","last_peered":"2024-03-17T23:47:01.944784+0000","last_clean":"2024-03-17T23:47:01.944784+0000","last_became_active":"2024-03-17T23:46:55.933516+0000","last_became_peered":"2024-03-17T23:46:55.933516+0000","last_unstale":"2024-03-17T23:47:01.944784+0000","last_undegraded":"2024-03-17T23:47:01.944784+0000","last_fullsized":"2024-03-17T23:47:01.944784+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:47:30.023063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713902+0000","last_change":"2024-03-17T23:46:55.932426+0000","last_active":"2024-03-17T23:47:02.713902+0000","last_peered":"2024-03-17T23:47:02.713902+0000","last_clean":"2024-03-17T23:47:02.713902+0000","last_became_active":"2024-03-17T23:46:55.932045+0000","last_became_peered":"2024-03-17T23:46:55.932045+0000","last_unstale":"2024-03-17T23:47:02.713902+0000","last_undegraded":"2024-03-17T23:47:02.713902+0000","last_fullsized":"2024-03-17T23:47:02.713902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:20:21.507725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.942978+0000","last_change":"2024-03-17T23:46:55.932245+0000","last_active":"2024-03-17T23:47:01.942978+0000","last_peered":"2024-03-17T23:47:01.942978+0000","last_clean":"2024-03-17T23:47:01.942978+0000","last_became_active":"2024-03-17T23:46:55.932001+0000","last_became_peered":"2024-03-17T23:46:55.932001+0000","last_unstale":"2024-03-17T23:47:01.942978+0000","last_undegraded":"2024-03-17T23:47:01.942978+0000","last_fullsized":"2024-03-17T23:47:01.942978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:44:47.002489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713629+0000","last_change":"2024-03-17T23:46:55.935868+0000","last_active":"2024-03-17T23:47:02.713629+0000","last_peered":"2024-03-17T23:47:02.713629+0000","last_clean":"2024-03-17T23:47:02.713629+0000","last_became_active":"2024-03-17T23:46:55.935180+0000","last_became_peered":"2024-03-17T23:46:55.935180+0000","last_unstale":"2024-03-17T23:47:02.713629+0000","last_undegraded":"2024-03-17T23:47:02.713629+0000","last_fullsized":"2024-03-17T23:47:02.713629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:44:47.002489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943696+0000","last_change":"2024-03-17T23:46:55.936947+0000","last_active":"2024-03-17T23:47:01.943696+0000","last_peered":"2024-03-17T23:47:01.943696+0000","last_clean":"2024-03-17T23:47:01.943696+0000","last_became_active":"2024-03-17T23:46:55.936694+0000","last_became_peered":"2024-03-17T23:46:55.936694+0000","last_unstale":"2024-03-17T23:47:01.943696+0000","last_undegraded":"2024-03-17T23:47:01.943696+0000","last_fullsized":"2024-03-17T23:47:01.943696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:39.972607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713525+0000","last_change":"2024-03-17T23:46:55.927487+0000","last_active":"2024-03-17T23:47:02.713525+0000","last_peered":"2024-03-17T23:47:02.713525+0000","last_clean":"2024-03-17T23:47:02.713525+0000","last_became_active":"2024-03-17T23:46:55.927225+0000","last_became_peered":"2024-03-17T23:46:55.927225+0000","last_unstale":"2024-03-17T23:47:02.713525+0000","last_undegraded":"2024-03-17T23:47:02.713525+0000","last_fullsized":"2024-03-17T23:47:02.713525+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:22:12.841390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943865+0000","last_change":"2024-03-17T23:46:55.933146+0000","last_active":"2024-03-17T23:47:01.943865+0000","last_peered":"2024-03-17T23:47:01.943865+0000","last_clean":"2024-03-17T23:47:01.943865+0000","last_became_active":"2024-03-17T23:46:55.932861+0000","last_became_peered":"2024-03-17T23:46:55.932861+0000","last_unstale":"2024-03-17T23:47:01.943865+0000","last_undegraded":"2024-03-17T23:47:01.943865+0000","last_fullsized":"2024-03-17T23:47:01.943865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:22:12.841390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713455+0000","last_change":"2024-03-17T23:46:55.924118+0000","last_active":"2024-03-17T23:47:02.713455+0000","last_peered":"2024-03-17T23:47:02.713455+0000","last_clean":"2024-03-17T23:47:02.713455+0000","last_became_active":"2024-03-17T23:46:55.923844+0000","last_became_peered":"2024-03-17T23:46:55.923844+0000","last_unstale":"2024-03-17T23:47:02.713455+0000","last_undegraded":"2024-03-17T23:47:02.713455+0000","last_fullsized":"2024-03-17T23:47:02.713455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:30:49.518932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713741+0000","last_change":"2024-03-17T23:46:55.925191+0000","last_active":"2024-03-17T23:47:02.713741+0000","last_peered":"2024-03-17T23:47:02.713741+0000","last_clean":"2024-03-17T23:47:02.713741+0000","last_became_active":"2024-03-17T23:46:55.924822+0000","last_became_peered":"2024-03-17T23:46:55.924822+0000","last_unstale":"2024-03-17T23:47:02.713741+0000","last_undegraded":"2024-03-17T23:47:02.713741+0000","last_fullsized":"2024-03-17T23:47:02.713741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:45:20.327436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713837+0000","last_change":"2024-03-17T23:46:55.928943+0000","last_active":"2024-03-17T23:47:02.713837+0000","last_peered":"2024-03-17T23:47:02.713837+0000","last_clean":"2024-03-17T23:47:02.713837+0000","last_became_active":"2024-03-17T23:46:55.928669+0000","last_became_peered":"2024-03-17T23:46:55.928669+0000","last_unstale":"2024-03-17T23:47:02.713837+0000","last_undegraded":"2024-03-17T23:47:02.713837+0000","last_fullsized":"2024-03-17T23:47:02.713837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:52:45.163567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4499,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":14,"seq":60129542152,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":286720,"data_stored":114925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542152,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":111224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542152,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":880640,"data_stored":702230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":872448,"data_stored":702174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31460,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340380,"statfs":{"total":96636764160,"available":96604549120,"internally_reserved":0,"allocated":286720,"data_stored":114925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":278528,"data_stored":111314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":286720,"data_stored":111350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":111384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-17T23:47:12.935 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-17T23:47:13.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.069+0000 7fbcdd2f1640 1 Processor -- start 2024-03-17T23:47:13.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.069+0000 7fbcdd2f1640 1 -- start start 2024-03-17T23:47:13.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.070+0000 7fbcdd2f1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbcd815e850 0x7fbcd815ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:13.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.070+0000 7fbcdd2f1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcd815b140 con 0x7fbcd815ad60 2024-03-17T23:47:13.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.070+0000 7fbcdd2f1640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcd815f310 con 0x7fbcd80c6310 2024-03-17T23:47:13.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.070+0000 7fbcdd2f1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbcd80b0540 con 0x7fbcd815e850 2024-03-17T23:47:13.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.070+0000 7fbcd6ffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fbcd815ad60 0x7fbcd80ced10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59344/0 (socket says 172.21.15.150:59344) 2024-03-17T23:47:13.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.070+0000 7fbcd6ffd640 1 -- 172.21.15.150:0/1784031343 learned_addr learned my addr 172.21.15.150:0/1784031343 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:13.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.071+0000 7fbcd67fc640 1 --2- 172.21.15.150:0/1784031343 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbcd815e850 0x7fbcd815ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:13.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.071+0000 7fbcd5ffb640 1 -- 172.21.15.150:0/1784031343 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4178534081 0 0) 0x7fbcd815b140 con 0x7fbcd815ad60 2024-03-17T23:47:13.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.071+0000 7fbcd5ffb640 1 -- 172.21.15.150:0/1784031343 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcc0003520 con 0x7fbcd815ad60 2024-03-17T23:47:13.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.071+0000 7fbcd5ffb640 1 -- 172.21.15.150:0/1784031343 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3892297478 0 0) 0x7fbcd815f310 con 0x7fbcd80c6310 2024-03-17T23:47:13.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.071+0000 7fbcd5ffb640 1 -- 172.21.15.150:0/1784031343 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcd815b140 con 0x7fbcd80c6310 2024-03-17T23:47:13.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.071+0000 7fbcd67fc640 1 -- 172.21.15.150:0/1784031343 >> v1:172.21.15.157:6789/0 conn(0x7fbcd80c6310 legacy=0x7fbcd80b0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:13.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.071+0000 7fbcd67fc640 1 -- 172.21.15.150:0/1784031343 >> v1:172.21.15.150:6789/0 conn(0x7fbcd815ad60 legacy=0x7fbcd80ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:13.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.071+0000 7fbcd67fc640 1 -- 172.21.15.150:0/1784031343 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd815f5f0 con 0x7fbcd815e850 2024-03-17T23:47:13.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.072+0000 7fbcd67fc640 1 --2- 172.21.15.150:0/1784031343 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbcd815e850 0x7fbcd815ecd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbccc0097d0 tx=0x7fbccc02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ddee262f046dc61 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:13.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.072+0000 7fbcd5ffb640 1 -- 172.21.15.150:0/1784031343 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbccc035020 con 0x7fbcd815e850 2024-03-17T23:47:13.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.072+0000 7fbcd5ffb640 1 -- 172.21.15.150:0/1784031343 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbccc031890 con 0x7fbcd815e850 2024-03-17T23:47:13.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.072+0000 7fbcd5ffb640 1 -- 172.21.15.150:0/1784031343 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbccc031c40 con 0x7fbcd815e850 2024-03-17T23:47:13.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.072+0000 7fbcdd2f1640 1 -- 172.21.15.150:0/1784031343 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbcd815e850 msgr2=0x7fbcd815ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:13.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.072+0000 7fbcdd2f1640 1 --2- 172.21.15.150:0/1784031343 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbcd815e850 0x7fbcd815ecd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbccc0097d0 tx=0x7fbccc02b930 comp rx=0 tx=0).stop 2024-03-17T23:47:13.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.072+0000 7fbcdd2f1640 1 -- 172.21.15.150:0/1784031343 shutdown_connections 2024-03-17T23:47:13.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.072+0000 7fbcdd2f1640 1 --2- 172.21.15.150:0/1784031343 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbcd815e850 0x7fbcd815ecd0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:13.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.072+0000 7fbcdd2f1640 1 -- 172.21.15.150:0/1784031343 >> 172.21.15.150:0/1784031343 conn(0x7fbcd8099a70 msgr2=0x7fbcd8099e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:13.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.072+0000 7fbcdd2f1640 1 -- 172.21.15.150:0/1784031343 shutdown_connections 2024-03-17T23:47:13.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.072+0000 7fbcdd2f1640 1 -- 172.21.15.150:0/1784031343 wait complete. 2024-03-17T23:47:13.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.074+0000 7fbcdd2f1640 1 Processor -- start 2024-03-17T23:47:13.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.074+0000 7fbcdd2f1640 1 -- start start 2024-03-17T23:47:13.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.074+0000 7fbcdd2f1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcd80c6310 0x7fbcd81f87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:13.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.074+0000 7fbcdd2f1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbcd815ad60 0x7fbcd8158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:13.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.074+0000 7fbcdd2f1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbcd815e850 0x7fbcd8158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:13.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.074+0000 7fbcdd2f1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd80c5b40 con 0x7fbcd80c6310 2024-03-17T23:47:13.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.074+0000 7fbcdd2f1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd80b0a70 con 0x7fbcd815e850 2024-03-17T23:47:13.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.074+0000 7fbcdd2f1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbcd80b16a0 con 0x7fbcd815ad60 2024-03-17T23:47:13.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.074+0000 7fbcd6ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcd80c6310 0x7fbcd81f87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:13.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.074+0000 7fbcd6ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcd80c6310 0x7fbcd81f87a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36280/0 (socket says 172.21.15.150:36280) 2024-03-17T23:47:13.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.074+0000 7fbcd6ffd640 1 -- 172.21.15.150:0/381177524 learned_addr learned my addr 172.21.15.150:0/381177524 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:13.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.074+0000 7fbcd67fc640 1 --2- 172.21.15.150:0/381177524 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbcd815ad60 0x7fbcd8158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:13.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.075+0000 7fbcd77fe640 1 --2- 172.21.15.150:0/381177524 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbcd815e850 0x7fbcd8158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:13.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.075+0000 7fbcd6ffd640 1 -- 172.21.15.150:0/381177524 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbcd815ad60 msgr2=0x7fbcd8158350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:13.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.075+0000 7fbcd6ffd640 1 --2- 172.21.15.150:0/381177524 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbcd815ad60 0x7fbcd8158350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:13.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.075+0000 7fbcd6ffd640 1 -- 172.21.15.150:0/381177524 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbcd815e850 msgr2=0x7fbcd8158e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:13.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.075+0000 7fbcd6ffd640 1 --2- 172.21.15.150:0/381177524 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbcd815e850 0x7fbcd8158e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:13.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.075+0000 7fbcd6ffd640 1 -- 172.21.15.150:0/381177524 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd80cd200 con 0x7fbcd80c6310 2024-03-17T23:47:13.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.075+0000 7fbcd6ffd640 1 --2- 172.21.15.150:0/381177524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcd80c6310 0x7fbcd81f87a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbcc800cbc0 tx=0x7fbcc800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:13.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.075+0000 7fbcbf7fe640 1 -- 172.21.15.150:0/381177524 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc8017070 con 0x7fbcd80c6310 2024-03-17T23:47:13.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.075+0000 7fbcbf7fe640 1 -- 172.21.15.150:0/381177524 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcc800ad40 con 0x7fbcd80c6310 2024-03-17T23:47:13.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.075+0000 7fbcdd2f1640 1 -- 172.21.15.150:0/381177524 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcd80ccf20 con 0x7fbcd80c6310 2024-03-17T23:47:13.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.075+0000 7fbcbf7fe640 1 -- 172.21.15.150:0/381177524 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc8012600 con 0x7fbcd80c6310 2024-03-17T23:47:13.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.075+0000 7fbcdd2f1640 1 -- 172.21.15.150:0/381177524 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcd8159d20 con 0x7fbcd80c6310 2024-03-17T23:47:13.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.076+0000 7fbcdd2f1640 1 -- 172.21.15.150:0/381177524 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcd80688b0 con 0x7fbcd80c6310 2024-03-17T23:47:13.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.077+0000 7fbcbf7fe640 1 -- 172.21.15.150:0/381177524 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbcc801a070 con 0x7fbcd80c6310 2024-03-17T23:47:13.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.077+0000 7fbcbf7fe640 1 --2- 172.21.15.150:0/381177524 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbcc406bbd0 0x7fbcc406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:13.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.077+0000 7fbcbf7fe640 1 -- 172.21.15.150:0/381177524 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fbcc808ab00 con 0x7fbcd80c6310 2024-03-17T23:47:13.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.077+0000 7fbcd67fc640 1 --2- 172.21.15.150:0/381177524 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbcc406bbd0 0x7fbcc406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:13.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.078+0000 7fbcd67fc640 1 --2- 172.21.15.150:0/381177524 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbcc406bbd0 0x7fbcc406e090 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbccc036000 tx=0x7fbccc03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:13.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.081+0000 7fbcbf7fe640 1 -- 172.21.15.150:0/381177524 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbcc8058c60 con 0x7fbcd80c6310 2024-03-17T23:47:13.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.253+0000 7fbcdd2f1640 1 -- 172.21.15.150:0/381177524 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbcd815a090 con 0x7fbcc406bbd0 2024-03-17T23:47:13.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.260+0000 7fbcbf7fe640 1 -- 172.21.15.150:0/381177524 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7fbccc099310 con 0x7fbcc406bbd0 2024-03-17T23:47:13.264 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:13.266 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-03-17T23:47:13.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.264+0000 7fbcdd2f1640 1 -- 172.21.15.150:0/381177524 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbcc406bbd0 msgr2=0x7fbcc406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:13.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.264+0000 7fbcdd2f1640 1 --2- 172.21.15.150:0/381177524 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbcc406bbd0 0x7fbcc406e090 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbccc036000 tx=0x7fbccc03c040 comp rx=0 tx=0).stop 2024-03-17T23:47:13.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.264+0000 7fbcdd2f1640 1 -- 172.21.15.150:0/381177524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcd80c6310 msgr2=0x7fbcd81f87a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:13.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.264+0000 7fbcdd2f1640 1 --2- 172.21.15.150:0/381177524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcd80c6310 0x7fbcd81f87a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbcc800cbc0 tx=0x7fbcc800a430 comp rx=0 tx=0).stop 2024-03-17T23:47:13.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.264+0000 7fbcdd2f1640 1 -- 172.21.15.150:0/381177524 shutdown_connections 2024-03-17T23:47:13.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.264+0000 7fbcdd2f1640 1 --2- 172.21.15.150:0/381177524 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbcc406bbd0 0x7fbcc406e090 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:13.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.264+0000 7fbcdd2f1640 1 --2- 172.21.15.150:0/381177524 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbcd815e850 0x7fbcd8158e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:13.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.264+0000 7fbcdd2f1640 1 --2- 172.21.15.150:0/381177524 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbcd815ad60 0x7fbcd8158350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:13.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.264+0000 7fbcdd2f1640 1 --2- 172.21.15.150:0/381177524 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcd80c6310 0x7fbcd81f87a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:13.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.264+0000 7fbcdd2f1640 1 -- 172.21.15.150:0/381177524 >> 172.21.15.150:0/381177524 conn(0x7fbcd8099a70 msgr2=0x7fbcd80cd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:13.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.264+0000 7fbcdd2f1640 1 -- 172.21.15.150:0/381177524 shutdown_connections 2024-03-17T23:47:13.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.264+0000 7fbcdd2f1640 1 -- 172.21.15.150:0/381177524 wait complete. 2024-03-17T23:47:13.275 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-17T23:47:11.599293+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594867,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":206,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":232008,"kb_used_data":3368,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754742712,"statfs":{"total":773094113280,"available":772856537088,"internally_reserved":0,"allocated":3448832,"data_stored":2079526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2089,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002137"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713953+0000","last_change":"2024-03-17T23:46:54.923706+0000","last_active":"2024-03-17T23:47:02.713953+0000","last_peered":"2024-03-17T23:47:02.713953+0000","last_clean":"2024-03-17T23:47:02.713953+0000","last_became_active":"2024-03-17T23:46:54.923462+0000","last_became_peered":"2024-03-17T23:46:54.923462+0000","last_unstale":"2024-03-17T23:47:02.713953+0000","last_undegraded":"2024-03-17T23:47:02.713953+0000","last_fullsized":"2024-03-17T23:47:02.713953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:34:48.054365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943580+0000","last_change":"2024-03-17T23:46:55.932962+0000","last_active":"2024-03-17T23:47:01.943580+0000","last_peered":"2024-03-17T23:47:01.943580+0000","last_clean":"2024-03-17T23:47:01.943580+0000","last_became_active":"2024-03-17T23:46:55.932316+0000","last_became_peered":"2024-03-17T23:46:55.932316+0000","last_unstale":"2024-03-17T23:47:01.943580+0000","last_undegraded":"2024-03-17T23:47:01.943580+0000","last_fullsized":"2024-03-17T23:47:01.943580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:45:20.327436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943046+0000","last_change":"2024-03-17T23:47:00.412319+0000","last_active":"2024-03-17T23:47:01.943046+0000","last_peered":"2024-03-17T23:47:01.943046+0000","last_clean":"2024-03-17T23:47:01.943046+0000","last_became_active":"2024-03-17T23:47:00.411544+0000","last_became_peered":"2024-03-17T23:47:00.411544+0000","last_unstale":"2024-03-17T23:47:01.943046+0000","last_undegraded":"2024-03-17T23:47:01.943046+0000","last_fullsized":"2024-03-17T23:47:01.943046+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:24:18.544213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943689+0000","last_change":"2024-03-17T23:46:54.921077+0000","last_active":"2024-03-17T23:47:01.943689+0000","last_peered":"2024-03-17T23:47:01.943689+0000","last_clean":"2024-03-17T23:47:01.943689+0000","last_became_active":"2024-03-17T23:46:54.920797+0000","last_became_peered":"2024-03-17T23:46:54.920797+0000","last_unstale":"2024-03-17T23:47:01.943689+0000","last_undegraded":"2024-03-17T23:47:01.943689+0000","last_fullsized":"2024-03-17T23:47:01.943689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:18.195935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713708+0000","last_change":"2024-03-17T23:46:55.943217+0000","last_active":"2024-03-17T23:47:02.713708+0000","last_peered":"2024-03-17T23:47:02.713708+0000","last_clean":"2024-03-17T23:47:02.713708+0000","last_became_active":"2024-03-17T23:46:55.942641+0000","last_became_peered":"2024-03-17T23:46:55.942641+0000","last_unstale":"2024-03-17T23:47:02.713708+0000","last_undegraded":"2024-03-17T23:47:02.713708+0000","last_fullsized":"2024-03-17T23:47:02.713708+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:45:20.327436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106123+0000","last_change":"2024-03-17T23:46:59.957747+0000","last_active":"2024-03-17T23:47:02.106123+0000","last_peered":"2024-03-17T23:47:02.106123+0000","last_clean":"2024-03-17T23:47:02.106123+0000","last_became_active":"2024-03-17T23:46:59.957329+0000","last_became_peered":"2024-03-17T23:46:59.957329+0000","last_unstale":"2024-03-17T23:47:02.106123+0000","last_undegraded":"2024-03-17T23:47:02.106123+0000","last_fullsized":"2024-03-17T23:47:02.106123+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:27:51.964285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713914+0000","last_change":"2024-03-17T23:46:54.926580+0000","last_active":"2024-03-17T23:47:02.713914+0000","last_peered":"2024-03-17T23:47:02.713914+0000","last_clean":"2024-03-17T23:47:02.713914+0000","last_became_active":"2024-03-17T23:46:54.926254+0000","last_became_peered":"2024-03-17T23:46:54.926254+0000","last_unstale":"2024-03-17T23:47:02.713914+0000","last_undegraded":"2024-03-17T23:47:02.713914+0000","last_fullsized":"2024-03-17T23:47:02.713914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:11:01.353944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943965+0000","last_change":"2024-03-17T23:46:55.937073+0000","last_active":"2024-03-17T23:47:01.943965+0000","last_peered":"2024-03-17T23:47:01.943965+0000","last_clean":"2024-03-17T23:47:01.943965+0000","last_became_active":"2024-03-17T23:46:55.936068+0000","last_became_peered":"2024-03-17T23:46:55.936068+0000","last_unstale":"2024-03-17T23:47:01.943965+0000","last_undegraded":"2024-03-17T23:47:01.943965+0000","last_fullsized":"2024-03-17T23:47:01.943965+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:45:29.447600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106472+0000","last_change":"2024-03-17T23:46:59.957945+0000","last_active":"2024-03-17T23:47:02.106472+0000","last_peered":"2024-03-17T23:47:02.106472+0000","last_clean":"2024-03-17T23:47:02.106472+0000","last_became_active":"2024-03-17T23:46:59.957324+0000","last_became_peered":"2024-03-17T23:46:59.957324+0000","last_unstale":"2024-03-17T23:47:02.106472+0000","last_undegraded":"2024-03-17T23:47:02.106472+0000","last_fullsized":"2024-03-17T23:47:02.106472+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:52:17.152097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713691+0000","last_change":"2024-03-17T23:46:54.924488+0000","last_active":"2024-03-17T23:47:02.713691+0000","last_peered":"2024-03-17T23:47:02.713691+0000","last_clean":"2024-03-17T23:47:02.713691+0000","last_became_active":"2024-03-17T23:46:54.924170+0000","last_became_peered":"2024-03-17T23:46:54.924170+0000","last_unstale":"2024-03-17T23:47:02.713691+0000","last_undegraded":"2024-03-17T23:47:02.713691+0000","last_fullsized":"2024-03-17T23:47:02.713691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:06:54.433520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944190+0000","last_change":"2024-03-17T23:46:55.938268+0000","last_active":"2024-03-17T23:47:01.944190+0000","last_peered":"2024-03-17T23:47:01.944190+0000","last_clean":"2024-03-17T23:47:01.944190+0000","last_became_active":"2024-03-17T23:46:55.937957+0000","last_became_peered":"2024-03-17T23:46:55.937957+0000","last_unstale":"2024-03-17T23:47:01.944190+0000","last_undegraded":"2024-03-17T23:47:01.944190+0000","last_fullsized":"2024-03-17T23:47:01.944190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T23:52:25.543746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106183+0000","last_change":"2024-03-17T23:46:59.959993+0000","last_active":"2024-03-17T23:47:02.106183+0000","last_peered":"2024-03-17T23:47:02.106183+0000","last_clean":"2024-03-17T23:47:02.106183+0000","last_became_active":"2024-03-17T23:46:59.959636+0000","last_became_peered":"2024-03-17T23:46:59.959636+0000","last_unstale":"2024-03-17T23:47:02.106183+0000","last_undegraded":"2024-03-17T23:47:02.106183+0000","last_fullsized":"2024-03-17T23:47:02.106183+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:39:36.583214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943413+0000","last_change":"2024-03-17T23:46:55.917068+0000","last_active":"2024-03-17T23:47:01.943413+0000","last_peered":"2024-03-17T23:47:01.943413+0000","last_clean":"2024-03-17T23:47:01.943413+0000","last_became_active":"2024-03-17T23:46:55.916759+0000","last_became_peered":"2024-03-17T23:46:55.916759+0000","last_unstale":"2024-03-17T23:47:01.943413+0000","last_undegraded":"2024-03-17T23:47:01.943413+0000","last_fullsized":"2024-03-17T23:47:01.943413+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:45:20.327436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713785+0000","last_change":"2024-03-17T23:46:54.925485+0000","last_active":"2024-03-17T23:47:02.713785+0000","last_peered":"2024-03-17T23:47:02.713785+0000","last_clean":"2024-03-17T23:47:02.713785+0000","last_became_active":"2024-03-17T23:46:54.925173+0000","last_became_peered":"2024-03-17T23:46:54.925173+0000","last_unstale":"2024-03-17T23:47:02.713785+0000","last_undegraded":"2024-03-17T23:47:02.713785+0000","last_fullsized":"2024-03-17T23:47:02.713785+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:51:18.545842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106704+0000","last_change":"2024-03-17T23:46:59.961930+0000","last_active":"2024-03-17T23:47:02.106704+0000","last_peered":"2024-03-17T23:47:02.106704+0000","last_clean":"2024-03-17T23:47:02.106704+0000","last_became_active":"2024-03-17T23:46:59.961495+0000","last_became_peered":"2024-03-17T23:46:59.961495+0000","last_unstale":"2024-03-17T23:47:02.106704+0000","last_undegraded":"2024-03-17T23:47:02.106704+0000","last_fullsized":"2024-03-17T23:47:02.106704+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:59:20.183441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943415+0000","last_change":"2024-03-17T23:46:55.927904+0000","last_active":"2024-03-17T23:47:01.943415+0000","last_peered":"2024-03-17T23:47:01.943415+0000","last_clean":"2024-03-17T23:47:01.943415+0000","last_became_active":"2024-03-17T23:46:55.927637+0000","last_became_peered":"2024-03-17T23:46:55.927637+0000","last_unstale":"2024-03-17T23:47:01.943415+0000","last_undegraded":"2024-03-17T23:47:01.943415+0000","last_fullsized":"2024-03-17T23:47:01.943415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:03.353861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714207+0000","last_change":"2024-03-17T23:46:54.920822+0000","last_active":"2024-03-17T23:47:02.714207+0000","last_peered":"2024-03-17T23:47:02.714207+0000","last_clean":"2024-03-17T23:47:02.714207+0000","last_became_active":"2024-03-17T23:46:54.920534+0000","last_became_peered":"2024-03-17T23:46:54.920534+0000","last_unstale":"2024-03-17T23:47:02.714207+0000","last_undegraded":"2024-03-17T23:47:02.714207+0000","last_fullsized":"2024-03-17T23:47:02.714207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:56.665566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106433+0000","last_change":"2024-03-17T23:46:59.961245+0000","last_active":"2024-03-17T23:47:02.106433+0000","last_peered":"2024-03-17T23:47:02.106433+0000","last_clean":"2024-03-17T23:47:02.106433+0000","last_became_active":"2024-03-17T23:46:59.960895+0000","last_became_peered":"2024-03-17T23:46:59.960895+0000","last_unstale":"2024-03-17T23:47:02.106433+0000","last_undegraded":"2024-03-17T23:47:02.106433+0000","last_fullsized":"2024-03-17T23:47:02.106433+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:57.296991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943288+0000","last_change":"2024-03-17T23:46:55.924013+0000","last_active":"2024-03-17T23:47:01.943288+0000","last_peered":"2024-03-17T23:47:01.943288+0000","last_clean":"2024-03-17T23:47:01.943288+0000","last_became_active":"2024-03-17T23:46:55.923741+0000","last_became_peered":"2024-03-17T23:46:55.923741+0000","last_unstale":"2024-03-17T23:47:01.943288+0000","last_undegraded":"2024-03-17T23:47:01.943288+0000","last_fullsized":"2024-03-17T23:47:01.943288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:00.571611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713487+0000","last_change":"2024-03-17T23:46:54.918712+0000","last_active":"2024-03-17T23:47:02.713487+0000","last_peered":"2024-03-17T23:47:02.713487+0000","last_clean":"2024-03-17T23:47:02.713487+0000","last_became_active":"2024-03-17T23:46:54.918470+0000","last_became_peered":"2024-03-17T23:46:54.918470+0000","last_unstale":"2024-03-17T23:47:02.713487+0000","last_undegraded":"2024-03-17T23:47:02.713487+0000","last_fullsized":"2024-03-17T23:47:02.713487+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:18.195935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.715105+0000","last_change":"2024-03-17T23:47:00.410049+0000","last_active":"2024-03-17T23:47:02.715105+0000","last_peered":"2024-03-17T23:47:02.715105+0000","last_clean":"2024-03-17T23:47:02.715105+0000","last_became_active":"2024-03-17T23:47:00.409688+0000","last_became_peered":"2024-03-17T23:47:00.409688+0000","last_unstale":"2024-03-17T23:47:02.715105+0000","last_undegraded":"2024-03-17T23:47:02.715105+0000","last_fullsized":"2024-03-17T23:47:02.715105+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:39:36.583214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106313+0000","last_change":"2024-03-17T23:46:55.925699+0000","last_active":"2024-03-17T23:47:02.106313+0000","last_peered":"2024-03-17T23:47:02.106313+0000","last_clean":"2024-03-17T23:47:02.106313+0000","last_became_active":"2024-03-17T23:46:55.925467+0000","last_became_peered":"2024-03-17T23:46:55.925467+0000","last_unstale":"2024-03-17T23:47:02.106313+0000","last_undegraded":"2024-03-17T23:47:02.106313+0000","last_fullsized":"2024-03-17T23:47:02.106313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:11:02.357691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713372+0000","last_change":"2024-03-17T23:46:54.923992+0000","last_active":"2024-03-17T23:47:02.713372+0000","last_peered":"2024-03-17T23:47:02.713372+0000","last_clean":"2024-03-17T23:47:02.713372+0000","last_became_active":"2024-03-17T23:46:54.923721+0000","last_became_peered":"2024-03-17T23:46:54.923721+0000","last_unstale":"2024-03-17T23:47:02.713372+0000","last_undegraded":"2024-03-17T23:47:02.713372+0000","last_fullsized":"2024-03-17T23:47:02.713372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:46:16.782215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106661+0000","last_change":"2024-03-17T23:46:59.960940+0000","last_active":"2024-03-17T23:47:02.106661+0000","last_peered":"2024-03-17T23:47:02.106661+0000","last_clean":"2024-03-17T23:47:02.106661+0000","last_became_active":"2024-03-17T23:46:59.960472+0000","last_became_peered":"2024-03-17T23:46:59.960472+0000","last_unstale":"2024-03-17T23:47:02.106661+0000","last_undegraded":"2024-03-17T23:47:02.106661+0000","last_fullsized":"2024-03-17T23:47:02.106661+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:21:41.104850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714557+0000","last_change":"2024-03-17T23:46:55.927164+0000","last_active":"2024-03-17T23:47:02.714557+0000","last_peered":"2024-03-17T23:47:02.714557+0000","last_clean":"2024-03-17T23:47:02.714557+0000","last_became_active":"2024-03-17T23:46:55.926883+0000","last_became_peered":"2024-03-17T23:46:55.926883+0000","last_unstale":"2024-03-17T23:47:02.714557+0000","last_undegraded":"2024-03-17T23:47:02.714557+0000","last_fullsized":"2024-03-17T23:47:02.714557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:03.353861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944736+0000","last_change":"2024-03-17T23:46:54.921997+0000","last_active":"2024-03-17T23:47:01.944736+0000","last_peered":"2024-03-17T23:47:01.944736+0000","last_clean":"2024-03-17T23:47:01.944736+0000","last_became_active":"2024-03-17T23:46:54.921701+0000","last_became_peered":"2024-03-17T23:46:54.921701+0000","last_unstale":"2024-03-17T23:47:01.944736+0000","last_undegraded":"2024-03-17T23:47:01.944736+0000","last_fullsized":"2024-03-17T23:47:01.944736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:11:01.353944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106094+0000","last_change":"2024-03-17T23:47:00.410238+0000","last_active":"2024-03-17T23:47:02.106094+0000","last_peered":"2024-03-17T23:47:02.106094+0000","last_clean":"2024-03-17T23:47:02.106094+0000","last_became_active":"2024-03-17T23:47:00.409845+0000","last_became_peered":"2024-03-17T23:47:00.409845+0000","last_unstale":"2024-03-17T23:47:02.106094+0000","last_undegraded":"2024-03-17T23:47:02.106094+0000","last_fullsized":"2024-03-17T23:47:02.106094+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:43:57.153828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943841+0000","last_change":"2024-03-17T23:46:55.929991+0000","last_active":"2024-03-17T23:47:01.943841+0000","last_peered":"2024-03-17T23:47:01.943841+0000","last_clean":"2024-03-17T23:47:01.943841+0000","last_became_active":"2024-03-17T23:46:55.929722+0000","last_became_peered":"2024-03-17T23:46:55.929722+0000","last_unstale":"2024-03-17T23:47:01.943841+0000","last_undegraded":"2024-03-17T23:47:01.943841+0000","last_fullsized":"2024-03-17T23:47:01.943841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:11:08.829506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713963+0000","last_change":"2024-03-17T23:46:54.913493+0000","last_active":"2024-03-17T23:47:02.713963+0000","last_peered":"2024-03-17T23:47:02.713963+0000","last_clean":"2024-03-17T23:47:02.713963+0000","last_became_active":"2024-03-17T23:46:54.913233+0000","last_became_peered":"2024-03-17T23:46:54.913233+0000","last_unstale":"2024-03-17T23:47:02.713963+0000","last_undegraded":"2024-03-17T23:47:02.713963+0000","last_fullsized":"2024-03-17T23:47:02.713963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:58.417421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944060+0000","last_change":"2024-03-17T23:47:00.408616+0000","last_active":"2024-03-17T23:47:01.944060+0000","last_peered":"2024-03-17T23:47:01.944060+0000","last_clean":"2024-03-17T23:47:01.944060+0000","last_became_active":"2024-03-17T23:47:00.408251+0000","last_became_peered":"2024-03-17T23:47:00.408251+0000","last_unstale":"2024-03-17T23:47:01.944060+0000","last_undegraded":"2024-03-17T23:47:01.944060+0000","last_fullsized":"2024-03-17T23:47:01.944060+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:16:52.426485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714810+0000","last_change":"2024-03-17T23:46:55.921416+0000","last_active":"2024-03-17T23:47:02.714810+0000","last_peered":"2024-03-17T23:47:02.714810+0000","last_clean":"2024-03-17T23:47:02.714810+0000","last_became_active":"2024-03-17T23:46:55.921128+0000","last_became_peered":"2024-03-17T23:46:55.921128+0000","last_unstale":"2024-03-17T23:47:02.714810+0000","last_undegraded":"2024-03-17T23:47:02.714810+0000","last_fullsized":"2024-03-17T23:47:02.714810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:49:47.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714025+0000","last_change":"2024-03-17T23:46:54.926748+0000","last_active":"2024-03-17T23:47:02.714025+0000","last_peered":"2024-03-17T23:47:02.714025+0000","last_clean":"2024-03-17T23:47:02.714025+0000","last_became_active":"2024-03-17T23:46:54.926467+0000","last_became_peered":"2024-03-17T23:46:54.926467+0000","last_unstale":"2024-03-17T23:47:02.714025+0000","last_undegraded":"2024-03-17T23:47:02.714025+0000","last_fullsized":"2024-03-17T23:47:02.714025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:58.417421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714085+0000","last_change":"2024-03-17T23:47:00.409306+0000","last_active":"2024-03-17T23:47:02.714085+0000","last_peered":"2024-03-17T23:47:02.714085+0000","last_clean":"2024-03-17T23:47:02.714085+0000","last_became_active":"2024-03-17T23:47:00.408934+0000","last_became_peered":"2024-03-17T23:47:00.408934+0000","last_unstale":"2024-03-17T23:47:02.714085+0000","last_undegraded":"2024-03-17T23:47:02.714085+0000","last_fullsized":"2024-03-17T23:47:02.714085+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:35:49.838832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944302+0000","last_change":"2024-03-17T23:46:55.933418+0000","last_active":"2024-03-17T23:47:01.944302+0000","last_peered":"2024-03-17T23:47:01.944302+0000","last_clean":"2024-03-17T23:47:01.944302+0000","last_became_active":"2024-03-17T23:46:55.931133+0000","last_became_peered":"2024-03-17T23:46:55.931133+0000","last_unstale":"2024-03-17T23:47:01.944302+0000","last_undegraded":"2024-03-17T23:47:01.944302+0000","last_fullsized":"2024-03-17T23:47:01.944302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:32:27.349125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714826+0000","last_change":"2024-03-17T23:46:54.914746+0000","last_active":"2024-03-17T23:47:02.714826+0000","last_peered":"2024-03-17T23:47:02.714826+0000","last_clean":"2024-03-17T23:47:02.714826+0000","last_became_active":"2024-03-17T23:46:54.914482+0000","last_became_peered":"2024-03-17T23:46:54.914482+0000","last_unstale":"2024-03-17T23:47:02.714826+0000","last_undegraded":"2024-03-17T23:47:02.714826+0000","last_fullsized":"2024-03-17T23:47:02.714826+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:42:33.325150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713844+0000","last_change":"2024-03-17T23:46:59.961836+0000","last_active":"2024-03-17T23:47:02.713844+0000","last_peered":"2024-03-17T23:47:02.713844+0000","last_clean":"2024-03-17T23:47:02.713844+0000","last_became_active":"2024-03-17T23:46:59.961458+0000","last_became_peered":"2024-03-17T23:46:59.961458+0000","last_unstale":"2024-03-17T23:47:02.713844+0000","last_undegraded":"2024-03-17T23:47:02.713844+0000","last_fullsized":"2024-03-17T23:47:02.713844+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:43:57.153828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106542+0000","last_change":"2024-03-17T23:46:55.926629+0000","last_active":"2024-03-17T23:47:02.106542+0000","last_peered":"2024-03-17T23:47:02.106542+0000","last_clean":"2024-03-17T23:47:02.106542+0000","last_became_active":"2024-03-17T23:46:55.926391+0000","last_became_peered":"2024-03-17T23:46:55.926391+0000","last_unstale":"2024-03-17T23:47:02.106542+0000","last_undegraded":"2024-03-17T23:47:02.106542+0000","last_fullsized":"2024-03-17T23:47:02.106542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:59.421169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943876+0000","last_change":"2024-03-17T23:46:54.921058+0000","last_active":"2024-03-17T23:47:01.943876+0000","last_peered":"2024-03-17T23:47:01.943876+0000","last_clean":"2024-03-17T23:47:01.943876+0000","last_became_active":"2024-03-17T23:46:54.919926+0000","last_became_peered":"2024-03-17T23:46:54.919926+0000","last_unstale":"2024-03-17T23:47:01.943876+0000","last_undegraded":"2024-03-17T23:47:01.943876+0000","last_fullsized":"2024-03-17T23:47:01.943876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:11:01.353944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714725+0000","last_change":"2024-03-17T23:46:59.957516+0000","last_active":"2024-03-17T23:47:02.714725+0000","last_peered":"2024-03-17T23:47:02.714725+0000","last_clean":"2024-03-17T23:47:02.714725+0000","last_became_active":"2024-03-17T23:46:59.957168+0000","last_became_peered":"2024-03-17T23:46:59.957168+0000","last_unstale":"2024-03-17T23:47:02.714725+0000","last_undegraded":"2024-03-17T23:47:02.714725+0000","last_fullsized":"2024-03-17T23:47:02.714725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:35:49.838832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713602+0000","last_change":"2024-03-17T23:46:55.921369+0000","last_active":"2024-03-17T23:47:02.713602+0000","last_peered":"2024-03-17T23:47:02.713602+0000","last_clean":"2024-03-17T23:47:02.713602+0000","last_became_active":"2024-03-17T23:46:55.921109+0000","last_became_peered":"2024-03-17T23:46:55.921109+0000","last_unstale":"2024-03-17T23:47:02.713602+0000","last_undegraded":"2024-03-17T23:47:02.713602+0000","last_fullsized":"2024-03-17T23:47:02.713602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:24.419077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713228+0000","last_change":"2024-03-17T23:46:54.918722+0000","last_active":"2024-03-17T23:47:02.713228+0000","last_peered":"2024-03-17T23:47:02.713228+0000","last_clean":"2024-03-17T23:47:02.713228+0000","last_became_active":"2024-03-17T23:46:54.918440+0000","last_became_peered":"2024-03-17T23:46:54.918440+0000","last_unstale":"2024-03-17T23:47:02.713228+0000","last_undegraded":"2024-03-17T23:47:02.713228+0000","last_fullsized":"2024-03-17T23:47:02.713228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:56.665566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.715106+0000","last_change":"2024-03-17T23:46:59.962185+0000","last_active":"2024-03-17T23:47:02.715106+0000","last_peered":"2024-03-17T23:47:02.715106+0000","last_clean":"2024-03-17T23:47:02.715106+0000","last_became_active":"2024-03-17T23:46:59.961682+0000","last_became_peered":"2024-03-17T23:46:59.961682+0000","last_unstale":"2024-03-17T23:47:02.715106+0000","last_undegraded":"2024-03-17T23:47:02.715106+0000","last_fullsized":"2024-03-17T23:47:02.715106+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:47:52.502020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713596+0000","last_change":"2024-03-17T23:46:55.924464+0000","last_active":"2024-03-17T23:47:02.713596+0000","last_peered":"2024-03-17T23:47:02.713596+0000","last_clean":"2024-03-17T23:47:02.713596+0000","last_became_active":"2024-03-17T23:46:55.924193+0000","last_became_peered":"2024-03-17T23:46:55.924193+0000","last_unstale":"2024-03-17T23:47:02.713596+0000","last_undegraded":"2024-03-17T23:47:02.713596+0000","last_fullsized":"2024-03-17T23:47:02.713596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:54.348184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713381+0000","last_change":"2024-03-17T23:46:54.921188+0000","last_active":"2024-03-17T23:47:02.713381+0000","last_peered":"2024-03-17T23:47:02.713381+0000","last_clean":"2024-03-17T23:47:02.713381+0000","last_became_active":"2024-03-17T23:46:54.920926+0000","last_became_peered":"2024-03-17T23:46:54.920926+0000","last_unstale":"2024-03-17T23:47:02.713381+0000","last_undegraded":"2024-03-17T23:47:02.713381+0000","last_fullsized":"2024-03-17T23:47:02.713381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:01:23.150557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714399+0000","last_change":"2024-03-17T23:47:00.410855+0000","last_active":"2024-03-17T23:47:02.714399+0000","last_peered":"2024-03-17T23:47:02.714399+0000","last_clean":"2024-03-17T23:47:02.714399+0000","last_became_active":"2024-03-17T23:47:00.410485+0000","last_became_peered":"2024-03-17T23:47:00.410485+0000","last_unstale":"2024-03-17T23:47:02.714399+0000","last_undegraded":"2024-03-17T23:47:02.714399+0000","last_fullsized":"2024-03-17T23:47:02.714399+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:21:41.104850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943282+0000","last_change":"2024-03-17T23:46:55.931906+0000","last_active":"2024-03-17T23:47:01.943282+0000","last_peered":"2024-03-17T23:47:01.943282+0000","last_clean":"2024-03-17T23:47:01.943282+0000","last_became_active":"2024-03-17T23:46:55.931643+0000","last_became_peered":"2024-03-17T23:46:55.931643+0000","last_unstale":"2024-03-17T23:47:01.943282+0000","last_undegraded":"2024-03-17T23:47:01.943282+0000","last_fullsized":"2024-03-17T23:47:01.943282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:24.419077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943618+0000","last_change":"2024-03-17T23:46:54.920477+0000","last_active":"2024-03-17T23:47:01.943618+0000","last_peered":"2024-03-17T23:47:01.943618+0000","last_clean":"2024-03-17T23:47:01.943618+0000","last_became_active":"2024-03-17T23:46:54.920116+0000","last_became_peered":"2024-03-17T23:46:54.920116+0000","last_unstale":"2024-03-17T23:47:01.943618+0000","last_undegraded":"2024-03-17T23:47:01.943618+0000","last_fullsized":"2024-03-17T23:47:01.943618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:56.665566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":180,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713494+0000","last_change":"2024-03-17T23:46:44.768668+0000","last_active":"2024-03-17T23:47:02.713494+0000","last_peered":"2024-03-17T23:47:02.713494+0000","last_clean":"2024-03-17T23:47:02.713494+0000","last_became_active":"2024-03-17T23:46:44.767826+0000","last_became_peered":"2024-03-17T23:46:44.767826+0000","last_unstale":"2024-03-17T23:47:02.713494+0000","last_undegraded":"2024-03-17T23:47:02.713494+0000","last_fullsized":"2024-03-17T23:47:02.713494+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_clean_scrub_stamp":"2024-03-17T23:46:43.744878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:54:22.286155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944382+0000","last_change":"2024-03-17T23:47:00.410345+0000","last_active":"2024-03-17T23:47:01.944382+0000","last_peered":"2024-03-17T23:47:01.944382+0000","last_clean":"2024-03-17T23:47:01.944382+0000","last_became_active":"2024-03-17T23:47:00.409948+0000","last_became_peered":"2024-03-17T23:47:00.409948+0000","last_unstale":"2024-03-17T23:47:01.944382+0000","last_undegraded":"2024-03-17T23:47:01.944382+0000","last_fullsized":"2024-03-17T23:47:01.944382+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:20:25.538662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713826+0000","last_change":"2024-03-17T23:46:55.943072+0000","last_active":"2024-03-17T23:47:02.713826+0000","last_peered":"2024-03-17T23:47:02.713826+0000","last_clean":"2024-03-17T23:47:02.713826+0000","last_became_active":"2024-03-17T23:46:55.942516+0000","last_became_peered":"2024-03-17T23:46:55.942516+0000","last_unstale":"2024-03-17T23:47:02.713826+0000","last_undegraded":"2024-03-17T23:47:02.713826+0000","last_fullsized":"2024-03-17T23:47:02.713826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:00.571611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713510+0000","last_change":"2024-03-17T23:46:54.916398+0000","last_active":"2024-03-17T23:47:02.713510+0000","last_peered":"2024-03-17T23:47:02.713510+0000","last_clean":"2024-03-17T23:47:02.713510+0000","last_became_active":"2024-03-17T23:46:54.916167+0000","last_became_peered":"2024-03-17T23:46:54.916167+0000","last_unstale":"2024-03-17T23:47:02.713510+0000","last_undegraded":"2024-03-17T23:47:02.713510+0000","last_fullsized":"2024-03-17T23:47:02.713510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:42:33.325150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943370+0000","last_change":"2024-03-17T23:47:00.408551+0000","last_active":"2024-03-17T23:47:01.943370+0000","last_peered":"2024-03-17T23:47:01.943370+0000","last_clean":"2024-03-17T23:47:01.943370+0000","last_became_active":"2024-03-17T23:47:00.408223+0000","last_became_peered":"2024-03-17T23:47:00.408223+0000","last_unstale":"2024-03-17T23:47:01.943370+0000","last_undegraded":"2024-03-17T23:47:01.943370+0000","last_fullsized":"2024-03-17T23:47:01.943370+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:52:49.194504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944055+0000","last_change":"2024-03-17T23:46:55.933223+0000","last_active":"2024-03-17T23:47:01.944055+0000","last_peered":"2024-03-17T23:47:01.944055+0000","last_clean":"2024-03-17T23:47:01.944055+0000","last_became_active":"2024-03-17T23:46:55.932957+0000","last_became_peered":"2024-03-17T23:46:55.932957+0000","last_unstale":"2024-03-17T23:47:01.944055+0000","last_undegraded":"2024-03-17T23:47:01.944055+0000","last_fullsized":"2024-03-17T23:47:01.944055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:52:45.163567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713856+0000","last_change":"2024-03-17T23:46:54.919441+0000","last_active":"2024-03-17T23:47:02.713856+0000","last_peered":"2024-03-17T23:47:02.713856+0000","last_clean":"2024-03-17T23:47:02.713856+0000","last_became_active":"2024-03-17T23:46:54.919151+0000","last_became_peered":"2024-03-17T23:46:54.919151+0000","last_unstale":"2024-03-17T23:47:02.713856+0000","last_undegraded":"2024-03-17T23:47:02.713856+0000","last_fullsized":"2024-03-17T23:47:02.713856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:11:01.353944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714221+0000","last_change":"2024-03-17T23:46:59.963445+0000","last_active":"2024-03-17T23:47:02.714221+0000","last_peered":"2024-03-17T23:47:02.714221+0000","last_clean":"2024-03-17T23:47:02.714221+0000","last_became_active":"2024-03-17T23:46:59.963078+0000","last_became_peered":"2024-03-17T23:46:59.963078+0000","last_unstale":"2024-03-17T23:47:02.714221+0000","last_undegraded":"2024-03-17T23:47:02.714221+0000","last_fullsized":"2024-03-17T23:47:02.714221+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:47:52.502020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713966+0000","last_change":"2024-03-17T23:46:55.930951+0000","last_active":"2024-03-17T23:47:02.713966+0000","last_peered":"2024-03-17T23:47:02.713966+0000","last_clean":"2024-03-17T23:47:02.713966+0000","last_became_active":"2024-03-17T23:46:55.930689+0000","last_became_peered":"2024-03-17T23:46:55.930689+0000","last_unstale":"2024-03-17T23:47:02.713966+0000","last_undegraded":"2024-03-17T23:47:02.713966+0000","last_fullsized":"2024-03-17T23:47:02.713966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:43:53.122891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713358+0000","last_change":"2024-03-17T23:46:54.921408+0000","last_active":"2024-03-17T23:47:02.713358+0000","last_peered":"2024-03-17T23:47:02.713358+0000","last_clean":"2024-03-17T23:47:02.713358+0000","last_became_active":"2024-03-17T23:46:54.920825+0000","last_became_peered":"2024-03-17T23:46:54.920825+0000","last_unstale":"2024-03-17T23:47:02.713358+0000","last_undegraded":"2024-03-17T23:47:02.713358+0000","last_fullsized":"2024-03-17T23:47:02.713358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:18.195935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.942947+0000","last_change":"2024-03-17T23:47:00.411371+0000","last_active":"2024-03-17T23:47:01.942947+0000","last_peered":"2024-03-17T23:47:01.942947+0000","last_clean":"2024-03-17T23:47:01.942947+0000","last_became_active":"2024-03-17T23:47:00.410726+0000","last_became_peered":"2024-03-17T23:47:00.410726+0000","last_unstale":"2024-03-17T23:47:01.942947+0000","last_undegraded":"2024-03-17T23:47:01.942947+0000","last_fullsized":"2024-03-17T23:47:01.942947+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:47:52.502020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713787+0000","last_change":"2024-03-17T23:46:55.933053+0000","last_active":"2024-03-17T23:47:02.713787+0000","last_peered":"2024-03-17T23:47:02.713787+0000","last_clean":"2024-03-17T23:47:02.713787+0000","last_became_active":"2024-03-17T23:46:55.932756+0000","last_became_peered":"2024-03-17T23:46:55.932756+0000","last_unstale":"2024-03-17T23:47:02.713787+0000","last_undegraded":"2024-03-17T23:47:02.713787+0000","last_fullsized":"2024-03-17T23:47:02.713787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:45:29.447600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106603+0000","last_change":"2024-03-17T23:46:54.916992+0000","last_active":"2024-03-17T23:47:02.106603+0000","last_peered":"2024-03-17T23:47:02.106603+0000","last_clean":"2024-03-17T23:47:02.106603+0000","last_became_active":"2024-03-17T23:46:54.916296+0000","last_became_peered":"2024-03-17T23:46:54.916296+0000","last_unstale":"2024-03-17T23:47:02.106603+0000","last_undegraded":"2024-03-17T23:47:02.106603+0000","last_fullsized":"2024-03-17T23:47:02.106603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:17:26.217921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106127+0000","last_change":"2024-03-17T23:46:59.957769+0000","last_active":"2024-03-17T23:47:02.106127+0000","last_peered":"2024-03-17T23:47:02.106127+0000","last_clean":"2024-03-17T23:47:02.106127+0000","last_became_active":"2024-03-17T23:46:59.957385+0000","last_became_peered":"2024-03-17T23:46:59.957385+0000","last_unstale":"2024-03-17T23:47:02.106127+0000","last_undegraded":"2024-03-17T23:47:02.106127+0000","last_fullsized":"2024-03-17T23:47:02.106127+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:35:49.838832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713581+0000","last_change":"2024-03-17T23:46:55.935175+0000","last_active":"2024-03-17T23:47:02.713581+0000","last_peered":"2024-03-17T23:47:02.713581+0000","last_clean":"2024-03-17T23:47:02.713581+0000","last_became_active":"2024-03-17T23:46:55.934338+0000","last_became_peered":"2024-03-17T23:46:55.934338+0000","last_unstale":"2024-03-17T23:47:02.713581+0000","last_undegraded":"2024-03-17T23:47:02.713581+0000","last_fullsized":"2024-03-17T23:47:02.713581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:59:32.027430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713064+0000","last_change":"2024-03-17T23:46:54.913663+0000","last_active":"2024-03-17T23:47:02.713064+0000","last_peered":"2024-03-17T23:47:02.713064+0000","last_clean":"2024-03-17T23:47:02.713064+0000","last_became_active":"2024-03-17T23:46:54.913410+0000","last_became_peered":"2024-03-17T23:46:54.913410+0000","last_unstale":"2024-03-17T23:47:02.713064+0000","last_undegraded":"2024-03-17T23:47:02.713064+0000","last_fullsized":"2024-03-17T23:47:02.713064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:45:17.571056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714251+0000","last_change":"2024-03-17T23:46:59.967164+0000","last_active":"2024-03-17T23:47:02.714251+0000","last_peered":"2024-03-17T23:47:02.714251+0000","last_clean":"2024-03-17T23:47:02.714251+0000","last_became_active":"2024-03-17T23:46:59.966653+0000","last_became_peered":"2024-03-17T23:46:59.966653+0000","last_unstale":"2024-03-17T23:47:02.714251+0000","last_undegraded":"2024-03-17T23:47:02.714251+0000","last_fullsized":"2024-03-17T23:47:02.714251+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:57.296991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943052+0000","last_change":"2024-03-17T23:46:55.925322+0000","last_active":"2024-03-17T23:47:01.943052+0000","last_peered":"2024-03-17T23:47:01.943052+0000","last_clean":"2024-03-17T23:47:01.943052+0000","last_became_active":"2024-03-17T23:46:55.925037+0000","last_became_peered":"2024-03-17T23:46:55.925037+0000","last_unstale":"2024-03-17T23:47:01.943052+0000","last_undegraded":"2024-03-17T23:47:01.943052+0000","last_fullsized":"2024-03-17T23:47:01.943052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:39.972607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943437+0000","last_change":"2024-03-17T23:46:54.918033+0000","last_active":"2024-03-17T23:47:01.943437+0000","last_peered":"2024-03-17T23:47:01.943437+0000","last_clean":"2024-03-17T23:47:01.943437+0000","last_became_active":"2024-03-17T23:46:54.917735+0000","last_became_peered":"2024-03-17T23:46:54.917735+0000","last_unstale":"2024-03-17T23:47:01.943437+0000","last_undegraded":"2024-03-17T23:47:01.943437+0000","last_fullsized":"2024-03-17T23:47:01.943437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:42:33.325150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943778+0000","last_change":"2024-03-17T23:47:00.412019+0000","last_active":"2024-03-17T23:47:01.943778+0000","last_peered":"2024-03-17T23:47:01.943778+0000","last_clean":"2024-03-17T23:47:01.943778+0000","last_became_active":"2024-03-17T23:47:00.411603+0000","last_became_peered":"2024-03-17T23:47:00.411603+0000","last_unstale":"2024-03-17T23:47:01.943778+0000","last_undegraded":"2024-03-17T23:47:01.943778+0000","last_fullsized":"2024-03-17T23:47:01.943778+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:18:26.998562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713747+0000","last_change":"2024-03-17T23:46:55.935249+0000","last_active":"2024-03-17T23:47:02.713747+0000","last_peered":"2024-03-17T23:47:02.713747+0000","last_clean":"2024-03-17T23:47:02.713747+0000","last_became_active":"2024-03-17T23:46:55.934833+0000","last_became_peered":"2024-03-17T23:46:55.934833+0000","last_unstale":"2024-03-17T23:47:02.713747+0000","last_undegraded":"2024-03-17T23:47:02.713747+0000","last_fullsized":"2024-03-17T23:47:02.713747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:32:27.349125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943797+0000","last_change":"2024-03-17T23:46:54.924340+0000","last_active":"2024-03-17T23:47:01.943797+0000","last_peered":"2024-03-17T23:47:01.943797+0000","last_clean":"2024-03-17T23:47:01.943797+0000","last_became_active":"2024-03-17T23:46:54.924080+0000","last_became_peered":"2024-03-17T23:46:54.924080+0000","last_unstale":"2024-03-17T23:47:01.943797+0000","last_undegraded":"2024-03-17T23:47:01.943797+0000","last_fullsized":"2024-03-17T23:47:01.943797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:06:54.433520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943804+0000","last_change":"2024-03-17T23:47:00.408517+0000","last_active":"2024-03-17T23:47:01.943804+0000","last_peered":"2024-03-17T23:47:01.943804+0000","last_clean":"2024-03-17T23:47:01.943804+0000","last_became_active":"2024-03-17T23:47:00.408125+0000","last_became_peered":"2024-03-17T23:47:00.408125+0000","last_unstale":"2024-03-17T23:47:01.943804+0000","last_undegraded":"2024-03-17T23:47:01.943804+0000","last_fullsized":"2024-03-17T23:47:01.943804+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:12:39.327858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714486+0000","last_change":"2024-03-17T23:46:55.928837+0000","last_active":"2024-03-17T23:47:02.714486+0000","last_peered":"2024-03-17T23:47:02.714486+0000","last_clean":"2024-03-17T23:47:02.714486+0000","last_became_active":"2024-03-17T23:46:55.928557+0000","last_became_peered":"2024-03-17T23:46:55.928557+0000","last_unstale":"2024-03-17T23:47:02.714486+0000","last_undegraded":"2024-03-17T23:47:02.714486+0000","last_fullsized":"2024-03-17T23:47:02.714486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:23:58.221114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714395+0000","last_change":"2024-03-17T23:46:54.912215+0000","last_active":"2024-03-17T23:47:02.714395+0000","last_peered":"2024-03-17T23:47:02.714395+0000","last_clean":"2024-03-17T23:47:02.714395+0000","last_became_active":"2024-03-17T23:46:54.911909+0000","last_became_peered":"2024-03-17T23:46:54.911909+0000","last_unstale":"2024-03-17T23:47:02.714395+0000","last_undegraded":"2024-03-17T23:47:02.714395+0000","last_fullsized":"2024-03-17T23:47:02.714395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:45:17.571056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944243+0000","last_change":"2024-03-17T23:47:00.408798+0000","last_active":"2024-03-17T23:47:01.944243+0000","last_peered":"2024-03-17T23:47:01.944243+0000","last_clean":"2024-03-17T23:47:01.944243+0000","last_became_active":"2024-03-17T23:47:00.408420+0000","last_became_peered":"2024-03-17T23:47:00.408420+0000","last_unstale":"2024-03-17T23:47:01.944243+0000","last_undegraded":"2024-03-17T23:47:01.944243+0000","last_fullsized":"2024-03-17T23:47:01.944243+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:57.296991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713721+0000","last_change":"2024-03-17T23:46:55.935456+0000","last_active":"2024-03-17T23:47:02.713721+0000","last_peered":"2024-03-17T23:47:02.713721+0000","last_clean":"2024-03-17T23:47:02.713721+0000","last_became_active":"2024-03-17T23:46:55.935187+0000","last_became_peered":"2024-03-17T23:46:55.935187+0000","last_unstale":"2024-03-17T23:47:02.713721+0000","last_undegraded":"2024-03-17T23:47:02.713721+0000","last_fullsized":"2024-03-17T23:47:02.713721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:11:08.829506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"26'8","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:04.869244+0000","last_change":"2024-03-17T23:46:54.918644+0000","last_active":"2024-03-17T23:47:04.869244+0000","last_peered":"2024-03-17T23:47:04.869244+0000","last_clean":"2024-03-17T23:47:04.869244+0000","last_became_active":"2024-03-17T23:46:54.918290+0000","last_became_peered":"2024-03-17T23:46:54.918290+0000","last_unstale":"2024-03-17T23:47:04.869244+0000","last_undegraded":"2024-03-17T23:47:04.869244+0000","last_fullsized":"2024-03-17T23:47:04.869244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:58.417421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3701,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714038+0000","last_change":"2024-03-17T23:46:59.961721+0000","last_active":"2024-03-17T23:47:02.714038+0000","last_peered":"2024-03-17T23:47:02.714038+0000","last_clean":"2024-03-17T23:47:02.714038+0000","last_became_active":"2024-03-17T23:46:59.960948+0000","last_became_peered":"2024-03-17T23:46:59.960948+0000","last_unstale":"2024-03-17T23:47:02.714038+0000","last_undegraded":"2024-03-17T23:47:02.714038+0000","last_fullsized":"2024-03-17T23:47:02.714038+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:27:51.964285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106592+0000","last_change":"2024-03-17T23:46:55.928225+0000","last_active":"2024-03-17T23:47:02.106592+0000","last_peered":"2024-03-17T23:47:02.106592+0000","last_clean":"2024-03-17T23:47:02.106592+0000","last_became_active":"2024-03-17T23:46:55.927359+0000","last_became_peered":"2024-03-17T23:46:55.927359+0000","last_unstale":"2024-03-17T23:47:02.106592+0000","last_undegraded":"2024-03-17T23:47:02.106592+0000","last_fullsized":"2024-03-17T23:47:02.106592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:46:17.785962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713797+0000","last_change":"2024-03-17T23:46:54.917416+0000","last_active":"2024-03-17T23:47:02.713797+0000","last_peered":"2024-03-17T23:47:02.713797+0000","last_clean":"2024-03-17T23:47:02.713797+0000","last_became_active":"2024-03-17T23:46:54.917136+0000","last_became_peered":"2024-03-17T23:46:54.917136+0000","last_unstale":"2024-03-17T23:47:02.713797+0000","last_undegraded":"2024-03-17T23:47:02.713797+0000","last_fullsized":"2024-03-17T23:47:02.713797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:06:54.433520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106414+0000","last_change":"2024-03-17T23:46:59.957651+0000","last_active":"2024-03-17T23:47:02.106414+0000","last_peered":"2024-03-17T23:47:02.106414+0000","last_clean":"2024-03-17T23:47:02.106414+0000","last_became_active":"2024-03-17T23:46:59.957261+0000","last_became_peered":"2024-03-17T23:46:59.957261+0000","last_unstale":"2024-03-17T23:47:02.106414+0000","last_undegraded":"2024-03-17T23:47:02.106414+0000","last_fullsized":"2024-03-17T23:47:02.106414+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:47:52.502020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713868+0000","last_change":"2024-03-17T23:46:55.937105+0000","last_active":"2024-03-17T23:47:02.713868+0000","last_peered":"2024-03-17T23:47:02.713868+0000","last_clean":"2024-03-17T23:47:02.713868+0000","last_became_active":"2024-03-17T23:46:55.936762+0000","last_became_peered":"2024-03-17T23:46:55.936762+0000","last_unstale":"2024-03-17T23:47:02.713868+0000","last_undegraded":"2024-03-17T23:47:02.713868+0000","last_fullsized":"2024-03-17T23:47:02.713868+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:39.972607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944371+0000","last_change":"2024-03-17T23:46:54.922118+0000","last_active":"2024-03-17T23:47:01.944371+0000","last_peered":"2024-03-17T23:47:01.944371+0000","last_clean":"2024-03-17T23:47:01.944371+0000","last_became_active":"2024-03-17T23:46:54.921834+0000","last_became_peered":"2024-03-17T23:46:54.921834+0000","last_unstale":"2024-03-17T23:47:01.944371+0000","last_undegraded":"2024-03-17T23:47:01.944371+0000","last_fullsized":"2024-03-17T23:47:01.944371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:42:33.325150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943156+0000","last_change":"2024-03-17T23:46:59.957652+0000","last_active":"2024-03-17T23:47:01.943156+0000","last_peered":"2024-03-17T23:47:01.943156+0000","last_clean":"2024-03-17T23:47:01.943156+0000","last_became_active":"2024-03-17T23:46:59.957242+0000","last_became_peered":"2024-03-17T23:46:59.957242+0000","last_unstale":"2024-03-17T23:47:01.943156+0000","last_undegraded":"2024-03-17T23:47:01.943156+0000","last_fullsized":"2024-03-17T23:47:01.943156+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:48:08.469403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944280+0000","last_change":"2024-03-17T23:46:55.930428+0000","last_active":"2024-03-17T23:47:01.944280+0000","last_peered":"2024-03-17T23:47:01.944280+0000","last_clean":"2024-03-17T23:47:01.944280+0000","last_became_active":"2024-03-17T23:46:55.930119+0000","last_became_peered":"2024-03-17T23:46:55.930119+0000","last_unstale":"2024-03-17T23:47:01.944280+0000","last_undegraded":"2024-03-17T23:47:01.944280+0000","last_fullsized":"2024-03-17T23:47:01.944280+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:00.571611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944550+0000","last_change":"2024-03-17T23:46:54.918186+0000","last_active":"2024-03-17T23:47:01.944550+0000","last_peered":"2024-03-17T23:47:01.944550+0000","last_clean":"2024-03-17T23:47:01.944550+0000","last_became_active":"2024-03-17T23:46:54.917839+0000","last_became_peered":"2024-03-17T23:46:54.917839+0000","last_unstale":"2024-03-17T23:47:01.944550+0000","last_undegraded":"2024-03-17T23:47:01.944550+0000","last_fullsized":"2024-03-17T23:47:01.944550+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:06:54.433520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943131+0000","last_change":"2024-03-17T23:47:00.411924+0000","last_active":"2024-03-17T23:47:01.943131+0000","last_peered":"2024-03-17T23:47:01.943131+0000","last_clean":"2024-03-17T23:47:01.943131+0000","last_became_active":"2024-03-17T23:47:00.411410+0000","last_became_peered":"2024-03-17T23:47:00.411410+0000","last_unstale":"2024-03-17T23:47:01.943131+0000","last_undegraded":"2024-03-17T23:47:01.943131+0000","last_fullsized":"2024-03-17T23:47:01.943131+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:56:09.808314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714107+0000","last_change":"2024-03-17T23:47:00.410837+0000","last_active":"2024-03-17T23:47:02.714107+0000","last_peered":"2024-03-17T23:47:02.714107+0000","last_clean":"2024-03-17T23:47:02.714107+0000","last_became_active":"2024-03-17T23:47:00.410459+0000","last_became_peered":"2024-03-17T23:47:00.410459+0000","last_unstale":"2024-03-17T23:47:02.714107+0000","last_undegraded":"2024-03-17T23:47:02.714107+0000","last_fullsized":"2024-03-17T23:47:02.714107+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:39:36.583214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713305+0000","last_change":"2024-03-17T23:46:55.928213+0000","last_active":"2024-03-17T23:47:02.713305+0000","last_peered":"2024-03-17T23:47:02.713305+0000","last_clean":"2024-03-17T23:47:02.713305+0000","last_became_active":"2024-03-17T23:46:55.927957+0000","last_became_peered":"2024-03-17T23:46:55.927957+0000","last_unstale":"2024-03-17T23:47:02.713305+0000","last_undegraded":"2024-03-17T23:47:02.713305+0000","last_fullsized":"2024-03-17T23:47:02.713305+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:03.353861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943171+0000","last_change":"2024-03-17T23:46:54.919392+0000","last_active":"2024-03-17T23:47:01.943171+0000","last_peered":"2024-03-17T23:47:01.943171+0000","last_clean":"2024-03-17T23:47:01.943171+0000","last_became_active":"2024-03-17T23:46:54.918594+0000","last_became_peered":"2024-03-17T23:46:54.918594+0000","last_unstale":"2024-03-17T23:47:01.943171+0000","last_undegraded":"2024-03-17T23:47:01.943171+0000","last_fullsized":"2024-03-17T23:47:01.943171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:18.195935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944720+0000","last_change":"2024-03-17T23:46:59.959748+0000","last_active":"2024-03-17T23:47:01.944720+0000","last_peered":"2024-03-17T23:47:01.944720+0000","last_clean":"2024-03-17T23:47:01.944720+0000","last_became_active":"2024-03-17T23:46:59.959383+0000","last_became_peered":"2024-03-17T23:46:59.959383+0000","last_unstale":"2024-03-17T23:47:01.944720+0000","last_undegraded":"2024-03-17T23:47:01.944720+0000","last_fullsized":"2024-03-17T23:47:01.944720+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:43:57.153828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943202+0000","last_change":"2024-03-17T23:46:55.929140+0000","last_active":"2024-03-17T23:47:01.943202+0000","last_peered":"2024-03-17T23:47:01.943202+0000","last_clean":"2024-03-17T23:47:01.943202+0000","last_became_active":"2024-03-17T23:46:55.928874+0000","last_became_peered":"2024-03-17T23:46:55.928874+0000","last_unstale":"2024-03-17T23:47:01.943202+0000","last_undegraded":"2024-03-17T23:47:01.943202+0000","last_fullsized":"2024-03-17T23:47:01.943202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:59:32.027430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943228+0000","last_change":"2024-03-17T23:46:54.923319+0000","last_active":"2024-03-17T23:47:01.943228+0000","last_peered":"2024-03-17T23:47:01.943228+0000","last_clean":"2024-03-17T23:47:01.943228+0000","last_became_active":"2024-03-17T23:46:54.923060+0000","last_became_peered":"2024-03-17T23:46:54.923060+0000","last_unstale":"2024-03-17T23:47:01.943228+0000","last_undegraded":"2024-03-17T23:47:01.943228+0000","last_fullsized":"2024-03-17T23:47:01.943228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:45:17.571056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943597+0000","last_change":"2024-03-17T23:46:54.917589+0000","last_active":"2024-03-17T23:47:01.943597+0000","last_peered":"2024-03-17T23:47:01.943597+0000","last_clean":"2024-03-17T23:47:01.943597+0000","last_became_active":"2024-03-17T23:46:54.917183+0000","last_became_peered":"2024-03-17T23:46:54.917183+0000","last_unstale":"2024-03-17T23:47:01.943597+0000","last_undegraded":"2024-03-17T23:47:01.943597+0000","last_fullsized":"2024-03-17T23:47:01.943597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:18.195935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714330+0000","last_change":"2024-03-17T23:46:55.924339+0000","last_active":"2024-03-17T23:47:02.714330+0000","last_peered":"2024-03-17T23:47:02.714330+0000","last_clean":"2024-03-17T23:47:02.714330+0000","last_became_active":"2024-03-17T23:46:55.923891+0000","last_became_peered":"2024-03-17T23:46:55.923891+0000","last_unstale":"2024-03-17T23:47:02.714330+0000","last_undegraded":"2024-03-17T23:47:02.714330+0000","last_fullsized":"2024-03-17T23:47:02.714330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:11:08.829506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713322+0000","last_change":"2024-03-17T23:46:59.959425+0000","last_active":"2024-03-17T23:47:02.713322+0000","last_peered":"2024-03-17T23:47:02.713322+0000","last_clean":"2024-03-17T23:47:02.713322+0000","last_became_active":"2024-03-17T23:46:59.958851+0000","last_became_peered":"2024-03-17T23:46:59.958851+0000","last_unstale":"2024-03-17T23:47:02.713322+0000","last_undegraded":"2024-03-17T23:47:02.713322+0000","last_fullsized":"2024-03-17T23:47:02.713322+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:58.379120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943490+0000","last_change":"2024-03-17T23:46:54.921325+0000","last_active":"2024-03-17T23:47:01.943490+0000","last_peered":"2024-03-17T23:47:01.943490+0000","last_clean":"2024-03-17T23:47:01.943490+0000","last_became_active":"2024-03-17T23:46:54.921093+0000","last_became_peered":"2024-03-17T23:46:54.921093+0000","last_unstale":"2024-03-17T23:47:01.943490+0000","last_undegraded":"2024-03-17T23:47:01.943490+0000","last_fullsized":"2024-03-17T23:47:01.943490+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:56.665566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106239+0000","last_change":"2024-03-17T23:46:55.922093+0000","last_active":"2024-03-17T23:47:02.106239+0000","last_peered":"2024-03-17T23:47:02.106239+0000","last_clean":"2024-03-17T23:47:02.106239+0000","last_became_active":"2024-03-17T23:46:55.919778+0000","last_became_peered":"2024-03-17T23:46:55.919778+0000","last_unstale":"2024-03-17T23:47:02.106239+0000","last_undegraded":"2024-03-17T23:47:02.106239+0000","last_fullsized":"2024-03-17T23:47:02.106239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:06:55.437268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943978+0000","last_change":"2024-03-17T23:46:59.964222+0000","last_active":"2024-03-17T23:47:01.943978+0000","last_peered":"2024-03-17T23:47:01.943978+0000","last_clean":"2024-03-17T23:47:01.943978+0000","last_became_active":"2024-03-17T23:46:59.963807+0000","last_became_peered":"2024-03-17T23:46:59.963807+0000","last_unstale":"2024-03-17T23:47:01.943978+0000","last_undegraded":"2024-03-17T23:47:01.943978+0000","last_fullsized":"2024-03-17T23:47:01.943978+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:25:11.791005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.942962+0000","last_change":"2024-03-17T23:46:55.926785+0000","last_active":"2024-03-17T23:47:01.942962+0000","last_peered":"2024-03-17T23:47:01.942962+0000","last_clean":"2024-03-17T23:47:01.942962+0000","last_became_active":"2024-03-17T23:46:55.926494+0000","last_became_peered":"2024-03-17T23:46:55.926494+0000","last_unstale":"2024-03-17T23:47:01.942962+0000","last_undegraded":"2024-03-17T23:47:01.942962+0000","last_fullsized":"2024-03-17T23:47:01.942962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:24.419077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713141+0000","last_change":"2024-03-17T23:46:55.930094+0000","last_active":"2024-03-17T23:47:02.713141+0000","last_peered":"2024-03-17T23:47:02.713141+0000","last_clean":"2024-03-17T23:47:02.713141+0000","last_became_active":"2024-03-17T23:46:55.929834+0000","last_became_peered":"2024-03-17T23:46:55.929834+0000","last_unstale":"2024-03-17T23:47:02.713141+0000","last_undegraded":"2024-03-17T23:47:02.713141+0000","last_fullsized":"2024-03-17T23:47:02.713141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:59.421169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943791+0000","last_change":"2024-03-17T23:46:55.934265+0000","last_active":"2024-03-17T23:47:01.943791+0000","last_peered":"2024-03-17T23:47:01.943791+0000","last_clean":"2024-03-17T23:47:01.943791+0000","last_became_active":"2024-03-17T23:46:55.933735+0000","last_became_peered":"2024-03-17T23:46:55.933735+0000","last_unstale":"2024-03-17T23:47:01.943791+0000","last_undegraded":"2024-03-17T23:47:01.943791+0000","last_fullsized":"2024-03-17T23:47:01.943791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:59:32.027430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714250+0000","last_change":"2024-03-17T23:46:55.930627+0000","last_active":"2024-03-17T23:47:02.714250+0000","last_peered":"2024-03-17T23:47:02.714250+0000","last_clean":"2024-03-17T23:47:02.714250+0000","last_became_active":"2024-03-17T23:46:55.930368+0000","last_became_peered":"2024-03-17T23:46:55.930368+0000","last_unstale":"2024-03-17T23:47:02.714250+0000","last_undegraded":"2024-03-17T23:47:02.714250+0000","last_fullsized":"2024-03-17T23:47:02.714250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:47:30.023063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943914+0000","last_change":"2024-03-17T23:46:55.930544+0000","last_active":"2024-03-17T23:47:01.943914+0000","last_peered":"2024-03-17T23:47:01.943914+0000","last_clean":"2024-03-17T23:47:01.943914+0000","last_became_active":"2024-03-17T23:46:55.930277+0000","last_became_peered":"2024-03-17T23:46:55.930277+0000","last_unstale":"2024-03-17T23:47:01.943914+0000","last_undegraded":"2024-03-17T23:47:01.943914+0000","last_fullsized":"2024-03-17T23:47:01.943914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:23:58.221114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713632+0000","last_change":"2024-03-17T23:46:55.927054+0000","last_active":"2024-03-17T23:47:02.713632+0000","last_peered":"2024-03-17T23:47:02.713632+0000","last_clean":"2024-03-17T23:47:02.713632+0000","last_became_active":"2024-03-17T23:46:55.926762+0000","last_became_peered":"2024-03-17T23:46:55.926762+0000","last_unstale":"2024-03-17T23:47:02.713632+0000","last_undegraded":"2024-03-17T23:47:02.713632+0000","last_fullsized":"2024-03-17T23:47:02.713632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:39.972607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944787+0000","last_change":"2024-03-17T23:46:55.929740+0000","last_active":"2024-03-17T23:47:01.944787+0000","last_peered":"2024-03-17T23:47:01.944787+0000","last_clean":"2024-03-17T23:47:01.944787+0000","last_became_active":"2024-03-17T23:46:55.929433+0000","last_became_peered":"2024-03-17T23:46:55.929433+0000","last_unstale":"2024-03-17T23:47:01.944787+0000","last_undegraded":"2024-03-17T23:47:01.944787+0000","last_fullsized":"2024-03-17T23:47:01.944787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:11:08.829506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943753+0000","last_change":"2024-03-17T23:46:55.930851+0000","last_active":"2024-03-17T23:47:01.943753+0000","last_peered":"2024-03-17T23:47:01.943753+0000","last_clean":"2024-03-17T23:47:01.943753+0000","last_became_active":"2024-03-17T23:46:55.930305+0000","last_became_peered":"2024-03-17T23:46:55.930305+0000","last_unstale":"2024-03-17T23:47:01.943753+0000","last_undegraded":"2024-03-17T23:47:01.943753+0000","last_fullsized":"2024-03-17T23:47:01.943753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:47:30.023063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943134+0000","last_change":"2024-03-17T23:46:55.933139+0000","last_active":"2024-03-17T23:47:01.943134+0000","last_peered":"2024-03-17T23:47:01.943134+0000","last_clean":"2024-03-17T23:47:01.943134+0000","last_became_active":"2024-03-17T23:46:55.932843+0000","last_became_peered":"2024-03-17T23:46:55.932843+0000","last_unstale":"2024-03-17T23:47:01.943134+0000","last_undegraded":"2024-03-17T23:47:01.943134+0000","last_fullsized":"2024-03-17T23:47:01.943134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:44:47.002489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943147+0000","last_change":"2024-03-17T23:46:55.930881+0000","last_active":"2024-03-17T23:47:01.943147+0000","last_peered":"2024-03-17T23:47:01.943147+0000","last_clean":"2024-03-17T23:47:01.943147+0000","last_became_active":"2024-03-17T23:46:55.930629+0000","last_became_peered":"2024-03-17T23:46:55.930629+0000","last_unstale":"2024-03-17T23:47:01.943147+0000","last_undegraded":"2024-03-17T23:47:01.943147+0000","last_fullsized":"2024-03-17T23:47:01.943147+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:49:47.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943696+0000","last_change":"2024-03-17T23:46:55.936236+0000","last_active":"2024-03-17T23:47:01.943696+0000","last_peered":"2024-03-17T23:47:01.943696+0000","last_clean":"2024-03-17T23:47:01.943696+0000","last_became_active":"2024-03-17T23:46:55.933485+0000","last_became_peered":"2024-03-17T23:46:55.933485+0000","last_unstale":"2024-03-17T23:47:01.943696+0000","last_undegraded":"2024-03-17T23:47:01.943696+0000","last_fullsized":"2024-03-17T23:47:01.943696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:11:08.829506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714118+0000","last_change":"2024-03-17T23:46:55.930067+0000","last_active":"2024-03-17T23:47:02.714118+0000","last_peered":"2024-03-17T23:47:02.714118+0000","last_clean":"2024-03-17T23:47:02.714118+0000","last_became_active":"2024-03-17T23:46:55.929808+0000","last_became_peered":"2024-03-17T23:46:55.929808+0000","last_unstale":"2024-03-17T23:47:02.714118+0000","last_undegraded":"2024-03-17T23:47:02.714118+0000","last_fullsized":"2024-03-17T23:47:02.714118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:49:47.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943993+0000","last_change":"2024-03-17T23:46:55.917074+0000","last_active":"2024-03-17T23:47:01.943993+0000","last_peered":"2024-03-17T23:47:01.943993+0000","last_clean":"2024-03-17T23:47:01.943993+0000","last_became_active":"2024-03-17T23:46:55.916784+0000","last_became_peered":"2024-03-17T23:46:55.916784+0000","last_unstale":"2024-03-17T23:47:01.943993+0000","last_undegraded":"2024-03-17T23:47:01.943993+0000","last_fullsized":"2024-03-17T23:47:01.943993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:49:47.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713939+0000","last_change":"2024-03-17T23:46:55.928539+0000","last_active":"2024-03-17T23:47:02.713939+0000","last_peered":"2024-03-17T23:47:02.713939+0000","last_clean":"2024-03-17T23:47:02.713939+0000","last_became_active":"2024-03-17T23:46:55.928275+0000","last_became_peered":"2024-03-17T23:46:55.928275+0000","last_unstale":"2024-03-17T23:47:02.713939+0000","last_undegraded":"2024-03-17T23:47:02.713939+0000","last_fullsized":"2024-03-17T23:47:02.713939+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T23:52:25.543746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713273+0000","last_change":"2024-03-17T23:46:55.924366+0000","last_active":"2024-03-17T23:47:02.713273+0000","last_peered":"2024-03-17T23:47:02.713273+0000","last_clean":"2024-03-17T23:47:02.713273+0000","last_became_active":"2024-03-17T23:46:55.924100+0000","last_became_peered":"2024-03-17T23:46:55.924100+0000","last_unstale":"2024-03-17T23:47:02.713273+0000","last_undegraded":"2024-03-17T23:47:02.713273+0000","last_fullsized":"2024-03-17T23:47:02.713273+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:51:19.549589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944530+0000","last_change":"2024-03-17T23:46:55.932514+0000","last_active":"2024-03-17T23:47:01.944530+0000","last_peered":"2024-03-17T23:47:01.944530+0000","last_clean":"2024-03-17T23:47:01.944530+0000","last_became_active":"2024-03-17T23:46:55.930899+0000","last_became_peered":"2024-03-17T23:46:55.930899+0000","last_unstale":"2024-03-17T23:47:01.944530+0000","last_undegraded":"2024-03-17T23:47:01.944530+0000","last_fullsized":"2024-03-17T23:47:01.944530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:47:30.023063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943088+0000","last_change":"2024-03-17T23:46:55.940176+0000","last_active":"2024-03-17T23:47:01.943088+0000","last_peered":"2024-03-17T23:47:01.943088+0000","last_clean":"2024-03-17T23:47:01.943088+0000","last_became_active":"2024-03-17T23:46:55.939853+0000","last_became_peered":"2024-03-17T23:46:55.939853+0000","last_unstale":"2024-03-17T23:47:01.943088+0000","last_undegraded":"2024-03-17T23:47:01.943088+0000","last_fullsized":"2024-03-17T23:47:01.943088+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:49:47.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713396+0000","last_change":"2024-03-17T23:46:55.917430+0000","last_active":"2024-03-17T23:47:02.713396+0000","last_peered":"2024-03-17T23:47:02.713396+0000","last_clean":"2024-03-17T23:47:02.713396+0000","last_became_active":"2024-03-17T23:46:55.917173+0000","last_became_peered":"2024-03-17T23:46:55.917173+0000","last_unstale":"2024-03-17T23:47:02.713396+0000","last_undegraded":"2024-03-17T23:47:02.713396+0000","last_fullsized":"2024-03-17T23:47:02.713396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:23:58.221114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714399+0000","last_change":"2024-03-17T23:46:55.927198+0000","last_active":"2024-03-17T23:47:02.714399+0000","last_peered":"2024-03-17T23:47:02.714399+0000","last_clean":"2024-03-17T23:47:02.714399+0000","last_became_active":"2024-03-17T23:46:55.926919+0000","last_became_peered":"2024-03-17T23:46:55.926919+0000","last_unstale":"2024-03-17T23:47:02.714399+0000","last_undegraded":"2024-03-17T23:47:02.714399+0000","last_fullsized":"2024-03-17T23:47:02.714399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:24.419077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943395+0000","last_change":"2024-03-17T23:46:55.932835+0000","last_active":"2024-03-17T23:47:01.943395+0000","last_peered":"2024-03-17T23:47:01.943395+0000","last_clean":"2024-03-17T23:47:01.943395+0000","last_became_active":"2024-03-17T23:46:55.932339+0000","last_became_peered":"2024-03-17T23:46:55.932339+0000","last_unstale":"2024-03-17T23:47:01.943395+0000","last_undegraded":"2024-03-17T23:47:01.943395+0000","last_fullsized":"2024-03-17T23:47:01.943395+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:39.972607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106535+0000","last_change":"2024-03-17T23:46:55.916730+0000","last_active":"2024-03-17T23:47:02.106535+0000","last_peered":"2024-03-17T23:47:02.106535+0000","last_clean":"2024-03-17T23:47:02.106535+0000","last_became_active":"2024-03-17T23:46:55.916438+0000","last_became_peered":"2024-03-17T23:46:55.916438+0000","last_unstale":"2024-03-17T23:47:02.106535+0000","last_undegraded":"2024-03-17T23:47:02.106535+0000","last_fullsized":"2024-03-17T23:47:02.106535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:51:19.549589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944384+0000","last_change":"2024-03-17T23:46:55.932468+0000","last_active":"2024-03-17T23:47:01.944384+0000","last_peered":"2024-03-17T23:47:01.944384+0000","last_clean":"2024-03-17T23:47:01.944384+0000","last_became_active":"2024-03-17T23:46:55.932210+0000","last_became_peered":"2024-03-17T23:46:55.932210+0000","last_unstale":"2024-03-17T23:47:01.944384+0000","last_undegraded":"2024-03-17T23:47:01.944384+0000","last_fullsized":"2024-03-17T23:47:01.944384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:23:58.221114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944784+0000","last_change":"2024-03-17T23:46:55.934093+0000","last_active":"2024-03-17T23:47:01.944784+0000","last_peered":"2024-03-17T23:47:01.944784+0000","last_clean":"2024-03-17T23:47:01.944784+0000","last_became_active":"2024-03-17T23:46:55.933516+0000","last_became_peered":"2024-03-17T23:46:55.933516+0000","last_unstale":"2024-03-17T23:47:01.944784+0000","last_undegraded":"2024-03-17T23:47:01.944784+0000","last_fullsized":"2024-03-17T23:47:01.944784+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:47:30.023063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713902+0000","last_change":"2024-03-17T23:46:55.932426+0000","last_active":"2024-03-17T23:47:02.713902+0000","last_peered":"2024-03-17T23:47:02.713902+0000","last_clean":"2024-03-17T23:47:02.713902+0000","last_became_active":"2024-03-17T23:46:55.932045+0000","last_became_peered":"2024-03-17T23:46:55.932045+0000","last_unstale":"2024-03-17T23:47:02.713902+0000","last_undegraded":"2024-03-17T23:47:02.713902+0000","last_fullsized":"2024-03-17T23:47:02.713902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:20:21.507725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.942978+0000","last_change":"2024-03-17T23:46:55.932245+0000","last_active":"2024-03-17T23:47:01.942978+0000","last_peered":"2024-03-17T23:47:01.942978+0000","last_clean":"2024-03-17T23:47:01.942978+0000","last_became_active":"2024-03-17T23:46:55.932001+0000","last_became_peered":"2024-03-17T23:46:55.932001+0000","last_unstale":"2024-03-17T23:47:01.942978+0000","last_undegraded":"2024-03-17T23:47:01.942978+0000","last_fullsized":"2024-03-17T23:47:01.942978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:44:47.002489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713629+0000","last_change":"2024-03-17T23:46:55.935868+0000","last_active":"2024-03-17T23:47:02.713629+0000","last_peered":"2024-03-17T23:47:02.713629+0000","last_clean":"2024-03-17T23:47:02.713629+0000","last_became_active":"2024-03-17T23:46:55.935180+0000","last_became_peered":"2024-03-17T23:46:55.935180+0000","last_unstale":"2024-03-17T23:47:02.713629+0000","last_undegraded":"2024-03-17T23:47:02.713629+0000","last_fullsized":"2024-03-17T23:47:02.713629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:44:47.002489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943696+0000","last_change":"2024-03-17T23:46:55.936947+0000","last_active":"2024-03-17T23:47:01.943696+0000","last_peered":"2024-03-17T23:47:01.943696+0000","last_clean":"2024-03-17T23:47:01.943696+0000","last_became_active":"2024-03-17T23:46:55.936694+0000","last_became_peered":"2024-03-17T23:46:55.936694+0000","last_unstale":"2024-03-17T23:47:01.943696+0000","last_undegraded":"2024-03-17T23:47:01.943696+0000","last_fullsized":"2024-03-17T23:47:01.943696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:39.972607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713525+0000","last_change":"2024-03-17T23:46:55.927487+0000","last_active":"2024-03-17T23:47:02.713525+0000","last_peered":"2024-03-17T23:47:02.713525+0000","last_clean":"2024-03-17T23:47:02.713525+0000","last_became_active":"2024-03-17T23:46:55.927225+0000","last_became_peered":"2024-03-17T23:46:55.927225+0000","last_unstale":"2024-03-17T23:47:02.713525+0000","last_undegraded":"2024-03-17T23:47:02.713525+0000","last_fullsized":"2024-03-17T23:47:02.713525+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:22:12.841390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943865+0000","last_change":"2024-03-17T23:46:55.933146+0000","last_active":"2024-03-17T23:47:01.943865+0000","last_peered":"2024-03-17T23:47:01.943865+0000","last_clean":"2024-03-17T23:47:01.943865+0000","last_became_active":"2024-03-17T23:46:55.932861+0000","last_became_peered":"2024-03-17T23:46:55.932861+0000","last_unstale":"2024-03-17T23:47:01.943865+0000","last_undegraded":"2024-03-17T23:47:01.943865+0000","last_fullsized":"2024-03-17T23:47:01.943865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:22:12.841390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713455+0000","last_change":"2024-03-17T23:46:55.924118+0000","last_active":"2024-03-17T23:47:02.713455+0000","last_peered":"2024-03-17T23:47:02.713455+0000","last_clean":"2024-03-17T23:47:02.713455+0000","last_became_active":"2024-03-17T23:46:55.923844+0000","last_became_peered":"2024-03-17T23:46:55.923844+0000","last_unstale":"2024-03-17T23:47:02.713455+0000","last_undegraded":"2024-03-17T23:47:02.713455+0000","last_fullsized":"2024-03-17T23:47:02.713455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:30:49.518932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713741+0000","last_change":"2024-03-17T23:46:55.925191+0000","last_active":"2024-03-17T23:47:02.713741+0000","last_peered":"2024-03-17T23:47:02.713741+0000","last_clean":"2024-03-17T23:47:02.713741+0000","last_became_active":"2024-03-17T23:46:55.924822+0000","last_became_peered":"2024-03-17T23:46:55.924822+0000","last_unstale":"2024-03-17T23:47:02.713741+0000","last_undegraded":"2024-03-17T23:47:02.713741+0000","last_fullsized":"2024-03-17T23:47:02.713741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:45:20.327436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713837+0000","last_change":"2024-03-17T23:46:55.928943+0000","last_active":"2024-03-17T23:47:02.713837+0000","last_peered":"2024-03-17T23:47:02.713837+0000","last_clean":"2024-03-17T23:47:02.713837+0000","last_became_active":"2024-03-17T23:46:55.928669+0000","last_became_peered":"2024-03-17T23:46:55.928669+0000","last_unstale":"2024-03-17T23:47:02.713837+0000","last_undegraded":"2024-03-17T23:47:02.713837+0000","last_fullsized":"2024-03-17T23:47:02.713837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:52:45.163567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4499,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":14,"seq":60129542152,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":286720,"data_stored":114925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542152,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":111224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542152,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":880640,"data_stored":702230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":872448,"data_stored":702174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31460,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340380,"statfs":{"total":96636764160,"available":96604549120,"internally_reserved":0,"allocated":286720,"data_stored":114925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":278528,"data_stored":111314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":286720,"data_stored":111350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":111384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-17T23:47:13.280 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-17T23:47:13.280 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-17T23:47:13.280 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-17T23:47:13.281 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-17T23:47:13.281 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-17T23:47:13.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.416+0000 7f0bb3fc6640 1 Processor -- start 2024-03-17T23:47:13.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.416+0000 7f0bb3fc6640 1 -- start start 2024-03-17T23:47:13.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.417+0000 7f0bb3fc6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0bac09a470 0x7f0bac09a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:13.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.417+0000 7f0bb3fc6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0bac09ad90 0x7f0bac15f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:13.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.417+0000 7f0bb3fc6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bac0cfad0 con 0x7f0bac15fcd0 2024-03-17T23:47:13.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.417+0000 7f0bb3fc6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f0bac0cfdb0 con 0x7f0bac09ad90 2024-03-17T23:47:13.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.417+0000 7f0bb3fc6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0bac0d0090 con 0x7f0bac09a470 2024-03-17T23:47:13.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.417+0000 7f0bb1561640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0bac09ad90 0x7f0bac15f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:13.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.417+0000 7f0bb2563640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f0bac15fcd0 0x7f0bac15c150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59352/0 (socket says 172.21.15.150:59352) 2024-03-17T23:47:13.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.417+0000 7f0bb1561640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0bac09ad90 0x7f0bac15f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36282/0 (socket says 172.21.15.150:36282) 2024-03-17T23:47:13.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.417+0000 7f0bb2563640 1 -- 172.21.15.150:0/679580670 learned_addr learned my addr 172.21.15.150:0/679580670 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:13.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.417+0000 7f0bb1d62640 1 --2- 172.21.15.150:0/679580670 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0bac09a470 0x7f0bac09a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:13.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.418+0000 7f0bb1561640 1 -- 172.21.15.150:0/679580670 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0bac09a470 msgr2=0x7f0bac09a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:13.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.418+0000 7f0bb1561640 1 --2- 172.21.15.150:0/679580670 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0bac09a470 0x7f0bac09a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:13.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.418+0000 7f0bb1561640 1 -- 172.21.15.150:0/679580670 >> v1:172.21.15.150:6789/0 conn(0x7f0bac15fcd0 legacy=0x7f0bac15c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:13.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.418+0000 7f0bb0d60640 1 -- 172.21.15.150:0/679580670 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1495434950 0 0) 0x7f0bac0cfad0 con 0x7f0bac15fcd0 2024-03-17T23:47:13.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.418+0000 7f0bb1561640 1 -- 172.21.15.150:0/679580670 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bac0d0370 con 0x7f0bac09ad90 2024-03-17T23:47:13.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.418+0000 7f0bb1d62640 1 --2- 172.21.15.150:0/679580670 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0bac09a470 0x7f0bac09a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:13.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.418+0000 7f0bb1561640 1 --2- 172.21.15.150:0/679580670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bac09ad90 0x7f0bac15f790 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c0097a0 tx=0x7f0b9c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bada95ebdde9e72 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:13.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.419+0000 7f0bb0d60640 1 -- 172.21.15.150:0/679580670 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c035020 con 0x7f0bac09ad90 2024-03-17T23:47:13.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.419+0000 7f0bb0d60640 1 -- 172.21.15.150:0/679580670 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b9c031830 con 0x7f0bac09ad90 2024-03-17T23:47:13.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.419+0000 7f0bb0d60640 1 -- 172.21.15.150:0/679580670 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c031c00 con 0x7f0bac09ad90 2024-03-17T23:47:13.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.419+0000 7f0bb3fc6640 1 -- 172.21.15.150:0/679580670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bac09ad90 msgr2=0x7f0bac15f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:13.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.419+0000 7f0bb3fc6640 1 --2- 172.21.15.150:0/679580670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bac09ad90 0x7f0bac15f790 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c0097a0 tx=0x7f0b9c02b8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:13.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.419+0000 7f0bb3fc6640 1 -- 172.21.15.150:0/679580670 shutdown_connections 2024-03-17T23:47:13.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.419+0000 7f0bb3fc6640 1 --2- 172.21.15.150:0/679580670 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bac09ad90 0x7f0bac15f790 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:13.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.419+0000 7f0bb3fc6640 1 --2- 172.21.15.150:0/679580670 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0bac09a470 0x7f0bac09a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:13.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.419+0000 7f0bb3fc6640 1 -- 172.21.15.150:0/679580670 >> 172.21.15.150:0/679580670 conn(0x7f0bac0b3630 msgr2=0x7f0bac0b3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:13.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.419+0000 7f0bb3fc6640 1 -- 172.21.15.150:0/679580670 shutdown_connections 2024-03-17T23:47:13.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.419+0000 7f0bb3fc6640 1 -- 172.21.15.150:0/679580670 wait complete. 2024-03-17T23:47:13.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.420+0000 7f0bb3fc6640 1 Processor -- start 2024-03-17T23:47:13.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.420+0000 7f0bb3fc6640 1 -- start start 2024-03-17T23:47:13.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.421+0000 7f0bb3fc6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0bac09a470 0x7f0bac15c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:13.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.421+0000 7f0bb3fc6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bac09ad90 0x7f0bac0cb170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:13.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.421+0000 7f0bb3fc6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0bac15fcd0 0x7f0bac1543d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:13.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.421+0000 7f0bb3fc6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0bac0cf890 con 0x7f0bac09ad90 2024-03-17T23:47:13.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.421+0000 7f0bb3fc6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0bac0cf710 con 0x7f0bac15fcd0 2024-03-17T23:47:13.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.421+0000 7f0bb3fc6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0bac0cd5f0 con 0x7f0bac09a470 2024-03-17T23:47:13.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.421+0000 7f0bb1561640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bac09ad90 0x7f0bac0cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:13.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.421+0000 7f0bb1561640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bac09ad90 0x7f0bac0cb170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36296/0 (socket says 172.21.15.150:36296) 2024-03-17T23:47:13.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.421+0000 7f0bb1561640 1 -- 172.21.15.150:0/1243896880 learned_addr learned my addr 172.21.15.150:0/1243896880 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:13.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.421+0000 7f0bb1d62640 1 --2- 172.21.15.150:0/1243896880 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0bac09a470 0x7f0bac15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:13.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.421+0000 7f0bb2563640 1 --2- 172.21.15.150:0/1243896880 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0bac15fcd0 0x7f0bac1543d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:13.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.421+0000 7f0bb1561640 1 -- 172.21.15.150:0/1243896880 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0bac09a470 msgr2=0x7f0bac15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:13.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.421+0000 7f0bb1561640 1 --2- 172.21.15.150:0/1243896880 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0bac09a470 0x7f0bac15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:13.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.421+0000 7f0bb1561640 1 -- 172.21.15.150:0/1243896880 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0bac15fcd0 msgr2=0x7f0bac1543d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:13.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.421+0000 7f0bb1561640 1 --2- 172.21.15.150:0/1243896880 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0bac15fcd0 0x7f0bac1543d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:13.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.421+0000 7f0bb1561640 1 -- 172.21.15.150:0/1243896880 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bac15a8e0 con 0x7f0bac09ad90 2024-03-17T23:47:13.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.422+0000 7f0bb1561640 1 --2- 172.21.15.150:0/1243896880 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bac09ad90 0x7f0bac0cb170 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c02bee0 tx=0x7f0b9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:13.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.422+0000 7f0ba27fc640 1 -- 172.21.15.150:0/1243896880 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c035040 con 0x7f0bac09ad90 2024-03-17T23:47:13.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.422+0000 7f0ba27fc640 1 -- 172.21.15.150:0/1243896880 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b9c03b920 con 0x7f0bac09ad90 2024-03-17T23:47:13.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.422+0000 7f0ba27fc640 1 -- 172.21.15.150:0/1243896880 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c03bd10 con 0x7f0bac09ad90 2024-03-17T23:47:13.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.422+0000 7f0bb3fc6640 1 -- 172.21.15.150:0/1243896880 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bac155290 con 0x7f0bac09ad90 2024-03-17T23:47:13.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.422+0000 7f0bb3fc6640 1 -- 172.21.15.150:0/1243896880 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bac15ab80 con 0x7f0bac09ad90 2024-03-17T23:47:13.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.423+0000 7f0bb3fc6640 1 -- 172.21.15.150:0/1243896880 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bac0681a0 con 0x7f0bac09ad90 2024-03-17T23:47:13.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.423+0000 7f0ba27fc640 1 -- 172.21.15.150:0/1243896880 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0b9c038020 con 0x7f0bac09ad90 2024-03-17T23:47:13.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.424+0000 7f0ba27fc640 1 --2- 172.21.15.150:0/1243896880 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b9006bbd0 0x7f0b9006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:13.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.424+0000 7f0bb1d62640 1 --2- 172.21.15.150:0/1243896880 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b9006bbd0 0x7f0b9006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:13.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.424+0000 7f0ba27fc640 1 -- 172.21.15.150:0/1243896880 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f0b9c0ace30 con 0x7f0bac09ad90 2024-03-17T23:47:13.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.424+0000 7f0bb1d62640 1 --2- 172.21.15.150:0/1243896880 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b9006bbd0 0x7f0b9006e090 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8009560 tx=0x7f0ba8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:13.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.428+0000 7f0ba27fc640 1 -- 172.21.15.150:0/1243896880 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0b9c031800 con 0x7f0bac09ad90 2024-03-17T23:47:13.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.649+0000 7f0bb3fc6640 1 -- 172.21.15.150:0/1243896880 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f0bac071ce0 con 0x7f0bac09ad90 2024-03-17T23:47:13.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.652+0000 7f0ba27fc640 1 -- 172.21.15.150:0/1243896880 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+172586 (secure 0 0 0) 0x7f0b9c07afa0 con 0x7f0bac09ad90 2024-03-17T23:47:13.654 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:13.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.654+0000 7f0bb3fc6640 1 -- 172.21.15.150:0/1243896880 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b9006bbd0 msgr2=0x7f0b9006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:13.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.654+0000 7f0bb3fc6640 1 --2- 172.21.15.150:0/1243896880 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b9006bbd0 0x7f0b9006e090 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8009560 tx=0x7f0ba8009190 comp rx=0 tx=0).stop 2024-03-17T23:47:13.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.654+0000 7f0bb3fc6640 1 -- 172.21.15.150:0/1243896880 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bac09ad90 msgr2=0x7f0bac0cb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:13.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.654+0000 7f0bb3fc6640 1 --2- 172.21.15.150:0/1243896880 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bac09ad90 0x7f0bac0cb170 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c02bee0 tx=0x7f0b9c0046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:13.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.655+0000 7f0bb3fc6640 1 -- 172.21.15.150:0/1243896880 shutdown_connections 2024-03-17T23:47:13.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.655+0000 7f0bb3fc6640 1 --2- 172.21.15.150:0/1243896880 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b9006bbd0 0x7f0b9006e090 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:13.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.655+0000 7f0bb3fc6640 1 --2- 172.21.15.150:0/1243896880 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0bac15fcd0 0x7f0bac1543d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:13.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.655+0000 7f0bb3fc6640 1 --2- 172.21.15.150:0/1243896880 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0bac09ad90 0x7f0bac0cb170 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:13.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.655+0000 7f0bb3fc6640 1 --2- 172.21.15.150:0/1243896880 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0bac09a470 0x7f0bac15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:13.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.655+0000 7f0bb3fc6640 1 -- 172.21.15.150:0/1243896880 >> 172.21.15.150:0/1243896880 conn(0x7f0bac0b3630 msgr2=0x7f0bac15d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:13.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.655+0000 7f0bb3fc6640 1 -- 172.21.15.150:0/1243896880 shutdown_connections 2024-03-17T23:47:13.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.655+0000 7f0bb3fc6640 1 -- 172.21.15.150:0/1243896880 wait complete. 2024-03-17T23:47:13.666 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":5,"flags":0,"active_gid":4106,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":4239152602},{"type":"v1","addr":"172.21.15.150:6833","nonce":4239152602}]},"active_addr":"172.21.15.150:6833/4239152602","active_change":"2024-03-17T23:46:35.587316+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4100,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":3998022422}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":2490443399}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":3732018807}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":1800672503}]}]} 2024-03-17T23:47:13.669 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-17T23:47:13.669 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-17T23:47:13.669 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:13.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.770+0000 7f2c70759640 1 Processor -- start 2024-03-17T23:47:13.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.770+0000 7f2c70759640 1 -- start start 2024-03-17T23:47:13.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.771+0000 7f2c70759640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2c68155320 0x7f2c6815bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:13.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.771+0000 7f2c70759640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c680cf3b0 con 0x7f2c680cd420 2024-03-17T23:47:13.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.771+0000 7f2c70759640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c680cf690 con 0x7f2c6806dd30 2024-03-17T23:47:13.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.771+0000 7f2c70759640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f2c6815c550 con 0x7f2c68155320 2024-03-17T23:47:13.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.771+0000 7f2c6e4f5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2c68155320 0x7f2c6815bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:13.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.771+0000 7f2c6ecf6640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f2c680cd420 0x7f2c6815ccc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59354/0 (socket says 172.21.15.150:59354) 2024-03-17T23:47:13.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.771+0000 7f2c6ecf6640 1 -- 172.21.15.150:0/3838636911 learned_addr learned my addr 172.21.15.150:0/3838636911 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:13.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.771+0000 7f2c6e4f5640 1 -- 172.21.15.150:0/3838636911 >> v1:172.21.15.157:6789/0 conn(0x7f2c6806dd30 legacy=0x7f2c680ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:13.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.771+0000 7f2c6e4f5640 1 -- 172.21.15.150:0/3838636911 >> v1:172.21.15.150:6789/0 conn(0x7f2c680cd420 legacy=0x7f2c6815ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:13.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.771+0000 7f2c6d4f3640 1 -- 172.21.15.150:0/3838636911 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3118517747 0 0) 0x7f2c680cf690 con 0x7f2c6806dd30 2024-03-17T23:47:13.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.771+0000 7f2c6e4f5640 1 -- 172.21.15.150:0/3838636911 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6815c7a0 con 0x7f2c68155320 2024-03-17T23:47:13.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.772+0000 7f2c6e4f5640 1 --2- 172.21.15.150:0/3838636911 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c68155320 0x7f2c6815bfd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2c640097a0 tx=0x7f2c6402b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3c04c927cc87321 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:13.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.772+0000 7f2c6d4f3640 1 -- 172.21.15.150:0/3838636911 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c64037070 con 0x7f2c68155320 2024-03-17T23:47:13.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.772+0000 7f2c6d4f3640 1 -- 172.21.15.150:0/3838636911 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c64030710 con 0x7f2c68155320 2024-03-17T23:47:13.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.772+0000 7f2c6d4f3640 1 -- 172.21.15.150:0/3838636911 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c64030ae0 con 0x7f2c68155320 2024-03-17T23:47:13.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.773+0000 7f2c70759640 1 -- 172.21.15.150:0/3838636911 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c68155320 msgr2=0x7f2c6815bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:13.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.773+0000 7f2c70759640 1 --2- 172.21.15.150:0/3838636911 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c68155320 0x7f2c6815bfd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2c640097a0 tx=0x7f2c6402b790 comp rx=0 tx=0).stop 2024-03-17T23:47:13.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.773+0000 7f2c70759640 1 -- 172.21.15.150:0/3838636911 shutdown_connections 2024-03-17T23:47:13.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.773+0000 7f2c70759640 1 --2- 172.21.15.150:0/3838636911 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c68155320 0x7f2c6815bfd0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:13.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.773+0000 7f2c70759640 1 -- 172.21.15.150:0/3838636911 >> 172.21.15.150:0/3838636911 conn(0x7f2c6806b190 msgr2=0x7f2c6806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:13.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.773+0000 7f2c70759640 1 -- 172.21.15.150:0/3838636911 shutdown_connections 2024-03-17T23:47:13.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.773+0000 7f2c70759640 1 -- 172.21.15.150:0/3838636911 wait complete. 2024-03-17T23:47:13.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.774+0000 7f2c70759640 1 Processor -- start 2024-03-17T23:47:13.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.774+0000 7f2c70759640 1 -- start start 2024-03-17T23:47:13.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.775+0000 7f2c70759640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c6806dd30 0x7f2c681402e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:13.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.775+0000 7f2c70759640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c680cd420 0x7f2c6814def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:13.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.775+0000 7f2c70759640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c68155320 0x7f2c681f1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:13.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.775+0000 7f2c70759640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2c68155700 con 0x7f2c6806dd30 2024-03-17T23:47:13.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.775+0000 7f2c70759640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6806e190 con 0x7f2c680cd420 2024-03-17T23:47:13.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.775+0000 7f2c70759640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2c68155880 con 0x7f2c68155320 2024-03-17T23:47:13.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.775+0000 7f2c6e4f5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c6806dd30 0x7f2c681402e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:13.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.775+0000 7f2c6e4f5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c6806dd30 0x7f2c681402e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36314/0 (socket says 172.21.15.150:36314) 2024-03-17T23:47:13.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.775+0000 7f2c6e4f5640 1 -- 172.21.15.150:0/3243439756 learned_addr learned my addr 172.21.15.150:0/3243439756 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:13.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.775+0000 7f2c6ecf6640 1 --2- 172.21.15.150:0/3243439756 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c68155320 0x7f2c681f1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:13.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.775+0000 7f2c6dcf4640 1 --2- 172.21.15.150:0/3243439756 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c680cd420 0x7f2c6814def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:13.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.775+0000 7f2c6e4f5640 1 -- 172.21.15.150:0/3243439756 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c68155320 msgr2=0x7f2c681f1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:13.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.775+0000 7f2c6e4f5640 1 --2- 172.21.15.150:0/3243439756 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c68155320 0x7f2c681f1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:13.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.775+0000 7f2c6e4f5640 1 -- 172.21.15.150:0/3243439756 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c680cd420 msgr2=0x7f2c6814def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:13.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.775+0000 7f2c6e4f5640 1 --2- 172.21.15.150:0/3243439756 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c680cd420 0x7f2c6814def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:13.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.775+0000 7f2c6e4f5640 1 -- 172.21.15.150:0/3243439756 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6814d430 con 0x7f2c6806dd30 2024-03-17T23:47:13.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.775+0000 7f2c6e4f5640 1 --2- 172.21.15.150:0/3243439756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c6806dd30 0x7f2c681402e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2c64035040 tx=0x7f2c640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:13.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.775+0000 7f2c5effd640 1 -- 172.21.15.150:0/3243439756 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c64037040 con 0x7f2c6806dd30 2024-03-17T23:47:13.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.776+0000 7f2c5effd640 1 -- 172.21.15.150:0/3243439756 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c64004030 con 0x7f2c6806dd30 2024-03-17T23:47:13.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.776+0000 7f2c70759640 1 -- 172.21.15.150:0/3243439756 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c681f2a60 con 0x7f2c6806dd30 2024-03-17T23:47:13.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.776+0000 7f2c5effd640 1 -- 172.21.15.150:0/3243439756 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c64004420 con 0x7f2c6806dd30 2024-03-17T23:47:13.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.776+0000 7f2c70759640 1 -- 172.21.15.150:0/3243439756 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c6814d6f0 con 0x7f2c6806dd30 2024-03-17T23:47:13.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.777+0000 7f2c70759640 1 -- 172.21.15.150:0/3243439756 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c680681a0 con 0x7f2c6806dd30 2024-03-17T23:47:13.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.777+0000 7f2c5effd640 1 -- 172.21.15.150:0/3243439756 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2c64032940 con 0x7f2c6806dd30 2024-03-17T23:47:13.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.777+0000 7f2c5effd640 1 --2- 172.21.15.150:0/3243439756 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2c4c06bc00 0x7f2c4c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:13.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.777+0000 7f2c5effd640 1 -- 172.21.15.150:0/3243439756 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f2c640acaa0 con 0x7f2c6806dd30 2024-03-17T23:47:13.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.777+0000 7f2c6dcf4640 1 --2- 172.21.15.150:0/3243439756 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2c4c06bc00 0x7f2c4c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:13.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.778+0000 7f2c6dcf4640 1 --2- 172.21.15.150:0/3243439756 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2c4c06bc00 0x7f2c4c06e0c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2c681efa90 tx=0x7f2c58009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:13.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.782+0000 7f2c5effd640 1 -- 172.21.15.150:0/3243439756 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2c64042020 con 0x7f2c6806dd30 2024-03-17T23:47:13.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.957+0000 7f2c70759640 1 -- 172.21.15.150:0/3243439756 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2c6806f390 con 0x7f2c6806dd30 2024-03-17T23:47:13.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.958+0000 7f2c5effd640 1 -- 172.21.15.150:0/3243439756 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18550 (secure 0 0 0) 0x7f2c6407ac00 con 0x7f2c6806dd30 2024-03-17T23:47:13.959 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:13.959 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:01.940019+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:13.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.960+0000 7f2c70759640 1 -- 172.21.15.150:0/3243439756 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2c4c06bc00 msgr2=0x7f2c4c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:13.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.960+0000 7f2c70759640 1 --2- 172.21.15.150:0/3243439756 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2c4c06bc00 0x7f2c4c06e0c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2c681efa90 tx=0x7f2c58009040 comp rx=0 tx=0).stop 2024-03-17T23:47:13.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.960+0000 7f2c70759640 1 -- 172.21.15.150:0/3243439756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c6806dd30 msgr2=0x7f2c681402e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:13.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.960+0000 7f2c70759640 1 --2- 172.21.15.150:0/3243439756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c6806dd30 0x7f2c681402e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2c64035040 tx=0x7f2c640046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:13.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.960+0000 7f2c70759640 1 -- 172.21.15.150:0/3243439756 shutdown_connections 2024-03-17T23:47:13.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.960+0000 7f2c70759640 1 --2- 172.21.15.150:0/3243439756 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c68155320 0x7f2c681f1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:13.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.960+0000 7f2c70759640 1 --2- 172.21.15.150:0/3243439756 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2c4c06bc00 0x7f2c4c06e0c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:13.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.960+0000 7f2c70759640 1 --2- 172.21.15.150:0/3243439756 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c680cd420 0x7f2c6814def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:13.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.960+0000 7f2c70759640 1 --2- 172.21.15.150:0/3243439756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c6806dd30 0x7f2c681402e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:13.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.960+0000 7f2c70759640 1 -- 172.21.15.150:0/3243439756 >> 172.21.15.150:0/3243439756 conn(0x7f2c6806b190 msgr2=0x7f2c680b49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:13.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.960+0000 7f2c70759640 1 -- 172.21.15.150:0/3243439756 shutdown_connections 2024-03-17T23:47:13.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:13.960+0000 7f2c70759640 1 -- 172.21.15.150:0/3243439756 wait complete. 2024-03-17T23:47:13.971 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-17T23:47:13.972 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:14.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.071+0000 7f6db3d37640 1 Processor -- start 2024-03-17T23:47:14.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.072+0000 7f6db3d37640 1 -- start start 2024-03-17T23:47:14.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.072+0000 7f6db3d37640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6dac0b5730 0x7f6dac0b2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.072+0000 7f6db3d37640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6dac0b3c30 0x7f6dac0b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.072+0000 7f6db3d37640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6dac0d1230 con 0x7f6dac06b190 2024-03-17T23:47:14.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.072+0000 7f6db3d37640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f6dac0b2560 con 0x7f6dac0b3c30 2024-03-17T23:47:14.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.072+0000 7f6db3d37640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6dac0b5c10 con 0x7f6dac0b5730 2024-03-17T23:47:14.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.072+0000 7f6db22d4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6dac0b3c30 0x7f6dac0b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.073+0000 7f6db22d4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6dac0b3c30 0x7f6dac0b01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36326/0 (socket says 172.21.15.150:36326) 2024-03-17T23:47:14.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.073+0000 7f6db1ad3640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f6dac06b190 0x7f6dac06b570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59366/0 (socket says 172.21.15.150:59366) 2024-03-17T23:47:14.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.073+0000 7f6db22d4640 1 -- 172.21.15.150:0/1573759556 learned_addr learned my addr 172.21.15.150:0/1573759556 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.073+0000 7f6db12d2640 1 --2- 172.21.15.150:0/1573759556 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6dac0b5730 0x7f6dac0b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.073+0000 7f6db22d4640 1 -- 172.21.15.150:0/1573759556 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6dac0b5730 msgr2=0x7f6dac0b2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.073+0000 7f6db22d4640 1 --2- 172.21.15.150:0/1573759556 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6dac0b5730 0x7f6dac0b2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.073+0000 7f6db22d4640 1 -- 172.21.15.150:0/1573759556 >> v1:172.21.15.150:6789/0 conn(0x7f6dac06b190 legacy=0x7f6dac06b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.073+0000 7f6db22d4640 1 -- 172.21.15.150:0/1573759556 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dac0d14d0 con 0x7f6dac0b3c30 2024-03-17T23:47:14.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.073+0000 7f6db0ad1640 1 -- 172.21.15.150:0/1573759556 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4279659577 0 0) 0x7f6dac0d1230 con 0x7f6dac06b190 2024-03-17T23:47:14.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.073+0000 7f6db22d4640 1 --2- 172.21.15.150:0/1573759556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dac0b3c30 0x7f6dac0b01a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6da4009e50 tx=0x7f6da402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa0a96b0322fb323 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.074+0000 7f6db0ad1640 1 -- 172.21.15.150:0/1573759556 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da402ca70 con 0x7f6dac0b3c30 2024-03-17T23:47:14.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.074+0000 7f6db0ad1640 1 -- 172.21.15.150:0/1573759556 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6da402cc10 con 0x7f6dac0b3c30 2024-03-17T23:47:14.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.074+0000 7f6db0ad1640 1 -- 172.21.15.150:0/1573759556 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da40343f0 con 0x7f6dac0b3c30 2024-03-17T23:47:14.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.074+0000 7f6db3d37640 1 -- 172.21.15.150:0/1573759556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dac0b3c30 msgr2=0x7f6dac0b01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.074+0000 7f6db3d37640 1 --2- 172.21.15.150:0/1573759556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dac0b3c30 0x7f6dac0b01a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6da4009e50 tx=0x7f6da402bed0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.074+0000 7f6db3d37640 1 -- 172.21.15.150:0/1573759556 shutdown_connections 2024-03-17T23:47:14.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.074+0000 7f6db3d37640 1 --2- 172.21.15.150:0/1573759556 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6dac0b5730 0x7f6dac0b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.074+0000 7f6db3d37640 1 --2- 172.21.15.150:0/1573759556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dac0b3c30 0x7f6dac0b01a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.074+0000 7f6db3d37640 1 -- 172.21.15.150:0/1573759556 >> 172.21.15.150:0/1573759556 conn(0x7f6dac0b6170 msgr2=0x7f6dac099a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:14.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.074+0000 7f6db3d37640 1 -- 172.21.15.150:0/1573759556 shutdown_connections 2024-03-17T23:47:14.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.074+0000 7f6db3d37640 1 -- 172.21.15.150:0/1573759556 wait complete. 2024-03-17T23:47:14.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.075+0000 7f6db3d37640 1 Processor -- start 2024-03-17T23:47:14.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.075+0000 7f6db3d37640 1 -- start start 2024-03-17T23:47:14.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.076+0000 7f6db3d37640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6dac06b190 0x7f6dac15ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.076+0000 7f6db3d37640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6dac0b3c30 0x7f6dac1592e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.076+0000 7f6db3d37640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dac0b5730 0x7f6dac154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.076+0000 7f6db3d37640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6dac09a380 con 0x7f6dac0b5730 2024-03-17T23:47:14.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.076+0000 7f6db3d37640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6dac0b5ef0 con 0x7f6dac06b190 2024-03-17T23:47:14.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.076+0000 7f6db3d37640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6dac15e260 con 0x7f6dac0b3c30 2024-03-17T23:47:14.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.076+0000 7f6db22d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dac0b5730 0x7f6dac154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.076+0000 7f6db22d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dac0b5730 0x7f6dac154d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36336/0 (socket says 172.21.15.150:36336) 2024-03-17T23:47:14.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.076+0000 7f6db22d4640 1 -- 172.21.15.150:0/1841857635 learned_addr learned my addr 172.21.15.150:0/1841857635 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.076+0000 7f6db1ad3640 1 --2- 172.21.15.150:0/1841857635 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6dac06b190 0x7f6dac15ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.076+0000 7f6db12d2640 1 --2- 172.21.15.150:0/1841857635 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6dac0b3c30 0x7f6dac1592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.076+0000 7f6db22d4640 1 -- 172.21.15.150:0/1841857635 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6dac0b3c30 msgr2=0x7f6dac1592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.077+0000 7f6db22d4640 1 --2- 172.21.15.150:0/1841857635 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6dac0b3c30 0x7f6dac1592e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.077+0000 7f6db22d4640 1 -- 172.21.15.150:0/1841857635 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6dac06b190 msgr2=0x7f6dac15ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.077+0000 7f6db22d4640 1 --2- 172.21.15.150:0/1841857635 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6dac06b190 0x7f6dac15ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.077+0000 7f6db22d4640 1 -- 172.21.15.150:0/1841857635 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dac158240 con 0x7f6dac0b5730 2024-03-17T23:47:14.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.077+0000 7f6db22d4640 1 --2- 172.21.15.150:0/1841857635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dac0b5730 0x7f6dac154d30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6da4009e50 tx=0x7f6da4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.077+0000 7f6d9e7fc640 1 -- 172.21.15.150:0/1841857635 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da4043020 con 0x7f6dac0b5730 2024-03-17T23:47:14.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.077+0000 7f6d9e7fc640 1 -- 172.21.15.150:0/1841857635 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6da403e450 con 0x7f6dac0b5730 2024-03-17T23:47:14.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.077+0000 7f6db3d37640 1 -- 172.21.15.150:0/1841857635 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dac158520 con 0x7f6dac0b5730 2024-03-17T23:47:14.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.077+0000 7f6d9e7fc640 1 -- 172.21.15.150:0/1841857635 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da4009510 con 0x7f6dac0b5730 2024-03-17T23:47:14.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.077+0000 7f6db3d37640 1 -- 172.21.15.150:0/1841857635 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dac158ae0 con 0x7f6dac0b5730 2024-03-17T23:47:14.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.078+0000 7f6db3d37640 1 -- 172.21.15.150:0/1841857635 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dac0681a0 con 0x7f6dac0b5730 2024-03-17T23:47:14.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.078+0000 7f6d9e7fc640 1 -- 172.21.15.150:0/1841857635 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6da4046020 con 0x7f6dac0b5730 2024-03-17T23:47:14.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.079+0000 7f6d9e7fc640 1 --2- 172.21.15.150:0/1841857635 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6d9006bc00 0x7f6d9006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.079+0000 7f6d9e7fc640 1 -- 172.21.15.150:0/1841857635 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f6da40ad300 con 0x7f6dac0b5730 2024-03-17T23:47:14.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.079+0000 7f6db1ad3640 1 --2- 172.21.15.150:0/1841857635 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6d9006bc00 0x7f6d9006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.079+0000 7f6db1ad3640 1 --2- 172.21.15.150:0/1841857635 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6d9006bc00 0x7f6d9006e0c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6da8000f60 tx=0x7f6da8007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.083+0000 7f6d9e7fc640 1 -- 172.21.15.150:0/1841857635 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6da404d030 con 0x7f6dac0b5730 2024-03-17T23:47:14.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.256+0000 7f6db3d37640 1 -- 172.21.15.150:0/1841857635 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6dac06dd50 con 0x7f6dac0b5730 2024-03-17T23:47:14.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.257+0000 7f6d9e7fc640 1 -- 172.21.15.150:0/1841857635 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18550 (secure 0 0 0) 0x7f6da407b4f0 con 0x7f6dac0b5730 2024-03-17T23:47:14.258 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:14.258 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:01.940019+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:14.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.259+0000 7f6db3d37640 1 -- 172.21.15.150:0/1841857635 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6d9006bc00 msgr2=0x7f6d9006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.259+0000 7f6db3d37640 1 --2- 172.21.15.150:0/1841857635 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6d9006bc00 0x7f6d9006e0c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6da8000f60 tx=0x7f6da8007d50 comp rx=0 tx=0).stop 2024-03-17T23:47:14.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.259+0000 7f6db3d37640 1 -- 172.21.15.150:0/1841857635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dac0b5730 msgr2=0x7f6dac154d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.259+0000 7f6db3d37640 1 --2- 172.21.15.150:0/1841857635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dac0b5730 0x7f6dac154d30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6da4009e50 tx=0x7f6da4004a40 comp rx=0 tx=0).stop 2024-03-17T23:47:14.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.259+0000 7f6db3d37640 1 -- 172.21.15.150:0/1841857635 shutdown_connections 2024-03-17T23:47:14.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.259+0000 7f6db3d37640 1 --2- 172.21.15.150:0/1841857635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dac0b5730 0x7f6dac154d30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.259+0000 7f6db3d37640 1 --2- 172.21.15.150:0/1841857635 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6dac0b3c30 0x7f6dac1592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.259+0000 7f6db3d37640 1 --2- 172.21.15.150:0/1841857635 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6d9006bc00 0x7f6d9006e0c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.259+0000 7f6db3d37640 1 --2- 172.21.15.150:0/1841857635 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6dac06b190 0x7f6dac15ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.259+0000 7f6db3d37640 1 -- 172.21.15.150:0/1841857635 >> 172.21.15.150:0/1841857635 conn(0x7f6dac0b6170 msgr2=0x7f6dac15d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:14.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.259+0000 7f6db3d37640 1 -- 172.21.15.150:0/1841857635 shutdown_connections 2024-03-17T23:47:14.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.259+0000 7f6db3d37640 1 -- 172.21.15.150:0/1841857635 wait complete. 2024-03-17T23:47:14.271 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-17T23:47:14.272 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-17T23:47:14.272 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-17T23:47:14.273 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-17T23:47:14.273 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-17T23:47:14.273 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-17T23:47:14.274 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-17T23:47:14.274 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-17T23:47:14.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.445+0000 7f1b2c8bd640 1 Processor -- start 2024-03-17T23:47:14.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.446+0000 7f1b2c8bd640 1 -- start start 2024-03-17T23:47:14.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.446+0000 7f6be0116640 1 Processor -- start 2024-03-17T23:47:14.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.445+0000 7f9699064640 1 Processor -- start 2024-03-17T23:47:14.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.445+0000 7f9699064640 1 -- start start 2024-03-17T23:47:14.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.446+0000 7fafd49df640 1 Processor -- start 2024-03-17T23:47:14.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.446+0000 7f6be0116640 1 -- start start 2024-03-17T23:47:14.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.446+0000 7f054bd87640 1 Processor -- start 2024-03-17T23:47:14.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.446+0000 7fafd49df640 1 -- start start 2024-03-17T23:47:14.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.447+0000 7f054bd87640 1 -- start start 2024-03-17T23:47:14.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.447+0000 7fd567223640 1 Processor -- start 2024-03-17T23:47:14.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.447+0000 7f9622086640 1 Processor -- start 2024-03-17T23:47:14.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.448+0000 7f9622086640 1 -- start start 2024-03-17T23:47:14.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.446+0000 7f9699064640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f968c00c0a0 0x7f968c038080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.447+0000 7f9699064640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f968c0544c0 con 0x7f968c053c70 2024-03-17T23:47:14.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.447+0000 7f9699064640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f968c0541e0 con 0x7f968c033ef0 2024-03-17T23:47:14.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.447+0000 7f9699064640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f968c005f30 con 0x7f968c00c0a0 2024-03-17T23:47:14.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.447+0000 7f9693fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f968c00c0a0 0x7f968c038080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.447+0000 7f9693fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f968c00c0a0 0x7f968c038080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:51162/0 (socket says 172.21.15.150:51162) 2024-03-17T23:47:14.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.447+0000 7f9693fff640 1 -- 172.21.15.150:0/2508009913 learned_addr learned my addr 172.21.15.150:0/2508009913 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.447+0000 7f1b2c8bd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1b280d2c10 0x7f1b28166800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.447+0000 7f1b2659c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1b280d2c10 0x7f1b28166800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.447+0000 7f1b2659c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1b280d2c10 0x7f1b28166800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:33864/0 (socket says 172.21.15.150:33864) 2024-03-17T23:47:14.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.447+0000 7f1b2659c640 1 -- 172.21.15.150:0/107601993 learned_addr learned my addr 172.21.15.150:0/107601993 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.447+0000 7f1b2c8bd640 1 --2- 172.21.15.150:0/107601993 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b28162e30 0x7f1b28163240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.448+0000 7f1b2c8bd640 1 -- 172.21.15.150:0/107601993 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b280d20f0 con 0x7f1b280d3810 2024-03-17T23:47:14.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.448+0000 7f1b26d9d640 1 --2- 172.21.15.150:0/107601993 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b28162e30 0x7f1b28163240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.448+0000 7f1b2c8bd640 1 -- 172.21.15.150:0/107601993 --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f1b281629a0 con 0x7f1b280d2c10 2024-03-17T23:47:14.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.448+0000 7f1b2c8bd640 1 -- 172.21.15.150:0/107601993 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1b2815cce0 con 0x7f1b28162e30 2024-03-17T23:47:14.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.448+0000 7f054bd87640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f054415be30 0x7f05440cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.448+0000 7f054bd87640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05440ccb30 con 0x7f05440cc6c0 2024-03-17T23:47:14.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.448+0000 7f054bd87640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05440ccda0 con 0x7f05440b16c0 2024-03-17T23:47:14.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.448+0000 7f054a324640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f05440b16c0 0x7f05440cf9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:37840/0 (socket says 172.21.15.150:37840) 2024-03-17T23:47:14.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.448+0000 7f054a324640 1 -- 172.21.15.150:0/1649304853 learned_addr learned my addr 172.21.15.150:0/1649304853 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:14.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.448+0000 7f054bd87640 1 -- 172.21.15.150:0/1649304853 --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f05440cd080 con 0x7f054415be30 2024-03-17T23:47:14.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.447+0000 7fd567223640 1 -- start start 2024-03-17T23:47:14.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.448+0000 7fd567223640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd56015c860 0x7fd560166750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.448+0000 7fd567223640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd5600b45c0 0x7fd560071a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.448+0000 7fd567223640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5600b3910 con 0x7fd560071fa0 2024-03-17T23:47:14.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.448+0000 7fd567223640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd5600b4d60 con 0x7fd56015c860 2024-03-17T23:47:14.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.448+0000 7fd567223640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd560162740 con 0x7fd5600b45c0 2024-03-17T23:47:14.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.447+0000 7f9693fff640 1 -- 172.21.15.150:0/2508009913 >> v1:172.21.15.157:6789/0 conn(0x7f968c033ef0 legacy=0x7f968c04fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.447+0000 7f9693fff640 1 -- 172.21.15.150:0/2508009913 >> v1:172.21.15.150:6789/0 conn(0x7f968c053c70 legacy=0x7f968c0df220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.447+0000 7f9693fff640 1 -- 172.21.15.150:0/2508009913 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f968c006210 con 0x7f968c00c0a0 2024-03-17T23:47:14.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.448+0000 7f6be0116640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6bd815ae40 0x7f6bd80cb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.448+0000 7f6bddeb2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6bd815ae40 0x7f6bd80cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.448+0000 7f6bddeb2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6bd815ae40 0x7f6bd80cb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36344/0 (socket says 172.21.15.150:36344) 2024-03-17T23:47:14.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.448+0000 7f6bddeb2640 1 -- 172.21.15.150:0/3517937914 learned_addr learned my addr 172.21.15.150:0/3517937914 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.448+0000 7f6be0116640 1 --2- 172.21.15.150:0/3517937914 >> v2:172.21.15.157:3300/0 conn(0x7f6bd80cf2a0 0x7f6bd80ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.448+0000 7f6be0116640 1 -- 172.21.15.150:0/3517937914 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bd81639d0 con 0x7f6bd80cd100 2024-03-17T23:47:14.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.448+0000 7f6be0116640 1 -- 172.21.15.150:0/3517937914 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f6bd80cc730 con 0x7f6bd815ae40 2024-03-17T23:47:14.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.448+0000 7f6be0116640 1 -- 172.21.15.150:0/3517937914 --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f6bd80cbb20 con 0x7f6bd80cf2a0 2024-03-17T23:47:14.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7f9693fff640 1 --2- 172.21.15.150:0/2508009913 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f968c00c0a0 0x7f968c038080 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9694075ec0 tx=0x7f96940af7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5cc1791fd114ddf server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.447+0000 7fafd49df640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fafd0072ce0 0x7fafd013cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.447+0000 7fafd49df640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fafd00cf640 0x7fafd013f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.447+0000 7fafd49df640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafd00ce490 con 0x7fafd006dd30 2024-03-17T23:47:14.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.447+0000 7fafd49df640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fafd00cca10 con 0x7fafd0072ce0 2024-03-17T23:47:14.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.447+0000 7fafd49df640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fafd00cccf0 con 0x7fafd00cf640 2024-03-17T23:47:14.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7fafce59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fafd0072ce0 0x7fafd013cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7fafce59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fafd0072ce0 0x7fafd013cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36340/0 (socket says 172.21.15.150:36340) 2024-03-17T23:47:14.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7fafce59c640 1 -- 172.21.15.150:0/2611187550 learned_addr learned my addr 172.21.15.150:0/2611187550 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7fd566a22640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd560071fa0 0x7fd56006dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59406/0 (socket says 172.21.15.150:59406) 2024-03-17T23:47:14.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7fd566a22640 1 -- 172.21.15.150:0/549871701 learned_addr learned my addr 172.21.15.150:0/549871701 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:14.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7fd565a20640 1 --2- 172.21.15.150:0/549871701 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd5600b45c0 0x7fd560071a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7f9622086640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f961c140340 0x7f961c0cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7fafced9d640 1 --2- 172.21.15.150:0/2611187550 >> v2:172.21.15.157:3300/0 conn(0x7fafd00cf640 0x7fafd013f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7f1b26d9d640 1 -- 172.21.15.150:0/107601993 >> v2:172.21.15.157:3300/0 conn(0x7f1b280d2c10 msgr2=0x7f1b28166800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7f1b26d9d640 1 --2- 172.21.15.150:0/107601993 >> v2:172.21.15.157:3300/0 conn(0x7f1b280d2c10 0x7f1b28166800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7f1b26d9d640 1 -- 172.21.15.150:0/107601993 >> v1:172.21.15.150:6789/0 conn(0x7f1b280d3810 legacy=0x7f1b280d19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7f1b26d9d640 1 -- 172.21.15.150:0/107601993 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b28163780 con 0x7f1b28162e30 2024-03-17T23:47:14.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7fd565a20640 1 -- 172.21.15.150:0/549871701 >> v2:172.21.15.157:3300/0 conn(0x7fd56015c860 msgr2=0x7fd560166750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-17T23:47:14.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7fd565a20640 1 --2- 172.21.15.150:0/549871701 >> v2:172.21.15.157:3300/0 conn(0x7fd56015c860 0x7fd560166750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7fd565a20640 1 -- 172.21.15.150:0/549871701 >> v1:172.21.15.150:6789/0 conn(0x7fd560071fa0 legacy=0x7fd56006dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7fd565a20640 1 -- 172.21.15.150:0/549871701 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd560162a20 con 0x7fd5600b45c0 2024-03-17T23:47:14.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7f9622086640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f961c06e240 0x7f961c071ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7fafced9d640 1 -- 172.21.15.150:0/2611187550 >> v2:172.21.15.150:3300/0 conn(0x7fafd0072ce0 msgr2=0x7fafd013cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7fafced9d640 1 --2- 172.21.15.150:0/2611187550 >> v2:172.21.15.150:3300/0 conn(0x7fafd0072ce0 0x7fafd013cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7fafced9d640 1 -- 172.21.15.150:0/2611187550 >> v1:172.21.15.150:6789/0 conn(0x7fafd006dd30 legacy=0x7fafd00cef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7f6bdd6b1640 1 --2- 172.21.15.150:0/3517937914 >> v2:172.21.15.157:3300/0 conn(0x7f6bd80cf2a0 0x7f6bd80ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7f6bddeb2640 1 -- 172.21.15.150:0/3517937914 >> v2:172.21.15.157:3300/0 conn(0x7f6bd80cf2a0 msgr2=0x7f6bd80ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7f6bddeb2640 1 --2- 172.21.15.150:0/3517937914 >> v2:172.21.15.157:3300/0 conn(0x7f6bd80cf2a0 0x7f6bd80ccbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7f6bddeb2640 1 -- 172.21.15.150:0/3517937914 >> v1:172.21.15.157:6789/0 conn(0x7f6bd80cd100 legacy=0x7f6bd80d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7f6bddeb2640 1 -- 172.21.15.150:0/3517937914 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd80cbe00 con 0x7f6bd815ae40 2024-03-17T23:47:14.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7fafced9d640 1 -- 172.21.15.150:0/2611187550 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd00ccfd0 con 0x7fafd00cf640 2024-03-17T23:47:14.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7f9622086640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f961c06e8a0 con 0x7f961c071a60 2024-03-17T23:47:14.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7f9692ffd640 1 -- 172.21.15.150:0/2508009913 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f969406e860 con 0x7f968c00c0a0 2024-03-17T23:47:14.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7f9692ffd640 1 -- 172.21.15.150:0/2508009913 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f969406ea40 con 0x7f968c00c0a0 2024-03-17T23:47:14.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.449+0000 7f9692ffd640 1 -- 172.21.15.150:0/2508009913 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96940b7400 con 0x7f968c00c0a0 2024-03-17T23:47:14.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.450+0000 7f9620883640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f961c071a60 0x7f961c06db30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:37856/0 (socket says 172.21.15.150:37856) 2024-03-17T23:47:14.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.450+0000 7f9620883640 1 -- 172.21.15.150:0/634500986 learned_addr learned my addr 172.21.15.150:0/634500986 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:14.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.450+0000 7f0548b21640 1 -- 172.21.15.150:0/1649304853 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3545044539 0 0) 0x7f05440ccda0 con 0x7f05440b16c0 2024-03-17T23:47:14.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.450+0000 7f0548b21640 1 -- 172.21.15.150:0/1649304853 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0530003520 con 0x7f05440b16c0 2024-03-17T23:47:14.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.450+0000 7f9621885640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f961c06e240 0x7f961c071ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.450+0000 7f9621084640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f961c140340 0x7f961c0cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.450+0000 7f0548b21640 1 -- 172.21.15.150:0/1649304853 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3555608854 0 0) 0x7f05440ccb30 con 0x7f05440cc6c0 2024-03-17T23:47:14.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.450+0000 7f0548b21640 1 -- 172.21.15.150:0/1649304853 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05440ccda0 con 0x7f05440cc6c0 2024-03-17T23:47:14.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.450+0000 7f9622086640 1 -- 172.21.15.150:0/634500986 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f961c072400 con 0x7f961c140340 2024-03-17T23:47:14.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.450+0000 7f9622086640 1 -- 172.21.15.150:0/634500986 --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f961c0ce1e0 con 0x7f961c06e240 2024-03-17T23:47:14.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.450+0000 7f0549b23640 1 --2- 172.21.15.150:0/1649304853 >> v2:172.21.15.157:3300/0 conn(0x7f054415be30 0x7f05440cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.450+0000 7f6bddeb2640 1 --2- 172.21.15.150:0/3517937914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bd815ae40 0x7f6bd80cb440 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6bd80cc8b0 tx=0x7f6bd402b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2e82d3066405ac2 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.450+0000 7f9613fff640 1 -- 172.21.15.150:0/634500986 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 728442312 0 0) 0x7f961c06e8a0 con 0x7f961c071a60 2024-03-17T23:47:14.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.450+0000 7f0548b21640 1 -- 172.21.15.150:0/1649304853 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2948398756 0 0) 0x7f05440ccda0 con 0x7f05440cc6c0 2024-03-17T23:47:14.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.450+0000 7f0548b21640 1 -- 172.21.15.150:0/1649304853 >> v2:172.21.15.157:3300/0 conn(0x7f054415be30 msgr2=0x7f05440cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.450+0000 7f0548b21640 1 --2- 172.21.15.150:0/1649304853 >> v2:172.21.15.157:3300/0 conn(0x7f054415be30 0x7f05440cde40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.450+0000 7f0548b21640 1 -- 172.21.15.150:0/1649304853 >> v1:172.21.15.157:6789/0 conn(0x7f05440b16c0 legacy=0x7f05440cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.450+0000 7f1b26d9d640 1 --2- 172.21.15.150:0/107601993 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b28162e30 0x7f1b28163240 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c0097a0 tx=0x7f1b1c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbd8a6b529f1ded7 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f0548b21640 1 -- 172.21.15.150:0/1649304853 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05440cd360 con 0x7f05440cc6c0 2024-03-17T23:47:14.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f0548b21640 1 -- 172.21.15.150:0/1649304853 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f05340015a0 con 0x7f05440cc6c0 2024-03-17T23:47:14.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7fafced9d640 1 --2- 172.21.15.150:0/2611187550 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafd00cf640 0x7fafd013f6d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fafd006f930 tx=0x7fafc402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c39f7135a308731d server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f6bdceb0640 1 -- 172.21.15.150:0/3517937914 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4035020 con 0x7f6bd815ae40 2024-03-17T23:47:14.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f1b2559a640 1 -- 172.21.15.150:0/107601993 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c035020 con 0x7f1b28162e30 2024-03-17T23:47:14.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f0548b21640 1 -- 172.21.15.150:0/1649304853 >> v1:172.21.15.150:6789/0 conn(0x7f05440cc6c0 legacy=0x7f05440b0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f0548b21640 1 --2- 172.21.15.150:0/1649304853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0530003b90 0x7f0530003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f1b2559a640 1 -- 172.21.15.150:0/107601993 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b1c031830 con 0x7f1b28162e30 2024-03-17T23:47:14.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f1b2559a640 1 -- 172.21.15.150:0/107601993 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c031c00 con 0x7f1b28162e30 2024-03-17T23:47:14.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f6bdceb0640 1 -- 172.21.15.150:0/3517937914 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bd4031890 con 0x7f6bd815ae40 2024-03-17T23:47:14.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f0548b21640 1 --2- 172.21.15.150:0/1649304853 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0530004800 0x7f0530004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f6bdceb0640 1 -- 172.21.15.150:0/3517937914 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4031ca0 con 0x7f6bd815ae40 2024-03-17T23:47:14.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7fd565a20640 1 --2- 172.21.15.150:0/549871701 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd5600b45c0 0x7fd560071a60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd5500097a0 tx=0x7fd55002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0b13a9264377db server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f0548b21640 1 --2- 172.21.15.150:0/1649304853 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05300052e0 0x7f053000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f9613fff640 1 -- 172.21.15.150:0/634500986 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9608004630 con 0x7f961c071a60 2024-03-17T23:47:14.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7fafcd59a640 1 -- 172.21.15.150:0/2611187550 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc4034070 con 0x7fafd00cf640 2024-03-17T23:47:14.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f0549b23640 1 --2- 172.21.15.150:0/1649304853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0530003b90 0x7f0530003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f0548b21640 1 -- 172.21.15.150:0/1649304853 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f05440cd080 con 0x7f0530003b90 2024-03-17T23:47:14.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f0548b21640 1 -- 172.21.15.150:0/1649304853 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f053000c230 con 0x7f05300052e0 2024-03-17T23:47:14.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f0548b21640 1 -- 172.21.15.150:0/1649304853 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f053000c430 con 0x7f0530004800 2024-03-17T23:47:14.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f0549322640 1 --2- 172.21.15.150:0/1649304853 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0530004800 0x7f0530004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7fafcd59a640 1 -- 172.21.15.150:0/2611187550 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafc4030610 con 0x7fafd00cf640 2024-03-17T23:47:14.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7fafcd59a640 1 -- 172.21.15.150:0/2611187550 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc40309e0 con 0x7fafd00cf640 2024-03-17T23:47:14.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7fd56521f640 1 -- 172.21.15.150:0/549871701 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd550035020 con 0x7fd5600b45c0 2024-03-17T23:47:14.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7fd56521f640 1 -- 172.21.15.150:0/549871701 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd550031830 con 0x7fd5600b45c0 2024-03-17T23:47:14.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f9613fff640 1 -- 172.21.15.150:0/634500986 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2010999563 0 0) 0x7f9608004630 con 0x7f961c071a60 2024-03-17T23:47:14.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7fd56521f640 1 -- 172.21.15.150:0/549871701 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd550031c00 con 0x7fd5600b45c0 2024-03-17T23:47:14.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f9613fff640 1 -- 172.21.15.150:0/634500986 >> v2:172.21.15.157:3300/0 conn(0x7f961c06e240 msgr2=0x7f961c071ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f9613fff640 1 --2- 172.21.15.150:0/634500986 >> v2:172.21.15.157:3300/0 conn(0x7f961c06e240 0x7f961c071ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f9613fff640 1 -- 172.21.15.150:0/634500986 >> v2:172.21.15.150:3300/0 conn(0x7f961c140340 msgr2=0x7f961c0cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f9613fff640 1 --2- 172.21.15.150:0/634500986 >> v2:172.21.15.150:3300/0 conn(0x7f961c140340 0x7f961c0cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7ff36ef50640 1 Processor -- start 2024-03-17T23:47:14.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f9613fff640 1 -- 172.21.15.150:0/634500986 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f961c0ce4c0 con 0x7f961c071a60 2024-03-17T23:47:14.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f054a324640 1 --2- 172.21.15.150:0/1649304853 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05300052e0 0x7f053000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f0549322640 1 -- 172.21.15.150:0/1649304853 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05300052e0 msgr2=0x7f053000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f0549322640 1 --2- 172.21.15.150:0/1649304853 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05300052e0 0x7f053000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f0549322640 1 -- 172.21.15.150:0/1649304853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0530003b90 msgr2=0x7f0530003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f0549322640 1 --2- 172.21.15.150:0/1649304853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0530003b90 0x7f0530003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f0549322640 1 -- 172.21.15.150:0/1649304853 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05440ccda0 con 0x7f0530004800 2024-03-17T23:47:14.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.452+0000 7f054a324640 1 --2- 172.21.15.150:0/1649304853 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f05300052e0 0x7f053000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:14.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.452+0000 7f054a324640 1 -- 172.21.15.150:0/1649304853 reap_dead start 2024-03-17T23:47:14.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f9613fff640 1 -- 172.21.15.150:0/634500986 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f9618001670 con 0x7f961c071a60 2024-03-17T23:47:14.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f9613fff640 1 -- 172.21.15.150:0/634500986 >> v1:172.21.15.157:6789/0 conn(0x7f961c071a60 legacy=0x7f961c06db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.452+0000 7f9613fff640 1 --2- 172.21.15.150:0/634500986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9608004b10 0x7f9608004f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.452+0000 7f9613fff640 1 --2- 172.21.15.150:0/634500986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9608005910 0x7f9608005d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.452+0000 7f9613fff640 1 --2- 172.21.15.150:0/634500986 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9608006320 0x7f9608003440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.452+0000 7f9613fff640 1 -- 172.21.15.150:0/634500986 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f961c0ce1e0 con 0x7f9608005910 2024-03-17T23:47:14.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.452+0000 7f9613fff640 1 -- 172.21.15.150:0/634500986 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9608003be0 con 0x7f9608006320 2024-03-17T23:47:14.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.452+0000 7f9613fff640 1 -- 172.21.15.150:0/634500986 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9608003de0 con 0x7f9608004b10 2024-03-17T23:47:14.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.452+0000 7fafd49df640 1 -- 172.21.15.150:0/2611187550 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafd00cf640 msgr2=0x7fafd013f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f9699064640 1 -- 172.21.15.150:0/2508009913 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f968c00c0a0 msgr2=0x7f968c038080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.451+0000 7f9699064640 1 --2- 172.21.15.150:0/2508009913 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f968c00c0a0 0x7f968c038080 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9694075ec0 tx=0x7f96940af7b0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.452+0000 7fafd49df640 1 --2- 172.21.15.150:0/2611187550 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafd00cf640 0x7fafd013f6d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fafd006f930 tx=0x7fafc402b960 comp rx=0 tx=0).stop 2024-03-17T23:47:14.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.452+0000 7f9620883640 1 --2- 172.21.15.150:0/634500986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9608005910 0x7f9608005d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.452+0000 7fafd49df640 1 -- 172.21.15.150:0/2611187550 shutdown_connections 2024-03-17T23:47:14.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.452+0000 7fafd49df640 1 --2- 172.21.15.150:0/2611187550 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafd00cf640 0x7fafd013f6d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.452+0000 7fafd49df640 1 --2- 172.21.15.150:0/2611187550 >> v2:172.21.15.150:3300/0 conn(0x7fafd0072ce0 0x7fafd013cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.452+0000 7fafd49df640 1 -- 172.21.15.150:0/2611187550 >> 172.21.15.150:0/2611187550 conn(0x7fafd006b190 msgr2=0x7fafd006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:14.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.452+0000 7ff36ef50640 1 -- start start 2024-03-17T23:47:14.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7f9620883640 1 -- 172.21.15.150:0/634500986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9608004b10 msgr2=0x7f9608004f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-17T23:47:14.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7f9620883640 1 --2- 172.21.15.150:0/634500986 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9608004b10 0x7f9608004f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7f9620883640 1 -- 172.21.15.150:0/634500986 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9608006320 msgr2=0x7f9608003440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7f9620883640 1 --2- 172.21.15.150:0/634500986 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9608006320 0x7f9608003440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7f9620883640 1 -- 172.21.15.150:0/634500986 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9608004630 con 0x7f9608005910 2024-03-17T23:47:14.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7fafd49df640 1 -- 172.21.15.150:0/2611187550 shutdown_connections 2024-03-17T23:47:14.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7fafd49df640 1 -- 172.21.15.150:0/2611187550 wait complete. 2024-03-17T23:47:14.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7f9621885640 1 -- 172.21.15.150:0/634500986 reap_dead start 2024-03-17T23:47:14.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7f0549322640 1 --2- 172.21.15.150:0/1649304853 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0530004800 0x7f0530004c80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0534007e80 tx=0x7f0530005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4999146b1bb3fa05 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.452+0000 7fd567223640 1 -- 172.21.15.150:0/549871701 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd5600b45c0 msgr2=0x7fd560071a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7ff36ef50640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3600ead00 0x7ff36004fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7f0548b21640 1 -- 172.21.15.150:0/1649304853 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0534039070 con 0x7f0530004800 2024-03-17T23:47:14.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.452+0000 7fd567223640 1 --2- 172.21.15.150:0/549871701 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd5600b45c0 0x7fd560071a60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd5500097a0 tx=0x7fd55002b8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.452+0000 7fd567223640 1 -- 172.21.15.150:0/549871701 shutdown_connections 2024-03-17T23:47:14.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.452+0000 7fd567223640 1 --2- 172.21.15.150:0/549871701 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd5600b45c0 0x7fd560071a60 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd5500097a0 tx=0x7fd55002b8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.452+0000 7fd567223640 1 --2- 172.21.15.150:0/549871701 >> v2:172.21.15.157:3300/0 conn(0x7fd56015c860 0x7fd560166750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.452+0000 7fd567223640 1 -- 172.21.15.150:0/549871701 >> 172.21.15.150:0/549871701 conn(0x7fd5600b17e0 msgr2=0x7fd5600b1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:14.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7f6be0116640 1 -- 172.21.15.150:0/3517937914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bd815ae40 msgr2=0x7f6bd80cb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7f6be0116640 1 --2- 172.21.15.150:0/3517937914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bd815ae40 0x7f6bd80cb440 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6bd80cc8b0 tx=0x7f6bd402b930 comp rx=0 tx=0).stop 2024-03-17T23:47:14.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7f0548b21640 1 -- 172.21.15.150:0/1649304853 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f053402c830 con 0x7f0530004800 2024-03-17T23:47:14.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7ff36df4e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3600ead00 0x7ff36004fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7ff36df4e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3600ead00 0x7ff36004fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:51192/0 (socket says 172.21.15.150:51192) 2024-03-17T23:47:14.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7ff36df4e640 1 -- 172.21.15.150:0/703655305 learned_addr learned my addr 172.21.15.150:0/703655305 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7f0548b21640 1 -- 172.21.15.150:0/1649304853 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0534007ad0 con 0x7f0530004800 2024-03-17T23:47:14.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7f6be0116640 1 -- 172.21.15.150:0/3517937914 shutdown_connections 2024-03-17T23:47:14.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7f6be0116640 1 --2- 172.21.15.150:0/3517937914 >> v2:172.21.15.157:3300/0 conn(0x7f6bd80cf2a0 0x7f6bd80ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7f6be0116640 1 --2- 172.21.15.150:0/3517937914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bd815ae40 0x7f6bd80cb440 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7f6be0116640 1 -- 172.21.15.150:0/3517937914 >> 172.21.15.150:0/3517937914 conn(0x7f6bd8099a70 msgr2=0x7f6bd8099e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:14.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7f1b2c8bd640 1 -- 172.21.15.150:0/107601993 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b28162e30 msgr2=0x7f1b28163240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7f1b2c8bd640 1 --2- 172.21.15.150:0/107601993 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b28162e30 0x7f1b28163240 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c0097a0 tx=0x7f1b1c02b8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7ff36ef50640 1 --2- 172.21.15.150:0/703655305 >> v2:172.21.15.150:3300/0 conn(0x7ff360006ea0 0x7ff360007330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7ff36ef50640 1 -- 172.21.15.150:0/703655305 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff360053720 con 0x7ff36004ffd0 2024-03-17T23:47:14.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7ff36ef50640 1 -- 172.21.15.150:0/703655305 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7ff360054330 con 0x7ff360006ea0 2024-03-17T23:47:14.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7ff36ef50640 1 -- 172.21.15.150:0/703655305 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff360053a00 con 0x7ff3600ead00 2024-03-17T23:47:14.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7f9699064640 1 -- 172.21.15.150:0/2508009913 shutdown_connections 2024-03-17T23:47:14.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7f9699064640 1 --2- 172.21.15.150:0/2508009913 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f968c00c0a0 0x7f968c038080 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.453+0000 7f9699064640 1 -- 172.21.15.150:0/2508009913 >> 172.21.15.150:0/2508009913 conn(0x7f968c009fb0 msgr2=0x7f968c00a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:14.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.454+0000 7ff36e74f640 1 --2- 172.21.15.150:0/703655305 >> v2:172.21.15.150:3300/0 conn(0x7ff360006ea0 0x7ff360007330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.454+0000 7f6be0116640 1 -- 172.21.15.150:0/3517937914 shutdown_connections 2024-03-17T23:47:14.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.454+0000 7f1b2c8bd640 1 -- 172.21.15.150:0/107601993 shutdown_connections 2024-03-17T23:47:14.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.454+0000 7f6be0116640 1 -- 172.21.15.150:0/3517937914 wait complete. 2024-03-17T23:47:14.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.454+0000 7f1b2c8bd640 1 --2- 172.21.15.150:0/107601993 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b28162e30 0x7f1b28163240 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.454+0000 7f1b2c8bd640 1 --2- 172.21.15.150:0/107601993 >> v2:172.21.15.157:3300/0 conn(0x7f1b280d2c10 0x7f1b28166800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.454+0000 7f1b2c8bd640 1 -- 172.21.15.150:0/107601993 >> 172.21.15.150:0/107601993 conn(0x7f1b28072ef0 msgr2=0x7f1b280d3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:14.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.454+0000 7ff36cf4c640 1 -- 172.21.15.150:0/703655305 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1910333854 0 0) 0x7ff360053720 con 0x7ff36004ffd0 2024-03-17T23:47:14.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.454+0000 7ff36cf4c640 1 -- 172.21.15.150:0/703655305 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff354003520 con 0x7ff36004ffd0 2024-03-17T23:47:14.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.454+0000 7fd567223640 1 -- 172.21.15.150:0/549871701 shutdown_connections 2024-03-17T23:47:14.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.454+0000 7ff36df4e640 1 -- 172.21.15.150:0/703655305 >> v2:172.21.15.150:3300/0 conn(0x7ff360006ea0 msgr2=0x7ff360007330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.454+0000 7f9699064640 1 -- 172.21.15.150:0/2508009913 shutdown_connections 2024-03-17T23:47:14.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.454+0000 7f9699064640 1 -- 172.21.15.150:0/2508009913 wait complete. 2024-03-17T23:47:14.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.454+0000 7f1b2c8bd640 1 -- 172.21.15.150:0/107601993 shutdown_connections 2024-03-17T23:47:14.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.454+0000 7ff36df4e640 1 --2- 172.21.15.150:0/703655305 >> v2:172.21.15.150:3300/0 conn(0x7ff360006ea0 0x7ff360007330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.454+0000 7ff36df4e640 1 -- 172.21.15.150:0/703655305 >> v1:172.21.15.157:6789/0 conn(0x7ff36004ffd0 legacy=0x7ff360037bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.454+0000 7ff36df4e640 1 -- 172.21.15.150:0/703655305 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff360053ce0 con 0x7ff3600ead00 2024-03-17T23:47:14.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.454+0000 7fafd49df640 1 Processor -- start 2024-03-17T23:47:14.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.454+0000 7f054bd87640 1 -- 172.21.15.150:0/1649304853 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0530004800 msgr2=0x7f0530004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.454+0000 7f054bd87640 1 --2- 172.21.15.150:0/1649304853 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0530004800 0x7f0530004c80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0534007e80 tx=0x7f0530005830 comp rx=0 tx=0).stop 2024-03-17T23:47:14.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.454+0000 7f054bd87640 1 -- 172.21.15.150:0/1649304853 shutdown_connections 2024-03-17T23:47:14.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.454+0000 7f054bd87640 1 --2- 172.21.15.150:0/1649304853 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0530004800 0x7f0530004c80 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0534007e80 tx=0x7f0530005830 comp rx=0 tx=0).stop 2024-03-17T23:47:14.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.454+0000 7f054bd87640 1 -- 172.21.15.150:0/1649304853 >> 172.21.15.150:0/1649304853 conn(0x7f05440b36a0 msgr2=0x7f05440b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:14.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.455+0000 7ff36e74f640 1 --2- 172.21.15.150:0/703655305 >> v2:172.21.15.150:3300/0 conn(0x7ff360006ea0 0x7ff360007330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:47:14.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.455+0000 7fafd49df640 1 -- start start 2024-03-17T23:47:14.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.455+0000 7f9620883640 1 --2- 172.21.15.150:0/634500986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9608005910 0x7f9608005d90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9618007fd0 tx=0x7f9608006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a270c5b8a9ece9d server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.455+0000 7f1b2c8bd640 1 -- 172.21.15.150:0/107601993 wait complete. 2024-03-17T23:47:14.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.455+0000 7f054bd87640 1 -- 172.21.15.150:0/1649304853 shutdown_connections 2024-03-17T23:47:14.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.455+0000 7f9613fff640 1 -- 172.21.15.150:0/634500986 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9618007c20 con 0x7f9608005910 2024-03-17T23:47:14.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.455+0000 7f054bd87640 1 -- 172.21.15.150:0/1649304853 wait complete. 2024-03-17T23:47:14.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.455+0000 7f9613fff640 1 -- 172.21.15.150:0/634500986 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9618030070 con 0x7f9608005910 2024-03-17T23:47:14.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.455+0000 7f9613fff640 1 -- 172.21.15.150:0/634500986 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9618005dd0 con 0x7f9608005910 2024-03-17T23:47:14.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.456+0000 7fafd49df640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fafd006dd30 0x7fafd0154540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.456+0000 7fafd49df640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafd0072ce0 0x7fafd013bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.456+0000 7fafd49df640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafd00cf640 0x7fafd013bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.456+0000 7fafd49df640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fafd006e190 con 0x7fafd00cf640 2024-03-17T23:47:14.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.456+0000 7fafd49df640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fafd00cea10 con 0x7fafd0072ce0 2024-03-17T23:47:14.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.456+0000 7fafd49df640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fafd006e310 con 0x7fafd006dd30 2024-03-17T23:47:14.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.456+0000 7f6be0116640 1 Processor -- start 2024-03-17T23:47:14.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.456+0000 7fafcdd9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafd0072ce0 0x7fafd013bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.456+0000 7f6be0116640 1 -- start start 2024-03-17T23:47:14.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.456+0000 7fafcdd9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafd0072ce0 0x7fafd013bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:33922/0 (socket says 172.21.15.150:33922) 2024-03-17T23:47:14.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.456+0000 7fafcdd9b640 1 -- 172.21.15.150:0/1871589140 learned_addr learned my addr 172.21.15.150:0/1871589140 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.456+0000 7fafced9d640 1 --2- 172.21.15.150:0/1871589140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafd00cf640 0x7fafd013bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.456+0000 7f9622086640 1 -- 172.21.15.150:0/634500986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9608005910 msgr2=0x7f9608005d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.456+0000 7f9622086640 1 --2- 172.21.15.150:0/634500986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9608005910 0x7f9608005d90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9618007fd0 tx=0x7f9608006940 comp rx=0 tx=0).stop 2024-03-17T23:47:14.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.456+0000 7fafce59c640 1 --2- 172.21.15.150:0/1871589140 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fafd006dd30 0x7fafd0154540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.457+0000 7f1b2c8bd640 1 Processor -- start 2024-03-17T23:47:14.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.457+0000 7fafcdd9b640 1 -- 172.21.15.150:0/1871589140 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fafd006dd30 msgr2=0x7fafd0154540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.457+0000 7fafcdd9b640 1 --2- 172.21.15.150:0/1871589140 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fafd006dd30 0x7fafd0154540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.457+0000 7fafcdd9b640 1 -- 172.21.15.150:0/1871589140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafd00cf640 msgr2=0x7fafd013bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.457+0000 7fafcdd9b640 1 --2- 172.21.15.150:0/1871589140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fafd00cf640 0x7fafd013bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.457+0000 7f1b2c8bd640 1 -- start start 2024-03-17T23:47:14.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.457+0000 7fafcdd9b640 1 -- 172.21.15.150:0/1871589140 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd014d140 con 0x7fafd0072ce0 2024-03-17T23:47:14.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.457+0000 7ff36df4e640 1 --2- 172.21.15.150:0/703655305 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3600ead00 0x7ff36004fa90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff368075ec0 tx=0x7ff3680a70d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca2c90b7bef31b69 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.457+0000 7fafcdd9b640 1 --2- 172.21.15.150:0/1871589140 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafd0072ce0 0x7fafd013bab0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fafc000c480 tx=0x7fafc000c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.457+0000 7fd567223640 1 -- 172.21.15.150:0/549871701 wait complete. 2024-03-17T23:47:14.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.457+0000 7f6be0116640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bd80cf2a0 0x7f6bd814edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.457+0000 7ff36cf4c640 1 -- 172.21.15.150:0/703655305 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3680b4070 con 0x7ff3600ead00 2024-03-17T23:47:14.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.457+0000 7ff36cf4c640 1 -- 172.21.15.150:0/703655305 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3680af440 con 0x7ff3600ead00 2024-03-17T23:47:14.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.457+0000 7ff36cf4c640 1 -- 172.21.15.150:0/703655305 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3680af8b0 con 0x7ff3600ead00 2024-03-17T23:47:14.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.456+0000 7f9699064640 1 Processor -- start 2024-03-17T23:47:14.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.456+0000 7f9699064640 1 -- start start 2024-03-17T23:47:14.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.457+0000 7f9699064640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f968c00c0a0 0x7f968c167b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.457+0000 7f6be0116640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6bd815ae40 0x7f6bd80c57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.457+0000 7fafbeffd640 1 -- 172.21.15.150:0/1871589140 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc00052d0 con 0x7fafd0072ce0 2024-03-17T23:47:14.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.458+0000 7fafbeffd640 1 -- 172.21.15.150:0/1871589140 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafc0005470 con 0x7fafd0072ce0 2024-03-17T23:47:14.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.458+0000 7fafbeffd640 1 -- 172.21.15.150:0/1871589140 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc0012850 con 0x7fafd0072ce0 2024-03-17T23:47:14.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.458+0000 7f054bd87640 1 Processor -- start 2024-03-17T23:47:14.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.458+0000 7fafd49df640 1 -- 172.21.15.150:0/1871589140 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafd014d420 con 0x7fafd0072ce0 2024-03-17T23:47:14.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.458+0000 7f9622086640 1 -- 172.21.15.150:0/634500986 shutdown_connections 2024-03-17T23:47:14.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.458+0000 7f9622086640 1 --2- 172.21.15.150:0/634500986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9608005910 0x7f9608005d90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.458+0000 7f054bd87640 1 -- start start 2024-03-17T23:47:14.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.458+0000 7f9622086640 1 -- 172.21.15.150:0/634500986 >> 172.21.15.150:0/634500986 conn(0x7f961c0b1600 msgr2=0x7f961c0b19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:14.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.458+0000 7fafd49df640 1 -- 172.21.15.150:0/1871589140 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafd014c080 con 0x7fafd0072ce0 2024-03-17T23:47:14.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.457+0000 7f9699064640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f968c033ef0 0x7f968c0357e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.458+0000 7f1b2c8bd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b280d3810 0x7f1b281fcac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.458+0000 7f1b2c8bd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b28162e30 0x7f1b280cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.458+0000 7f1b2c8bd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b281f8be0 0x7f1b280b7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.458+0000 7f1b2c8bd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1b280ce840 con 0x7f1b281f8be0 2024-03-17T23:47:14.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.458+0000 7f1b2c8bd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1b280ce560 con 0x7f1b28162e30 2024-03-17T23:47:14.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.458+0000 7f1b2c8bd640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1b280d04a0 con 0x7f1b280d3810 2024-03-17T23:47:14.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.458+0000 7f1b25d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b28162e30 0x7f1b280cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.458+0000 7f1b25d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b28162e30 0x7f1b280cd640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:33946/0 (socket says 172.21.15.150:33946) 2024-03-17T23:47:14.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.458+0000 7f1b25d9b640 1 -- 172.21.15.150:0/2626332126 learned_addr learned my addr 172.21.15.150:0/2626332126 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.458+0000 7f1b26d9d640 1 --2- 172.21.15.150:0/2626332126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b281f8be0 0x7f1b280b7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.458+0000 7f96937fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f968c033ef0 0x7f968c0357e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.459+0000 7f96937fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f968c033ef0 0x7f968c0357e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36408/0 (socket says 172.21.15.150:36408) 2024-03-17T23:47:14.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.459+0000 7f1b25d9b640 1 -- 172.21.15.150:0/2626332126 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b280d3810 msgr2=0x7f1b281fcac0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-17T23:47:14.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.459+0000 7f9693fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f968c00c0a0 0x7f968c167b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.459+0000 7f9693fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f968c00c0a0 0x7f968c167b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:33934/0 (socket says 172.21.15.150:33934) 2024-03-17T23:47:14.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.459+0000 7fd567223640 1 Processor -- start 2024-03-17T23:47:14.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.459+0000 7f1b25d9b640 1 --2- 172.21.15.150:0/2626332126 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b280d3810 0x7f1b281fcac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.459+0000 7fd567223640 1 -- start start 2024-03-17T23:47:14.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.459+0000 7f1b25d9b640 1 -- 172.21.15.150:0/2626332126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b281f8be0 msgr2=0x7f1b280b7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.459+0000 7fafd49df640 1 -- 172.21.15.150:0/1871589140 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fafd01494f0 con 0x7fafd0072ce0 2024-03-17T23:47:14.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.459+0000 7f1b25d9b640 1 --2- 172.21.15.150:0/2626332126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b281f8be0 0x7f1b280b7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.459+0000 7f1b25d9b640 1 -- 172.21.15.150:0/2626332126 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b281f95c0 con 0x7f1b28162e30 2024-03-17T23:47:14.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.459+0000 7f9622086640 1 -- 172.21.15.150:0/634500986 shutdown_connections 2024-03-17T23:47:14.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.458+0000 7f6bddeb2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bd80cf2a0 0x7f6bd814edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.459+0000 7f6be0116640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6bd80c5ff0 0x7f6bd806b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.459+0000 7f6be0116640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd815b220 con 0x7f6bd80cf2a0 2024-03-17T23:47:14.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.459+0000 7f6bddeb2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bd80cf2a0 0x7f6bd814edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36398/0 (socket says 172.21.15.150:36398) 2024-03-17T23:47:14.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.459+0000 7f6bddeb2640 1 -- 172.21.15.150:0/1890661654 learned_addr learned my addr 172.21.15.150:0/1890661654 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.459+0000 7f6be0116640 1 -- 172.21.15.150:0/1890661654 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd80cc3c0 con 0x7f6bd815ae40 2024-03-17T23:47:14.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.459+0000 7f6be0116640 1 -- 172.21.15.150:0/1890661654 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6bd80d11d0 con 0x7f6bd80c5ff0 2024-03-17T23:47:14.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.459+0000 7f9622086640 1 -- 172.21.15.150:0/634500986 wait complete. 2024-03-17T23:47:14.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.459+0000 7f9699064640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f968c053c70 0x7f968c035d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.459+0000 7f9699064640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f968c037a90 con 0x7f968c033ef0 2024-03-17T23:47:14.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.459+0000 7f9699064640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f968c037910 con 0x7f968c00c0a0 2024-03-17T23:47:14.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.459+0000 7f9699064640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f968c0501a0 con 0x7f968c053c70 2024-03-17T23:47:14.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.459+0000 7f96937fe640 1 -- 172.21.15.150:0/148270250 learned_addr learned my addr 172.21.15.150:0/148270250 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.459+0000 7f054bd87640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05441f81d0 0x7f05441f85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7ff36ef50640 1 -- 172.21.15.150:0/703655305 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3600ead00 msgr2=0x7ff36004fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7ff36ef50640 1 --2- 172.21.15.150:0/703655305 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3600ead00 0x7ff36004fa90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff368075ec0 tx=0x7ff3680a70d0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.459+0000 7f9698863640 1 --2- 172.21.15.150:0/148270250 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f968c053c70 0x7f968c035d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7f9693fff640 1 -- 172.21.15.150:0/148270250 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f968c053c70 msgr2=0x7f968c035d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7f9693fff640 1 --2- 172.21.15.150:0/148270250 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f968c053c70 0x7f968c035d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7f9693fff640 1 -- 172.21.15.150:0/148270250 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f968c033ef0 msgr2=0x7f968c0357e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7f9693fff640 1 --2- 172.21.15.150:0/148270250 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f968c033ef0 0x7f968c0357e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7f9693fff640 1 -- 172.21.15.150:0/148270250 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f968c189940 con 0x7f968c00c0a0 2024-03-17T23:47:14.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7f9693fff640 1 --2- 172.21.15.150:0/148270250 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f968c00c0a0 0x7f968c167b10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9694050e40 tx=0x7f9694068f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7ff36ef50640 1 -- 172.21.15.150:0/703655305 shutdown_connections 2024-03-17T23:47:14.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7ff36ef50640 1 --2- 172.21.15.150:0/703655305 >> v2:172.21.15.150:3300/0 conn(0x7ff360006ea0 0x7ff360007330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7ff36ef50640 1 --2- 172.21.15.150:0/703655305 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3600ead00 0x7ff36004fa90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7ff36ef50640 1 -- 172.21.15.150:0/703655305 >> 172.21.15.150:0/703655305 conn(0x7ff360009fb0 msgr2=0x7ff36000a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:14.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7f6bde6b3640 1 --2- 172.21.15.150:0/1890661654 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6bd80c5ff0 0x7f6bd806b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7fd567223640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd560071fa0 0x7fd560161d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7fd567223640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56015c860 0x7fd56015ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7fd567223640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd560149e40 0x7fd560160b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7fd567223640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd5600b4a40 con 0x7fd56015c860 2024-03-17T23:47:14.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7fd567223640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd560162fe0 con 0x7fd560149e40 2024-03-17T23:47:14.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7fd567223640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd560163500 con 0x7fd560071fa0 2024-03-17T23:47:14.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7f9690ff9640 1 -- 172.21.15.150:0/148270250 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96940933a0 con 0x7f968c00c0a0 2024-03-17T23:47:14.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7f9690ff9640 1 -- 172.21.15.150:0/148270250 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9694093540 con 0x7f968c00c0a0 2024-03-17T23:47:14.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7f9690ff9640 1 -- 172.21.15.150:0/148270250 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96940b8bc0 con 0x7f968c00c0a0 2024-03-17T23:47:14.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7ff36ef50640 1 -- 172.21.15.150:0/703655305 shutdown_connections 2024-03-17T23:47:14.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7ff36ef50640 1 -- 172.21.15.150:0/703655305 wait complete. 2024-03-17T23:47:14.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7f9699064640 1 -- 172.21.15.150:0/148270250 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f968c189c20 con 0x7f968c00c0a0 2024-03-17T23:47:14.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7f9699064640 1 -- 172.21.15.150:0/148270250 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f968c18a1e0 con 0x7f968c00c0a0 2024-03-17T23:47:14.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7fd566221640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd560071fa0 0x7fd560161d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7fd566221640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd560071fa0 0x7fd560161d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:51218/0 (socket says 172.21.15.150:51218) 2024-03-17T23:47:14.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7fd566221640 1 -- 172.21.15.150:0/572098332 learned_addr learned my addr 172.21.15.150:0/572098332 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.461+0000 7fd566a22640 1 --2- 172.21.15.150:0/572098332 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd560149e40 0x7fd560160b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7f6bddeb2640 1 -- 172.21.15.150:0/1890661654 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6bd80c5ff0 msgr2=0x7f6bd806b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7f6bddeb2640 1 --2- 172.21.15.150:0/1890661654 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6bd80c5ff0 0x7f6bd806b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.461+0000 7fafbeffd640 1 -- 172.21.15.150:0/1871589140 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fafc00129f0 con 0x7fafd0072ce0 2024-03-17T23:47:14.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.461+0000 7f054bd87640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f054406d5f0 0x7f05440cb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.461+0000 7f054bd87640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0544070ff0 0x7f05440cdb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.461+0000 7f054bd87640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f054415c210 con 0x7f0544070ff0 2024-03-17T23:47:14.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.461+0000 7f054bd87640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f05440cd920 con 0x7f054406d5f0 2024-03-17T23:47:14.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.461+0000 7f054bd87640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f054415c390 con 0x7f05441f81d0 2024-03-17T23:47:14.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.461+0000 7fd566a22640 1 -- 172.21.15.150:0/572098332 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd560071fa0 msgr2=0x7fd560161d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.461+0000 7fd566a22640 1 --2- 172.21.15.150:0/572098332 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd560071fa0 0x7fd560161d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.461+0000 7fd566a22640 1 -- 172.21.15.150:0/572098332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56015c860 msgr2=0x7fd56015ccf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-17T23:47:14.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.461+0000 7fd566a22640 1 --2- 172.21.15.150:0/572098332 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd56015c860 0x7fd56015ccf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.461+0000 7fd566a22640 1 -- 172.21.15.150:0/572098332 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd56015d230 con 0x7fd560149e40 2024-03-17T23:47:14.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7f6bddeb2640 1 -- 172.21.15.150:0/1890661654 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6bd815ae40 msgr2=0x7f6bd80c57e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-17T23:47:14.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7f6bddeb2640 1 --2- 172.21.15.150:0/1890661654 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6bd815ae40 0x7f6bd80c57e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.460+0000 7f6bddeb2640 1 -- 172.21.15.150:0/1890661654 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd80c6760 con 0x7f6bd80cf2a0 2024-03-17T23:47:14.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.461+0000 7f9699064640 1 -- 172.21.15.150:0/148270250 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f968c005390 con 0x7f968c00c0a0 2024-03-17T23:47:14.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.461+0000 7fafbeffd640 1 --2- 172.21.15.150:0/1871589140 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fafb006bca0 0x7fafb006e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.461+0000 7f1b25d9b640 1 --2- 172.21.15.150:0/2626332126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b28162e30 0x7f1b280cd640 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1b2000c7a0 tx=0x7f1b2000cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.462+0000 7f9622086640 1 Processor -- start 2024-03-17T23:47:14.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.462+0000 7f9622086640 1 -- start start 2024-03-17T23:47:14.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.462+0000 7ff36ef50640 1 Processor -- start 2024-03-17T23:47:14.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.462+0000 7f1b16ffd640 1 -- 172.21.15.150:0/2626332126 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b20003b70 con 0x7f1b28162e30 2024-03-17T23:47:14.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.462+0000 7f0549b23640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05441f81d0 0x7f05441f85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.462+0000 7ff36ef50640 1 -- start start 2024-03-17T23:47:14.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.462+0000 7f1b16ffd640 1 -- 172.21.15.150:0/2626332126 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b20003d10 con 0x7f1b28162e30 2024-03-17T23:47:14.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.462+0000 7f0549b23640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05441f81d0 0x7f05441f85b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:51230/0 (socket says 172.21.15.150:51230) 2024-03-17T23:47:14.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.462+0000 7fd566a22640 1 --2- 172.21.15.150:0/572098332 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd560149e40 0x7fd560160b60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd55800ca70 tx=0x7fd55800cf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.462+0000 7f0549b23640 1 -- 172.21.15.150:0/2872659399 learned_addr learned my addr 172.21.15.150:0/2872659399 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.462+0000 7f054a324640 1 --2- 172.21.15.150:0/2872659399 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0544070ff0 0x7f05440cdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.462+0000 7f0549322640 1 --2- 172.21.15.150:0/2872659399 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f054406d5f0 0x7f05440cb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.462+0000 7f0549b23640 1 -- 172.21.15.150:0/2872659399 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f054406d5f0 msgr2=0x7f05440cb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.462+0000 7f0549b23640 1 --2- 172.21.15.150:0/2872659399 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f054406d5f0 0x7f05440cb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.462+0000 7f0549b23640 1 -- 172.21.15.150:0/2872659399 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0544070ff0 msgr2=0x7f05440cdb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.462+0000 7fd556ffd640 1 -- 172.21.15.150:0/572098332 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd558017070 con 0x7fd560149e40 2024-03-17T23:47:14.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.463+0000 7fd556ffd640 1 -- 172.21.15.150:0/572098332 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd55800ab70 con 0x7fd560149e40 2024-03-17T23:47:14.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.463+0000 7ff36ef50640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff360006ea0 0x7ff3600e8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.463+0000 7ff36ef50640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff36004ffd0 0x7ff36017d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.463+0000 7ff36ef50640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3600ead00 0x7ff36004dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.463+0000 7ff36ef50640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff360054940 con 0x7ff36004ffd0 2024-03-17T23:47:14.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.463+0000 7ff36ef50640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff3600547c0 con 0x7ff3600ead00 2024-03-17T23:47:14.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.462+0000 7fafce59c640 1 --2- 172.21.15.150:0/1871589140 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fafb006bca0 0x7fafb006e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.463+0000 7ff36ef50640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff36004f550 con 0x7ff360006ea0 2024-03-17T23:47:14.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.462+0000 7f1b16ffd640 1 -- 172.21.15.150:0/2626332126 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2000aa90 con 0x7f1b28162e30 2024-03-17T23:47:14.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.462+0000 7f1b2c8bd640 1 -- 172.21.15.150:0/2626332126 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b281f98a0 con 0x7f1b28162e30 2024-03-17T23:47:14.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.462+0000 7f1b2c8bd640 1 -- 172.21.15.150:0/2626332126 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b281fd000 con 0x7f1b28162e30 2024-03-17T23:47:14.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.463+0000 7f9622086640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9608005910 0x7f961c13f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.463+0000 7ff36d74d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff36004ffd0 0x7ff36017d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.463+0000 7ff36d74d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff36004ffd0 0x7ff36017d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36438/0 (socket says 172.21.15.150:36438) 2024-03-17T23:47:14.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.463+0000 7ff36d74d640 1 -- 172.21.15.150:0/2538225631 learned_addr learned my addr 172.21.15.150:0/2538225631 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.463+0000 7f9621084640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9608005910 0x7f961c13f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.463+0000 7f9621084640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9608005910 0x7f961c13f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36448/0 (socket says 172.21.15.150:36448) 2024-03-17T23:47:14.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7f9621084640 1 -- 172.21.15.150:0/3014085748 learned_addr learned my addr 172.21.15.150:0/3014085748 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.463+0000 7f9690ff9640 1 -- 172.21.15.150:0/148270250 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f96940c7060 con 0x7f968c00c0a0 2024-03-17T23:47:14.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7f1b2c8bd640 1 -- 172.21.15.150:0/2626332126 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1b281fc6a0 con 0x7f1b28162e30 2024-03-17T23:47:14.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.462+0000 7f6bddeb2640 1 --2- 172.21.15.150:0/1890661654 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bd80cf2a0 0x7f6bd814edd0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40090b0 tx=0x7f6bd40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7ff36d74d640 1 -- 172.21.15.150:0/2538225631 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff360006ea0 msgr2=0x7ff3600e8ff0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-17T23:47:14.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7ff36d74d640 1 --2- 172.21.15.150:0/2538225631 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff360006ea0 0x7ff3600e8ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7ff36d74d640 1 -- 172.21.15.150:0/2538225631 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3600ead00 msgr2=0x7ff36004dac0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-17T23:47:14.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7ff36d74d640 1 --2- 172.21.15.150:0/2538225631 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3600ead00 0x7ff36004dac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7ff36d74d640 1 -- 172.21.15.150:0/2538225631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff36004ce80 con 0x7ff36004ffd0 2024-03-17T23:47:14.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7f0549322640 1 --2- 172.21.15.150:0/2872659399 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f054406d5f0 0x7f05440cb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:47:14.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7fd556ffd640 1 -- 172.21.15.150:0/572098332 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd558012400 con 0x7fd560149e40 2024-03-17T23:47:14.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7f9690ff9640 1 --2- 172.21.15.150:0/148270250 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f967806bc20 0x7f967806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7ff36d74d640 1 --2- 172.21.15.150:0/2538225631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff36004ffd0 0x7ff36017d130 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff35800c490 tx=0x7ff35800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7f9690ff9640 1 -- 172.21.15.150:0/148270250 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f9694128670 con 0x7f968c00c0a0 2024-03-17T23:47:14.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7fd567223640 1 -- 172.21.15.150:0/572098332 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd56015d4d0 con 0x7fd560149e40 2024-03-17T23:47:14.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.463+0000 7fafbeffd640 1 -- 172.21.15.150:0/1871589140 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fafc008b450 con 0x7fafd0072ce0 2024-03-17T23:47:14.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7f6bce7fc640 1 -- 172.21.15.150:0/1890661654 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4040070 con 0x7f6bd80cf2a0 2024-03-17T23:47:14.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7f9690ff9640 1 --2- 172.21.15.150:0/148270250 >> [v2:172.21.15.157:6824/1224877787,v1:172.21.15.157:6825/1224877787] conn(0x7f96780753a0 0x7f96780777e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7f9690ff9640 1 -- 172.21.15.150:0/148270250 --> [v2:172.21.15.157:6824/1224877787,v1:172.21.15.157:6825/1224877787] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9694072280 con 0x7f96780753a0 2024-03-17T23:47:14.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7f9690ff9640 1 -- 172.21.15.150:0/148270250 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9694128b50 con 0x7f968c00c0a0 2024-03-17T23:47:14.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7fd567223640 1 -- 172.21.15.150:0/572098332 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd560160360 con 0x7fd560149e40 2024-03-17T23:47:14.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7ff35e7fc640 1 -- 172.21.15.150:0/2538225631 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3580052d0 con 0x7ff36004ffd0 2024-03-17T23:47:14.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7ff36ef50640 1 -- 172.21.15.150:0/2538225631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff360043890 con 0x7ff36004ffd0 2024-03-17T23:47:14.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7ff35e7fc640 1 -- 172.21.15.150:0/2538225631 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff358005470 con 0x7ff36004ffd0 2024-03-17T23:47:14.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7ff35e7fc640 1 -- 172.21.15.150:0/2538225631 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3580052d0 con 0x7ff36004ffd0 2024-03-17T23:47:14.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.465+0000 7ff36ef50640 1 -- 172.21.15.150:0/2538225631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff360043e50 con 0x7ff36004ffd0 2024-03-17T23:47:14.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7f6bce7fc640 1 -- 172.21.15.150:0/1890661654 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bd4030d20 con 0x7f6bd80cf2a0 2024-03-17T23:47:14.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7f6be0116640 1 -- 172.21.15.150:0/1890661654 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bd814e010 con 0x7f6bd80cf2a0 2024-03-17T23:47:14.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.465+0000 7f6be0116640 1 -- 172.21.15.150:0/1890661654 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bd814e5d0 con 0x7f6bd80cf2a0 2024-03-17T23:47:14.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7f96937fe640 1 --2- 172.21.15.150:0/148270250 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f967806bc20 0x7f967806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.465+0000 7f9698863640 1 --2- 172.21.15.150:0/148270250 >> [v2:172.21.15.157:6824/1224877787,v1:172.21.15.157:6825/1224877787] conn(0x7f96780753a0 0x7f96780777e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7fafbeffd640 1 --2- 172.21.15.150:0/1871589140 >> [v2:172.21.15.157:6809/2764449896,v1:172.21.15.157:6811/2764449896] conn(0x7fafb0075410 0x7fafb0077850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7fafbeffd640 1 -- 172.21.15.150:0/1871589140 --> [v2:172.21.15.157:6809/2764449896,v1:172.21.15.157:6811/2764449896] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fafb0077f20 con 0x7fafb0075410 2024-03-17T23:47:14.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7fafbeffd640 1 -- 172.21.15.150:0/1871589140 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fafc008b840 con 0x7fafd0072ce0 2024-03-17T23:47:14.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.465+0000 7fafced9d640 1 --2- 172.21.15.150:0/1871589140 >> [v2:172.21.15.157:6809/2764449896,v1:172.21.15.157:6811/2764449896] conn(0x7fafb0075410 0x7fafb0077850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.465+0000 7f1b16ffd640 1 -- 172.21.15.150:0/2626332126 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1b2000ac30 con 0x7f1b28162e30 2024-03-17T23:47:14.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.465+0000 7f9698863640 1 --2- 172.21.15.150:0/148270250 >> [v2:172.21.15.157:6824/1224877787,v1:172.21.15.157:6825/1224877787] conn(0x7f96780753a0 0x7f96780777e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.465+0000 7f96937fe640 1 --2- 172.21.15.150:0/148270250 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f967806bc20 0x7f967806e0e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f968c04e540 tx=0x7f96800090e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.466+0000 7f6bce7fc640 1 -- 172.21.15.150:0/1890661654 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd4035040 con 0x7f6bd80cf2a0 2024-03-17T23:47:14.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7f0549b23640 1 --2- 172.21.15.150:0/2872659399 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0544070ff0 0x7f05440cdb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.464+0000 7f0549b23640 1 -- 172.21.15.150:0/2872659399 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0544148050 con 0x7f05441f81d0 2024-03-17T23:47:14.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.466+0000 7fd556ffd640 1 -- 172.21.15.150:0/572098332 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd55801a070 con 0x7fd560149e40 2024-03-17T23:47:14.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.466+0000 7f1b16ffd640 1 --2- 172.21.15.150:0/2626332126 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1b0806bca0 0x7f1b0806e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.466+0000 7ff35e7fc640 1 -- 172.21.15.150:0/2538225631 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff358012630 con 0x7ff36004ffd0 2024-03-17T23:47:14.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.465+0000 7f0549b23640 1 --2- 172.21.15.150:0/2872659399 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05441f81d0 0x7f05441f85b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f054000f940 tx=0x7f054000fd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.466+0000 7f054a324640 1 --2- 172.21.15.150:0/2872659399 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0544070ff0 0x7f05440cdb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:14.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.466+0000 7f9690ff9640 1 -- 172.21.15.150:0/148270250 <== osd.5 v2:172.21.15.157:6824/1224877787 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9694072280 con 0x7f96780753a0 2024-03-17T23:47:14.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.467+0000 7f053a7fc640 1 -- 172.21.15.150:0/2872659399 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054000dbb0 con 0x7f05441f81d0 2024-03-17T23:47:14.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.467+0000 7fd556ffd640 1 --2- 172.21.15.150:0/572098332 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd54406bca0 0x7fd54406e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.467+0000 7f1b16ffd640 1 -- 172.21.15.150:0/2626332126 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f1b2008ac10 con 0x7f1b28162e30 2024-03-17T23:47:14.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.467+0000 7f053a7fc640 1 -- 172.21.15.150:0/2872659399 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f054000dd50 con 0x7f05441f81d0 2024-03-17T23:47:14.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.467+0000 7f054bd87640 1 -- 172.21.15.150:0/2872659399 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05440c9e50 con 0x7f05441f81d0 2024-03-17T23:47:14.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.467+0000 7f053a7fc640 1 -- 172.21.15.150:0/2872659399 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0540013660 con 0x7f05441f81d0 2024-03-17T23:47:14.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.467+0000 7fd566221640 1 --2- 172.21.15.150:0/572098332 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd54406bca0 0x7fd54406e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.467+0000 7fd556ffd640 1 -- 172.21.15.150:0/572098332 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fd55808a9e0 con 0x7fd560149e40 2024-03-17T23:47:14.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.467+0000 7f6bce7fc640 1 -- 172.21.15.150:0/1890661654 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6bd404b060 con 0x7f6bd80cf2a0 2024-03-17T23:47:14.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.467+0000 7f054bd87640 1 -- 172.21.15.150:0/2872659399 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f05440af880 con 0x7f05441f81d0 2024-03-17T23:47:14.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.468+0000 7f6bce7fc640 1 --2- 172.21.15.150:0/1890661654 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6bbc06bca0 0x7f6bbc06e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.468+0000 7fd567223640 1 --2- 172.21.15.150:0/572098332 >> [v2:172.21.15.150:6808/1014841375,v1:172.21.15.150:6809/1014841375] conn(0x7fd56014a2f0 0x7fd560165bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.468+0000 7fd567223640 1 -- 172.21.15.150:0/572098332 --> [v2:172.21.15.150:6808/1014841375,v1:172.21.15.150:6809/1014841375] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd560160920 con 0x7fd56014a2f0 2024-03-17T23:47:14.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.467+0000 7f1b2659c640 1 --2- 172.21.15.150:0/2626332126 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1b0806bca0 0x7f1b0806e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.467+0000 7f1b16ffd640 1 --2- 172.21.15.150:0/2626332126 >> [v2:172.21.15.157:6800/624642970,v1:172.21.15.157:6801/624642970] conn(0x7f1b080753a0 0x7f1b080777e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.467+0000 7f1b16ffd640 1 -- 172.21.15.150:0/2626332126 --> [v2:172.21.15.157:6800/624642970,v1:172.21.15.157:6801/624642970] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1b08077eb0 con 0x7f1b080753a0 2024-03-17T23:47:14.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.467+0000 7f1b16ffd640 1 -- 172.21.15.150:0/2626332126 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1b20054fd0 con 0x7f1b28162e30 2024-03-17T23:47:14.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.467+0000 7ff35e7fc640 1 --2- 172.21.15.150:0/2538225631 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff34c06bfd0 0x7ff34c06e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.467+0000 7ff35e7fc640 1 -- 172.21.15.150:0/2538225631 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7ff3580050d0 con 0x7ff36004ffd0 2024-03-17T23:47:14.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.465+0000 7f9622086640 1 --2- 172.21.15.150:0/3014085748 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f961c157610 0x7f961c14d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.465+0000 7f9622086640 1 --2- 172.21.15.150:0/3014085748 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f961c1efc90 0x7f961c1f0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.465+0000 7f9622086640 1 -- 172.21.15.150:0/3014085748 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f961c0b20f0 con 0x7f9608005910 2024-03-17T23:47:14.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.465+0000 7f9622086640 1 -- 172.21.15.150:0/3014085748 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f961c0cde90 con 0x7f961c157610 2024-03-17T23:47:14.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.465+0000 7f9622086640 1 -- 172.21.15.150:0/3014085748 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f961c0b2270 con 0x7f961c1efc90 2024-03-17T23:47:14.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.468+0000 7f6bce7fc640 1 -- 172.21.15.150:0/1890661654 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f6bd40acfb0 con 0x7f6bd80cf2a0 2024-03-17T23:47:14.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.468+0000 7f1b26d9d640 1 --2- 172.21.15.150:0/2626332126 >> [v2:172.21.15.157:6800/624642970,v1:172.21.15.157:6801/624642970] conn(0x7f1b080753a0 0x7f1b080777e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.468+0000 7fd565a20640 1 --2- 172.21.15.150:0/572098332 >> [v2:172.21.15.150:6808/1014841375,v1:172.21.15.150:6809/1014841375] conn(0x7fd56014a2f0 0x7fd560165bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.469+0000 7f9621885640 1 --2- 172.21.15.150:0/3014085748 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f961c1efc90 0x7f961c1f0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.469+0000 7f1b26d9d640 1 --2- 172.21.15.150:0/2626332126 >> [v2:172.21.15.157:6800/624642970,v1:172.21.15.157:6801/624642970] conn(0x7f1b080753a0 0x7f1b080777e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.469+0000 7fd566221640 1 --2- 172.21.15.150:0/572098332 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd54406bca0 0x7fd54406e160 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd56006bf20 tx=0x7fd55c0090e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.469+0000 7ff36df4e640 1 --2- 172.21.15.150:0/2538225631 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff34c06bfd0 0x7ff34c06e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.468+0000 7fafce59c640 1 --2- 172.21.15.150:0/1871589140 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fafb006bca0 0x7fafb006e160 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009ce0 tx=0x7fafc800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.469+0000 7f9620883640 1 --2- 172.21.15.150:0/3014085748 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f961c157610 0x7f961c14d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.469+0000 7f053a7fc640 1 -- 172.21.15.150:0/2872659399 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0540013800 con 0x7f05441f81d0 2024-03-17T23:47:14.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.469+0000 7f9621084640 1 -- 172.21.15.150:0/3014085748 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f961c1efc90 msgr2=0x7f961c1f0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.469+0000 7fafced9d640 1 --2- 172.21.15.150:0/1871589140 >> [v2:172.21.15.157:6809/2764449896,v1:172.21.15.157:6811/2764449896] conn(0x7fafb0075410 0x7fafb0077850 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.469+0000 7f6be0116640 1 --2- 172.21.15.150:0/1890661654 >> [v2:172.21.15.157:6808/4022520191,v1:172.21.15.157:6810/4022520191] conn(0x7f6bd80cd100 0x7f6bd8146850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.469+0000 7f6be0116640 1 -- 172.21.15.150:0/1890661654 --> [v2:172.21.15.157:6808/4022520191,v1:172.21.15.157:6810/4022520191] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6bd80cf890 con 0x7f6bd80cd100 2024-03-17T23:47:14.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.469+0000 7f9621084640 1 --2- 172.21.15.150:0/3014085748 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f961c1efc90 0x7f961c1f0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.469+0000 7fd565a20640 1 --2- 172.21.15.150:0/572098332 >> [v2:172.21.15.150:6808/1014841375,v1:172.21.15.150:6809/1014841375] conn(0x7fd56014a2f0 0x7fd560165bb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.469+0000 7f9621084640 1 -- 172.21.15.150:0/3014085748 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f961c157610 msgr2=0x7f961c14d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.469+0000 7f9621084640 1 --2- 172.21.15.150:0/3014085748 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f961c157610 0x7f961c14d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.469+0000 7ff36ef50640 1 --2- 172.21.15.150:0/2538225631 >> [v2:172.21.15.150:6817/347494201,v1:172.21.15.150:6819/347494201] conn(0x7ff36017d670 0x7ff360037b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.469+0000 7f9621084640 1 -- 172.21.15.150:0/3014085748 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f961c141e90 con 0x7f9608005910 2024-03-17T23:47:14.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.469+0000 7ff36ef50640 1 -- 172.21.15.150:0/2538225631 --> [v2:172.21.15.150:6817/347494201,v1:172.21.15.150:6819/347494201] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff3600ea2d0 con 0x7ff36017d670 2024-03-17T23:47:14.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.470+0000 7f9620883640 1 --2- 172.21.15.150:0/3014085748 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f961c157610 0x7f961c14d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:47:14.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.470+0000 7f9621885640 1 --2- 172.21.15.150:0/3014085748 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f961c1efc90 0x7f961c1f0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:14.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.470+0000 7f6bde6b3640 1 --2- 172.21.15.150:0/1890661654 >> [v2:172.21.15.157:6808/4022520191,v1:172.21.15.157:6810/4022520191] conn(0x7f6bd80cd100 0x7f6bd8146850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.470+0000 7f053a7fc640 1 --2- 172.21.15.150:0/2872659399 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f052806bcd0 0x7f052806e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.470+0000 7f053a7fc640 1 -- 172.21.15.150:0/2872659399 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f054008b5a0 con 0x7f05441f81d0 2024-03-17T23:47:14.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.470+0000 7ff36df4e640 1 --2- 172.21.15.150:0/2538225631 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff34c06bfd0 0x7ff34c06e490 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff3600544b0 tx=0x7ff3680ab000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.470+0000 7f6bdd6b1640 1 --2- 172.21.15.150:0/1890661654 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6bbc06bca0 0x7f6bbc06e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.470+0000 7f9621084640 1 --2- 172.21.15.150:0/3014085748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9608005910 0x7f961c13f3e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f960c00ee50 tx=0x7f960c00c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.470+0000 7fafbeffd640 1 -- 172.21.15.150:0/1871589140 <== osd.6 v2:172.21.15.157:6809/2764449896 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fafb0077f20 con 0x7fafb0075410 2024-03-17T23:47:14.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.470+0000 7ff36e74f640 1 --2- 172.21.15.150:0/2538225631 >> [v2:172.21.15.150:6817/347494201,v1:172.21.15.150:6819/347494201] conn(0x7ff36017d670 0x7ff360037b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.470+0000 7f054bd87640 1 --2- 172.21.15.150:0/2872659399 >> [v2:172.21.15.150:6800/1962922469,v1:172.21.15.150:6801/1962922469] conn(0x7f052c001650 0x7f052c003b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.470+0000 7f9611ffb640 1 -- 172.21.15.150:0/3014085748 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f960c00a410 con 0x7f9608005910 2024-03-17T23:47:14.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.470+0000 7f1b2659c640 1 --2- 172.21.15.150:0/2626332126 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1b0806bca0 0x7f1b0806e160 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1b28156cc0 tx=0x7f1b18008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.470+0000 7f1b16ffd640 1 -- 172.21.15.150:0/2626332126 <== osd.7 v2:172.21.15.157:6800/624642970 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1b08077eb0 con 0x7f1b080753a0 2024-03-17T23:47:14.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.470+0000 7f9611ffb640 1 -- 172.21.15.150:0/3014085748 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f960c00a5b0 con 0x7f9608005910 2024-03-17T23:47:14.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.470+0000 7fd556ffd640 1 -- 172.21.15.150:0/572098332 <== osd.1 v2:172.21.15.150:6808/1014841375 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd560160920 con 0x7fd56014a2f0 2024-03-17T23:47:14.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.471+0000 7f9622086640 1 -- 172.21.15.150:0/3014085748 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f961c142170 con 0x7f9608005910 2024-03-17T23:47:14.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.471+0000 7f054a324640 1 --2- 172.21.15.150:0/2872659399 >> [v2:172.21.15.150:6800/1962922469,v1:172.21.15.150:6801/1962922469] conn(0x7f052c001650 0x7f052c003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.471+0000 7f054a324640 1 --2- 172.21.15.150:0/2872659399 >> [v2:172.21.15.150:6800/1962922469,v1:172.21.15.150:6801/1962922469] conn(0x7f052c001650 0x7f052c003b10 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.471+0000 7f054bd87640 1 -- 172.21.15.150:0/2872659399 --> [v2:172.21.15.150:6800/1962922469,v1:172.21.15.150:6801/1962922469] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f052c006bb0 con 0x7f052c001650 2024-03-17T23:47:14.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.471+0000 7f6bdd6b1640 1 --2- 172.21.15.150:0/1890661654 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6bbc06bca0 0x7f6bbc06e160 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6bd814f310 tx=0x7f6bc8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.473+0000 7f9622086640 1 -- 172.21.15.150:0/3014085748 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f961c142730 con 0x7f9608005910 2024-03-17T23:47:14.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.473+0000 7f053a7fc640 1 -- 172.21.15.150:0/2872659399 <== osd.0 v2:172.21.15.150:6800/1962922469 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f052c006bb0 con 0x7f052c001650 2024-03-17T23:47:14.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.472+0000 7ff36e74f640 1 --2- 172.21.15.150:0/2538225631 >> [v2:172.21.15.150:6817/347494201,v1:172.21.15.150:6819/347494201] conn(0x7ff36017d670 0x7ff360037b40 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.473+0000 7ff35e7fc640 1 -- 172.21.15.150:0/2538225631 <== osd.2 v2:172.21.15.150:6817/347494201 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff3600ea2d0 con 0x7ff36017d670 2024-03-17T23:47:14.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.475+0000 7f9611ffb640 1 -- 172.21.15.150:0/3014085748 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f960c00a9a0 con 0x7f9608005910 2024-03-17T23:47:14.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.475+0000 7f9611ffb640 1 -- 172.21.15.150:0/3014085748 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f960c00ab80 con 0x7f9608005910 2024-03-17T23:47:14.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.476+0000 7f9611ffb640 1 --2- 172.21.15.150:0/3014085748 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f960006bc70 0x7f960006e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.476+0000 7f9611ffb640 1 -- 172.21.15.150:0/3014085748 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f960c090060 con 0x7f9608005910 2024-03-17T23:47:14.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.477+0000 7f6bde6b3640 1 --2- 172.21.15.150:0/1890661654 >> [v2:172.21.15.157:6808/4022520191,v1:172.21.15.157:6810/4022520191] conn(0x7f6bd80cd100 0x7f6bd8146850 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.476+0000 7f0549322640 1 --2- 172.21.15.150:0/2872659399 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f052806bcd0 0x7f052806e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.477+0000 7f0549322640 1 --2- 172.21.15.150:0/2872659399 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f052806bcd0 0x7f052806e190 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0534036000 tx=0x7f0534030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.477+0000 7f9620883640 1 --2- 172.21.15.150:0/3014085748 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f960006bc70 0x7f960006e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.478+0000 7f9622086640 1 --2- 172.21.15.150:0/3014085748 >> [v2:172.21.15.150:6816/3072406303,v1:172.21.15.150:6818/3072406303] conn(0x7f961c142930 0x7f961c15d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.478+0000 7f6bce7fc640 1 -- 172.21.15.150:0/1890661654 <== osd.4 v2:172.21.15.157:6808/4022520191 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6bd80cf890 con 0x7f6bd80cd100 2024-03-17T23:47:14.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.478+0000 7f9620883640 1 --2- 172.21.15.150:0/3014085748 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f960006bc70 0x7f960006e130 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9618034000 tx=0x7f9618007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.478+0000 7f9622086640 1 -- 172.21.15.150:0/3014085748 --> [v2:172.21.15.150:6816/3072406303,v1:172.21.15.150:6818/3072406303] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f961c141c40 con 0x7f961c142930 2024-03-17T23:47:14.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.478+0000 7f9621885640 1 --2- 172.21.15.150:0/3014085748 >> [v2:172.21.15.150:6816/3072406303,v1:172.21.15.150:6818/3072406303] conn(0x7f961c142930 0x7f961c15d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.479+0000 7f9621885640 1 --2- 172.21.15.150:0/3014085748 >> [v2:172.21.15.150:6816/3072406303,v1:172.21.15.150:6818/3072406303] conn(0x7f961c142930 0x7f961c15d7e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.480+0000 7f9611ffb640 1 -- 172.21.15.150:0/3014085748 <== osd.3 v2:172.21.15.150:6816/3072406303 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f961c141c40 con 0x7f961c142930 2024-03-17T23:47:14.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.496+0000 7f9699064640 1 -- 172.21.15.150:0/148270250 --> [v2:172.21.15.157:6824/1224877787,v1:172.21.15.157:6825/1224877787] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f968c052be0 con 0x7f96780753a0 2024-03-17T23:47:14.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.500+0000 7f9690ff9640 1 -- 172.21.15.150:0/148270250 <== osd.5 v2:172.21.15.157:6824/1224877787 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f968c052be0 con 0x7f96780753a0 2024-03-17T23:47:14.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.501+0000 7f967e7fc640 1 -- 172.21.15.150:0/148270250 >> [v2:172.21.15.157:6824/1224877787,v1:172.21.15.157:6825/1224877787] conn(0x7f96780753a0 msgr2=0x7f96780777e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.501+0000 7f967e7fc640 1 --2- 172.21.15.150:0/148270250 >> [v2:172.21.15.157:6824/1224877787,v1:172.21.15.157:6825/1224877787] conn(0x7f96780753a0 0x7f96780777e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.501+0000 7f967e7fc640 1 -- 172.21.15.150:0/148270250 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f967806bc20 msgr2=0x7f967806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.501+0000 7f967e7fc640 1 --2- 172.21.15.150:0/148270250 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f967806bc20 0x7f967806e0e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f968c04e540 tx=0x7f96800090e0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.501+0000 7f967e7fc640 1 -- 172.21.15.150:0/148270250 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f968c00c0a0 msgr2=0x7f968c167b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.501+0000 7f967e7fc640 1 --2- 172.21.15.150:0/148270250 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f968c00c0a0 0x7f968c167b10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9694050e40 tx=0x7f9694068f20 comp rx=0 tx=0).stop 2024-03-17T23:47:14.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.502+0000 7f9698863640 1 -- 172.21.15.150:0/148270250 reap_dead start 2024-03-17T23:47:14.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.502+0000 7f967e7fc640 1 -- 172.21.15.150:0/148270250 shutdown_connections 2024-03-17T23:47:14.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.502+0000 7f967e7fc640 1 -- 172.21.15.150:0/148270250 >> 172.21.15.150:0/148270250 conn(0x7f968c009fb0 msgr2=0x7f968c036590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:14.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.502+0000 7f967e7fc640 1 -- 172.21.15.150:0/148270250 shutdown_connections 2024-03-17T23:47:14.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.502+0000 7f967e7fc640 1 -- 172.21.15.150:0/148270250 wait complete. 2024-03-17T23:47:14.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.504+0000 7f1b2c8bd640 1 -- 172.21.15.150:0/2626332126 --> [v2:172.21.15.157:6800/624642970,v1:172.21.15.157:6801/624642970] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1b280681a0 con 0x7f1b080753a0 2024-03-17T23:47:14.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.504+0000 7fd567223640 1 -- 172.21.15.150:0/572098332 --> [v2:172.21.15.150:6808/1014841375,v1:172.21.15.150:6809/1014841375] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd56006efd0 con 0x7fd56014a2f0 2024-03-17T23:47:14.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.504+0000 7f1b16ffd640 1 -- 172.21.15.150:0/2626332126 <== osd.7 v2:172.21.15.157:6800/624642970 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1b280681a0 con 0x7f1b080753a0 2024-03-17T23:47:14.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.504+0000 7fafd49df640 1 -- 172.21.15.150:0/1871589140 --> [v2:172.21.15.157:6809/2764449896,v1:172.21.15.157:6811/2764449896] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fafd00681a0 con 0x7fafb0075410 2024-03-17T23:47:14.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.504+0000 7fd556ffd640 1 -- 172.21.15.150:0/572098332 <== osd.1 v2:172.21.15.150:6808/1014841375 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd56006efd0 con 0x7fd56014a2f0 2024-03-17T23:47:14.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.505+0000 7f1b2c8bd640 1 -- 172.21.15.150:0/2626332126 >> [v2:172.21.15.157:6800/624642970,v1:172.21.15.157:6801/624642970] conn(0x7f1b080753a0 msgr2=0x7f1b080777e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.505+0000 7f1b2c8bd640 1 --2- 172.21.15.150:0/2626332126 >> [v2:172.21.15.157:6800/624642970,v1:172.21.15.157:6801/624642970] conn(0x7f1b080753a0 0x7f1b080777e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.505+0000 7f1b2c8bd640 1 -- 172.21.15.150:0/2626332126 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1b0806bca0 msgr2=0x7f1b0806e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.505+0000 7f1b2c8bd640 1 --2- 172.21.15.150:0/2626332126 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1b0806bca0 0x7f1b0806e160 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1b28156cc0 tx=0x7f1b18008040 comp rx=0 tx=0).stop 2024-03-17T23:47:14.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.505+0000 7f1b2c8bd640 1 -- 172.21.15.150:0/2626332126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b28162e30 msgr2=0x7f1b280cd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.505+0000 7f1b2c8bd640 1 --2- 172.21.15.150:0/2626332126 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b28162e30 0x7f1b280cd640 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1b2000c7a0 tx=0x7f1b2000cc60 comp rx=0 tx=0).stop 2024-03-17T23:47:14.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.505+0000 7f1b26d9d640 1 -- 172.21.15.150:0/2626332126 reap_dead start 2024-03-17T23:47:14.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.505+0000 7f1b2c8bd640 1 -- 172.21.15.150:0/2626332126 shutdown_connections 2024-03-17T23:47:14.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.505+0000 7f1b2c8bd640 1 -- 172.21.15.150:0/2626332126 >> 172.21.15.150:0/2626332126 conn(0x7f1b28072ef0 msgr2=0x7f1b2806c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:14.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.505+0000 7fd567223640 1 -- 172.21.15.150:0/572098332 >> [v2:172.21.15.150:6808/1014841375,v1:172.21.15.150:6809/1014841375] conn(0x7fd56014a2f0 msgr2=0x7fd560165bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.505+0000 7fd567223640 1 --2- 172.21.15.150:0/572098332 >> [v2:172.21.15.150:6808/1014841375,v1:172.21.15.150:6809/1014841375] conn(0x7fd56014a2f0 0x7fd560165bb0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.505+0000 7fd567223640 1 -- 172.21.15.150:0/572098332 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd54406bca0 msgr2=0x7fd54406e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.505+0000 7f1b2c8bd640 1 -- 172.21.15.150:0/2626332126 shutdown_connections 2024-03-17T23:47:14.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.505+0000 7fd567223640 1 --2- 172.21.15.150:0/572098332 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd54406bca0 0x7fd54406e160 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd56006bf20 tx=0x7fd55c0090e0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.505+0000 7fafbeffd640 1 -- 172.21.15.150:0/1871589140 <== osd.6 v2:172.21.15.157:6809/2764449896 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fafd00681a0 con 0x7fafb0075410 2024-03-17T23:47:14.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.505+0000 7fd567223640 1 -- 172.21.15.150:0/572098332 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd560149e40 msgr2=0x7fd560160b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.505+0000 7fd567223640 1 --2- 172.21.15.150:0/572098332 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd560149e40 0x7fd560160b60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd55800ca70 tx=0x7fd55800cf30 comp rx=0 tx=0).stop 2024-03-17T23:47:14.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.505+0000 7f1b2c8bd640 1 -- 172.21.15.150:0/2626332126 wait complete. 2024-03-17T23:47:14.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.505+0000 7fd566a22640 1 -- 172.21.15.150:0/572098332 reap_dead start 2024-03-17T23:47:14.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.506+0000 7f054bd87640 1 -- 172.21.15.150:0/2872659399 --> [v2:172.21.15.150:6800/1962922469,v1:172.21.15.150:6801/1962922469] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f052c005c30 con 0x7f052c001650 2024-03-17T23:47:14.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.506+0000 7fd567223640 1 -- 172.21.15.150:0/572098332 shutdown_connections 2024-03-17T23:47:14.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.506+0000 7fd567223640 1 -- 172.21.15.150:0/572098332 >> 172.21.15.150:0/572098332 conn(0x7fd5600b17e0 msgr2=0x7fd56014e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:14.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.506+0000 7f6be0116640 1 -- 172.21.15.150:0/1890661654 --> [v2:172.21.15.157:6808/4022520191,v1:172.21.15.157:6810/4022520191] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6bd8163760 con 0x7f6bd80cd100 2024-03-17T23:47:14.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.506+0000 7fafd49df640 1 -- 172.21.15.150:0/1871589140 >> [v2:172.21.15.157:6809/2764449896,v1:172.21.15.157:6811/2764449896] conn(0x7fafb0075410 msgr2=0x7fafb0077850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.506+0000 7fafd49df640 1 --2- 172.21.15.150:0/1871589140 >> [v2:172.21.15.157:6809/2764449896,v1:172.21.15.157:6811/2764449896] conn(0x7fafb0075410 0x7fafb0077850 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.506+0000 7fafd49df640 1 -- 172.21.15.150:0/1871589140 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fafb006bca0 msgr2=0x7fafb006e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.506+0000 7fafd49df640 1 --2- 172.21.15.150:0/1871589140 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fafb006bca0 0x7fafb006e160 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009ce0 tx=0x7fafc800b040 comp rx=0 tx=0).stop 2024-03-17T23:47:14.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.506+0000 7fafd49df640 1 -- 172.21.15.150:0/1871589140 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafd0072ce0 msgr2=0x7fafd013bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.506+0000 7fafd49df640 1 --2- 172.21.15.150:0/1871589140 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fafd0072ce0 0x7fafd013bab0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fafc000c480 tx=0x7fafc000c940 comp rx=0 tx=0).stop 2024-03-17T23:47:14.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.506+0000 7fafced9d640 1 -- 172.21.15.150:0/1871589140 reap_dead start 2024-03-17T23:47:14.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.506+0000 7fafd49df640 1 -- 172.21.15.150:0/1871589140 shutdown_connections 2024-03-17T23:47:14.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.506+0000 7fafd49df640 1 -- 172.21.15.150:0/1871589140 >> 172.21.15.150:0/1871589140 conn(0x7fafd006b190 msgr2=0x7fafd00afbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:14.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.506+0000 7fd567223640 1 -- 172.21.15.150:0/572098332 shutdown_connections 2024-03-17T23:47:14.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.506+0000 7fd567223640 1 -- 172.21.15.150:0/572098332 wait complete. 2024-03-17T23:47:14.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.506+0000 7fafd49df640 1 -- 172.21.15.150:0/1871589140 shutdown_connections 2024-03-17T23:47:14.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.507+0000 7f6bce7fc640 1 -- 172.21.15.150:0/1890661654 <== osd.4 v2:172.21.15.157:6808/4022520191 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6bd8163760 con 0x7f6bd80cd100 2024-03-17T23:47:14.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.507+0000 7fafd49df640 1 -- 172.21.15.150:0/1871589140 wait complete. 2024-03-17T23:47:14.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.507+0000 7f053a7fc640 1 -- 172.21.15.150:0/2872659399 <== osd.0 v2:172.21.15.150:6800/1962922469 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f052c005c30 con 0x7f052c001650 2024-03-17T23:47:14.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.508+0000 7f6be0116640 1 -- 172.21.15.150:0/1890661654 >> [v2:172.21.15.157:6808/4022520191,v1:172.21.15.157:6810/4022520191] conn(0x7f6bd80cd100 msgr2=0x7f6bd8146850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.508+0000 7f6be0116640 1 --2- 172.21.15.150:0/1890661654 >> [v2:172.21.15.157:6808/4022520191,v1:172.21.15.157:6810/4022520191] conn(0x7f6bd80cd100 0x7f6bd8146850 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.508+0000 7f054bd87640 1 -- 172.21.15.150:0/2872659399 >> [v2:172.21.15.150:6800/1962922469,v1:172.21.15.150:6801/1962922469] conn(0x7f052c001650 msgr2=0x7f052c003b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.508+0000 7f054bd87640 1 --2- 172.21.15.150:0/2872659399 >> [v2:172.21.15.150:6800/1962922469,v1:172.21.15.150:6801/1962922469] conn(0x7f052c001650 0x7f052c003b10 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.508+0000 7f054bd87640 1 -- 172.21.15.150:0/2872659399 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f052806bcd0 msgr2=0x7f052806e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.508+0000 7f054bd87640 1 --2- 172.21.15.150:0/2872659399 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f052806bcd0 0x7f052806e190 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0534036000 tx=0x7f0534030000 comp rx=0 tx=0).stop 2024-03-17T23:47:14.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.508+0000 7f054bd87640 1 -- 172.21.15.150:0/2872659399 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05441f81d0 msgr2=0x7f05441f85b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.508+0000 7f054bd87640 1 --2- 172.21.15.150:0/2872659399 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f05441f81d0 0x7f05441f85b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f054000f940 tx=0x7f054000fd40 comp rx=0 tx=0).stop 2024-03-17T23:47:14.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.508+0000 7f054a324640 1 -- 172.21.15.150:0/2872659399 reap_dead start 2024-03-17T23:47:14.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.508+0000 7f054bd87640 1 -- 172.21.15.150:0/2872659399 shutdown_connections 2024-03-17T23:47:14.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.508+0000 7f054bd87640 1 -- 172.21.15.150:0/2872659399 >> 172.21.15.150:0/2872659399 conn(0x7f05440b36a0 msgr2=0x7f05440aefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:14.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.508+0000 7f054bd87640 1 -- 172.21.15.150:0/2872659399 shutdown_connections 2024-03-17T23:47:14.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.508+0000 7f054bd87640 1 -- 172.21.15.150:0/2872659399 wait complete. 2024-03-17T23:47:14.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.509+0000 7f6be0116640 1 -- 172.21.15.150:0/1890661654 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6bbc06bca0 msgr2=0x7f6bbc06e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.509+0000 7f6be0116640 1 --2- 172.21.15.150:0/1890661654 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6bbc06bca0 0x7f6bbc06e160 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6bd814f310 tx=0x7f6bc8008040 comp rx=0 tx=0).stop 2024-03-17T23:47:14.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.509+0000 7f6be0116640 1 -- 172.21.15.150:0/1890661654 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bd80cf2a0 msgr2=0x7f6bd814edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.509+0000 7f6be0116640 1 --2- 172.21.15.150:0/1890661654 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bd80cf2a0 0x7f6bd814edd0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40090b0 tx=0x7f6bd40046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.509+0000 7f6bde6b3640 1 -- 172.21.15.150:0/1890661654 reap_dead start 2024-03-17T23:47:14.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.509+0000 7f6be0116640 1 -- 172.21.15.150:0/1890661654 shutdown_connections 2024-03-17T23:47:14.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.509+0000 7f6be0116640 1 -- 172.21.15.150:0/1890661654 >> 172.21.15.150:0/1890661654 conn(0x7f6bd8099a70 msgr2=0x7f6bd80c90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:14.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.510+0000 7f6be0116640 1 -- 172.21.15.150:0/1890661654 shutdown_connections 2024-03-17T23:47:14.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.510+0000 7f6be0116640 1 -- 172.21.15.150:0/1890661654 wait complete. 2024-03-17T23:47:14.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.516+0000 7f9622086640 1 -- 172.21.15.150:0/3014085748 --> [v2:172.21.15.150:6816/3072406303,v1:172.21.15.150:6818/3072406303] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f961c074bd0 con 0x7f961c142930 2024-03-17T23:47:14.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.517+0000 7f9611ffb640 1 -- 172.21.15.150:0/3014085748 <== osd.3 v2:172.21.15.150:6816/3072406303 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f961c074bd0 con 0x7f961c142930 2024-03-17T23:47:14.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.518+0000 7f96077fe640 1 -- 172.21.15.150:0/3014085748 >> [v2:172.21.15.150:6816/3072406303,v1:172.21.15.150:6818/3072406303] conn(0x7f961c142930 msgr2=0x7f961c15d7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.518+0000 7f96077fe640 1 --2- 172.21.15.150:0/3014085748 >> [v2:172.21.15.150:6816/3072406303,v1:172.21.15.150:6818/3072406303] conn(0x7f961c142930 0x7f961c15d7e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.518+0000 7f96077fe640 1 -- 172.21.15.150:0/3014085748 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f960006bc70 msgr2=0x7f960006e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.518+0000 7f96077fe640 1 --2- 172.21.15.150:0/3014085748 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f960006bc70 0x7f960006e130 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9618034000 tx=0x7f9618007000 comp rx=0 tx=0).stop 2024-03-17T23:47:14.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.518+0000 7f96077fe640 1 -- 172.21.15.150:0/3014085748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9608005910 msgr2=0x7f961c13f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.518+0000 7f96077fe640 1 --2- 172.21.15.150:0/3014085748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9608005910 0x7f961c13f3e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f960c00ee50 tx=0x7f960c00c790 comp rx=0 tx=0).stop 2024-03-17T23:47:14.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.518+0000 7f9621885640 1 -- 172.21.15.150:0/3014085748 reap_dead start 2024-03-17T23:47:14.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.519+0000 7f96077fe640 1 -- 172.21.15.150:0/3014085748 shutdown_connections 2024-03-17T23:47:14.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.519+0000 7f96077fe640 1 -- 172.21.15.150:0/3014085748 >> 172.21.15.150:0/3014085748 conn(0x7f961c0b1600 msgr2=0x7f961c0cebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:14.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.519+0000 7f96077fe640 1 -- 172.21.15.150:0/3014085748 shutdown_connections 2024-03-17T23:47:14.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.519+0000 7ff36ef50640 1 -- 172.21.15.150:0/2538225631 --> [v2:172.21.15.150:6817/347494201,v1:172.21.15.150:6819/347494201] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff360005070 con 0x7ff36017d670 2024-03-17T23:47:14.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.519+0000 7f96077fe640 1 -- 172.21.15.150:0/3014085748 wait complete. 2024-03-17T23:47:14.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.519+0000 7ff35e7fc640 1 -- 172.21.15.150:0/2538225631 <== osd.2 v2:172.21.15.150:6817/347494201 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff360005070 con 0x7ff36017d670 2024-03-17T23:47:14.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.521+0000 7ff34bfff640 1 -- 172.21.15.150:0/2538225631 >> [v2:172.21.15.150:6817/347494201,v1:172.21.15.150:6819/347494201] conn(0x7ff36017d670 msgr2=0x7ff360037b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.521+0000 7ff34bfff640 1 --2- 172.21.15.150:0/2538225631 >> [v2:172.21.15.150:6817/347494201,v1:172.21.15.150:6819/347494201] conn(0x7ff36017d670 0x7ff360037b40 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.522+0000 7ff34bfff640 1 -- 172.21.15.150:0/2538225631 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff34c06bfd0 msgr2=0x7ff34c06e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.522+0000 7ff34bfff640 1 --2- 172.21.15.150:0/2538225631 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff34c06bfd0 0x7ff34c06e490 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff3600544b0 tx=0x7ff3680ab000 comp rx=0 tx=0).stop 2024-03-17T23:47:14.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.522+0000 7ff34bfff640 1 -- 172.21.15.150:0/2538225631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff36004ffd0 msgr2=0x7ff36017d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.522+0000 7ff34bfff640 1 --2- 172.21.15.150:0/2538225631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff36004ffd0 0x7ff36017d130 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff35800c490 tx=0x7ff35800c950 comp rx=0 tx=0).stop 2024-03-17T23:47:14.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.522+0000 7ff36e74f640 1 -- 172.21.15.150:0/2538225631 reap_dead start 2024-03-17T23:47:14.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.522+0000 7ff34bfff640 1 -- 172.21.15.150:0/2538225631 shutdown_connections 2024-03-17T23:47:14.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.522+0000 7ff34bfff640 1 -- 172.21.15.150:0/2538225631 >> 172.21.15.150:0/2538225631 conn(0x7ff360009fb0 msgr2=0x7ff36003a320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:14.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.522+0000 7ff34bfff640 1 -- 172.21.15.150:0/2538225631 shutdown_connections 2024-03-17T23:47:14.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.523+0000 7ff34bfff640 1 -- 172.21.15.150:0/2538225631 wait complete. 2024-03-17T23:47:14.571 INFO:teuthology.orchestra.run.smithi150.stdout:55834574857 2024-03-17T23:47:14.571 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-17T23:47:14.572 INFO:teuthology.orchestra.run.smithi150.stdout:55834574858 2024-03-17T23:47:14.573 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-17T23:47:14.573 INFO:teuthology.orchestra.run.smithi150.stdout:55834574857 2024-03-17T23:47:14.574 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-17T23:47:14.574 INFO:teuthology.orchestra.run.smithi150.stdout:55834574857 2024-03-17T23:47:14.575 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-17T23:47:14.575 INFO:teuthology.orchestra.run.smithi150.stdout:60129542154 2024-03-17T23:47:14.576 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-17T23:47:14.576 INFO:teuthology.orchestra.run.smithi150.stdout:55834574857 2024-03-17T23:47:14.577 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-17T23:47:14.577 INFO:teuthology.orchestra.run.smithi150.stdout:60129542154 2024-03-17T23:47:14.578 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-17T23:47:14.578 INFO:teuthology.orchestra.run.smithi150.stdout:60129542154 2024-03-17T23:47:14.579 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-17T23:47:14.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.782+0000 7f292575b640 1 Processor -- start 2024-03-17T23:47:14.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.782+0000 7f292575b640 1 -- start start 2024-03-17T23:47:14.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.779+0000 7f1105df4640 1 Processor -- start 2024-03-17T23:47:14.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.779+0000 7f1105df4640 1 -- start start 2024-03-17T23:47:14.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.780+0000 7f1105df4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11000b3b40 0x7f11000cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.780+0000 7f1105df4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11000ccb80 con 0x7f11000ce5a0 2024-03-17T23:47:14.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.780+0000 7f1105df4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11000b4120 con 0x7f11000ce1c0 2024-03-17T23:47:14.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.780+0000 7f1105df4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f11000cf500 con 0x7f11000b3b40 2024-03-17T23:47:14.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.781+0000 7f10ff7fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f11000ce1c0 0x7f11001629e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:37880/0 (socket says 172.21.15.150:37880) 2024-03-17T23:47:14.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.781+0000 7f10ff7fe640 1 -- 172.21.15.150:0/57358240 learned_addr learned my addr 172.21.15.150:0/57358240 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:14.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.781+0000 7f10feffd640 1 --2- 172.21.15.150:0/57358240 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11000b3b40 0x7f11000cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.781+0000 7f10fe7fc640 1 -- 172.21.15.150:0/57358240 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 168356193 0 0) 0x7f11000b4120 con 0x7f11000ce1c0 2024-03-17T23:47:14.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.781+0000 7f10fe7fc640 1 -- 172.21.15.150:0/57358240 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10e8003520 con 0x7f11000ce1c0 2024-03-17T23:47:14.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.781+0000 7f10fe7fc640 1 -- 172.21.15.150:0/57358240 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2653606527 0 0) 0x7f10e8003520 con 0x7f11000ce1c0 2024-03-17T23:47:14.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.781+0000 7f10fe7fc640 1 -- 172.21.15.150:0/57358240 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11000b3b40 msgr2=0x7f11000cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.781+0000 7f10fe7fc640 1 --2- 172.21.15.150:0/57358240 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11000b3b40 0x7f11000cc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.781+0000 7f10fe7fc640 1 -- 172.21.15.150:0/57358240 >> v1:172.21.15.150:6789/0 conn(0x7f11000ce5a0 legacy=0x7f110006eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.782+0000 7f10fe7fc640 1 -- 172.21.15.150:0/57358240 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11000cef40 con 0x7f11000ce1c0 2024-03-17T23:47:14.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.782+0000 7f10fe7fc640 1 -- 172.21.15.150:0/57358240 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f10f80019f0 con 0x7f11000ce1c0 2024-03-17T23:47:14.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.782+0000 7f10fe7fc640 1 -- 172.21.15.150:0/57358240 >> v1:172.21.15.157:6789/0 conn(0x7f11000ce1c0 legacy=0x7f11001629e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.782+0000 7f10fe7fc640 1 --2- 172.21.15.150:0/57358240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10e80039b0 0x7f10e8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.782+0000 7f10fe7fc640 1 --2- 172.21.15.150:0/57358240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10e8004800 0x7f10e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.782+0000 7f10fe7fc640 1 --2- 172.21.15.150:0/57358240 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11000b3b40 0x7f10e80051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.782+0000 7f10fe7fc640 1 -- 172.21.15.150:0/57358240 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f11000cf500 con 0x7f10e80039b0 2024-03-17T23:47:14.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.782+0000 7f10fe7fc640 1 -- 172.21.15.150:0/57358240 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f10e802dae0 con 0x7f10e8004800 2024-03-17T23:47:14.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.782+0000 7f10fe7fc640 1 -- 172.21.15.150:0/57358240 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f10e802dce0 con 0x7f11000b3b40 2024-03-17T23:47:14.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.783+0000 7f10fffff640 1 --2- 172.21.15.150:0/57358240 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11000b3b40 0x7f10e80051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.783+0000 7f10feffd640 1 --2- 172.21.15.150:0/57358240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10e8004800 0x7f10e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.783+0000 7f10ff7fe640 1 --2- 172.21.15.150:0/57358240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10e80039b0 0x7f10e8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.784+0000 7f292575b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2920162450 0x7f292006f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.784+0000 7f292575b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f29200cf2c0 0x7f292006fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.784+0000 7f292575b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29200b5000 con 0x7f29200cc930 2024-03-17T23:47:14.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.784+0000 7f292575b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f29200b52e0 con 0x7f2920162450 2024-03-17T23:47:14.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.784+0000 7f292575b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f292006eb70 con 0x7f29200cf2c0 2024-03-17T23:47:14.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.784+0000 7f291f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f29200cf2c0 0x7f292006fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.784+0000 7f291f7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f29200cf2c0 0x7f292006fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:51266/0 (socket says 172.21.15.150:51266) 2024-03-17T23:47:14.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.784+0000 7f291f7fe640 1 -- 172.21.15.150:0/1720195848 learned_addr learned my addr 172.21.15.150:0/1720195848 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.784+0000 7f76c5547640 1 Processor -- start 2024-03-17T23:47:14.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.784+0000 7f10ff7fe640 1 -- 172.21.15.150:0/57358240 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11000b3b40 msgr2=0x7f10e80051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.784+0000 7f291f7fe640 1 -- 172.21.15.150:0/1720195848 >> v2:172.21.15.157:3300/0 conn(0x7f2920162450 msgr2=0x7f292006f4c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-17T23:47:14.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.784+0000 7f291f7fe640 1 --2- 172.21.15.150:0/1720195848 >> v2:172.21.15.157:3300/0 conn(0x7f2920162450 0x7f292006f4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.784+0000 7f0a8279d640 1 Processor -- start 2024-03-17T23:47:14.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.784+0000 7f291f7fe640 1 -- 172.21.15.150:0/1720195848 >> v1:172.21.15.150:6789/0 conn(0x7f29200cc930 legacy=0x7f29200b3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.784+0000 7f291f7fe640 1 -- 172.21.15.150:0/1720195848 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f292006ee50 con 0x7f29200cf2c0 2024-03-17T23:47:14.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.784+0000 7f10ff7fe640 1 --2- 172.21.15.150:0/57358240 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11000b3b40 0x7f10e80051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.784+0000 7f10ff7fe640 1 -- 172.21.15.150:0/57358240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10e8004800 msgr2=0x7f10e8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.784+0000 7f10ff7fe640 1 --2- 172.21.15.150:0/57358240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10e8004800 0x7f10e8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.784+0000 7f10ff7fe640 1 -- 172.21.15.150:0/57358240 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10e8003520 con 0x7f10e80039b0 2024-03-17T23:47:14.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.785+0000 7fd62759e640 1 Processor -- start 2024-03-17T23:47:14.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.785+0000 7f0a8279d640 1 -- start start 2024-03-17T23:47:14.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.785+0000 7fd62759e640 1 -- start start 2024-03-17T23:47:14.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.785+0000 7f76c5547640 1 -- start start 2024-03-17T23:47:14.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.784+0000 7fd73eaf0640 1 Processor -- start 2024-03-17T23:47:14.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.784+0000 7fd73eaf0640 1 -- start start 2024-03-17T23:47:14.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.785+0000 7fd73eaf0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd730053c60 0x7fd730038070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.785+0000 7fd73eaf0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7300544b0 con 0x7fd730033ee0 2024-03-17T23:47:14.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.785+0000 7fd73eaf0640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7300541d0 con 0x7fd73000c090 2024-03-17T23:47:14.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.785+0000 7fd73eaf0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd730005f20 con 0x7fd730053c60 2024-03-17T23:47:14.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7f0a8279d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0a740e30b0 0x7f0a74057d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7f0a8279d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0a7403e830 0x7f0a74053be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7f0a8279d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a74054120 0x7f0a740e6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7f0a8279d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f0a74054590 con 0x7f0a7403e830 2024-03-17T23:47:14.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7f0a8279d640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f0a740582c0 con 0x7f0a740e30b0 2024-03-17T23:47:14.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7f0a8279d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0a74058560 con 0x7f0a74054120 2024-03-17T23:47:14.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7fd73e2ef640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd730033ee0 0x7fd73004fa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59414/0 (socket says 172.21.15.150:59414) 2024-03-17T23:47:14.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7f0a81f9c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a74054120 0x7f0a740e6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7f76c5547640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f76c01621a0 0x7f76c006eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7f76c5547640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f76c0165cc0 0x7f76c0162580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7fd73e2ef640 1 -- 172.21.15.150:0/4170214923 learned_addr learned my addr 172.21.15.150:0/4170214923 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:14.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7fd73d2ed640 1 --2- 172.21.15.150:0/4170214923 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd730053c60 0x7fd730038070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7f76c5547640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76c00d14d0 con 0x7f76c00c3810 2024-03-17T23:47:14.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7f76c5547640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f76c0162ac0 con 0x7f76c01621a0 2024-03-17T23:47:14.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7f76c5547640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f76c00cf0d0 con 0x7f76c0165cc0 2024-03-17T23:47:14.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.785+0000 7fa19c644640 1 Processor -- start 2024-03-17T23:47:14.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.785+0000 7fa19c644640 1 -- start start 2024-03-17T23:47:14.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7f0a80f9a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0a7403e830 0x7f0a74053be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7f0a81f9c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a74054120 0x7f0a740e6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:51270/0 (socket says 172.21.15.150:51270) 2024-03-17T23:47:14.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7f76c4d46640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f76c00c3810 0x7f76c00cf830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:37910/0 (socket says 172.21.15.150:37910) 2024-03-17T23:47:14.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7f76c4d46640 1 -- 172.21.15.150:0/2185052089 learned_addr learned my addr 172.21.15.150:0/2185052089 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:14.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7f76bf7fe640 1 --2- 172.21.15.150:0/2185052089 >> v2:172.21.15.157:3300/0 conn(0x7f76c0165cc0 0x7f76c0162580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7f291f7fe640 1 --2- 172.21.15.150:0/1720195848 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f29200cf2c0 0x7f292006fa00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2914009870 tx=0x7f291402b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eda6fdcb26280396 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7f0a81f9c640 1 -- 172.21.15.150:0/400999188 learned_addr learned my addr 172.21.15.150:0/400999188 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7fd73caec640 1 -- 172.21.15.150:0/4170214923 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3610717640 0 0) 0x7fd7300541d0 con 0x7fd73000c090 2024-03-17T23:47:14.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7f0a80f9a640 1 -- 172.21.15.150:0/400999188 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a74054120 msgr2=0x7f0a740e6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7f0a80f9a640 1 --2- 172.21.15.150:0/400999188 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a74054120 0x7f0a740e6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7f0a80f9a640 1 -- 172.21.15.150:0/400999188 >> v2:172.21.15.157:3300/0 conn(0x7f0a740e30b0 msgr2=0x7f0a74057d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7f0a8179b640 1 --2- 172.21.15.150:0/400999188 >> v2:172.21.15.157:3300/0 conn(0x7f0a740e30b0 0x7f0a74057d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7f0a80f9a640 1 --2- 172.21.15.150:0/400999188 >> v2:172.21.15.157:3300/0 conn(0x7f0a740e30b0 0x7f0a74057d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.785+0000 7f10ff7fe640 1 --2- 172.21.15.150:0/57358240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10e80039b0 0x7f10e8003d90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f10f800a7b0 tx=0x7f10e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ded143a5fe1ef507 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.785+0000 7f10fe7fc640 1 -- 172.21.15.150:0/57358240 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f800a3d0 con 0x7f10e80039b0 2024-03-17T23:47:14.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.785+0000 7f10fe7fc640 1 -- 172.21.15.150:0/57358240 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10f8034630 con 0x7f10e80039b0 2024-03-17T23:47:14.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd62759e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd620050190 0x7fd620038070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7f0a80f9a640 1 -- 172.21.15.150:0/400999188 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a74058840 con 0x7f0a7403e830 2024-03-17T23:47:14.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7f10fe7fc640 1 -- 172.21.15.150:0/57358240 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f8034aa0 con 0x7f10e80039b0 2024-03-17T23:47:14.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7f76bffff640 1 --2- 172.21.15.150:0/2185052089 >> v2:172.21.15.150:3300/0 conn(0x7f76c01621a0 0x7f76c006eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7f76bf7fe640 1 -- 172.21.15.150:0/2185052089 >> v2:172.21.15.150:3300/0 conn(0x7f76c01621a0 msgr2=0x7f76c006eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7f76bf7fe640 1 --2- 172.21.15.150:0/2185052089 >> v2:172.21.15.150:3300/0 conn(0x7f76c01621a0 0x7f76c006eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7f76bf7fe640 1 -- 172.21.15.150:0/2185052089 >> v1:172.21.15.157:6789/0 conn(0x7f76c00c3810 legacy=0x7f76c00cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7f76bf7fe640 1 -- 172.21.15.150:0/2185052089 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76c0166120 con 0x7f76c0165cc0 2024-03-17T23:47:14.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.786+0000 7fd73caec640 1 -- 172.21.15.150:0/4170214923 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd724003520 con 0x7fd73000c090 2024-03-17T23:47:14.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd73caec640 1 -- 172.21.15.150:0/4170214923 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2433720229 0 0) 0x7fd7300544b0 con 0x7fd730033ee0 2024-03-17T23:47:14.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd62759e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6200544b0 con 0x7fd620053c60 2024-03-17T23:47:14.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd62659c640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fd6200df210 0x7fd62004fa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:37920/0 (socket says 172.21.15.150:37920) 2024-03-17T23:47:14.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd73caec640 1 -- 172.21.15.150:0/4170214923 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7300541d0 con 0x7fd730033ee0 2024-03-17T23:47:14.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd62659c640 1 -- 172.21.15.150:0/3206027951 learned_addr learned my addr 172.21.15.150:0/3206027951 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:14.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd62759e640 1 -- 172.21.15.150:0/3206027951 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6200541d0 con 0x7fd6200df210 2024-03-17T23:47:14.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd62759e640 1 -- 172.21.15.150:0/3206027951 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd620005f20 con 0x7fd620050190 2024-03-17T23:47:14.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd626d9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd620053c60 0x7fd620006a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59420/0 (socket says 172.21.15.150:59420) 2024-03-17T23:47:14.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd625d9b640 1 --2- 172.21.15.150:0/3206027951 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd620050190 0x7fd620038070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd73caec640 1 -- 172.21.15.150:0/4170214923 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 162163892 0 0) 0x7fd7300541d0 con 0x7fd730033ee0 2024-03-17T23:47:14.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7f291dffb640 1 -- 172.21.15.150:0/1720195848 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2914034070 con 0x7f29200cf2c0 2024-03-17T23:47:14.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd73caec640 1 -- 172.21.15.150:0/4170214923 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd730053c60 msgr2=0x7fd730038070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7f291dffb640 1 -- 172.21.15.150:0/1720195848 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2914030860 con 0x7f29200cf2c0 2024-03-17T23:47:14.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7f291dffb640 1 -- 172.21.15.150:0/1720195848 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2914030c30 con 0x7f29200cf2c0 2024-03-17T23:47:14.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd73caec640 1 --2- 172.21.15.150:0/4170214923 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd730053c60 0x7fd730038070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd73caec640 1 -- 172.21.15.150:0/4170214923 >> v1:172.21.15.157:6789/0 conn(0x7fd73000c090 legacy=0x7fd7300df210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd73caec640 1 -- 172.21.15.150:0/4170214923 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd730006200 con 0x7fd730033ee0 2024-03-17T23:47:14.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd62559a640 1 -- 172.21.15.150:0/3206027951 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3953598751 0 0) 0x7fd6200541d0 con 0x7fd6200df210 2024-03-17T23:47:14.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd73caec640 1 -- 172.21.15.150:0/4170214923 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7fd738075300 con 0x7fd730033ee0 2024-03-17T23:47:14.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd73caec640 1 -- 172.21.15.150:0/4170214923 >> v1:172.21.15.150:6789/0 conn(0x7fd730033ee0 legacy=0x7fd73004fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd73caec640 1 --2- 172.21.15.150:0/4170214923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd724004bd0 0x7fd724004fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd73caec640 1 --2- 172.21.15.150:0/4170214923 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7240037a0 0x7fd724003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd73caec640 1 --2- 172.21.15.150:0/4170214923 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd724004160 0x7fd72400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd73caec640 1 -- 172.21.15.150:0/4170214923 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd730005f20 con 0x7fd724004bd0 2024-03-17T23:47:14.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd73caec640 1 -- 172.21.15.150:0/4170214923 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd72400bf30 con 0x7fd7240037a0 2024-03-17T23:47:14.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd73caec640 1 -- 172.21.15.150:0/4170214923 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd72400c130 con 0x7fd724004160 2024-03-17T23:47:14.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd73e2ef640 1 --2- 172.21.15.150:0/4170214923 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd724004160 0x7fd72400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fd73d2ed640 1 --2- 172.21.15.150:0/4170214923 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7240037a0 0x7fd724003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fd73daee640 1 --2- 172.21.15.150:0/4170214923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd724004bd0 0x7fd724004fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7f76bffff640 1 --2- 172.21.15.150:0/2185052089 >> v2:172.21.15.150:3300/0 conn(0x7f76c01621a0 0x7f76c006eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:47:14.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fa19c644640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa1940b9250 0x7fa1940d3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fa19c644640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa1940b9630 0x7fa1940b8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fa199bdf640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa1940b9630 0x7fa1940b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fa199bdf640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa1940b9630 0x7fa1940b8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36476/0 (socket says 172.21.15.150:36476) 2024-03-17T23:47:14.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd62559a640 1 -- 172.21.15.150:0/3206027951 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd610003520 con 0x7fd6200df210 2024-03-17T23:47:14.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.787+0000 7fd62559a640 1 -- 172.21.15.150:0/3206027951 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 684311667 0 0) 0x7fd6200544b0 con 0x7fd620053c60 2024-03-17T23:47:14.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fd62559a640 1 -- 172.21.15.150:0/3206027951 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6200541d0 con 0x7fd620053c60 2024-03-17T23:47:14.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fd62559a640 1 -- 172.21.15.150:0/3206027951 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3812852170 0 0) 0x7fd610003520 con 0x7fd6200df210 2024-03-17T23:47:14.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fd62559a640 1 -- 172.21.15.150:0/3206027951 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd620050190 msgr2=0x7fd620038070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fd62559a640 1 --2- 172.21.15.150:0/3206027951 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd620050190 0x7fd620038070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fd62559a640 1 -- 172.21.15.150:0/3206027951 >> v1:172.21.15.150:6789/0 conn(0x7fd620053c60 legacy=0x7fd620006a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fd62559a640 1 -- 172.21.15.150:0/3206027951 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd620006200 con 0x7fd6200df210 2024-03-17T23:47:14.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fd62559a640 1 -- 172.21.15.150:0/3206027951 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7fd628075300 con 0x7fd6200df210 2024-03-17T23:47:14.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7f5ce1463640 1 Processor -- start 2024-03-17T23:47:14.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fd62559a640 1 -- 172.21.15.150:0/3206027951 >> v1:172.21.15.157:6789/0 conn(0x7fd6200df210 legacy=0x7fd62004fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7f5ce1463640 1 -- start start 2024-03-17T23:47:14.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fd62559a640 1 --2- 172.21.15.150:0/3206027951 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd620050190 0x7fd610003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fd62559a640 1 --2- 172.21.15.150:0/3206027951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6100040a0 0x7fd6100099b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fd62559a640 1 --2- 172.21.15.150:0/3206027951 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd610009ef0 0x7fd61000c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fa19c644640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1940d3a90 con 0x7fa1940b89d0 2024-03-17T23:47:14.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fa19c644640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fa1940d1da0 con 0x7fa1940b9630 2024-03-17T23:47:14.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fa19c644640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fa1940d2080 con 0x7fa1940b9250 2024-03-17T23:47:14.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fd62559a640 1 -- 172.21.15.150:0/3206027951 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd620005f20 con 0x7fd6100040a0 2024-03-17T23:47:14.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fd62559a640 1 -- 172.21.15.150:0/3206027951 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd61000ca60 con 0x7fd610009ef0 2024-03-17T23:47:14.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fa199bdf640 1 -- 172.21.15.150:0/2874387998 learned_addr learned my addr 172.21.15.150:0/2874387998 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fd62559a640 1 -- 172.21.15.150:0/3206027951 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd61000cc60 con 0x7fd620050190 2024-03-17T23:47:14.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7f0a80f9a640 1 --2- 172.21.15.150:0/400999188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a7403e830 0x7f0a74053be0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0a74056950 tx=0x7f0a7802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1638188320c2774f server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7f292575b640 1 -- 172.21.15.150:0/1720195848 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f29200cf2c0 msgr2=0x7f292006fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7f292575b640 1 --2- 172.21.15.150:0/1720195848 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f29200cf2c0 0x7f292006fa00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2914009870 tx=0x7f291402b9a0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7f1105df4640 1 -- 172.21.15.150:0/57358240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10e80039b0 msgr2=0x7f10e8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7f1105df4640 1 --2- 172.21.15.150:0/57358240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10e80039b0 0x7f10e8003d90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f10f800a7b0 tx=0x7f10e8005830 comp rx=0 tx=0).stop 2024-03-17T23:47:14.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7f10fffff640 1 -- 172.21.15.150:0/57358240 reap_dead start 2024-03-17T23:47:14.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7f292575b640 1 -- 172.21.15.150:0/1720195848 shutdown_connections 2024-03-17T23:47:14.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7f292575b640 1 --2- 172.21.15.150:0/1720195848 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f29200cf2c0 0x7f292006fa00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7f292575b640 1 --2- 172.21.15.150:0/1720195848 >> v2:172.21.15.157:3300/0 conn(0x7f2920162450 0x7f292006f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7f292575b640 1 -- 172.21.15.150:0/1720195848 >> 172.21.15.150:0/1720195848 conn(0x7f292006de70 msgr2=0x7f292006e250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:14.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fa19a3e0640 1 --2- 172.21.15.150:0/2874387998 >> v2:172.21.15.157:3300/0 conn(0x7fa1940b9250 0x7fa1940d3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7f1105df4640 1 -- 172.21.15.150:0/57358240 shutdown_connections 2024-03-17T23:47:14.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7f1105df4640 1 -- 172.21.15.150:0/57358240 >> 172.21.15.150:0/57358240 conn(0x7f110006b190 msgr2=0x7f110006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:14.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fd626d9d640 1 --2- 172.21.15.150:0/3206027951 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd610009ef0 0x7fd61000c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.789+0000 7fd62659c640 1 --2- 172.21.15.150:0/3206027951 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd620050190 0x7fd610003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.789+0000 7f0a73fff640 1 -- 172.21.15.150:0/400999188 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a78035020 con 0x7f0a7403e830 2024-03-17T23:47:14.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.789+0000 7fd626d9d640 1 -- 172.21.15.150:0/3206027951 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd620050190 msgr2=0x7fd610003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.789+0000 7fd626d9d640 1 --2- 172.21.15.150:0/3206027951 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd620050190 0x7fd610003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.789+0000 7fd626d9d640 1 -- 172.21.15.150:0/3206027951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6100040a0 msgr2=0x7fd6100099b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-17T23:47:14.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.789+0000 7f0a73fff640 1 -- 172.21.15.150:0/400999188 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a78031830 con 0x7f0a7403e830 2024-03-17T23:47:14.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.789+0000 7f0a73fff640 1 -- 172.21.15.150:0/400999188 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a78031c00 con 0x7f0a7403e830 2024-03-17T23:47:14.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.789+0000 7fd625d9b640 1 --2- 172.21.15.150:0/3206027951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6100040a0 0x7fd6100099b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.789+0000 7f292575b640 1 -- 172.21.15.150:0/1720195848 shutdown_connections 2024-03-17T23:47:14.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7f76bf7fe640 1 --2- 172.21.15.150:0/2185052089 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76c0165cc0 0x7f76c0162580 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f76c0070df0 tx=0x7f76b002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f17b99940d54ea9e server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.789+0000 7f76beffd640 1 -- 172.21.15.150:0/2185052089 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b0035020 con 0x7f76c0165cc0 2024-03-17T23:47:14.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.789+0000 7f76beffd640 1 -- 172.21.15.150:0/2185052089 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76b0031640 con 0x7f76c0165cc0 2024-03-17T23:47:14.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.789+0000 7f76beffd640 1 -- 172.21.15.150:0/2185052089 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b002d040 con 0x7f76c0165cc0 2024-03-17T23:47:14.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.789+0000 7f5ce1463640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5cdc13d080 0x7f5cdc06dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.789+0000 7f1105df4640 1 -- 172.21.15.150:0/57358240 shutdown_connections 2024-03-17T23:47:14.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.789+0000 7f1105df4640 1 -- 172.21.15.150:0/57358240 wait complete. 2024-03-17T23:47:14.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f5ce1463640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5cdc0cf640 0x7f5cdc13f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f5cdaffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5cdc13d080 0x7f5cdc06dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f5cdaffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5cdc13d080 0x7f5cdc06dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:51310/0 (socket says 172.21.15.150:51310) 2024-03-17T23:47:14.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f0a8279d640 1 -- 172.21.15.150:0/400999188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a7403e830 msgr2=0x7f0a74053be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7fd626d9d640 1 --2- 172.21.15.150:0/3206027951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6100040a0 0x7fd6100099b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f5cda7fc640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f5cdc06e270 0x7f5cdc0cef30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:37936/0 (socket says 172.21.15.150:37936) 2024-03-17T23:47:14.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f5cdaffd640 1 -- 172.21.15.150:0/3445452277 learned_addr learned my addr 172.21.15.150:0/3445452277 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f5ce1463640 1 -- 172.21.15.150:0/3445452277 --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cdc0ce490 con 0x7f5cdc06e270 2024-03-17T23:47:14.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7fd626d9d640 1 -- 172.21.15.150:0/3206027951 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd610003520 con 0x7fd610009ef0 2024-03-17T23:47:14.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f5ce1463640 1 -- 172.21.15.150:0/3445452277 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f5cdc0cca10 con 0x7f5cdc0cf640 2024-03-17T23:47:14.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f5ce1463640 1 -- 172.21.15.150:0/3445452277 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5cdc0cccf0 con 0x7f5cdc13d080 2024-03-17T23:47:14.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.789+0000 7f292575b640 1 -- 172.21.15.150:0/1720195848 wait complete. 2024-03-17T23:47:14.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f0a8279d640 1 --2- 172.21.15.150:0/400999188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a7403e830 0x7f0a74053be0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0a74056950 tx=0x7f0a7802b8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7fd625d9b640 1 --2- 172.21.15.150:0/3206027951 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6100040a0 0x7fd6100099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:47:14.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fd73daee640 1 -- 172.21.15.150:0/4170214923 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd724004160 msgr2=0x7fd72400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fd73daee640 1 --2- 172.21.15.150:0/4170214923 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd724004160 0x7fd72400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fd73daee640 1 -- 172.21.15.150:0/4170214923 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7240037a0 msgr2=0x7fd724003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fd73daee640 1 --2- 172.21.15.150:0/4170214923 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7240037a0 0x7fd724003c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fd73daee640 1 -- 172.21.15.150:0/4170214923 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd730006200 con 0x7fd724004bd0 2024-03-17T23:47:14.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.788+0000 7fd73e2ef640 1 --2- 172.21.15.150:0/4170214923 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd724004160 0x7fd72400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:47:14.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.789+0000 7fd73daee640 1 --2- 172.21.15.150:0/4170214923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd724004bd0 0x7fd724004fb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd734001450 tx=0x7fd724005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e1fab40a9cb8d68 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f0a8279d640 1 -- 172.21.15.150:0/400999188 shutdown_connections 2024-03-17T23:47:14.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f0a8279d640 1 --2- 172.21.15.150:0/400999188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a7403e830 0x7f0a74053be0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f0a8279d640 1 --2- 172.21.15.150:0/400999188 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a74054120 0x7f0a740e6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f0a8279d640 1 --2- 172.21.15.150:0/400999188 >> v2:172.21.15.157:3300/0 conn(0x7f0a740e30b0 0x7f0a74057d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f0a8279d640 1 -- 172.21.15.150:0/400999188 >> 172.21.15.150:0/400999188 conn(0x7f0a740559d0 msgr2=0x7f0a74055de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:14.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f5cd9ffb640 1 -- 172.21.15.150:0/3445452277 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2949380343 0 0) 0x7f5cdc0ce490 con 0x7f5cdc06e270 2024-03-17T23:47:14.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f5cd9ffb640 1 -- 172.21.15.150:0/3445452277 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cc4003520 con 0x7f5cdc06e270 2024-03-17T23:47:14.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f76c5547640 1 -- 172.21.15.150:0/2185052089 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76c0165cc0 msgr2=0x7f76c0162580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7fd73caec640 1 -- 172.21.15.150:0/4170214923 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73403d070 con 0x7fd724004bd0 2024-03-17T23:47:14.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7fd73caec640 1 -- 172.21.15.150:0/4170214923 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7340308f0 con 0x7fd724004bd0 2024-03-17T23:47:14.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7fd73caec640 1 -- 172.21.15.150:0/4170214923 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd734001490 con 0x7fd724004bd0 2024-03-17T23:47:14.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f0a8279d640 1 -- 172.21.15.150:0/400999188 shutdown_connections 2024-03-17T23:47:14.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f5cd9ffb640 1 -- 172.21.15.150:0/3445452277 <== mon.1 v1:172.21.15.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3270568832 0 0) 0x7f5cc4003520 con 0x7f5cdc06e270 2024-03-17T23:47:14.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f5cd9ffb640 1 -- 172.21.15.150:0/3445452277 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5cdc13d080 msgr2=0x7f5cdc06dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f5cd9ffb640 1 --2- 172.21.15.150:0/3445452277 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5cdc13d080 0x7f5cdc06dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f5cd9ffb640 1 -- 172.21.15.150:0/3445452277 >> v2:172.21.15.150:3300/0 conn(0x7f5cdc0cf640 msgr2=0x7f5cdc13f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f5cd9ffb640 1 --2- 172.21.15.150:0/3445452277 >> v2:172.21.15.150:3300/0 conn(0x7f5cdc0cf640 0x7f5cdc13f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f5cd9ffb640 1 -- 172.21.15.150:0/3445452277 --> v1:172.21.15.157:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cdc0ccfd0 con 0x7f5cdc06e270 2024-03-17T23:47:14.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f5cd9ffb640 1 -- 172.21.15.150:0/3445452277 <== mon.1 v1:172.21.15.157:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f5cd0001670 con 0x7f5cdc06e270 2024-03-17T23:47:14.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f5cd9ffb640 1 -- 172.21.15.150:0/3445452277 >> v1:172.21.15.157:6789/0 conn(0x7f5cdc06e270 legacy=0x7f5cdc0cef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f5cd9ffb640 1 --2- 172.21.15.150:0/3445452277 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5cc4004b90 0x7f5cc40037a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f5cd9ffb640 1 --2- 172.21.15.150:0/3445452277 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5cc4003da0 0x7f5cc4004220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.791+0000 7f5cd9ffb640 1 --2- 172.21.15.150:0/3445452277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cc400d640 0x7f5cc400fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.791+0000 7f5cd9ffb640 1 -- 172.21.15.150:0/3445452277 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc0cca10 con 0x7f5cc400d640 2024-03-17T23:47:14.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.791+0000 7f5cd9ffb640 1 -- 172.21.15.150:0/3445452277 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc0cccf0 con 0x7f5cc4004b90 2024-03-17T23:47:14.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.791+0000 7f5cd9ffb640 1 -- 172.21.15.150:0/3445452277 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5cc4010430 con 0x7f5cc4003da0 2024-03-17T23:47:14.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f1105df4640 1 Processor -- start 2024-03-17T23:47:14.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f76c5547640 1 --2- 172.21.15.150:0/2185052089 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76c0165cc0 0x7f76c0162580 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f76c0070df0 tx=0x7f76b002b8f0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f76c5547640 1 -- 172.21.15.150:0/2185052089 shutdown_connections 2024-03-17T23:47:14.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f76c5547640 1 --2- 172.21.15.150:0/2185052089 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76c0165cc0 0x7f76c0162580 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.791+0000 7f1105df4640 1 -- start start 2024-03-17T23:47:14.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f76c5547640 1 --2- 172.21.15.150:0/2185052089 >> v2:172.21.15.150:3300/0 conn(0x7f76c01621a0 0x7f76c006eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f76c5547640 1 -- 172.21.15.150:0/2185052089 >> 172.21.15.150:0/2185052089 conn(0x7f76c0099a70 msgr2=0x7f76c0099e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:14.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f76c5547640 1 -- 172.21.15.150:0/2185052089 shutdown_connections 2024-03-17T23:47:14.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.791+0000 7fa199bdf640 1 -- 172.21.15.150:0/2874387998 >> v2:172.21.15.157:3300/0 conn(0x7fa1940b9250 msgr2=0x7fa1940d3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f76c5547640 1 -- 172.21.15.150:0/2185052089 wait complete. 2024-03-17T23:47:14.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.791+0000 7fd73eaf0640 1 -- 172.21.15.150:0/4170214923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd724004bd0 msgr2=0x7fd724004fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.791+0000 7fd73eaf0640 1 --2- 172.21.15.150:0/4170214923 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd724004bd0 0x7fd724004fb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd734001450 tx=0x7fd724005830 comp rx=0 tx=0).stop 2024-03-17T23:47:14.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.791+0000 7f5cda7fc640 1 --2- 172.21.15.150:0/3445452277 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5cc4003da0 0x7f5cc4004220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.791+0000 7f5cdaffd640 1 --2- 172.21.15.150:0/3445452277 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5cc4004b90 0x7f5cc40037a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.791+0000 7fd73e2ef640 1 -- 172.21.15.150:0/4170214923 reap_dead start 2024-03-17T23:47:14.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.792+0000 7f292575b640 1 Processor -- start 2024-03-17T23:47:14.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.792+0000 7f292575b640 1 -- start start 2024-03-17T23:47:14.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.792+0000 7fd73eaf0640 1 -- 172.21.15.150:0/4170214923 shutdown_connections 2024-03-17T23:47:14.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.792+0000 7fd73eaf0640 1 -- 172.21.15.150:0/4170214923 >> 172.21.15.150:0/4170214923 conn(0x7fd730009fa0 msgr2=0x7fd73000a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:14.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.792+0000 7fd73eaf0640 1 -- 172.21.15.150:0/4170214923 shutdown_connections 2024-03-17T23:47:14.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.792+0000 7fd73eaf0640 1 -- 172.21.15.150:0/4170214923 wait complete. 2024-03-17T23:47:14.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.792+0000 7f5cdb7fe640 1 --2- 172.21.15.150:0/3445452277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cc400d640 0x7f5cc400fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.792+0000 7f5cda7fc640 1 -- 172.21.15.150:0/3445452277 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5cc4004b90 msgr2=0x7f5cc40037a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.792+0000 7f5cda7fc640 1 --2- 172.21.15.150:0/3445452277 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5cc4004b90 0x7f5cc40037a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.792+0000 7f5cda7fc640 1 -- 172.21.15.150:0/3445452277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cc400d640 msgr2=0x7f5cc400fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:14.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.792+0000 7f5cda7fc640 1 --2- 172.21.15.150:0/3445452277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cc400d640 0x7f5cc400fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.792+0000 7f5cda7fc640 1 -- 172.21.15.150:0/3445452277 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cdc0ccfd0 con 0x7f5cc4003da0 2024-03-17T23:47:14.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.792+0000 7f5cdb7fe640 1 --2- 172.21.15.150:0/3445452277 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cc400d640 0x7f5cc400fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:47:14.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.791+0000 7fa199bdf640 1 --2- 172.21.15.150:0/2874387998 >> v2:172.21.15.157:3300/0 conn(0x7fa1940b9250 0x7fa1940d3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.791+0000 7fa199bdf640 1 -- 172.21.15.150:0/2874387998 >> v1:172.21.15.150:6789/0 conn(0x7fa1940b89d0 legacy=0x7fa1940b5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.791+0000 7fa199bdf640 1 -- 172.21.15.150:0/2874387998 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1940d2360 con 0x7fa1940b9630 2024-03-17T23:47:14.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.792+0000 7fa199bdf640 1 --2- 172.21.15.150:0/2874387998 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1940b9630 0x7fa1940b8490 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa1940d2b60 tx=0x7fa18c02bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f4065dd9cbf84e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.793+0000 7fa19a3e0640 1 --2- 172.21.15.150:0/2874387998 >> v2:172.21.15.157:3300/0 conn(0x7fa1940b9250 0x7fa1940d3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:14.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.793+0000 7fa1993de640 1 -- 172.21.15.150:0/2874387998 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa18c02cb30 con 0x7fa1940b9630 2024-03-17T23:47:14.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.793+0000 7fa1993de640 1 -- 172.21.15.150:0/2874387998 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa18c02ccd0 con 0x7fa1940b9630 2024-03-17T23:47:14.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.790+0000 7f0a8279d640 1 -- 172.21.15.150:0/400999188 wait complete. 2024-03-17T23:47:14.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.792+0000 7f0a8279d640 1 Processor -- start 2024-03-17T23:47:14.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.793+0000 7fa1993de640 1 -- 172.21.15.150:0/2874387998 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa18c034610 con 0x7fa1940b9630 2024-03-17T23:47:14.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.793+0000 7f76c5547640 1 Processor -- start 2024-03-17T23:47:14.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.792+0000 7f1105df4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f110014b080 0x7f110013daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.792+0000 7f1105df4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110014b460 0x7f1100147200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.792+0000 7f1105df4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11001ef870 0x7f11001efcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.792+0000 7f1105df4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f11000ce9f0 con 0x7f110014b460 2024-03-17T23:47:14.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.792+0000 7f1105df4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f110014c070 con 0x7f11001ef870 2024-03-17T23:47:14.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.792+0000 7f1105df4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f11000ceb70 con 0x7f110014b080 2024-03-17T23:47:14.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.792+0000 7f10feffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110014b460 0x7f1100147200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.792+0000 7f10ff7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f110014b080 0x7f110013daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.792+0000 7f10feffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110014b460 0x7f1100147200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36500/0 (socket says 172.21.15.150:36500) 2024-03-17T23:47:14.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.792+0000 7f10feffd640 1 -- 172.21.15.150:0/1007862328 learned_addr learned my addr 172.21.15.150:0/1007862328 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.793+0000 7f10feffd640 1 -- 172.21.15.150:0/1007862328 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f110014b080 msgr2=0x7f110013daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.793+0000 7f10feffd640 1 --2- 172.21.15.150:0/1007862328 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f110014b080 0x7f110013daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.793+0000 7f10feffd640 1 -- 172.21.15.150:0/1007862328 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11001ef870 msgr2=0x7f11001efcf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-17T23:47:14.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.793+0000 7f10feffd640 1 --2- 172.21.15.150:0/1007862328 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11001ef870 0x7f11001efcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.793+0000 7f10feffd640 1 -- 172.21.15.150:0/1007862328 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f110013e1f0 con 0x7f110014b460 2024-03-17T23:47:14.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.793+0000 7f10feffd640 1 --2- 172.21.15.150:0/1007862328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110014b460 0x7f1100147200 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f10f4011cd0 tx=0x7f10f400f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.793+0000 7f292575b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f29200cc930 0x7f292014e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.793+0000 7f292575b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29200cf2c0 0x7f2920163c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.793+0000 7f10e7fff640 1 -- 172.21.15.150:0/1007862328 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f4011dc0 con 0x7f110014b460 2024-03-17T23:47:14.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.793+0000 7f291effd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f29200cc930 0x7f292014e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.793+0000 7f76c5547640 1 -- start start 2024-03-17T23:47:14.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.793+0000 7f291effd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f29200cc930 0x7f292014e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:51344/0 (socket says 172.21.15.150:51344) 2024-03-17T23:47:14.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.793+0000 7f291effd640 1 -- 172.21.15.150:0/2600587854 learned_addr learned my addr 172.21.15.150:0/2600587854 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.792+0000 7f0a8279d640 1 -- start start 2024-03-17T23:47:14.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7f0a8279d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a7403e830 0x7f0a740d8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7f292575b640 1 --2- 172.21.15.150:0/2600587854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2920162450 0x7f29200b0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7f292575b640 1 -- 172.21.15.150:0/2600587854 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f29200afa20 con 0x7f2920162450 2024-03-17T23:47:14.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7f292575b640 1 -- 172.21.15.150:0/2600587854 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f29200af8a0 con 0x7f29200cf2c0 2024-03-17T23:47:14.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7f291e7fc640 1 --2- 172.21.15.150:0/2600587854 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29200cf2c0 0x7f2920163c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7f292575b640 1 -- 172.21.15.150:0/2600587854 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f29200b4200 con 0x7f29200cc930 2024-03-17T23:47:14.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7fa19c644640 1 -- 172.21.15.150:0/2874387998 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1940b9630 msgr2=0x7fa1940b8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7fa19c644640 1 --2- 172.21.15.150:0/2874387998 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1940b9630 0x7fa1940b8490 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa1940d2b60 tx=0x7fa18c02bf90 comp rx=0 tx=0).stop 2024-03-17T23:47:14.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7fd626d9d640 1 --2- 172.21.15.150:0/3206027951 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd610009ef0 0x7fd61000c2e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd6180018d0 tx=0x7fd610005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=797479df360a5def server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7f0a8279d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a74054120 0x7f0a740502c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7f0a8279d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a740e30b0 0x7f0a740e1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7f0a8179b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a7403e830 0x7f0a740d8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7f0a8179b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a7403e830 0x7f0a740d8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:51360/0 (socket says 172.21.15.150:51360) 2024-03-17T23:47:14.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7f291e7fc640 1 -- 172.21.15.150:0/2600587854 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f29200cc930 msgr2=0x7f292014e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7f291e7fc640 1 --2- 172.21.15.150:0/2600587854 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f29200cc930 0x7f292014e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7f291f7fe640 1 --2- 172.21.15.150:0/2600587854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2920162450 0x7f29200b0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7f291e7fc640 1 -- 172.21.15.150:0/2600587854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2920162450 msgr2=0x7f29200b0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7f0a8179b640 1 -- 172.21.15.150:0/295227591 learned_addr learned my addr 172.21.15.150:0/295227591 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7f0a8279d640 1 -- 172.21.15.150:0/295227591 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7403bfb0 con 0x7f0a74054120 2024-03-17T23:47:14.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7f0a8279d640 1 -- 172.21.15.150:0/295227591 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0a740063a0 con 0x7f0a740e30b0 2024-03-17T23:47:14.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7f291e7fc640 1 --2- 172.21.15.150:0/2600587854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2920162450 0x7f29200b0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7f291e7fc640 1 -- 172.21.15.150:0/2600587854 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2920164560 con 0x7f29200cf2c0 2024-03-17T23:47:14.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7f291f7fe640 1 --2- 172.21.15.150:0/2600587854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2920162450 0x7f29200b0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:47:14.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7f0a81f9c640 1 --2- 172.21.15.150:0/295227591 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a740e30b0 0x7f0a740e1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7fd62559a640 1 -- 172.21.15.150:0/3206027951 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd618037070 con 0x7fd610009ef0 2024-03-17T23:47:14.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7f0a8279d640 1 -- 172.21.15.150:0/295227591 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0a740e3490 con 0x7f0a7403e830 2024-03-17T23:47:14.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7fd62559a640 1 -- 172.21.15.150:0/3206027951 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd618007270 con 0x7fd610009ef0 2024-03-17T23:47:14.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7fd62559a640 1 -- 172.21.15.150:0/3206027951 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd618037070 con 0x7fd610009ef0 2024-03-17T23:47:14.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7f76c5547640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76c00c3810 0x7f76c0150a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7f0a8179b640 1 -- 172.21.15.150:0/295227591 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a740e30b0 msgr2=0x7f0a740e1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7f0a8179b640 1 --2- 172.21.15.150:0/295227591 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a740e30b0 0x7f0a740e1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7f0a8179b640 1 -- 172.21.15.150:0/295227591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a74054120 msgr2=0x7f0a740502c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7f0a80f9a640 1 --2- 172.21.15.150:0/295227591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a74054120 0x7f0a740502c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7f76c5547640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76c01621a0 0x7f76c014d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.793+0000 7f1105df4640 1 -- 172.21.15.150:0/1007862328 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f110013e4d0 con 0x7f110014b460 2024-03-17T23:47:14.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.793+0000 7f1105df4640 1 -- 172.21.15.150:0/1007862328 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f110013ea90 con 0x7f110014b460 2024-03-17T23:47:14.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7f10e7fff640 1 -- 172.21.15.150:0/1007862328 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10f400fd20 con 0x7f110014b460 2024-03-17T23:47:14.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7f10e7fff640 1 -- 172.21.15.150:0/1007862328 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f4014640 con 0x7f110014b460 2024-03-17T23:47:14.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7f0a8179b640 1 --2- 172.21.15.150:0/295227591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a74054120 0x7f0a740502c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7f0a8179b640 1 -- 172.21.15.150:0/295227591 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a74052190 con 0x7f0a7403e830 2024-03-17T23:47:14.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7f291e7fc640 1 --2- 172.21.15.150:0/2600587854 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29200cf2c0 0x7f2920163c60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f29200ccdb0 tx=0x7f291000c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7f5cda7fc640 1 --2- 172.21.15.150:0/3445452277 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5cc4003da0 0x7f5cc4004220 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0001eb0 tx=0x7f5cc4004820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b848b8737b6b361b server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7f0a8179b640 1 --2- 172.21.15.150:0/295227591 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a7403e830 0x7f0a740d8630 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c00c7f0 tx=0x7f0a6c00ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7f5cd9ffb640 1 -- 172.21.15.150:0/3445452277 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd0007cd0 con 0x7f5cc4003da0 2024-03-17T23:47:14.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7f5cd9ffb640 1 -- 172.21.15.150:0/3445452277 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cd0030070 con 0x7f5cc4003da0 2024-03-17T23:47:14.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7f5cd9ffb640 1 -- 172.21.15.150:0/3445452277 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd002cc90 con 0x7f5cc4003da0 2024-03-17T23:47:14.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7f0a71ffb640 1 -- 172.21.15.150:0/295227591 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c015070 con 0x7f0a7403e830 2024-03-17T23:47:14.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7f76bffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76c00c3810 0x7f76c0150a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7f76c5547640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f76c0165cc0 0x7f76c00c6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7f76bf7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76c01621a0 0x7f76c014d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7fd62759e640 1 -- 172.21.15.150:0/3206027951 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd610009ef0 msgr2=0x7fd61000c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7fd62759e640 1 --2- 172.21.15.150:0/3206027951 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd610009ef0 0x7fd61000c2e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd6180018d0 tx=0x7fd610005830 comp rx=0 tx=0).stop 2024-03-17T23:47:14.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7fd626d9d640 1 -- 172.21.15.150:0/3206027951 reap_dead start 2024-03-17T23:47:14.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7f0a8279d640 1 -- 172.21.15.150:0/295227591 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a74052470 con 0x7f0a7403e830 2024-03-17T23:47:14.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7f0a8279d640 1 -- 172.21.15.150:0/295227591 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a74052a30 con 0x7f0a7403e830 2024-03-17T23:47:14.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7f76c4d46640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f76c0165cc0 0x7f76c00c6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7f0a71ffb640 1 -- 172.21.15.150:0/295227591 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a6c004220 con 0x7f0a7403e830 2024-03-17T23:47:14.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7f76c4d46640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f76c0165cc0 0x7f76c00c6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:51372/0 (socket says 172.21.15.150:51372) 2024-03-17T23:47:14.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7f76c4d46640 1 -- 172.21.15.150:0/977017985 learned_addr learned my addr 172.21.15.150:0/977017985 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7f0a71ffb640 1 -- 172.21.15.150:0/295227591 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c007620 con 0x7f0a7403e830 2024-03-17T23:47:14.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7f76bffff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76c00c3810 0x7f76c0150a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:34094/0 (socket says 172.21.15.150:34094) 2024-03-17T23:47:14.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7f10e7fff640 1 -- 172.21.15.150:0/1007862328 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f10f4012040 con 0x7f110014b460 2024-03-17T23:47:14.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7f76c5547640 1 -- 172.21.15.150:0/977017985 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f76c01666a0 con 0x7f76c01621a0 2024-03-17T23:47:14.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7f76c5547640 1 -- 172.21.15.150:0/977017985 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f76c01663c0 con 0x7f76c00c3810 2024-03-17T23:47:14.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7f76c5547640 1 -- 172.21.15.150:0/977017985 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f76c0166980 con 0x7f76c0165cc0 2024-03-17T23:47:14.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7f5ce1463640 1 -- 172.21.15.150:0/3445452277 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5cc4003da0 msgr2=0x7f5cc4004220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7f5ce1463640 1 --2- 172.21.15.150:0/3445452277 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5cc4003da0 0x7f5cc4004220 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0001eb0 tx=0x7f5cc4004820 comp rx=0 tx=0).stop 2024-03-17T23:47:14.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7f5cdb7fe640 1 -- 172.21.15.150:0/3445452277 reap_dead start 2024-03-17T23:47:14.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7f5ce1463640 1 -- 172.21.15.150:0/3445452277 shutdown_connections 2024-03-17T23:47:14.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7f5ce1463640 1 -- 172.21.15.150:0/3445452277 >> 172.21.15.150:0/3445452277 conn(0x7f5cdc06b190 msgr2=0x7f5cdc06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:14.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7fd73eaf0640 1 Processor -- start 2024-03-17T23:47:14.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.794+0000 7fd73eaf0640 1 -- start start 2024-03-17T23:47:14.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7fd73eaf0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd724004bd0 0x7fd7300d86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7fd73eaf0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd73004bb80 0x7fd7300e11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7fd73eaf0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7300d5210 0x7fd7300d45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7fd73eaf0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd730037a80 con 0x7fd73004bb80 2024-03-17T23:47:14.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7fd73eaf0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd730037900 con 0x7fd7300d5210 2024-03-17T23:47:14.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7fd73eaf0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd730050190 con 0x7fd724004bd0 2024-03-17T23:47:14.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7fd73d2ed640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd73004bb80 0x7fd7300e11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7fd73d2ed640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd73004bb80 0x7fd7300e11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36526/0 (socket says 172.21.15.150:36526) 2024-03-17T23:47:14.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7fd73d2ed640 1 -- 172.21.15.150:0/1459374248 learned_addr learned my addr 172.21.15.150:0/1459374248 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.795+0000 7fd73e2ef640 1 --2- 172.21.15.150:0/1459374248 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7300d5210 0x7fd7300d45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7fd73d2ed640 1 -- 172.21.15.150:0/1459374248 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd724004bd0 msgr2=0x7fd7300d86c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-17T23:47:14.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7fd73d2ed640 1 --2- 172.21.15.150:0/1459374248 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd724004bd0 0x7fd7300d86c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7fd73d2ed640 1 -- 172.21.15.150:0/1459374248 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7300d5210 msgr2=0x7fd7300d45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7fd73d2ed640 1 --2- 172.21.15.150:0/1459374248 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7300d5210 0x7fd7300d45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7fd73d2ed640 1 -- 172.21.15.150:0/1459374248 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd73004c010 con 0x7fd73004bb80 2024-03-17T23:47:14.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7fd73d2ed640 1 --2- 172.21.15.150:0/1459374248 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd73004bb80 0x7fd7300e11a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd72800be20 tx=0x7fd72800d720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7fd72e7fc640 1 -- 172.21.15.150:0/1459374248 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd72803a5f0 con 0x7fd73004bb80 2024-03-17T23:47:14.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7f10e5ffb640 1 -- 172.21.15.150:0/1007862328 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11000681a0 con 0x7f110014b460 2024-03-17T23:47:14.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7f29077fe640 1 -- 172.21.15.150:0/2600587854 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2910017070 con 0x7f29200cf2c0 2024-03-17T23:47:14.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7f292575b640 1 -- 172.21.15.150:0/2600587854 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2920153b40 con 0x7f29200cf2c0 2024-03-17T23:47:14.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7f292575b640 1 -- 172.21.15.150:0/2600587854 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29201641a0 con 0x7f29200cf2c0 2024-03-17T23:47:14.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.797+0000 7fd73eaf0640 1 -- 172.21.15.150:0/1459374248 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd73004c2f0 con 0x7fd73004bb80 2024-03-17T23:47:14.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.797+0000 7fd72e7fc640 1 -- 172.21.15.150:0/1459374248 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd72800fe90 con 0x7fd73004bb80 2024-03-17T23:47:14.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.797+0000 7fa19c644640 1 -- 172.21.15.150:0/2874387998 shutdown_connections 2024-03-17T23:47:14.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.797+0000 7fa19c644640 1 --2- 172.21.15.150:0/2874387998 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1940b9630 0x7fa1940b8490 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.797+0000 7fa19c644640 1 --2- 172.21.15.150:0/2874387998 >> v2:172.21.15.157:3300/0 conn(0x7fa1940b9250 0x7fa1940d3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.797+0000 7fa19c644640 1 -- 172.21.15.150:0/2874387998 >> 172.21.15.150:0/2874387998 conn(0x7fa1940d4770 msgr2=0x7fa1940d0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:14.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.797+0000 7f10e7fff640 1 --2- 172.21.15.150:0/1007862328 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f10ec06bc00 0x7f10ec06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.797+0000 7f10e7fff640 1 -- 172.21.15.150:0/1007862328 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f10f408b420 con 0x7f110014b460 2024-03-17T23:47:14.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.797+0000 7fd62759e640 1 -- 172.21.15.150:0/3206027951 shutdown_connections 2024-03-17T23:47:14.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.797+0000 7fd62759e640 1 -- 172.21.15.150:0/3206027951 >> 172.21.15.150:0/3206027951 conn(0x7fd620009fa0 msgr2=0x7fd62000a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:14.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.797+0000 7f0a8279d640 1 -- 172.21.15.150:0/295227591 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a7403c3f0 con 0x7f0a7403e830 2024-03-17T23:47:14.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.797+0000 7f29077fe640 1 -- 172.21.15.150:0/2600587854 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29100052d0 con 0x7f29200cf2c0 2024-03-17T23:47:14.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.797+0000 7f29077fe640 1 -- 172.21.15.150:0/2600587854 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2910003810 con 0x7f29200cf2c0 2024-03-17T23:47:14.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.797+0000 7f292575b640 1 -- 172.21.15.150:0/2600587854 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29200688b0 con 0x7f29200cf2c0 2024-03-17T23:47:14.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.796+0000 7f5ce1463640 1 -- 172.21.15.150:0/3445452277 shutdown_connections 2024-03-17T23:47:14.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.797+0000 7f5ce1463640 1 -- 172.21.15.150:0/3445452277 wait complete. 2024-03-17T23:47:14.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.798+0000 7f76c4d46640 1 -- 172.21.15.150:0/977017985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76c00c3810 msgr2=0x7f76c0150a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.798+0000 7f76c4d46640 1 --2- 172.21.15.150:0/977017985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76c00c3810 0x7f76c0150a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.798+0000 7f76c4d46640 1 -- 172.21.15.150:0/977017985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76c01621a0 msgr2=0x7f76c014d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.798+0000 7f76c4d46640 1 --2- 172.21.15.150:0/977017985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76c01621a0 0x7f76c014d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.798+0000 7f76c4d46640 1 -- 172.21.15.150:0/977017985 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76c00b5ea0 con 0x7f76c0165cc0 2024-03-17T23:47:14.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.798+0000 7f76bf7fe640 1 --2- 172.21.15.150:0/977017985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76c01621a0 0x7f76c014d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:14.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.797+0000 7fd73eaf0640 1 -- 172.21.15.150:0/1459374248 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd73004c8b0 con 0x7fd73004bb80 2024-03-17T23:47:14.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.797+0000 7fd72e7fc640 1 -- 172.21.15.150:0/1459374248 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd72803a9b0 con 0x7fd73004bb80 2024-03-17T23:47:14.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.798+0000 7f0a71ffb640 1 -- 172.21.15.150:0/295227591 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0a6c0077c0 con 0x7f0a7403e830 2024-03-17T23:47:14.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.798+0000 7fa19c644640 1 -- 172.21.15.150:0/2874387998 shutdown_connections 2024-03-17T23:47:14.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.798+0000 7f76c4d46640 1 --2- 172.21.15.150:0/977017985 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f76c0165cc0 0x7f76c00c6520 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f76b800caa0 tx=0x7f76b800cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.798+0000 7fd73eaf0640 1 -- 172.21.15.150:0/1459374248 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd730005380 con 0x7fd73004bb80 2024-03-17T23:47:14.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.798+0000 7fa19c644640 1 -- 172.21.15.150:0/2874387998 wait complete. 2024-03-17T23:47:14.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.798+0000 7f10ff7fe640 1 --2- 172.21.15.150:0/1007862328 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f10ec06bc00 0x7f10ec06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.798+0000 7f0a71ffb640 1 --2- 172.21.15.150:0/295227591 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0a6006bc20 0x7f0a6006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.799+0000 7f0a71ffb640 1 -- 172.21.15.150:0/295227591 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f0a6c08a960 con 0x7f0a7403e830 2024-03-17T23:47:14.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.799+0000 7f5ce1463640 1 Processor -- start 2024-03-17T23:47:14.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.799+0000 7f5ce1463640 1 -- start start 2024-03-17T23:47:14.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.799+0000 7f0a80f9a640 1 --2- 172.21.15.150:0/295227591 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0a6006bc20 0x7f0a6006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.799+0000 7f0a80f9a640 1 --2- 172.21.15.150:0/295227591 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0a6006bc20 0x7f0a6006e0e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0a78036000 tx=0x7f0a78008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.799+0000 7f29077fe640 1 -- 172.21.15.150:0/2600587854 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f291001a070 con 0x7f29200cf2c0 2024-03-17T23:47:14.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.800+0000 7f10ff7fe640 1 --2- 172.21.15.150:0/1007862328 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f10ec06bc00 0x7f10ec06e0c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f10f8034000 tx=0x7f10f802f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.800+0000 7fd62759e640 1 -- 172.21.15.150:0/3206027951 shutdown_connections 2024-03-17T23:47:14.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.800+0000 7fd62759e640 1 -- 172.21.15.150:0/3206027951 wait complete. 2024-03-17T23:47:14.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.800+0000 7f29077fe640 1 --2- 172.21.15.150:0/2600587854 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f290c06bc50 0x7f290c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.800+0000 7fa19c644640 1 Processor -- start 2024-03-17T23:47:14.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.800+0000 7fa19c644640 1 -- start start 2024-03-17T23:47:14.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.800+0000 7f291effd640 1 --2- 172.21.15.150:0/2600587854 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f290c06bc50 0x7f290c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.800+0000 7f29077fe640 1 -- 172.21.15.150:0/2600587854 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f291008a430 con 0x7f29200cf2c0 2024-03-17T23:47:14.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.800+0000 7f5ce1463640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cc4003da0 0x7f5cdc12e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.800+0000 7f5cdaffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cc4003da0 0x7f5cdc12e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.800+0000 7f5cdaffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cc4003da0 0x7f5cdc12e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36556/0 (socket says 172.21.15.150:36556) 2024-03-17T23:47:14.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.800+0000 7f5cdaffd640 1 -- 172.21.15.150:0/3968065713 learned_addr learned my addr 172.21.15.150:0/3968065713 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.801+0000 7f291effd640 1 --2- 172.21.15.150:0/2600587854 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f290c06bc50 0x7f290c06e110 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2920155330 tx=0x7f291800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.801+0000 7f5ce1463640 1 --2- 172.21.15.150:0/3968065713 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5cdc154c40 0x7f5cdc156730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.801+0000 7f5ce1463640 1 --2- 172.21.15.150:0/3968065713 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5cdc1550a0 0x7f5cdc156c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.801+0000 7f5ce1463640 1 -- 172.21.15.150:0/3968065713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc072ce0 con 0x7f5cc4003da0 2024-03-17T23:47:14.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.801+0000 7f5ce1463640 1 -- 172.21.15.150:0/3968065713 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc0cea10 con 0x7f5cdc1550a0 2024-03-17T23:47:14.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.801+0000 7f5cda7fc640 1 --2- 172.21.15.150:0/3968065713 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5cdc154c40 0x7f5cdc156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.801+0000 7f5ce1463640 1 -- 172.21.15.150:0/3968065713 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5cdc072e60 con 0x7f5cdc154c40 2024-03-17T23:47:14.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.801+0000 7f5cdb7fe640 1 --2- 172.21.15.150:0/3968065713 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5cdc1550a0 0x7f5cdc156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.801+0000 7fa19c644640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa1940b89d0 0x7fa1940cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.801+0000 7fa19c644640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa1940b9250 0x7fa1940ce610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.801+0000 7fa19c644640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1940b9630 0x7fa194164ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.801+0000 7f5cdaffd640 1 -- 172.21.15.150:0/3968065713 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5cdc154c40 msgr2=0x7f5cdc156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.801+0000 7fa19c644640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa1940b5e80 con 0x7fa1940b9630 2024-03-17T23:47:14.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.801+0000 7f5cdaffd640 1 --2- 172.21.15.150:0/3968065713 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5cdc154c40 0x7f5cdc156730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.801+0000 7fa19c644640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa194165f90 con 0x7fa1940b9250 2024-03-17T23:47:14.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.801+0000 7fa19c644640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa1940b6000 con 0x7fa1940b89d0 2024-03-17T23:47:14.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.801+0000 7fa19a3e0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa1940b89d0 0x7fa1940cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.801+0000 7fa19abe1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1940b9630 0x7fa194164ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.801+0000 7f5cdaffd640 1 -- 172.21.15.150:0/3968065713 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5cdc1550a0 msgr2=0x7f5cdc156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.801+0000 7fa19a3e0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa1940b89d0 0x7fa1940cb1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:51392/0 (socket says 172.21.15.150:51392) 2024-03-17T23:47:14.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.801+0000 7f5cdaffd640 1 --2- 172.21.15.150:0/3968065713 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5cdc1550a0 0x7f5cdc156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.801+0000 7fa19a3e0640 1 -- 172.21.15.150:0/3884142058 learned_addr learned my addr 172.21.15.150:0/3884142058 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.801+0000 7fd62759e640 1 Processor -- start 2024-03-17T23:47:14.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.801+0000 7fa199bdf640 1 --2- 172.21.15.150:0/3884142058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa1940b9250 0x7fa1940ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.801+0000 7f5cdaffd640 1 -- 172.21.15.150:0/3968065713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cdc157620 con 0x7f5cc4003da0 2024-03-17T23:47:14.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.801+0000 7fd62759e640 1 -- start start 2024-03-17T23:47:14.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.802+0000 7f5cdb7fe640 1 --2- 172.21.15.150:0/3968065713 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5cdc1550a0 0x7f5cdc156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:14.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.802+0000 7fa19abe1640 1 -- 172.21.15.150:0/3884142058 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa1940b89d0 msgr2=0x7fa1940cb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.802+0000 7fa19abe1640 1 --2- 172.21.15.150:0/3884142058 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa1940b89d0 0x7fa1940cb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.802+0000 7fa19abe1640 1 -- 172.21.15.150:0/3884142058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa1940b9250 msgr2=0x7fa1940ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.802+0000 7fa19abe1640 1 --2- 172.21.15.150:0/3884142058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa1940b9250 0x7fa1940ce610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.802+0000 7fa19abe1640 1 -- 172.21.15.150:0/3884142058 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa19415ff20 con 0x7fa1940b9630 2024-03-17T23:47:14.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.802+0000 7f5cdaffd640 1 --2- 172.21.15.150:0/3968065713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cc4003da0 0x7f5cdc12e090 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4007d30 tx=0x7f5cd4010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.802+0000 7fa19abe1640 1 --2- 172.21.15.150:0/3884142058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1940b9630 0x7fa194164ee0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa19000c3f0 tx=0x7fa19000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.802+0000 7fd62759e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd610009ef0 0x7fd6200eb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.802+0000 7fd62759e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6200cf470 0x7fd6200dc070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.802+0000 7fd62759e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6200ee290 0x7fd6200ee740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.802+0000 7fd62759e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd620037a80 con 0x7fd6200ee290 2024-03-17T23:47:14.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.802+0000 7fd62759e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd620037900 con 0x7fd6200cf470 2024-03-17T23:47:14.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.802+0000 7fd62759e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd6200df5f0 con 0x7fd610009ef0 2024-03-17T23:47:14.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.802+0000 7fd625d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6200cf470 0x7fd6200dc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.802+0000 7fd626d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6200ee290 0x7fd6200ee740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.802+0000 7fd625d9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6200cf470 0x7fd6200dc070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:34120/0 (socket says 172.21.15.150:34120) 2024-03-17T23:47:14.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.802+0000 7fd625d9b640 1 -- 172.21.15.150:0/1175870505 learned_addr learned my addr 172.21.15.150:0/1175870505 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:14.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.803+0000 7fd62659c640 1 --2- 172.21.15.150:0/1175870505 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd610009ef0 0x7fd6200eb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.803+0000 7fd625d9b640 1 -- 172.21.15.150:0/1175870505 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd610009ef0 msgr2=0x7fd6200eb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.803+0000 7fd625d9b640 1 --2- 172.21.15.150:0/1175870505 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd610009ef0 0x7fd6200eb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.803+0000 7fd625d9b640 1 -- 172.21.15.150:0/1175870505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6200ee290 msgr2=0x7fd6200ee740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:14.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.803+0000 7fd625d9b640 1 --2- 172.21.15.150:0/1175870505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6200ee290 0x7fd6200ee740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:14.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.803+0000 7fd625d9b640 1 -- 172.21.15.150:0/1175870505 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6200ef330 con 0x7fd6200cf470 2024-03-17T23:47:14.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.803+0000 7f29077fe640 1 -- 172.21.15.150:0/2600587854 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2910021020 con 0x7f29200cf2c0 2024-03-17T23:47:14.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.803+0000 7fd625d9b640 1 --2- 172.21.15.150:0/1175870505 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6200cf470 0x7fd6200dc070 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd61c010910 tx=0x7fd61c010d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.803+0000 7f76bcff9640 1 -- 172.21.15.150:0/977017985 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b8017070 con 0x7f76c0165cc0 2024-03-17T23:47:14.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.803+0000 7f76bcff9640 1 -- 172.21.15.150:0/977017985 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76b800ac00 con 0x7f76c0165cc0 2024-03-17T23:47:14.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.803+0000 7f76bcff9640 1 -- 172.21.15.150:0/977017985 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b8012600 con 0x7f76c0165cc0 2024-03-17T23:47:14.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.803+0000 7f76c5547640 1 -- 172.21.15.150:0/977017985 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76c00b63f0 con 0x7f76c0165cc0 2024-03-17T23:47:14.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.803+0000 7f76c5547640 1 -- 172.21.15.150:0/977017985 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f76c00b1eb0 con 0x7f76c0165cc0 2024-03-17T23:47:14.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.803+0000 7f0a71ffb640 1 -- 172.21.15.150:0/295227591 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0a6c058ba0 con 0x7f0a7403e830 2024-03-17T23:47:14.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.803+0000 7fd616ffd640 1 -- 172.21.15.150:0/1175870505 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd61c00f040 con 0x7fd6200cf470 2024-03-17T23:47:14.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.803+0000 7fd616ffd640 1 -- 172.21.15.150:0/1175870505 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd61c00eb60 con 0x7fd6200cf470 2024-03-17T23:47:14.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.803+0000 7fd616ffd640 1 -- 172.21.15.150:0/1175870505 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd61c01c400 con 0x7fd6200cf470 2024-03-17T23:47:14.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.803+0000 7fd62759e640 1 -- 172.21.15.150:0/1175870505 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6200dc5b0 con 0x7fd6200cf470 2024-03-17T23:47:14.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.803+0000 7fd62759e640 1 -- 172.21.15.150:0/1175870505 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6200ec1d0 con 0x7fd6200cf470 2024-03-17T23:47:14.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.805+0000 7f10e7fff640 1 -- 172.21.15.150:0/1007862328 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f10f4019030 con 0x7f110014b460 2024-03-17T23:47:14.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.805+0000 7fd616ffd640 1 -- 172.21.15.150:0/1175870505 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd61c01c5a0 con 0x7fd6200cf470 2024-03-17T23:47:14.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.805+0000 7fd62759e640 1 -- 172.21.15.150:0/1175870505 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd620005380 con 0x7fd6200cf470 2024-03-17T23:47:14.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.805+0000 7f76a67fc640 1 -- 172.21.15.150:0/977017985 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7698005290 con 0x7f76c0165cc0 2024-03-17T23:47:14.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.805+0000 7fd616ffd640 1 --2- 172.21.15.150:0/1175870505 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd60806bc00 0x7fd60806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.806+0000 7fd616ffd640 1 -- 172.21.15.150:0/1175870505 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fd61c08b390 con 0x7fd6200cf470 2024-03-17T23:47:14.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.806+0000 7fd62659c640 1 --2- 172.21.15.150:0/1175870505 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd60806bc00 0x7fd60806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.806+0000 7fa18affd640 1 -- 172.21.15.150:0/3884142058 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1900052d0 con 0x7fa1940b9630 2024-03-17T23:47:14.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.806+0000 7fa18affd640 1 -- 172.21.15.150:0/3884142058 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa190005470 con 0x7fa1940b9630 2024-03-17T23:47:14.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.806+0000 7fa18affd640 1 -- 172.21.15.150:0/3884142058 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1900052d0 con 0x7fa1940b9630 2024-03-17T23:47:14.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.806+0000 7fa19c644640 1 -- 172.21.15.150:0/3884142058 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1940cba20 con 0x7fa1940b9630 2024-03-17T23:47:14.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.806+0000 7fa19c644640 1 -- 172.21.15.150:0/3884142058 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1940cbfe0 con 0x7fa1940b9630 2024-03-17T23:47:14.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.807+0000 7fd62659c640 1 --2- 172.21.15.150:0/1175870505 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd60806bc00 0x7fd60806e0c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd628062da0 tx=0x7fd62806af30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.806+0000 7f76bcff9640 1 -- 172.21.15.150:0/977017985 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f76b801a070 con 0x7f76c0165cc0 2024-03-17T23:47:14.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.806+0000 7f76bcff9640 1 --2- 172.21.15.150:0/977017985 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f76ac06bbd0 0x7f76ac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.806+0000 7f76bcff9640 1 -- 172.21.15.150:0/977017985 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f76b808ab10 con 0x7f76c0165cc0 2024-03-17T23:47:14.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.807+0000 7f76bffff640 1 --2- 172.21.15.150:0/977017985 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f76ac06bbd0 0x7f76ac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.807+0000 7f76bffff640 1 --2- 172.21.15.150:0/977017985 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f76ac06bbd0 0x7f76ac06e090 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f76b4006fd0 tx=0x7f76b4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.807+0000 7f5cc37fe640 1 -- 172.21.15.150:0/3968065713 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd401a030 con 0x7f5cc4003da0 2024-03-17T23:47:14.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.807+0000 7f5cc37fe640 1 -- 172.21.15.150:0/3968065713 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cd4010db0 con 0x7f5cc4003da0 2024-03-17T23:47:14.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.807+0000 7f5cc37fe640 1 -- 172.21.15.150:0/3968065713 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4003bd0 con 0x7f5cc4003da0 2024-03-17T23:47:14.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.807+0000 7fd72e7fc640 1 -- 172.21.15.150:0/1459374248 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd72803ab50 con 0x7fd73004bb80 2024-03-17T23:47:14.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.808+0000 7f5ce1463640 1 -- 172.21.15.150:0/3968065713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cdc157900 con 0x7f5cc4003da0 2024-03-17T23:47:14.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.808+0000 7f5ce1463640 1 -- 172.21.15.150:0/3968065713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cdc151cf0 con 0x7f5cc4003da0 2024-03-17T23:47:14.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.808+0000 7fd72e7fc640 1 --2- 172.21.15.150:0/1459374248 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd71c06bbd0 0x7fd71c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.808+0000 7fd72e7fc640 1 -- 172.21.15.150:0/1459374248 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fd7280adb30 con 0x7fd73004bb80 2024-03-17T23:47:14.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.808+0000 7fa19c644640 1 -- 172.21.15.150:0/3884142058 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa17c005290 con 0x7fa1940b9630 2024-03-17T23:47:14.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.809+0000 7fd73daee640 1 --2- 172.21.15.150:0/1459374248 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd71c06bbd0 0x7fd71c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.809+0000 7fd73daee640 1 --2- 172.21.15.150:0/1459374248 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd71c06bbd0 0x7fd71c06e090 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd73403a000 tx=0x7fd734034000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.809+0000 7f5ce1463640 1 -- 172.21.15.150:0/3968065713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cdc0681a0 con 0x7f5cc4003da0 2024-03-17T23:47:14.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.810+0000 7fd616ffd640 1 -- 172.21.15.150:0/1175870505 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd61c018030 con 0x7fd6200cf470 2024-03-17T23:47:14.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.811+0000 7f76bcff9640 1 -- 172.21.15.150:0/977017985 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f76b8058b40 con 0x7f76c0165cc0 2024-03-17T23:47:14.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.812+0000 7fd72e7fc640 1 -- 172.21.15.150:0/1459374248 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd728048020 con 0x7fd73004bb80 2024-03-17T23:47:14.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.814+0000 7f5cc37fe640 1 -- 172.21.15.150:0/3968065713 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5cd4005d10 con 0x7f5cc4003da0 2024-03-17T23:47:14.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.814+0000 7fa18affd640 1 -- 172.21.15.150:0/3884142058 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa1900067d0 con 0x7fa1940b9630 2024-03-17T23:47:14.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.815+0000 7fa18affd640 1 --2- 172.21.15.150:0/3884142058 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa17806bfd0 0x7fa17806e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.815+0000 7f5cc37fe640 1 --2- 172.21.15.150:0/3968065713 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5cc806bca0 0x7f5cc806e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:14.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.815+0000 7f5cc37fe640 1 -- 172.21.15.150:0/3968065713 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f5cd408bf90 con 0x7f5cc4003da0 2024-03-17T23:47:14.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.816+0000 7fa18affd640 1 -- 172.21.15.150:0/3884142058 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fa1900050d0 con 0x7fa1940b9630 2024-03-17T23:47:14.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.816+0000 7fa19a3e0640 1 --2- 172.21.15.150:0/3884142058 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa17806bfd0 0x7fa17806e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.817+0000 7fa19a3e0640 1 --2- 172.21.15.150:0/3884142058 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa17806bfd0 0x7fa17806e490 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa184009560 tx=0x7fa184009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.815+0000 7f5cda7fc640 1 --2- 172.21.15.150:0/3968065713 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5cc806bca0 0x7f5cc806e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:14.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.816+0000 7f5cda7fc640 1 --2- 172.21.15.150:0/3968065713 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5cc806bca0 0x7f5cc806e160 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0034000 tx=0x7f5cd0007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:14.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.820+0000 7fa18affd640 1 -- 172.21.15.150:0/3884142058 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa19415ff20 con 0x7fa1940b9630 2024-03-17T23:47:14.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:14.829+0000 7f5cc37fe640 1 -- 172.21.15.150:0/3968065713 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5cd405a0d0 con 0x7f5cc4003da0 2024-03-17T23:47:15.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.213+0000 7f0a8279d640 1 -- 172.21.15.150:0/295227591 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f0a740057d0 con 0x7f0a7403e830 2024-03-17T23:47:15.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.214+0000 7f0a71ffb640 1 -- 172.21.15.150:0/295227591 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0a6c05cc40 con 0x7f0a7403e830 2024-03-17T23:47:15.215 INFO:teuthology.orchestra.run.smithi150.stdout:60129542152 2024-03-17T23:47:15.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.217+0000 7f0a677fe640 1 -- 172.21.15.150:0/295227591 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0a6006bc20 msgr2=0x7f0a6006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:15.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.217+0000 7f0a677fe640 1 --2- 172.21.15.150:0/295227591 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0a6006bc20 0x7f0a6006e0e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0a78036000 tx=0x7f0a78008030 comp rx=0 tx=0).stop 2024-03-17T23:47:15.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.217+0000 7f0a677fe640 1 -- 172.21.15.150:0/295227591 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a7403e830 msgr2=0x7f0a740d8630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:15.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.218+0000 7f0a677fe640 1 --2- 172.21.15.150:0/295227591 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a7403e830 0x7f0a740d8630 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c00c7f0 tx=0x7f0a6c00ccb0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.219+0000 7f0a677fe640 1 -- 172.21.15.150:0/295227591 shutdown_connections 2024-03-17T23:47:15.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.219+0000 7f0a677fe640 1 --2- 172.21.15.150:0/295227591 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0a6006bc20 0x7f0a6006e0e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.219+0000 7f0a677fe640 1 --2- 172.21.15.150:0/295227591 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0a740e30b0 0x7f0a740e1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.219+0000 7f0a677fe640 1 --2- 172.21.15.150:0/295227591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0a74054120 0x7f0a740502c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.219+0000 7f0a677fe640 1 --2- 172.21.15.150:0/295227591 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0a7403e830 0x7f0a740d8630 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.219+0000 7f0a677fe640 1 -- 172.21.15.150:0/295227591 >> 172.21.15.150:0/295227591 conn(0x7f0a740559d0 msgr2=0x7f0a740073f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:15.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.219+0000 7f0a677fe640 1 -- 172.21.15.150:0/295227591 shutdown_connections 2024-03-17T23:47:15.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.219+0000 7f0a677fe640 1 -- 172.21.15.150:0/295227591 wait complete. 2024-03-17T23:47:15.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.223+0000 7f10e5ffb640 1 -- 172.21.15.150:0/1007862328 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f1100143080 con 0x7f110014b460 2024-03-17T23:47:15.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.223+0000 7f292575b640 1 -- 172.21.15.150:0/2600587854 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f29200b3c00 con 0x7f29200cf2c0 2024-03-17T23:47:15.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.223+0000 7f29077fe640 1 -- 172.21.15.150:0/2600587854 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2910058670 con 0x7f29200cf2c0 2024-03-17T23:47:15.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.224+0000 7f10e7fff640 1 -- 172.21.15.150:0/1007862328 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f10f4059aa0 con 0x7f110014b460 2024-03-17T23:47:15.224 INFO:teuthology.orchestra.run.smithi150.stdout:55834574856 2024-03-17T23:47:15.225 INFO:teuthology.orchestra.run.smithi150.stdout:55834574857 2024-03-17T23:47:15.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.226+0000 7fd73eaf0640 1 -- 172.21.15.150:0/1459374248 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fd73004f190 con 0x7fd73004bb80 2024-03-17T23:47:15.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.227+0000 7fd72e7fc640 1 -- 172.21.15.150:0/1459374248 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd728038300 con 0x7fd73004bb80 2024-03-17T23:47:15.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.227+0000 7f29057fa640 1 -- 172.21.15.150:0/2600587854 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f290c06bc50 msgr2=0x7f290c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:15.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.227+0000 7f29057fa640 1 --2- 172.21.15.150:0/2600587854 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f290c06bc50 0x7f290c06e110 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2920155330 tx=0x7f291800b040 comp rx=0 tx=0).stop 2024-03-17T23:47:15.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.227+0000 7f29057fa640 1 -- 172.21.15.150:0/2600587854 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29200cf2c0 msgr2=0x7f2920163c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:15.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.227+0000 7f29057fa640 1 --2- 172.21.15.150:0/2600587854 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29200cf2c0 0x7f2920163c60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f29200ccdb0 tx=0x7f291000c820 comp rx=0 tx=0).stop 2024-03-17T23:47:15.230 INFO:teuthology.orchestra.run.smithi150.stdout:55834574856 2024-03-17T23:47:15.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.227+0000 7f29057fa640 1 -- 172.21.15.150:0/2600587854 shutdown_connections 2024-03-17T23:47:15.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.227+0000 7f29057fa640 1 --2- 172.21.15.150:0/2600587854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2920162450 0x7f29200b0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.227+0000 7f29057fa640 1 --2- 172.21.15.150:0/2600587854 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f290c06bc50 0x7f290c06e110 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2920155330 tx=0x7f291800b040 comp rx=0 tx=0).stop 2024-03-17T23:47:15.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.227+0000 7f29057fa640 1 --2- 172.21.15.150:0/2600587854 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29200cf2c0 0x7f2920163c60 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f29200ccdb0 tx=0x7f291000c820 comp rx=0 tx=0).stop 2024-03-17T23:47:15.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.227+0000 7f29057fa640 1 --2- 172.21.15.150:0/2600587854 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f29200cc930 0x7f292014e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.227+0000 7f29057fa640 1 -- 172.21.15.150:0/2600587854 >> 172.21.15.150:0/2600587854 conn(0x7f292006de70 msgr2=0x7f2920160a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:15.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.227+0000 7f1105df4640 1 -- 172.21.15.150:0/1007862328 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f10ec06bc00 msgr2=0x7f10ec06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:15.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.227+0000 7f1105df4640 1 --2- 172.21.15.150:0/1007862328 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f10ec06bc00 0x7f10ec06e0c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f10f8034000 tx=0x7f10f802f000 comp rx=0 tx=0).stop 2024-03-17T23:47:15.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.227+0000 7f1105df4640 1 -- 172.21.15.150:0/1007862328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110014b460 msgr2=0x7f1100147200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:15.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.227+0000 7f1105df4640 1 --2- 172.21.15.150:0/1007862328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110014b460 0x7f1100147200 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f10f4011cd0 tx=0x7f10f400f610 comp rx=0 tx=0).stop 2024-03-17T23:47:15.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.227+0000 7f1105df4640 1 -- 172.21.15.150:0/1007862328 shutdown_connections 2024-03-17T23:47:15.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.227+0000 7f1105df4640 1 --2- 172.21.15.150:0/1007862328 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f10ec06bc00 0x7f10ec06e0c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.227+0000 7f1105df4640 1 --2- 172.21.15.150:0/1007862328 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11001ef870 0x7f11001efcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.227+0000 7f1105df4640 1 --2- 172.21.15.150:0/1007862328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110014b460 0x7f1100147200 secure :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f10f4011cd0 tx=0x7f10f400f610 comp rx=0 tx=0).stop 2024-03-17T23:47:15.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.227+0000 7f1105df4640 1 --2- 172.21.15.150:0/1007862328 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f110014b080 0x7f110013daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.227+0000 7f1105df4640 1 -- 172.21.15.150:0/1007862328 >> 172.21.15.150:0/1007862328 conn(0x7f110006b190 msgr2=0x7f11001434c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:15.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.228+0000 7f1105df4640 1 -- 172.21.15.150:0/1007862328 shutdown_connections 2024-03-17T23:47:15.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.228+0000 7f1105df4640 1 -- 172.21.15.150:0/1007862328 wait complete. 2024-03-17T23:47:15.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.228+0000 7fd62759e640 1 -- 172.21.15.150:0/1175870505 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fd6200075d0 con 0x7fd6200cf470 2024-03-17T23:47:15.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.228+0000 7f29057fa640 1 -- 172.21.15.150:0/2600587854 shutdown_connections 2024-03-17T23:47:15.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.229+0000 7fd616ffd640 1 -- 172.21.15.150:0/1175870505 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd61c059aa0 con 0x7fd6200cf470 2024-03-17T23:47:15.235 INFO:teuthology.orchestra.run.smithi150.stdout:60129542152 2024-03-17T23:47:15.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.229+0000 7f29057fa640 1 -- 172.21.15.150:0/2600587854 wait complete. 2024-03-17T23:47:15.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.231+0000 7fd73eaf0640 1 -- 172.21.15.150:0/1459374248 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd71c06bbd0 msgr2=0x7fd71c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:15.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.231+0000 7fd73eaf0640 1 --2- 172.21.15.150:0/1459374248 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd71c06bbd0 0x7fd71c06e090 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd73403a000 tx=0x7fd734034000 comp rx=0 tx=0).stop 2024-03-17T23:47:15.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.231+0000 7fd73eaf0640 1 -- 172.21.15.150:0/1459374248 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd73004bb80 msgr2=0x7fd7300e11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:15.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.231+0000 7fd73eaf0640 1 --2- 172.21.15.150:0/1459374248 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd73004bb80 0x7fd7300e11a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd72800be20 tx=0x7fd72800d720 comp rx=0 tx=0).stop 2024-03-17T23:47:15.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.231+0000 7fd73eaf0640 1 -- 172.21.15.150:0/1459374248 shutdown_connections 2024-03-17T23:47:15.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.231+0000 7fd73eaf0640 1 --2- 172.21.15.150:0/1459374248 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd71c06bbd0 0x7fd71c06e090 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.231+0000 7fd73eaf0640 1 --2- 172.21.15.150:0/1459374248 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7300d5210 0x7fd7300d45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.231+0000 7fd73eaf0640 1 --2- 172.21.15.150:0/1459374248 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd73004bb80 0x7fd7300e11a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.231+0000 7fd73eaf0640 1 --2- 172.21.15.150:0/1459374248 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd724004bd0 0x7fd7300d86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.231+0000 7fd73eaf0640 1 -- 172.21.15.150:0/1459374248 >> 172.21.15.150:0/1459374248 conn(0x7fd730009fa0 msgr2=0x7fd7300351e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:15.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.231+0000 7f76a67fc640 1 -- 172.21.15.150:0/977017985 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f7698005580 con 0x7f76c0165cc0 2024-03-17T23:47:15.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.231+0000 7f76bcff9640 1 -- 172.21.15.150:0/977017985 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f76b805cbe0 con 0x7f76c0165cc0 2024-03-17T23:47:15.237 INFO:teuthology.orchestra.run.smithi150.stdout:55834574856 2024-03-17T23:47:15.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.232+0000 7fd73eaf0640 1 -- 172.21.15.150:0/1459374248 shutdown_connections 2024-03-17T23:47:15.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.232+0000 7fd73eaf0640 1 -- 172.21.15.150:0/1459374248 wait complete. 2024-03-17T23:47:15.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.232+0000 7fd62759e640 1 -- 172.21.15.150:0/1175870505 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd60806bc00 msgr2=0x7fd60806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:15.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.232+0000 7fd62759e640 1 --2- 172.21.15.150:0/1175870505 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd60806bc00 0x7fd60806e0c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd628062da0 tx=0x7fd62806af30 comp rx=0 tx=0).stop 2024-03-17T23:47:15.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.232+0000 7fd62759e640 1 -- 172.21.15.150:0/1175870505 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6200cf470 msgr2=0x7fd6200dc070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:15.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.232+0000 7fd62759e640 1 --2- 172.21.15.150:0/1175870505 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6200cf470 0x7fd6200dc070 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd61c010910 tx=0x7fd61c010d10 comp rx=0 tx=0).stop 2024-03-17T23:47:15.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.233+0000 7fd62759e640 1 -- 172.21.15.150:0/1175870505 shutdown_connections 2024-03-17T23:47:15.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.233+0000 7fd62759e640 1 --2- 172.21.15.150:0/1175870505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6200ee290 0x7fd6200ee740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.233+0000 7fd62759e640 1 --2- 172.21.15.150:0/1175870505 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd60806bc00 0x7fd60806e0c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.233+0000 7fd62759e640 1 --2- 172.21.15.150:0/1175870505 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6200cf470 0x7fd6200dc070 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.233+0000 7fd62759e640 1 --2- 172.21.15.150:0/1175870505 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd610009ef0 0x7fd6200eb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.233+0000 7fd62759e640 1 -- 172.21.15.150:0/1175870505 >> 172.21.15.150:0/1175870505 conn(0x7fd620009fa0 msgr2=0x7fd620033b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:15.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.235+0000 7fd62759e640 1 -- 172.21.15.150:0/1175870505 shutdown_connections 2024-03-17T23:47:15.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.235+0000 7fd62759e640 1 -- 172.21.15.150:0/1175870505 wait complete. 2024-03-17T23:47:15.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.235+0000 7f76a67fc640 1 -- 172.21.15.150:0/977017985 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f76ac06bbd0 msgr2=0x7f76ac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:15.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.235+0000 7f76a67fc640 1 --2- 172.21.15.150:0/977017985 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f76ac06bbd0 0x7f76ac06e090 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f76b4006fd0 tx=0x7f76b4008040 comp rx=0 tx=0).stop 2024-03-17T23:47:15.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.235+0000 7f76a67fc640 1 -- 172.21.15.150:0/977017985 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f76c0165cc0 msgr2=0x7f76c00c6520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:15.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.235+0000 7f76a67fc640 1 --2- 172.21.15.150:0/977017985 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f76c0165cc0 0x7f76c00c6520 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f76b800caa0 tx=0x7f76b800cf60 comp rx=0 tx=0).stop 2024-03-17T23:47:15.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.236+0000 7f76a67fc640 1 -- 172.21.15.150:0/977017985 shutdown_connections 2024-03-17T23:47:15.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.236+0000 7f76a67fc640 1 --2- 172.21.15.150:0/977017985 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f76c0165cc0 0x7f76c00c6520 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.236+0000 7f76a67fc640 1 --2- 172.21.15.150:0/977017985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76c01621a0 0x7f76c014d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.236+0000 7f76a67fc640 1 --2- 172.21.15.150:0/977017985 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f76ac06bbd0 0x7f76ac06e090 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.236+0000 7f76a67fc640 1 --2- 172.21.15.150:0/977017985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76c00c3810 0x7f76c0150a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.236+0000 7f76a67fc640 1 -- 172.21.15.150:0/977017985 >> 172.21.15.150:0/977017985 conn(0x7f76c0099a70 msgr2=0x7f76c0160c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:15.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.236+0000 7f76a67fc640 1 -- 172.21.15.150:0/977017985 shutdown_connections 2024-03-17T23:47:15.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.236+0000 7f76a67fc640 1 -- 172.21.15.150:0/977017985 wait complete. 2024-03-17T23:47:15.242 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542152 for osd.3 2024-03-17T23:47:15.249 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.4 2024-03-17T23:47:15.251 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.5 2024-03-17T23:47:15.251 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.1 2024-03-17T23:47:15.254 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542152 for osd.2 2024-03-17T23:47:15.256 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.6 2024-03-17T23:47:15.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.264+0000 7fa19c644640 1 -- 172.21.15.150:0/3884142058 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fa17c005850 con 0x7fa1940b9630 2024-03-17T23:47:15.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.264+0000 7fa18affd640 1 -- 172.21.15.150:0/3884142058 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa19415ff20 con 0x7fa1940b9630 2024-03-17T23:47:15.265 INFO:teuthology.orchestra.run.smithi150.stdout:60129542152 2024-03-17T23:47:15.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.267+0000 7fa19c644640 1 -- 172.21.15.150:0/3884142058 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa17806bfd0 msgr2=0x7fa17806e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:15.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.267+0000 7fa19c644640 1 --2- 172.21.15.150:0/3884142058 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa17806bfd0 0x7fa17806e490 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa184009560 tx=0x7fa184009190 comp rx=0 tx=0).stop 2024-03-17T23:47:15.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.267+0000 7fa19c644640 1 -- 172.21.15.150:0/3884142058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1940b9630 msgr2=0x7fa194164ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:15.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.267+0000 7fa19c644640 1 --2- 172.21.15.150:0/3884142058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1940b9630 0x7fa194164ee0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa19000c3f0 tx=0x7fa19000c8b0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.267+0000 7fa19c644640 1 -- 172.21.15.150:0/3884142058 shutdown_connections 2024-03-17T23:47:15.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.267+0000 7fa19c644640 1 --2- 172.21.15.150:0/3884142058 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1940b9630 0x7fa194164ee0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.267+0000 7fa19c644640 1 --2- 172.21.15.150:0/3884142058 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa17806bfd0 0x7fa17806e490 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.267+0000 7fa19c644640 1 --2- 172.21.15.150:0/3884142058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa1940b9250 0x7fa1940ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.267+0000 7fa19c644640 1 --2- 172.21.15.150:0/3884142058 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa1940b89d0 0x7fa1940cb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.267+0000 7fa19c644640 1 -- 172.21.15.150:0/3884142058 >> 172.21.15.150:0/3884142058 conn(0x7fa1940d4770 msgr2=0x7fa1940cf450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:15.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.267+0000 7fa19c644640 1 -- 172.21.15.150:0/3884142058 shutdown_connections 2024-03-17T23:47:15.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.267+0000 7fa19c644640 1 -- 172.21.15.150:0/3884142058 wait complete. 2024-03-17T23:47:15.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.275+0000 7f5ce1463640 1 -- 172.21.15.150:0/3968065713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f5cdc06bef0 con 0x7f5cc4003da0 2024-03-17T23:47:15.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.275+0000 7f5cc37fe640 1 -- 172.21.15.150:0/3968065713 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5cd405e170 con 0x7f5cc4003da0 2024-03-17T23:47:15.276 INFO:teuthology.orchestra.run.smithi150.stdout:55834574856 2024-03-17T23:47:15.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.277+0000 7f5ce1463640 1 -- 172.21.15.150:0/3968065713 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5cc806bca0 msgr2=0x7f5cc806e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:15.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.277+0000 7f5ce1463640 1 --2- 172.21.15.150:0/3968065713 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5cc806bca0 0x7f5cc806e160 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0034000 tx=0x7f5cd0007000 comp rx=0 tx=0).stop 2024-03-17T23:47:15.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.277+0000 7f5ce1463640 1 -- 172.21.15.150:0/3968065713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cc4003da0 msgr2=0x7f5cdc12e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:15.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.277+0000 7f5ce1463640 1 --2- 172.21.15.150:0/3968065713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cc4003da0 0x7f5cdc12e090 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4007d30 tx=0x7f5cd4010590 comp rx=0 tx=0).stop 2024-03-17T23:47:15.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.277+0000 7f5ce1463640 1 -- 172.21.15.150:0/3968065713 shutdown_connections 2024-03-17T23:47:15.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.277+0000 7f5ce1463640 1 --2- 172.21.15.150:0/3968065713 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5cc806bca0 0x7f5cc806e160 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.277+0000 7f5ce1463640 1 --2- 172.21.15.150:0/3968065713 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5cdc1550a0 0x7f5cdc156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.277+0000 7f5ce1463640 1 --2- 172.21.15.150:0/3968065713 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5cdc154c40 0x7f5cdc156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.277+0000 7f5ce1463640 1 --2- 172.21.15.150:0/3968065713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cc4003da0 0x7f5cdc12e090 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:15.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.277+0000 7f5ce1463640 1 -- 172.21.15.150:0/3968065713 >> 172.21.15.150:0/3968065713 conn(0x7f5cdc06b190 msgr2=0x7f5cdc0b4000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:15.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.277+0000 7f5ce1463640 1 -- 172.21.15.150:0/3968065713 shutdown_connections 2024-03-17T23:47:15.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:15.277+0000 7f5ce1463640 1 -- 172.21.15.150:0/3968065713 wait complete. 2024-03-17T23:47:15.280 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542152 for osd.0 2024-03-17T23:47:15.289 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.7 2024-03-17T23:47:16.243 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-17T23:47:16.250 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-17T23:47:16.251 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-17T23:47:16.252 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-17T23:47:16.255 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-17T23:47:16.257 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-17T23:47:16.281 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-17T23:47:16.289 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-17T23:47:16.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.419+0000 7f77c0777640 1 Processor -- start 2024-03-17T23:47:16.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.423+0000 7f77c0777640 1 -- start start 2024-03-17T23:47:16.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.424+0000 7f77c0777640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f77b8162530 0x7f77b80b2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.424+0000 7f77c0777640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f77b80670b0 0x7f77b80cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.424+0000 7f77c0777640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77b80ccfb0 0x7f77b80b0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.424+0000 7f77c0777640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f77b8162910 con 0x7f77b8162530 2024-03-17T23:47:16.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.424+0000 7f77c0777640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f77b80b62b0 con 0x7f77b80670b0 2024-03-17T23:47:16.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.424+0000 7f77c0777640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f77b80b2c40 con 0x7f77b80ccfb0 2024-03-17T23:47:16.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.425+0000 7f77be513640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f77b8162530 0x7f77b80b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.425+0000 7f77be513640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f77b8162530 0x7f77b80b2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40228/0 (socket says 172.21.15.150:40228) 2024-03-17T23:47:16.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.425+0000 7f77be513640 1 -- 172.21.15.150:0/638141095 learned_addr learned my addr 172.21.15.150:0/638141095 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:16.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.425+0000 7f77bed14640 1 --2- 172.21.15.150:0/638141095 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77b80ccfb0 0x7f77b80b0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.425+0000 7f77bdd12640 1 --2- 172.21.15.150:0/638141095 >> v2:172.21.15.157:3300/0 conn(0x7f77b80670b0 0x7f77b80cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.425+0000 7f77be513640 1 -- 172.21.15.150:0/638141095 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77b80ccfb0 msgr2=0x7f77b80b0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:16.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.425+0000 7f77be513640 1 --2- 172.21.15.150:0/638141095 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77b80ccfb0 0x7f77b80b0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.425+0000 7f77be513640 1 -- 172.21.15.150:0/638141095 >> v2:172.21.15.157:3300/0 conn(0x7f77b80670b0 msgr2=0x7f77b80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:16.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.425+0000 7f77be513640 1 --2- 172.21.15.150:0/638141095 >> v2:172.21.15.157:3300/0 conn(0x7f77b80670b0 0x7f77b80cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.425+0000 7f77be513640 1 -- 172.21.15.150:0/638141095 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77b8067670 con 0x7f77b8162530 2024-03-17T23:47:16.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.430+0000 7f77be513640 1 --2- 172.21.15.150:0/638141095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b8162530 0x7f77b80b2700 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f77b80709c0 tx=0x7f77b004dea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dc56bd821415646 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:16.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.430+0000 7f77bd511640 1 -- 172.21.15.150:0/638141095 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b004ea40 con 0x7f77b8162530 2024-03-17T23:47:16.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.430+0000 7f77bd511640 1 -- 172.21.15.150:0/638141095 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77b004ebe0 con 0x7f77b8162530 2024-03-17T23:47:16.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.430+0000 7f77bd511640 1 -- 172.21.15.150:0/638141095 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b00563f0 con 0x7f77b8162530 2024-03-17T23:47:16.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.431+0000 7f77c0777640 1 -- 172.21.15.150:0/638141095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b8162530 msgr2=0x7f77b80b2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.431+0000 7f77c0777640 1 --2- 172.21.15.150:0/638141095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b8162530 0x7f77b80b2700 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f77b80709c0 tx=0x7f77b004dea0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.438+0000 7f77c0777640 1 -- 172.21.15.150:0/638141095 shutdown_connections 2024-03-17T23:47:16.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.438+0000 7f77c0777640 1 --2- 172.21.15.150:0/638141095 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77b80ccfb0 0x7f77b80b0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.438+0000 7f77c0777640 1 --2- 172.21.15.150:0/638141095 >> v2:172.21.15.157:3300/0 conn(0x7f77b80670b0 0x7f77b80cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.438+0000 7f77c0777640 1 --2- 172.21.15.150:0/638141095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b8162530 0x7f77b80b2700 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.438+0000 7f77c0777640 1 -- 172.21.15.150:0/638141095 >> 172.21.15.150:0/638141095 conn(0x7f77b8099a70 msgr2=0x7f77b80b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:16.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.444+0000 7f77c0777640 1 -- 172.21.15.150:0/638141095 shutdown_connections 2024-03-17T23:47:16.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.452+0000 7f77c0777640 1 -- 172.21.15.150:0/638141095 wait complete. 2024-03-17T23:47:16.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.453+0000 7f77c0777640 1 Processor -- start 2024-03-17T23:47:16.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.454+0000 7f77c0777640 1 -- start start 2024-03-17T23:47:16.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.455+0000 7f2d9561c640 1 Processor -- start 2024-03-17T23:47:16.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.454+0000 7f4ccc91a640 1 Processor -- start 2024-03-17T23:47:16.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.455+0000 7f4ccc91a640 1 -- start start 2024-03-17T23:47:16.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.455+0000 7f2d9561c640 1 -- start start 2024-03-17T23:47:16.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.456+0000 7f2d9561c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2d901443a0 0x7f2d90071a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.456+0000 7f2d9561c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d90161fc0 0x7f2d90071fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.456+0000 7f2d9561c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d9015e250 con 0x7f2d9015eff0 2024-03-17T23:47:16.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.456+0000 7f2d9561c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f2d900aeff0 con 0x7f2d901443a0 2024-03-17T23:47:16.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.456+0000 7f2d9561c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2d90162600 con 0x7f2d90161fc0 2024-03-17T23:47:16.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.456+0000 7f4ccc91a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4cc80cf320 0x7f4cc80b36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.456+0000 7f4ccc91a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4cc8071a60 0x7f4cc8071ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.456+0000 7f4ccc91a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cc81595c0 con 0x7f4cc81632f0 2024-03-17T23:47:16.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.456+0000 7f4ccc91a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f4cc8072410 con 0x7f4cc80cf320 2024-03-17T23:47:16.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.456+0000 7f4ccc91a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4cc80b3230 con 0x7f4cc8071a60 2024-03-17T23:47:16.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.456+0000 7f4cc659c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f4cc81632f0 0x7f4cc80b44c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34680/0 (socket says 172.21.15.150:34680) 2024-03-17T23:47:16.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.456+0000 7f4cc659c640 1 -- 172.21.15.150:0/4146213633 learned_addr learned my addr 172.21.15.150:0/4146213633 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:16.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.457+0000 7f4cc6d9d640 1 --2- 172.21.15.150:0/4146213633 >> v2:172.21.15.150:3300/0 conn(0x7f4cc80cf320 0x7f4cc80b36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.457+0000 7f2d94e1b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d90161fc0 0x7f2d90071fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.457+0000 7f2d94e1b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d90161fc0 0x7f2d90071fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:55302/0 (socket says 172.21.15.150:55302) 2024-03-17T23:47:16.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.457+0000 7f2d94e1b640 1 -- 172.21.15.150:0/1953679795 learned_addr learned my addr 172.21.15.150:0/1953679795 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:16.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.457+0000 7f2d8ffff640 1 --2- 172.21.15.150:0/1953679795 >> v2:172.21.15.150:3300/0 conn(0x7f2d901443a0 0x7f2d90071a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.457+0000 7f4cc759e640 1 --2- 172.21.15.150:0/4146213633 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4cc8071a60 0x7f4cc8071ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.458+0000 7f4cc759e640 1 -- 172.21.15.150:0/4146213633 >> v2:172.21.15.150:3300/0 conn(0x7f4cc80cf320 msgr2=0x7f4cc80b36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:16.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.458+0000 7f4cc759e640 1 --2- 172.21.15.150:0/4146213633 >> v2:172.21.15.150:3300/0 conn(0x7f4cc80cf320 0x7f4cc80b36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.458+0000 7f2d8ffff640 1 -- 172.21.15.150:0/1953679795 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d90161fc0 msgr2=0x7f2d90071fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:16.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.458+0000 7f2d8ffff640 1 --2- 172.21.15.150:0/1953679795 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d90161fc0 0x7f2d90071fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.458+0000 7f2d8ffff640 1 -- 172.21.15.150:0/1953679795 >> v1:172.21.15.150:6789/0 conn(0x7f2d9015eff0 legacy=0x7f2d901618b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.458+0000 7f2d8ffff640 1 -- 172.21.15.150:0/1953679795 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d9015e510 con 0x7f2d901443a0 2024-03-17T23:47:16.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.458+0000 7f4cc759e640 1 -- 172.21.15.150:0/4146213633 >> v1:172.21.15.150:6789/0 conn(0x7f4cc81632f0 legacy=0x7f4cc80b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.458+0000 7f4cc759e640 1 -- 172.21.15.150:0/4146213633 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cc80b4d10 con 0x7f4cc8071a60 2024-03-17T23:47:16.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.459+0000 7f4cc759e640 1 --2- 172.21.15.150:0/4146213633 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4cc8071a60 0x7f4cc8071ed0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4cb80097a0 tx=0x7f4cb802b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=957de0a0a05774c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:16.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.459+0000 7f4cc5d9b640 1 -- 172.21.15.150:0/4146213633 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8037070 con 0x7f4cc8071a60 2024-03-17T23:47:16.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.460+0000 7f4cc5d9b640 1 -- 172.21.15.150:0/4146213633 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cb8030710 con 0x7f4cc8071a60 2024-03-17T23:47:16.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.460+0000 7f77c0777640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77b80670b0 0x7f77b8155740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.460+0000 7f2d8ffff640 1 --2- 172.21.15.150:0/1953679795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d901443a0 0x7f2d90071a60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2d780097a0 tx=0x7f2d7802b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c195d331a09bdb server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:16.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.460+0000 7f77c0777640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b80ccfb0 0x7f77b8151b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.460+0000 7f77c0777640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77b8162530 0x7f77b81616e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.460+0000 7f77c0777640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f77b80b5fd0 con 0x7f77b80ccfb0 2024-03-17T23:47:16.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.460+0000 7f77c0777640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f77b80b5cf0 con 0x7f77b8162530 2024-03-17T23:47:16.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.460+0000 7f77c0777640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f77b80cdd40 con 0x7f77b80670b0 2024-03-17T23:47:16.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.460+0000 7f4ccc91a640 1 -- 172.21.15.150:0/4146213633 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4cc8071a60 msgr2=0x7f4cc8071ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.460+0000 7f4ccc91a640 1 --2- 172.21.15.150:0/4146213633 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4cc8071a60 0x7f4cc8071ed0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4cb80097a0 tx=0x7f4cb802b730 comp rx=0 tx=0).stop 2024-03-17T23:47:16.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.461+0000 7f77be513640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77b80670b0 0x7f77b8155740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.461+0000 7f77be513640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77b80670b0 0x7f77b8155740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:55312/0 (socket says 172.21.15.150:55312) 2024-03-17T23:47:16.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.461+0000 7f77be513640 1 -- 172.21.15.150:0/246728261 learned_addr learned my addr 172.21.15.150:0/246728261 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:16.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.461+0000 7f2d8effd640 1 -- 172.21.15.150:0/1953679795 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d78038070 con 0x7f2d901443a0 2024-03-17T23:47:16.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.461+0000 7f4ccc91a640 1 -- 172.21.15.150:0/4146213633 shutdown_connections 2024-03-17T23:47:16.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.461+0000 7f4ccc91a640 1 --2- 172.21.15.150:0/4146213633 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4cc8071a60 0x7f4cc8071ed0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.461+0000 7f4ccc91a640 1 --2- 172.21.15.150:0/4146213633 >> v2:172.21.15.150:3300/0 conn(0x7f4cc80cf320 0x7f4cc80b36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.461+0000 7f4ccc91a640 1 -- 172.21.15.150:0/4146213633 >> 172.21.15.150:0/4146213633 conn(0x7f4cc80b1600 msgr2=0x7f4cc80b19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:16.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.461+0000 7f2d8effd640 1 -- 172.21.15.150:0/1953679795 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d78031520 con 0x7f2d901443a0 2024-03-17T23:47:16.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.461+0000 7f77bed14640 1 --2- 172.21.15.150:0/246728261 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77b8162530 0x7f77b81616e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.461+0000 7f77be513640 1 -- 172.21.15.150:0/246728261 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77b8162530 msgr2=0x7f77b81616e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.461+0000 7f77be513640 1 --2- 172.21.15.150:0/246728261 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77b8162530 0x7f77b81616e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.461+0000 7f77be513640 1 -- 172.21.15.150:0/246728261 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b80ccfb0 msgr2=0x7f77b8151b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.461+0000 7f77be513640 1 --2- 172.21.15.150:0/246728261 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b80ccfb0 0x7f77b8151b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.461+0000 7f77be513640 1 -- 172.21.15.150:0/246728261 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77b8163640 con 0x7f77b80670b0 2024-03-17T23:47:16.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.462+0000 7f77be513640 1 --2- 172.21.15.150:0/246728261 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77b80670b0 0x7f77b8155740 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f77b0058040 tx=0x7f77b0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:16.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.462+0000 7f77aeffd640 1 -- 172.21.15.150:0/246728261 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b0065020 con 0x7f77b80670b0 2024-03-17T23:47:16.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.462+0000 7f2d9561c640 1 -- 172.21.15.150:0/1953679795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d901443a0 msgr2=0x7f2d90071a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.462+0000 7f2d9561c640 1 --2- 172.21.15.150:0/1953679795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d901443a0 0x7f2d90071a60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2d780097a0 tx=0x7f2d7802b7c0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.462+0000 7f91e80dc640 1 Processor -- start 2024-03-17T23:47:16.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.462+0000 7f91e80dc640 1 -- start start 2024-03-17T23:47:16.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.462+0000 7f77aeffd640 1 -- 172.21.15.150:0/246728261 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77b0060450 con 0x7f77b80670b0 2024-03-17T23:47:16.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.462+0000 7f77c0777640 1 -- 172.21.15.150:0/246728261 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77b8163900 con 0x7f77b80670b0 2024-03-17T23:47:16.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.463+0000 7f4ccc91a640 1 -- 172.21.15.150:0/4146213633 shutdown_connections 2024-03-17T23:47:16.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.462+0000 7f77c0777640 1 -- 172.21.15.150:0/246728261 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f77b806eb90 con 0x7f77b80670b0 2024-03-17T23:47:16.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.463+0000 7f77aeffd640 1 -- 172.21.15.150:0/246728261 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b0065020 con 0x7f77b80670b0 2024-03-17T23:47:16.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.463+0000 7f77c0777640 1 -- 172.21.15.150:0/246728261 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77b80688b0 con 0x7f77b80670b0 2024-03-17T23:47:16.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.464+0000 7f91e80dc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f91e0072ce0 0x7f91e013ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.464+0000 7f91e80dc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f91e00cf640 0x7f91e013f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.464+0000 7f91e80dc640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91e013d350 con 0x7f91e006dd30 2024-03-17T23:47:16.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.464+0000 7f91e80dc640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f91e00cca10 con 0x7f91e00cf640 2024-03-17T23:47:16.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.464+0000 7f91e80dc640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f91e00cccf0 con 0x7f91e0072ce0 2024-03-17T23:47:16.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.464+0000 7f91e5677640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f91e006dd30 0x7f91e00cef30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:52282/0 (socket says 172.21.15.150:52282) 2024-03-17T23:47:16.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.464+0000 7f91e5677640 1 -- 172.21.15.150:0/1671073734 learned_addr learned my addr 172.21.15.150:0/1671073734 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:16.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.464+0000 7f91e5e78640 1 --2- 172.21.15.150:0/1671073734 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f91e0072ce0 0x7f91e013ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.464+0000 7f91e5e78640 1 -- 172.21.15.150:0/1671073734 >> v2:172.21.15.150:3300/0 conn(0x7f91e00cf640 msgr2=0x7f91e013f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:16.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.464+0000 7f2d9561c640 1 -- 172.21.15.150:0/1953679795 shutdown_connections 2024-03-17T23:47:16.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.464+0000 7f2d9561c640 1 --2- 172.21.15.150:0/1953679795 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d90161fc0 0x7f2d90071fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.464+0000 7f2d9561c640 1 --2- 172.21.15.150:0/1953679795 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d901443a0 0x7f2d90071a60 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.464+0000 7f2d9561c640 1 -- 172.21.15.150:0/1953679795 >> 172.21.15.150:0/1953679795 conn(0x7f2d900b1600 msgr2=0x7f2d900b19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:16.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.465+0000 7f91e6679640 1 --2- 172.21.15.150:0/1671073734 >> v2:172.21.15.150:3300/0 conn(0x7f91e00cf640 0x7f91e013f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.465+0000 7f2d9561c640 1 -- 172.21.15.150:0/1953679795 shutdown_connections 2024-03-17T23:47:16.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.465+0000 7f77aeffd640 1 -- 172.21.15.150:0/246728261 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f77b0068020 con 0x7f77b80670b0 2024-03-17T23:47:16.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.465+0000 7f77aeffd640 1 --2- 172.21.15.150:0/246728261 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f779c06bdd0 0x7f779c06e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.465+0000 7f91e5e78640 1 --2- 172.21.15.150:0/1671073734 >> v2:172.21.15.150:3300/0 conn(0x7f91e00cf640 0x7f91e013f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.465+0000 7f91e5e78640 1 -- 172.21.15.150:0/1671073734 >> v1:172.21.15.157:6789/0 conn(0x7f91e006dd30 legacy=0x7f91e00cef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.465+0000 7f91e5e78640 1 -- 172.21.15.150:0/1671073734 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e00ccfd0 con 0x7f91e0072ce0 2024-03-17T23:47:16.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.465+0000 7f77bdd12640 1 --2- 172.21.15.150:0/246728261 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f779c06bdd0 0x7f779c06e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.466+0000 7f77aeffd640 1 -- 172.21.15.150:0/246728261 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f77b009afa0 con 0x7f77b80670b0 2024-03-17T23:47:16.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.465+0000 7f2d9561c640 1 -- 172.21.15.150:0/1953679795 wait complete. 2024-03-17T23:47:16.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.463+0000 7f4ccc91a640 1 -- 172.21.15.150:0/4146213633 wait complete. 2024-03-17T23:47:16.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.465+0000 7f4ccc91a640 1 Processor -- start 2024-03-17T23:47:16.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.465+0000 7f4ccc91a640 1 -- start start 2024-03-17T23:47:16.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.466+0000 7f91e6679640 1 --2- 172.21.15.150:0/1671073734 >> v2:172.21.15.150:3300/0 conn(0x7f91e00cf640 0x7f91e013f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:47:16.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.466+0000 7f77bdd12640 1 --2- 172.21.15.150:0/246728261 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f779c06bdd0 0x7f779c06e290 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f77b8158920 tx=0x7f77a8009460 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:16.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.466+0000 7f2d9561c640 1 Processor -- start 2024-03-17T23:47:16.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.466+0000 7f91e5e78640 1 --2- 172.21.15.150:0/1671073734 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f91e0072ce0 0x7f91e013ce10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f91d80097a0 tx=0x7f91d802b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b60a5553e8b9d7c server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:16.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.466+0000 7f4ccc91a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4cc80cf320 0x7f4cc815b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.466+0000 7f4ccc91a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4cc81632f0 0x7f4cc81f55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.466+0000 7f4ccc91a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cc81514b0 0x7f4cc814d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.466+0000 7f4ccc91a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc80b2ec0 con 0x7f4cc81514b0 2024-03-17T23:47:16.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.466+0000 7f4ccc91a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4cc80b2be0 con 0x7f4cc81632f0 2024-03-17T23:47:16.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.466+0000 7f4ccc91a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4cc80cd9f0 con 0x7f4cc80cf320 2024-03-17T23:47:16.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.467+0000 7f4cc659c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4cc81632f0 0x7f4cc81f55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.467+0000 7f4cc659c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4cc81632f0 0x7f4cc81f55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:42140/0 (socket says 172.21.15.150:42140) 2024-03-17T23:47:16.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.467+0000 7f4cc759e640 1 --2- 172.21.15.150:0/2110063973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cc81514b0 0x7f4cc814d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.467+0000 7f4cc6d9d640 1 --2- 172.21.15.150:0/2110063973 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4cc80cf320 0x7f4cc815b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.467+0000 7f4cc659c640 1 -- 172.21.15.150:0/2110063973 learned_addr learned my addr 172.21.15.150:0/2110063973 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:16.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.467+0000 7f4cc759e640 1 -- 172.21.15.150:0/2110063973 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4cc80cf320 msgr2=0x7f4cc815b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.467+0000 7f4cc759e640 1 --2- 172.21.15.150:0/2110063973 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4cc80cf320 0x7f4cc815b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.467+0000 7f4cc759e640 1 -- 172.21.15.150:0/2110063973 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4cc81632f0 msgr2=0x7f4cc81f55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.467+0000 7f4cc759e640 1 --2- 172.21.15.150:0/2110063973 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4cc81632f0 0x7f4cc81f55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.467+0000 7f4cc759e640 1 -- 172.21.15.150:0/2110063973 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cc81519b0 con 0x7f4cc81514b0 2024-03-17T23:47:16.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.467+0000 7f4cc659c640 1 --2- 172.21.15.150:0/2110063973 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4cc81632f0 0x7f4cc81f55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:16.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.467+0000 7f2d9561c640 1 -- start start 2024-03-17T23:47:16.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.467+0000 7f4cc759e640 1 --2- 172.21.15.150:0/2110063973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cc81514b0 0x7f4cc814d130 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8032000 tx=0x7f4cb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:16.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.468+0000 7f4caf7fe640 1 -- 172.21.15.150:0/2110063973 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8040020 con 0x7f4cc81514b0 2024-03-17T23:47:16.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.468+0000 7f2d9561c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d901443a0 0x7f2d902006b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.468+0000 7f2d9561c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d9015eff0 0x7f2d90160440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.468+0000 7f2d9561c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d90161fc0 0x7f2d90160980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.468+0000 7f2d9561c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2d900adfe0 con 0x7f2d901443a0 2024-03-17T23:47:16.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.468+0000 7f2d9561c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9015ead0 con 0x7f2d9015eff0 2024-03-17T23:47:16.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.468+0000 7f2d9561c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2d900ae160 con 0x7f2d90161fc0 2024-03-17T23:47:16.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.468+0000 7f2d94e1b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d90161fc0 0x7f2d90160980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.468+0000 7f4ccc91a640 1 -- 172.21.15.150:0/2110063973 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cc815ace0 con 0x7f4cc81514b0 2024-03-17T23:47:16.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.468+0000 7f4ccc91a640 1 -- 172.21.15.150:0/2110063973 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cc815b2a0 con 0x7f4cc81514b0 2024-03-17T23:47:16.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.468+0000 7f4caf7fe640 1 -- 172.21.15.150:0/2110063973 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cb8003610 con 0x7f4cc81514b0 2024-03-17T23:47:16.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.468+0000 7f4caf7fe640 1 -- 172.21.15.150:0/2110063973 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8037040 con 0x7f4cc81514b0 2024-03-17T23:47:16.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.468+0000 7f2d94e1b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d90161fc0 0x7f2d90160980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:55356/0 (socket says 172.21.15.150:55356) 2024-03-17T23:47:16.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.468+0000 7f2d94e1b640 1 -- 172.21.15.150:0/3506457757 learned_addr learned my addr 172.21.15.150:0/3506457757 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:16.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.468+0000 7f2d8ffff640 1 --2- 172.21.15.150:0/3506457757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d901443a0 0x7f2d902006b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.468+0000 7f2d94e1b640 1 -- 172.21.15.150:0/3506457757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d9015eff0 msgr2=0x7f2d90160440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.468+0000 7f2d94e1b640 1 --2- 172.21.15.150:0/3506457757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d9015eff0 0x7f2d90160440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.468+0000 7f2d94e1b640 1 -- 172.21.15.150:0/3506457757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d901443a0 msgr2=0x7f2d902006b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.469+0000 7f2d94e1b640 1 --2- 172.21.15.150:0/3506457757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d901443a0 0x7f2d902006b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.469+0000 7f2d94e1b640 1 -- 172.21.15.150:0/3506457757 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d9014ff20 con 0x7f2d90161fc0 2024-03-17T23:47:16.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.469+0000 7f2d8ffff640 1 --2- 172.21.15.150:0/3506457757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d901443a0 0x7f2d902006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:47:16.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.469+0000 7f2d94e1b640 1 --2- 172.21.15.150:0/3506457757 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d90161fc0 0x7f2d90160980 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2d8800eed0 tx=0x7f2d8800b480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:16.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.470+0000 7f4ccc91a640 1 -- 172.21.15.150:0/2110063973 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ca8005290 con 0x7f4cc81514b0 2024-03-17T23:47:16.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.470+0000 7f4caf7fe640 1 -- 172.21.15.150:0/2110063973 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4cb8043020 con 0x7f4cc81514b0 2024-03-17T23:47:16.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.471+0000 7f4caf7fe640 1 --2- 172.21.15.150:0/2110063973 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4cb406bbd0 0x7f4cb406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.471+0000 7f4caf7fe640 1 -- 172.21.15.150:0/2110063973 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f4cb80abfe0 con 0x7f4cc81514b0 2024-03-17T23:47:16.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.471+0000 7f4cc6d9d640 1 --2- 172.21.15.150:0/2110063973 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4cb406bbd0 0x7f4cb406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.472+0000 7f4cc6d9d640 1 --2- 172.21.15.150:0/2110063973 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4cb406bbd0 0x7f4cb406e090 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4cc81f6d70 tx=0x7f4cbc009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:16.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.475+0000 7f77aeffd640 1 -- 172.21.15.150:0/246728261 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f77b006f070 con 0x7f77b80670b0 2024-03-17T23:47:16.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.475+0000 7f91e4e76640 1 -- 172.21.15.150:0/1671073734 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d8037070 con 0x7f91e0072ce0 2024-03-17T23:47:16.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.475+0000 7f91e4e76640 1 -- 172.21.15.150:0/1671073734 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91d8030710 con 0x7f91e0072ce0 2024-03-17T23:47:16.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.476+0000 7f34c215b640 1 Processor -- start 2024-03-17T23:47:16.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.476+0000 7f85ebfff640 1 Processor -- start 2024-03-17T23:47:16.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.476+0000 7f91e80dc640 1 -- 172.21.15.150:0/1671073734 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f91e0072ce0 msgr2=0x7f91e013ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.476+0000 7f91e80dc640 1 --2- 172.21.15.150:0/1671073734 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f91e0072ce0 0x7f91e013ce10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f91d80097a0 tx=0x7f91d802b7b0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.476+0000 7f85ebfff640 1 -- start start 2024-03-17T23:47:16.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.477+0000 7f91e80dc640 1 -- 172.21.15.150:0/1671073734 shutdown_connections 2024-03-17T23:47:16.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.477+0000 7f91e80dc640 1 --2- 172.21.15.150:0/1671073734 >> v2:172.21.15.150:3300/0 conn(0x7f91e00cf640 0x7f91e013f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.477+0000 7f91e80dc640 1 --2- 172.21.15.150:0/1671073734 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f91e0072ce0 0x7f91e013ce10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.477+0000 7f91e80dc640 1 -- 172.21.15.150:0/1671073734 >> 172.21.15.150:0/1671073734 conn(0x7f91e006b190 msgr2=0x7f91e006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:16.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.475+0000 7f2d8cff9640 1 -- 172.21.15.150:0/3506457757 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d8801a070 con 0x7f2d90161fc0 2024-03-17T23:47:16.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.475+0000 7f2d8cff9640 1 -- 172.21.15.150:0/3506457757 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d8800bb10 con 0x7f2d90161fc0 2024-03-17T23:47:16.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.477+0000 7f91e80dc640 1 -- 172.21.15.150:0/1671073734 shutdown_connections 2024-03-17T23:47:16.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.476+0000 7f2d8cff9640 1 -- 172.21.15.150:0/3506457757 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d88003990 con 0x7f2d90161fc0 2024-03-17T23:47:16.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.477+0000 7f34c215b640 1 -- start start 2024-03-17T23:47:16.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.477+0000 7f91e80dc640 1 -- 172.21.15.150:0/1671073734 wait complete. 2024-03-17T23:47:16.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.477+0000 7f85ebfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f85e4048ac0 0x7f85e4052670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.477+0000 7f85ebfff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f85e403b0a0 0x7f85e4053790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.477+0000 7f85ebfff640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85e403a2e0 con 0x7f85e40eac60 2024-03-17T23:47:16.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.477+0000 7f85ebfff640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f85e4052bb0 con 0x7f85e4048ac0 2024-03-17T23:47:16.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.477+0000 7f85ebfff640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f85e403a090 con 0x7f85e403b0a0 2024-03-17T23:47:16.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.478+0000 7f2d9561c640 1 -- 172.21.15.150:0/3506457757 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d90150200 con 0x7f2d90161fc0 2024-03-17T23:47:16.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.478+0000 7f2d9561c640 1 -- 172.21.15.150:0/3506457757 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d901507c0 con 0x7f2d90161fc0 2024-03-17T23:47:16.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.478+0000 7f85eb7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f85e403b0a0 0x7f85e4053790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.478+0000 7f85eb7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f85e403b0a0 0x7f85e4053790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:42150/0 (socket says 172.21.15.150:42150) 2024-03-17T23:47:16.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.478+0000 7f85eb7fe640 1 -- 172.21.15.150:0/2545109264 learned_addr learned my addr 172.21.15.150:0/2545109264 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:16.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.478+0000 7f34c215b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f34bc15ae40 0x7f34bc0cb440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.478+0000 7f34c215b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34bc0cf2a0 0x7f34bc0ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.478+0000 7f34c215b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f34bc0cd100 0x7f34bc0d0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.478+0000 7f34bb7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f34bc15ae40 0x7f34bc0cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.478+0000 7f34c215b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f34bc0cc730 con 0x7f34bc0cd100 2024-03-17T23:47:16.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.478+0000 7f34c215b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f34bc0cbb20 con 0x7f34bc15ae40 2024-03-17T23:47:16.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.478+0000 7f34bb7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f34bc15ae40 0x7f34bc0cb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:42166/0 (socket says 172.21.15.150:42166) 2024-03-17T23:47:16.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.478+0000 7f34c215b640 1 -- 172.21.15.150:0/1090559368 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f34bc0cbe00 con 0x7f34bc0cf2a0 2024-03-17T23:47:16.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.478+0000 7f85eb7fe640 1 -- 172.21.15.150:0/2545109264 >> v2:172.21.15.150:3300/0 conn(0x7f85e4048ac0 msgr2=0x7f85e4052670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:16.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.478+0000 7f85eb7fe640 1 --2- 172.21.15.150:0/2545109264 >> v2:172.21.15.150:3300/0 conn(0x7f85e4048ac0 0x7f85e4052670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.478+0000 7f85eb7fe640 1 -- 172.21.15.150:0/2545109264 >> v1:172.21.15.150:6789/0 conn(0x7f85e40eac60 legacy=0x7f85e40483b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:16.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.478+0000 7f85eb7fe640 1 -- 172.21.15.150:0/2545109264 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85e403a5c0 con 0x7f85e403b0a0 2024-03-17T23:47:16.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.479+0000 7f91e80dc640 1 Processor -- start 2024-03-17T23:47:16.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.479+0000 7f34bbfff640 1 --2- 172.21.15.150:0/1090559368 >> v2:172.21.15.150:3300/0 conn(0x7f34bc0cd100 0x7f34bc0d0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.479+0000 7f91e80dc640 1 -- start start 2024-03-17T23:47:16.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.479+0000 7f85eb7fe640 1 --2- 172.21.15.150:0/2545109264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f85e403b0a0 0x7f85e4053790 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f85ec06b030 tx=0x7f85ec0a6f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=582eb799e5764191 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:16.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.479+0000 7f85e9ffb640 1 -- 172.21.15.150:0/2545109264 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ec0b4070 con 0x7f85e403b0a0 2024-03-17T23:47:16.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.479+0000 7f85e9ffb640 1 -- 172.21.15.150:0/2545109264 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85ec0a7dd0 con 0x7f85e403b0a0 2024-03-17T23:47:16.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.479+0000 7f85e9ffb640 1 -- 172.21.15.150:0/2545109264 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ec0af640 con 0x7f85e403b0a0 2024-03-17T23:47:16.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.478+0000 7f34bb7fe640 1 -- 172.21.15.150:0/1090559368 learned_addr learned my addr 172.21.15.150:0/1090559368 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:16.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.479+0000 7f34bbfff640 1 -- 172.21.15.150:0/1090559368 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34bc0cf2a0 msgr2=0x7f34bc0ccbc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-17T23:47:16.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.479+0000 7f34bbfff640 1 --2- 172.21.15.150:0/1090559368 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34bc0cf2a0 0x7f34bc0ccbc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.479+0000 7f34bbfff640 1 -- 172.21.15.150:0/1090559368 >> v2:172.21.15.157:3300/0 conn(0x7f34bc15ae40 msgr2=0x7f34bc0cb440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:16.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.479+0000 7f34bbfff640 1 --2- 172.21.15.150:0/1090559368 >> v2:172.21.15.157:3300/0 conn(0x7f34bc15ae40 0x7f34bc0cb440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.479+0000 7f34bbfff640 1 -- 172.21.15.150:0/1090559368 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34bc0cc0e0 con 0x7f34bc0cd100 2024-03-17T23:47:16.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.480+0000 7f2d9561c640 1 -- 172.21.15.150:0/3506457757 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d900b2c40 con 0x7f2d90161fc0 2024-03-17T23:47:16.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.480+0000 7f91e80dc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91e006dd30 0x7f91e01577e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.480+0000 7f91e80dc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91e00cf640 0x7f91e015ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.480+0000 7f4caf7fe640 1 -- 172.21.15.150:0/2110063973 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4cb804a030 con 0x7f4cc81514b0 2024-03-17T23:47:16.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.480+0000 7f2d8cff9640 1 -- 172.21.15.150:0/3506457757 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2d88005d10 con 0x7f2d90161fc0 2024-03-17T23:47:16.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.480+0000 7f91e80dc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f91e0152160 0x7f91e0159640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.480+0000 7f91e5677640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91e00cf640 0x7f91e015ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.480+0000 7f91e5677640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91e00cf640 0x7f91e015ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40298/0 (socket says 172.21.15.150:40298) 2024-03-17T23:47:16.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.480+0000 7f91e5677640 1 -- 172.21.15.150:0/838118530 learned_addr learned my addr 172.21.15.150:0/838118530 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:16.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.480+0000 7f91e5e78640 1 --2- 172.21.15.150:0/838118530 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91e006dd30 0x7f91e01577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.480+0000 7f85ebfff640 1 -- 172.21.15.150:0/2545109264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f85e403b0a0 msgr2=0x7f85e4053790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.480+0000 7f85ebfff640 1 --2- 172.21.15.150:0/2545109264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f85e403b0a0 0x7f85e4053790 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f85ec06b030 tx=0x7f85ec0a6f20 comp rx=0 tx=0).stop 2024-03-17T23:47:16.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.480+0000 7f85ebfff640 1 -- 172.21.15.150:0/2545109264 shutdown_connections 2024-03-17T23:47:16.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.480+0000 7f91e6679640 1 --2- 172.21.15.150:0/838118530 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f91e0152160 0x7f91e0159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.481+0000 7f91e80dc640 1 -- 172.21.15.150:0/838118530 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f91e006e170 con 0x7f91e00cf640 2024-03-17T23:47:16.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.481+0000 7f91e80dc640 1 -- 172.21.15.150:0/838118530 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f91e00cea10 con 0x7f91e006dd30 2024-03-17T23:47:16.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.480+0000 7f34bbfff640 1 --2- 172.21.15.150:0/1090559368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34bc0cd100 0x7f34bc0d0ac0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f34bc06e620 tx=0x7f34b4030250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fe8a3ae8502a271 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:16.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.481+0000 7f91e80dc640 1 -- 172.21.15.150:0/838118530 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f91e006e2f0 con 0x7f91e0152160 2024-03-17T23:47:16.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.480+0000 7f85ebfff640 1 --2- 172.21.15.150:0/2545109264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f85e403b0a0 0x7f85e4053790 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.480+0000 7f85ebfff640 1 --2- 172.21.15.150:0/2545109264 >> v2:172.21.15.150:3300/0 conn(0x7f85e4048ac0 0x7f85e4052670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.481+0000 7f85ebfff640 1 -- 172.21.15.150:0/2545109264 >> 172.21.15.150:0/2545109264 conn(0x7f85e401e6b0 msgr2=0x7f85e40553e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:16.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.481+0000 7f85ebfff640 1 -- 172.21.15.150:0/2545109264 shutdown_connections 2024-03-17T23:47:16.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.481+0000 7f2d8cff9640 1 --2- 172.21.15.150:0/3506457757 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2d7006bca0 0x7f2d7006e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.481+0000 7f2d8cff9640 1 -- 172.21.15.150:0/3506457757 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f2d8808ad50 con 0x7f2d90161fc0 2024-03-17T23:47:16.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.481+0000 7f91e5677640 1 -- 172.21.15.150:0/838118530 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f91e0152160 msgr2=0x7f91e0159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.481+0000 7f91e5677640 1 --2- 172.21.15.150:0/838118530 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f91e0152160 0x7f91e0159640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.481+0000 7f91e5677640 1 -- 172.21.15.150:0/838118530 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91e006dd30 msgr2=0x7f91e01577e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.481+0000 7f91e5677640 1 --2- 172.21.15.150:0/838118530 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91e006dd30 0x7f91e01577e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.481+0000 7f91e5677640 1 -- 172.21.15.150:0/838118530 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e015cf90 con 0x7f91e00cf640 2024-03-17T23:47:16.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.481+0000 7f85ebfff640 1 -- 172.21.15.150:0/2545109264 wait complete. 2024-03-17T23:47:16.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.482+0000 7f91e5677640 1 --2- 172.21.15.150:0/838118530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91e00cf640 0x7f91e015ca50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f91d000cb20 tx=0x7f91d0002170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:16.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.482+0000 7f34ba7fc640 1 -- 172.21.15.150:0/1090559368 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b403d070 con 0x7f34bc0cd100 2024-03-17T23:47:16.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.482+0000 7f34ba7fc640 1 -- 172.21.15.150:0/1090559368 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34b4038440 con 0x7f34bc0cd100 2024-03-17T23:47:16.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.482+0000 7f34ba7fc640 1 -- 172.21.15.150:0/1090559368 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b4038810 con 0x7f34bc0cd100 2024-03-17T23:47:16.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.482+0000 7f2d8ffff640 1 --2- 172.21.15.150:0/3506457757 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2d7006bca0 0x7f2d7006e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.482+0000 7f91de7fc640 1 -- 172.21.15.150:0/838118530 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d0018070 con 0x7f91e00cf640 2024-03-17T23:47:16.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.482+0000 7f91e80dc640 1 -- 172.21.15.150:0/838118530 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91e01516a0 con 0x7f91e00cf640 2024-03-17T23:47:16.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.482+0000 7f91e80dc640 1 -- 172.21.15.150:0/838118530 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91e015d4f0 con 0x7f91e00cf640 2024-03-17T23:47:16.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.482+0000 7f91de7fc640 1 -- 172.21.15.150:0/838118530 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91d000a7a0 con 0x7f91e00cf640 2024-03-17T23:47:16.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.483+0000 7f91de7fc640 1 -- 172.21.15.150:0/838118530 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d000ab70 con 0x7f91e00cf640 2024-03-17T23:47:16.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.483+0000 7f34c215b640 1 -- 172.21.15.150:0/1090559368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34bc0cd100 msgr2=0x7f34bc0d0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.483+0000 7f34c215b640 1 --2- 172.21.15.150:0/1090559368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34bc0cd100 0x7f34bc0d0ac0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f34bc06e620 tx=0x7f34b4030250 comp rx=0 tx=0).stop 2024-03-17T23:47:16.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.483+0000 7f34c215b640 1 -- 172.21.15.150:0/1090559368 shutdown_connections 2024-03-17T23:47:16.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.483+0000 7f34c215b640 1 --2- 172.21.15.150:0/1090559368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34bc0cd100 0x7f34bc0d0ac0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.483+0000 7f34c215b640 1 --2- 172.21.15.150:0/1090559368 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34bc0cf2a0 0x7f34bc0ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.483+0000 7f34c215b640 1 --2- 172.21.15.150:0/1090559368 >> v2:172.21.15.157:3300/0 conn(0x7f34bc15ae40 0x7f34bc0cb440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.483+0000 7f34c215b640 1 -- 172.21.15.150:0/1090559368 >> 172.21.15.150:0/1090559368 conn(0x7f34bc099a70 msgr2=0x7f34bc099e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:16.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.483+0000 7f85ebfff640 1 Processor -- start 2024-03-17T23:47:16.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.483+0000 7f34c215b640 1 -- 172.21.15.150:0/1090559368 shutdown_connections 2024-03-17T23:47:16.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.483+0000 7f85ebfff640 1 -- start start 2024-03-17T23:47:16.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.484+0000 7f34c215b640 1 -- 172.21.15.150:0/1090559368 wait complete. 2024-03-17T23:47:16.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.484+0000 7f85ebfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f85e403b0a0 0x7f85e404e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.484+0000 7f85ebfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85e4048ac0 0x7f85e40dd010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.485+0000 7f85eaffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f85e403b0a0 0x7f85e404e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.485+0000 7f85eaffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f85e403b0a0 0x7f85e404e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:42188/0 (socket says 172.21.15.150:42188) 2024-03-17T23:47:16.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.485+0000 7f85eaffd640 1 -- 172.21.15.150:0/240956365 learned_addr learned my addr 172.21.15.150:0/240956365 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:16.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.485+0000 7f85ebfff640 1 --2- 172.21.15.150:0/240956365 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f85e40eac60 0x7f85e40dd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.485+0000 7f91de7fc640 1 -- 172.21.15.150:0/838118530 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f91d001b070 con 0x7f91e00cf640 2024-03-17T23:47:16.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.485+0000 7f85ebfff640 1 -- 172.21.15.150:0/240956365 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f85e40eb040 con 0x7f85e4048ac0 2024-03-17T23:47:16.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.485+0000 7f85ebfff640 1 -- 172.21.15.150:0/240956365 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f85e403ac10 con 0x7f85e403b0a0 2024-03-17T23:47:16.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.485+0000 7f85ebfff640 1 -- 172.21.15.150:0/240956365 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f85e40eb1c0 con 0x7f85e40eac60 2024-03-17T23:47:16.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.485+0000 7f85eb7fe640 1 --2- 172.21.15.150:0/240956365 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f85e40eac60 0x7f85e40dd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.485+0000 7f85ea7fc640 1 --2- 172.21.15.150:0/240956365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85e4048ac0 0x7f85e40dd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.486+0000 7f34c215b640 1 Processor -- start 2024-03-17T23:47:16.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.486+0000 7f85eaffd640 1 -- 172.21.15.150:0/240956365 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f85e40eac60 msgr2=0x7f85e40dd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.486+0000 7f85eaffd640 1 --2- 172.21.15.150:0/240956365 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f85e40eac60 0x7f85e40dd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.486+0000 7f85eaffd640 1 -- 172.21.15.150:0/240956365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85e4048ac0 msgr2=0x7f85e40dd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.486+0000 7f85eaffd640 1 --2- 172.21.15.150:0/240956365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85e4048ac0 0x7f85e40dd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.486+0000 7f85eaffd640 1 -- 172.21.15.150:0/240956365 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85e404d710 con 0x7f85e403b0a0 2024-03-17T23:47:16.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.486+0000 7f85ea7fc640 1 --2- 172.21.15.150:0/240956365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85e4048ac0 0x7f85e40dd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:16.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.487+0000 7f2d8ffff640 1 --2- 172.21.15.150:0/3506457757 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2d7006bca0 0x7f2d7006e160 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2d78035000 tx=0x7f2d78030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:16.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.487+0000 7f85eaffd640 1 --2- 172.21.15.150:0/240956365 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f85e403b0a0 0x7f85e404e7b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f85dc00b3e0 tx=0x7f85dc00b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:16.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.487+0000 7f91de7fc640 1 --2- 172.21.15.150:0/838118530 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f91cc06bbd0 0x7f91cc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.486+0000 7f34c215b640 1 -- start start 2024-03-17T23:47:16.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.487+0000 7f34c215b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34bc0cf2a0 0x7f34bc151b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.487+0000 7f85d37fe640 1 -- 172.21.15.150:0/240956365 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85dc015020 con 0x7f85e403b0a0 2024-03-17T23:47:16.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.487+0000 7f85d37fe640 1 -- 172.21.15.150:0/240956365 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85dc011850 con 0x7f85e403b0a0 2024-03-17T23:47:16.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.487+0000 7f34c215b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34bc15ae40 0x7f34bc1f7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.488+0000 7f34c215b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34bc14eb70 0x7f34bc158f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.488+0000 7f34c215b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f34bc0cf700 con 0x7f34bc14eb70 2024-03-17T23:47:16.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.488+0000 7f34c215b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f34bc15b220 con 0x7f34bc0cf2a0 2024-03-17T23:47:16.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.488+0000 7f34c215b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f34bc0cf880 con 0x7f34bc15ae40 2024-03-17T23:47:16.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.488+0000 7f34bbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34bc14eb70 0x7f34bc158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.488+0000 7f34bbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34bc14eb70 0x7f34bc158f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40316/0 (socket says 172.21.15.150:40316) 2024-03-17T23:47:16.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.488+0000 7f34bbfff640 1 -- 172.21.15.150:0/2355402762 learned_addr learned my addr 172.21.15.150:0/2355402762 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:16.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.488+0000 7f85d37fe640 1 -- 172.21.15.150:0/240956365 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85dc011c40 con 0x7f85e403b0a0 2024-03-17T23:47:16.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.488+0000 7f5390475640 1 Processor -- start 2024-03-17T23:47:16.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.488+0000 7f34bbfff640 1 -- 172.21.15.150:0/2355402762 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34bc15ae40 msgr2=0x7f34bc1f7e10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-17T23:47:16.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.488+0000 7f34bbfff640 1 --2- 172.21.15.150:0/2355402762 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34bc15ae40 0x7f34bc1f7e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.488+0000 7f34bbfff640 1 -- 172.21.15.150:0/2355402762 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34bc0cf2a0 msgr2=0x7f34bc151b60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-17T23:47:16.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.488+0000 7f34bbfff640 1 --2- 172.21.15.150:0/2355402762 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34bc0cf2a0 0x7f34bc151b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.488+0000 7f34bbfff640 1 -- 172.21.15.150:0/2355402762 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34bc1522f0 con 0x7f34bc14eb70 2024-03-17T23:47:16.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.489+0000 7f34bbfff640 1 --2- 172.21.15.150:0/2355402762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34bc14eb70 0x7f34bc158f00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f34b403b040 tx=0x7f34b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:16.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.488+0000 7f85ebfff640 1 -- 172.21.15.150:0/240956365 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85e404d9f0 con 0x7f85e403b0a0 2024-03-17T23:47:16.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.489+0000 7f85ebfff640 1 -- 172.21.15.150:0/240956365 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85e404dfb0 con 0x7f85e403b0a0 2024-03-17T23:47:16.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.489+0000 7f34a3fff640 1 -- 172.21.15.150:0/2355402762 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b4046070 con 0x7f34bc14eb70 2024-03-17T23:47:16.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.489+0000 7f34a3fff640 1 -- 172.21.15.150:0/2355402762 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34b4038e20 con 0x7f34bc14eb70 2024-03-17T23:47:16.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.489+0000 7f34a3fff640 1 -- 172.21.15.150:0/2355402762 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b403d030 con 0x7f34bc14eb70 2024-03-17T23:47:16.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.490+0000 7f2d8cff9640 1 -- 172.21.15.150:0/3506457757 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2d900b2c40 con 0x7f2d90161fc0 2024-03-17T23:47:16.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.490+0000 7f34c215b640 1 -- 172.21.15.150:0/2355402762 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34bc1525d0 con 0x7f34bc14eb70 2024-03-17T23:47:16.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.489+0000 7f5390475640 1 -- start start 2024-03-17T23:47:16.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.490+0000 7f34c215b640 1 -- 172.21.15.150:0/2355402762 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34bc152b90 con 0x7f34bc14eb70 2024-03-17T23:47:16.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.490+0000 7f85d17fa640 1 -- 172.21.15.150:0/240956365 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85e4055fc0 con 0x7f85e403b0a0 2024-03-17T23:47:16.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.490+0000 7f5390475640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f538815df40 0x7f53880b97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.490+0000 7f5390475640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f538815e7a0 0x7f53880d2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.490+0000 7f5390475640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f538815ec90 con 0x7f538815e320 2024-03-17T23:47:16.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.490+0000 7f5390475640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f53880b6d40 con 0x7f538815e7a0 2024-03-17T23:47:16.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.490+0000 7f5390475640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f538815ef70 con 0x7f538815df40 2024-03-17T23:47:16.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.491+0000 7f538e211640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f538815df40 0x7f53880b97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.491+0000 7f538e211640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f538815df40 0x7f53880b97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:55376/0 (socket says 172.21.15.150:55376) 2024-03-17T23:47:16.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.491+0000 7f538da10640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f538815e320 0x7f5388162160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:52296/0 (socket says 172.21.15.150:52296) 2024-03-17T23:47:16.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.491+0000 7f538da10640 1 -- 172.21.15.150:0/1840843067 learned_addr learned my addr 172.21.15.150:0/1840843067 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:16.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.491+0000 7f538e211640 1 -- 172.21.15.150:0/1840843067 >> v2:172.21.15.150:3300/0 conn(0x7f538815e7a0 msgr2=0x7f53880d2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:16.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.491+0000 7f538e211640 1 --2- 172.21.15.150:0/1840843067 >> v2:172.21.15.150:3300/0 conn(0x7f538815e7a0 0x7f53880d2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.491+0000 7f538e211640 1 -- 172.21.15.150:0/1840843067 >> v1:172.21.15.157:6789/0 conn(0x7f538815e320 legacy=0x7f5388162160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.491+0000 7f538e211640 1 -- 172.21.15.150:0/1840843067 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f538815f250 con 0x7f538815df40 2024-03-17T23:47:16.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.489+0000 7f91e80dc640 1 -- 172.21.15.150:0/838118530 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91c0005290 con 0x7f91e00cf640 2024-03-17T23:47:16.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.490+0000 7f91de7fc640 1 -- 172.21.15.150:0/838118530 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f91d008a610 con 0x7f91e00cf640 2024-03-17T23:47:16.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.491+0000 7f91e5e78640 1 --2- 172.21.15.150:0/838118530 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f91cc06bbd0 0x7f91cc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.491+0000 7f91e5e78640 1 --2- 172.21.15.150:0/838118530 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f91cc06bbd0 0x7f91cc06e090 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f91d8034000 tx=0x7f91d8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:16.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.491+0000 7f85d37fe640 1 -- 172.21.15.150:0/240956365 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f85dc018020 con 0x7f85e403b0a0 2024-03-17T23:47:16.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.493+0000 7f538e211640 1 --2- 172.21.15.150:0/1840843067 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f538815df40 0x7f53880b97f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f53780097a0 tx=0x7f537802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f365880d501008d server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:16.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.493+0000 7f538d20f640 1 -- 172.21.15.150:0/1840843067 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5378034070 con 0x7f538815df40 2024-03-17T23:47:16.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.493+0000 7f538d20f640 1 -- 172.21.15.150:0/1840843067 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5378030800 con 0x7f538815df40 2024-03-17T23:47:16.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.494+0000 7f5390475640 1 -- 172.21.15.150:0/1840843067 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f538815df40 msgr2=0x7f53880b97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.494+0000 7f5390475640 1 --2- 172.21.15.150:0/1840843067 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f538815df40 0x7f53880b97f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f53780097a0 tx=0x7f537802b940 comp rx=0 tx=0).stop 2024-03-17T23:47:16.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.493+0000 7f34a1ffb640 1 -- 172.21.15.150:0/2355402762 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3494005290 con 0x7f34bc14eb70 2024-03-17T23:47:16.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.495+0000 7f5390475640 1 -- 172.21.15.150:0/1840843067 shutdown_connections 2024-03-17T23:47:16.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.495+0000 7f5390475640 1 --2- 172.21.15.150:0/1840843067 >> v2:172.21.15.150:3300/0 conn(0x7f538815e7a0 0x7f53880d2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.495+0000 7f5390475640 1 --2- 172.21.15.150:0/1840843067 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f538815df40 0x7f53880b97f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.495+0000 7f5390475640 1 -- 172.21.15.150:0/1840843067 >> 172.21.15.150:0/1840843067 conn(0x7f5388072ef0 msgr2=0x7f53880d3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:16.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.495+0000 7f5390475640 1 -- 172.21.15.150:0/1840843067 shutdown_connections 2024-03-17T23:47:16.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.495+0000 7f5390475640 1 -- 172.21.15.150:0/1840843067 wait complete. 2024-03-17T23:47:16.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.496+0000 7f34a3fff640 1 -- 172.21.15.150:0/2355402762 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f34b4049070 con 0x7f34bc14eb70 2024-03-17T23:47:16.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.496+0000 7f34a3fff640 1 --2- 172.21.15.150:0/2355402762 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f34a806bca0 0x7f34a806e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.496+0000 7f34a3fff640 1 -- 172.21.15.150:0/2355402762 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f34b40ac3a0 con 0x7f34bc14eb70 2024-03-17T23:47:16.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.492+0000 7f85d37fe640 1 --2- 172.21.15.150:0/240956365 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f85d806bca0 0x7f85d806e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.492+0000 7f85d37fe640 1 -- 172.21.15.150:0/240956365 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f85dc08a950 con 0x7f85e403b0a0 2024-03-17T23:47:16.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.492+0000 7f85ea7fc640 1 --2- 172.21.15.150:0/240956365 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f85d806bca0 0x7f85d806e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.493+0000 7f85ea7fc640 1 --2- 172.21.15.150:0/240956365 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f85d806bca0 0x7f85d806e160 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f85e4180e90 tx=0x7f85e0009110 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:16.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.497+0000 7f5390475640 1 Processor -- start 2024-03-17T23:47:16.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.497+0000 7f5390475640 1 -- start start 2024-03-17T23:47:16.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.497+0000 7f85d37fe640 1 -- 172.21.15.150:0/240956365 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f85e4055fc0 con 0x7f85e403b0a0 2024-03-17T23:47:16.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.498+0000 7f5390475640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f538815df40 0x7f53880c9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.498+0000 7f5390475640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f538815e320 0x7f53881f89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.498+0000 7f5390475640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f538815e7a0 0x7f53881f8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.498+0000 7f5390475640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5388162870 con 0x7f538815e7a0 2024-03-17T23:47:16.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.498+0000 7f5390475640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f53880d2640 con 0x7f538815e320 2024-03-17T23:47:16.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.498+0000 7f5390475640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f53880b4dd0 con 0x7f538815df40 2024-03-17T23:47:16.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.499+0000 7f538e211640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f538815df40 0x7f53880c9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.499+0000 7f538e211640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f538815df40 0x7f53880c9ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:55388/0 (socket says 172.21.15.150:55388) 2024-03-17T23:47:16.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.499+0000 7f538e211640 1 -- 172.21.15.150:0/943539024 learned_addr learned my addr 172.21.15.150:0/943539024 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:16.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.499+0000 7f538da10640 1 --2- 172.21.15.150:0/943539024 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f538815e320 0x7f53881f89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.499+0000 7f538ea12640 1 --2- 172.21.15.150:0/943539024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f538815e7a0 0x7f53881f8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.499+0000 7f538e211640 1 -- 172.21.15.150:0/943539024 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f538815e320 msgr2=0x7f53881f89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.499+0000 7f538e211640 1 --2- 172.21.15.150:0/943539024 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f538815e320 0x7f53881f89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.499+0000 7f538e211640 1 -- 172.21.15.150:0/943539024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f538815e7a0 msgr2=0x7f53881f8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.499+0000 7f538e211640 1 --2- 172.21.15.150:0/943539024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f538815e7a0 0x7f53881f8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.499+0000 7f538e211640 1 -- 172.21.15.150:0/943539024 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53880ccde0 con 0x7f538815df40 2024-03-17T23:47:16.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.500+0000 7f538e211640 1 --2- 172.21.15.150:0/943539024 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f538815df40 0x7f53880c9ea0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f53780098c0 tx=0x7f53780046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:16.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.500+0000 7f537effd640 1 -- 172.21.15.150:0/943539024 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5378034040 con 0x7f538815df40 2024-03-17T23:47:16.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.500+0000 7f537effd640 1 -- 172.21.15.150:0/943539024 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5378039e50 con 0x7f538815df40 2024-03-17T23:47:16.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.500+0000 7f5390475640 1 -- 172.21.15.150:0/943539024 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53880cd0c0 con 0x7f538815df40 2024-03-17T23:47:16.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.500+0000 7f5390475640 1 -- 172.21.15.150:0/943539024 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f53881f9b00 con 0x7f538815df40 2024-03-17T23:47:16.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.501+0000 7f537effd640 1 -- 172.21.15.150:0/943539024 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5378030bd0 con 0x7f538815df40 2024-03-17T23:47:16.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.503+0000 7f5390475640 1 -- 172.21.15.150:0/943539024 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53880681a0 con 0x7f538815df40 2024-03-17T23:47:16.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.505+0000 7f91de7fc640 1 -- 172.21.15.150:0/838118530 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f91e015cf90 con 0x7f91e00cf640 2024-03-17T23:47:16.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.504+0000 7f537effd640 1 -- 172.21.15.150:0/943539024 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5378037070 con 0x7f538815df40 2024-03-17T23:47:16.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.505+0000 7f537effd640 1 --2- 172.21.15.150:0/943539024 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f536c06bc20 0x7f536c06e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.505+0000 7f537effd640 1 -- 172.21.15.150:0/943539024 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f53780ad8f0 con 0x7f538815df40 2024-03-17T23:47:16.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.499+0000 7f34bb7fe640 1 --2- 172.21.15.150:0/2355402762 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f34a806bca0 0x7f34a806e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.505+0000 7f34bb7fe640 1 --2- 172.21.15.150:0/2355402762 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f34a806bca0 0x7f34a806e160 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f34b000b7f0 tx=0x7f34b0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:16.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.506+0000 7f538da10640 1 --2- 172.21.15.150:0/943539024 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f536c06bc20 0x7f536c06e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.506+0000 7f538da10640 1 --2- 172.21.15.150:0/943539024 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f536c06bc20 0x7f536c06e0e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5388149ba0 tx=0x7f5384009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:16.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.514+0000 7f34a3fff640 1 -- 172.21.15.150:0/2355402762 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f34b407a4e0 con 0x7f34bc14eb70 2024-03-17T23:47:16.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.514+0000 7f537effd640 1 -- 172.21.15.150:0/943539024 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f537807c080 con 0x7f538815df40 2024-03-17T23:47:16.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.532+0000 7fb7f9c63640 1 Processor -- start 2024-03-17T23:47:16.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.532+0000 7fb7f9c63640 1 -- start start 2024-03-17T23:47:16.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.533+0000 7fb7f9c63640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb7ec049fa0 0x7fb7ec051840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.533+0000 7fb7f9c63640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7ec057470 0x7fb7ec051d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.533+0000 7fb7f9c63640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7ec0545b0 con 0x7fb7ec04a380 2024-03-17T23:47:16.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.533+0000 7fb7f9c63640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fb7ec0513b0 con 0x7fb7ec049fa0 2024-03-17T23:47:16.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.533+0000 7fb7f9c63640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb7ec007c50 con 0x7fb7ec057470 2024-03-17T23:47:16.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.534+0000 7fb7f2ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7ec057470 0x7fb7ec051d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.534+0000 7fb7f2ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7ec057470 0x7fb7ec051d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:55392/0 (socket says 172.21.15.150:55392) 2024-03-17T23:47:16.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.534+0000 7fb7f2ffd640 1 -- 172.21.15.150:0/3741622254 learned_addr learned my addr 172.21.15.150:0/3741622254 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:16.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.534+0000 7fb7f37fe640 1 --2- 172.21.15.150:0/3741622254 >> v2:172.21.15.157:3300/0 conn(0x7fb7ec049fa0 0x7fb7ec051840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.534+0000 7fb7f2ffd640 1 -- 172.21.15.150:0/3741622254 >> v2:172.21.15.157:3300/0 conn(0x7fb7ec049fa0 msgr2=0x7fb7ec051840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:16.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.535+0000 7fb7f2ffd640 1 --2- 172.21.15.150:0/3741622254 >> v2:172.21.15.157:3300/0 conn(0x7fb7ec049fa0 0x7fb7ec051840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.535+0000 7fb7f2ffd640 1 -- 172.21.15.150:0/3741622254 >> v1:172.21.15.157:6789/0 conn(0x7fb7ec04a380 legacy=0x7fb7ec0522c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.535+0000 7fb7f2ffd640 1 -- 172.21.15.150:0/3741622254 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7ec007f30 con 0x7fb7ec057470 2024-03-17T23:47:16.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.539+0000 7fb7f2ffd640 1 --2- 172.21.15.150:0/3741622254 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7ec057470 0x7fb7ec051d80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009870 tx=0x7fb7e802b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f07f7525bdcdd62a server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:16.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.539+0000 7fb7f27fc640 1 -- 172.21.15.150:0/3741622254 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8034070 con 0x7fb7ec057470 2024-03-17T23:47:16.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.539+0000 7fb7f27fc640 1 -- 172.21.15.150:0/3741622254 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7e8030880 con 0x7fb7ec057470 2024-03-17T23:47:16.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.540+0000 7fb7f9c63640 1 -- 172.21.15.150:0/3741622254 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7ec057470 msgr2=0x7fb7ec051d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.540+0000 7fb7f27fc640 1 -- 172.21.15.150:0/3741622254 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8034070 con 0x7fb7ec057470 2024-03-17T23:47:16.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.540+0000 7fb7f9c63640 1 --2- 172.21.15.150:0/3741622254 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7ec057470 0x7fb7ec051d80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009870 tx=0x7fb7e802b9c0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.541+0000 7fb7f9c63640 1 -- 172.21.15.150:0/3741622254 shutdown_connections 2024-03-17T23:47:16.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.541+0000 7fb7f9c63640 1 --2- 172.21.15.150:0/3741622254 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7ec057470 0x7fb7ec051d80 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009870 tx=0x7fb7e802b9c0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.541+0000 7fb7f9c63640 1 --2- 172.21.15.150:0/3741622254 >> v2:172.21.15.157:3300/0 conn(0x7fb7ec049fa0 0x7fb7ec051840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.541+0000 7fb7f9c63640 1 -- 172.21.15.150:0/3741622254 >> 172.21.15.150:0/3741622254 conn(0x7fb7ec020030 msgr2=0x7fb7ec056d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:16.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.547+0000 7fb7f9c63640 1 -- 172.21.15.150:0/3741622254 shutdown_connections 2024-03-17T23:47:16.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.547+0000 7fb7f9c63640 1 -- 172.21.15.150:0/3741622254 wait complete. 2024-03-17T23:47:16.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.549+0000 7fb7f9c63640 1 Processor -- start 2024-03-17T23:47:16.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.549+0000 7fb7f9c63640 1 -- start start 2024-03-17T23:47:16.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.550+0000 7fb7f9c63640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7ec049fa0 0x7fb7ec04c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.550+0000 7fb7f9c63640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7ec04a380 0x7fb7ec0339d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.550+0000 7fb7f9c63640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7ec035ec0 0x7fb7ec036370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.550+0000 7fb7f9c63640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ec0529d0 con 0x7fb7ec04a380 2024-03-17T23:47:16.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.550+0000 7fb7f9c63640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ec038570 con 0x7fb7ec035ec0 2024-03-17T23:47:16.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.550+0000 7fb7f9c63640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb7ec052b50 con 0x7fb7ec049fa0 2024-03-17T23:47:16.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.551+0000 7fb7f3fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7ec035ec0 0x7fb7ec036370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.551+0000 7fb7f37fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7ec049fa0 0x7fb7ec04c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.557+0000 7fb7f37fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7ec049fa0 0x7fb7ec04c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:55394/0 (socket says 172.21.15.150:55394) 2024-03-17T23:47:16.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.557+0000 7fb7f37fe640 1 -- 172.21.15.150:0/746292548 learned_addr learned my addr 172.21.15.150:0/746292548 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:16.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.557+0000 7fb7f2ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7ec04a380 0x7fb7ec0339d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.557+0000 7fb7f3fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7ec035ec0 0x7fb7ec036370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:42212/0 (socket says 172.21.15.150:42212) 2024-03-17T23:47:16.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.558+0000 7fb7f37fe640 1 -- 172.21.15.150:0/746292548 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7ec035ec0 msgr2=0x7fb7ec036370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.558+0000 7fb7f37fe640 1 --2- 172.21.15.150:0/746292548 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7ec035ec0 0x7fb7ec036370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.558+0000 7fb7f37fe640 1 -- 172.21.15.150:0/746292548 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7ec04a380 msgr2=0x7fb7ec0339d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.558+0000 7fb7f37fe640 1 --2- 172.21.15.150:0/746292548 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7ec04a380 0x7fb7ec0339d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.558+0000 7fb7f37fe640 1 -- 172.21.15.150:0/746292548 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7ec034360 con 0x7fb7ec049fa0 2024-03-17T23:47:16.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.558+0000 7fb7f3fff640 1 --2- 172.21.15.150:0/746292548 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7ec035ec0 0x7fb7ec036370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:16.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.558+0000 7fb7f2ffd640 1 --2- 172.21.15.150:0/746292548 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7ec04a380 0x7fb7ec0339d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:16.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.561+0000 7fb7f37fe640 1 --2- 172.21.15.150:0/746292548 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7ec049fa0 0x7fb7ec04c180 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400d800 tx=0x7fb7e400dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:16.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.566+0000 7fb7dbfff640 1 -- 172.21.15.150:0/746292548 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e400b6c0 con 0x7fb7ec049fa0 2024-03-17T23:47:16.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.566+0000 7fb7f9c63640 1 -- 172.21.15.150:0/746292548 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7ec0506d0 con 0x7fb7ec049fa0 2024-03-17T23:47:16.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.566+0000 7fb7f9c63640 1 -- 172.21.15.150:0/746292548 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7ec050c90 con 0x7fb7ec049fa0 2024-03-17T23:47:16.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.566+0000 7fb7dbfff640 1 -- 172.21.15.150:0/746292548 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7e4019070 con 0x7fb7ec049fa0 2024-03-17T23:47:16.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.566+0000 7fb7dbfff640 1 -- 172.21.15.150:0/746292548 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e400ba80 con 0x7fb7ec049fa0 2024-03-17T23:47:16.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.572+0000 7fb7dbfff640 1 -- 172.21.15.150:0/746292548 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb7e400bc20 con 0x7fb7ec049fa0 2024-03-17T23:47:16.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.572+0000 7fb7dbfff640 1 --2- 172.21.15.150:0/746292548 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb7e006bc20 0x7fb7e006e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:16.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.573+0000 7fb7dbfff640 1 -- 172.21.15.150:0/746292548 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fb7e4057300 con 0x7fb7ec049fa0 2024-03-17T23:47:16.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.574+0000 7fb7f9c63640 1 -- 172.21.15.150:0/746292548 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7d4005290 con 0x7fb7ec049fa0 2024-03-17T23:47:16.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.575+0000 7fb7f2ffd640 1 --2- 172.21.15.150:0/746292548 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb7e006bc20 0x7fb7e006e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:16.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.576+0000 7fb7f2ffd640 1 --2- 172.21.15.150:0/746292548 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb7e006bc20 0x7fb7e006e0e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8036000 tx=0x7fb7e803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:16.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.582+0000 7fb7dbfff640 1 -- 172.21.15.150:0/746292548 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb7e4059260 con 0x7fb7ec049fa0 2024-03-17T23:47:16.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.875+0000 7f91e80dc640 1 -- 172.21.15.150:0/838118530 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f91c0005580 con 0x7f91e00cf640 2024-03-17T23:47:16.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.876+0000 7f91de7fc640 1 -- 172.21.15.150:0/838118530 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f91d0058760 con 0x7f91e00cf640 2024-03-17T23:47:16.877 INFO:teuthology.orchestra.run.smithi150.stdout:55834574857 2024-03-17T23:47:16.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.886+0000 7f91c7fff640 1 -- 172.21.15.150:0/838118530 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f91cc06bbd0 msgr2=0x7f91cc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.886+0000 7f91c7fff640 1 --2- 172.21.15.150:0/838118530 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f91cc06bbd0 0x7f91cc06e090 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f91d8034000 tx=0x7f91d8008030 comp rx=0 tx=0).stop 2024-03-17T23:47:16.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.886+0000 7f91c7fff640 1 -- 172.21.15.150:0/838118530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91e00cf640 msgr2=0x7f91e015ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.886+0000 7f91c7fff640 1 --2- 172.21.15.150:0/838118530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91e00cf640 0x7f91e015ca50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f91d000cb20 tx=0x7f91d0002170 comp rx=0 tx=0).stop 2024-03-17T23:47:16.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.895+0000 7f91c7fff640 1 -- 172.21.15.150:0/838118530 shutdown_connections 2024-03-17T23:47:16.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.895+0000 7f91c7fff640 1 --2- 172.21.15.150:0/838118530 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f91e0152160 0x7f91e0159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.895+0000 7f91c7fff640 1 --2- 172.21.15.150:0/838118530 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91e00cf640 0x7f91e015ca50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.895+0000 7f91c7fff640 1 --2- 172.21.15.150:0/838118530 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f91cc06bbd0 0x7f91cc06e090 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.895+0000 7f91c7fff640 1 --2- 172.21.15.150:0/838118530 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91e006dd30 0x7f91e01577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.895+0000 7f91c7fff640 1 -- 172.21.15.150:0/838118530 >> 172.21.15.150:0/838118530 conn(0x7f91e006b190 msgr2=0x7f91e00af7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:16.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.905+0000 7f91c7fff640 1 -- 172.21.15.150:0/838118530 shutdown_connections 2024-03-17T23:47:16.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.906+0000 7f91c7fff640 1 -- 172.21.15.150:0/838118530 wait complete. 2024-03-17T23:47:16.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.907+0000 7f4ccc91a640 1 -- 172.21.15.150:0/2110063973 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f4ca8005850 con 0x7f4cc81514b0 2024-03-17T23:47:16.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.908+0000 7f4caf7fe640 1 -- 172.21.15.150:0/2110063973 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4cb807a120 con 0x7f4cc81514b0 2024-03-17T23:47:16.909 INFO:teuthology.orchestra.run.smithi150.stdout:60129542154 2024-03-17T23:47:16.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.921+0000 7f2d9561c640 1 -- 172.21.15.150:0/3506457757 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f2d9006f8d0 con 0x7f2d90161fc0 2024-03-17T23:47:16.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.921+0000 7f4ccc91a640 1 -- 172.21.15.150:0/2110063973 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4cb406bbd0 msgr2=0x7f4cb406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.921+0000 7f4ccc91a640 1 --2- 172.21.15.150:0/2110063973 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4cb406bbd0 0x7f4cb406e090 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4cc81f6d70 tx=0x7f4cbc009190 comp rx=0 tx=0).stop 2024-03-17T23:47:16.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.922+0000 7f4ccc91a640 1 -- 172.21.15.150:0/2110063973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cc81514b0 msgr2=0x7f4cc814d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.922+0000 7f4ccc91a640 1 --2- 172.21.15.150:0/2110063973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cc81514b0 0x7f4cc814d130 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8032000 tx=0x7f4cb80046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.925 INFO:teuthology.orchestra.run.smithi150.stdout:55834574857 2024-03-17T23:47:16.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.922+0000 7f2d8cff9640 1 -- 172.21.15.150:0/3506457757 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2d88059570 con 0x7f2d90161fc0 2024-03-17T23:47:16.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.922+0000 7f4ccc91a640 1 -- 172.21.15.150:0/2110063973 shutdown_connections 2024-03-17T23:47:16.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.922+0000 7f4ccc91a640 1 --2- 172.21.15.150:0/2110063973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cc81514b0 0x7f4cc814d130 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.922+0000 7f4ccc91a640 1 --2- 172.21.15.150:0/2110063973 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4cb406bbd0 0x7f4cb406e090 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.922+0000 7f4ccc91a640 1 --2- 172.21.15.150:0/2110063973 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4cc81632f0 0x7f4cc81f55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.922+0000 7f4ccc91a640 1 --2- 172.21.15.150:0/2110063973 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4cc80cf320 0x7f4cc815b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.922+0000 7f4ccc91a640 1 -- 172.21.15.150:0/2110063973 >> 172.21.15.150:0/2110063973 conn(0x7f4cc80b1600 msgr2=0x7f4cc8161070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:16.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.923+0000 7f4ccc91a640 1 -- 172.21.15.150:0/2110063973 shutdown_connections 2024-03-17T23:47:16.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.923+0000 7f4ccc91a640 1 -- 172.21.15.150:0/2110063973 wait complete. 2024-03-17T23:47:16.927 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.5 2024-03-17T23:47:16.927 DEBUG:teuthology.parallel:result is None 2024-03-17T23:47:16.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.925+0000 7f2d9561c640 1 -- 172.21.15.150:0/3506457757 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2d7006bca0 msgr2=0x7f2d7006e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.926+0000 7f2d9561c640 1 --2- 172.21.15.150:0/3506457757 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2d7006bca0 0x7f2d7006e160 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2d78035000 tx=0x7f2d78030000 comp rx=0 tx=0).stop 2024-03-17T23:47:16.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.926+0000 7f2d9561c640 1 -- 172.21.15.150:0/3506457757 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d90161fc0 msgr2=0x7f2d90160980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.926+0000 7f2d9561c640 1 --2- 172.21.15.150:0/3506457757 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d90161fc0 0x7f2d90160980 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2d8800eed0 tx=0x7f2d8800b480 comp rx=0 tx=0).stop 2024-03-17T23:47:16.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.926+0000 7f2d9561c640 1 -- 172.21.15.150:0/3506457757 shutdown_connections 2024-03-17T23:47:16.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.926+0000 7f2d9561c640 1 --2- 172.21.15.150:0/3506457757 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d90161fc0 0x7f2d90160980 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.926+0000 7f2d9561c640 1 --2- 172.21.15.150:0/3506457757 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2d7006bca0 0x7f2d7006e160 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.926+0000 7f2d9561c640 1 --2- 172.21.15.150:0/3506457757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d9015eff0 0x7f2d90160440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.926+0000 7f2d9561c640 1 --2- 172.21.15.150:0/3506457757 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d901443a0 0x7f2d902006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.926+0000 7f2d9561c640 1 -- 172.21.15.150:0/3506457757 >> 172.21.15.150:0/3506457757 conn(0x7f2d900b1600 msgr2=0x7f2d90073da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:16.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.927+0000 7f2d9561c640 1 -- 172.21.15.150:0/3506457757 shutdown_connections 2024-03-17T23:47:16.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.927+0000 7f2d9561c640 1 -- 172.21.15.150:0/3506457757 wait complete. 2024-03-17T23:47:16.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.928+0000 7f85d17fa640 1 -- 172.21.15.150:0/240956365 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f85e40057d0 con 0x7f85e403b0a0 2024-03-17T23:47:16.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.928+0000 7f85d37fe640 1 -- 172.21.15.150:0/240956365 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f85dc01f030 con 0x7f85e403b0a0 2024-03-17T23:47:16.930 INFO:teuthology.orchestra.run.smithi150.stdout:55834574857 2024-03-17T23:47:16.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.932+0000 7f85ebfff640 1 -- 172.21.15.150:0/240956365 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f85d806bca0 msgr2=0x7f85d806e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.932+0000 7f85ebfff640 1 --2- 172.21.15.150:0/240956365 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f85d806bca0 0x7f85d806e160 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f85e4180e90 tx=0x7f85e0009110 comp rx=0 tx=0).stop 2024-03-17T23:47:16.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.932+0000 7f85ebfff640 1 -- 172.21.15.150:0/240956365 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f85e403b0a0 msgr2=0x7f85e404e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.932+0000 7f85ebfff640 1 --2- 172.21.15.150:0/240956365 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f85e403b0a0 0x7f85e404e7b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f85dc00b3e0 tx=0x7f85dc00b7e0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.932+0000 7f85ebfff640 1 -- 172.21.15.150:0/240956365 shutdown_connections 2024-03-17T23:47:16.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.932+0000 7f85ebfff640 1 --2- 172.21.15.150:0/240956365 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f85e40eac60 0x7f85e40dd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.932+0000 7f85ebfff640 1 --2- 172.21.15.150:0/240956365 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85e4048ac0 0x7f85e40dd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.932+0000 7f85ebfff640 1 --2- 172.21.15.150:0/240956365 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f85d806bca0 0x7f85d806e160 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.932+0000 7f85ebfff640 1 --2- 172.21.15.150:0/240956365 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f85e403b0a0 0x7f85e404e7b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.933+0000 7f85ebfff640 1 -- 172.21.15.150:0/240956365 >> 172.21.15.150:0/240956365 conn(0x7f85e401e6b0 msgr2=0x7f85e40e9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:16.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.933+0000 7f85ebfff640 1 -- 172.21.15.150:0/240956365 shutdown_connections 2024-03-17T23:47:16.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.933+0000 7f85ebfff640 1 -- 172.21.15.150:0/240956365 wait complete. 2024-03-17T23:47:16.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.938+0000 7f34a1ffb640 1 -- 172.21.15.150:0/2355402762 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f3494005580 con 0x7f34bc14eb70 2024-03-17T23:47:16.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.939+0000 7f34a3fff640 1 -- 172.21.15.150:0/2355402762 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f34b407e580 con 0x7f34bc14eb70 2024-03-17T23:47:16.940 INFO:teuthology.orchestra.run.smithi150.stdout:55834574858 2024-03-17T23:47:16.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.943+0000 7f34a1ffb640 1 -- 172.21.15.150:0/2355402762 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f34a806bca0 msgr2=0x7f34a806e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.943+0000 7f34a1ffb640 1 --2- 172.21.15.150:0/2355402762 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f34a806bca0 0x7f34a806e160 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f34b000b7f0 tx=0x7f34b0008040 comp rx=0 tx=0).stop 2024-03-17T23:47:16.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.943+0000 7f34a1ffb640 1 -- 172.21.15.150:0/2355402762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34bc14eb70 msgr2=0x7f34bc158f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.943+0000 7f34a1ffb640 1 --2- 172.21.15.150:0/2355402762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34bc14eb70 0x7f34bc158f00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f34b403b040 tx=0x7f34b4004a40 comp rx=0 tx=0).stop 2024-03-17T23:47:16.944 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542154 for osd.2 2024-03-17T23:47:16.945 DEBUG:teuthology.parallel:result is None 2024-03-17T23:47:16.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.943+0000 7f34a1ffb640 1 -- 172.21.15.150:0/2355402762 shutdown_connections 2024-03-17T23:47:16.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.943+0000 7f34a1ffb640 1 --2- 172.21.15.150:0/2355402762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34bc14eb70 0x7f34bc158f00 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.943+0000 7f34a1ffb640 1 --2- 172.21.15.150:0/2355402762 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34bc15ae40 0x7f34bc1f7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.943+0000 7f34a1ffb640 1 --2- 172.21.15.150:0/2355402762 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f34a806bca0 0x7f34a806e160 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.943+0000 7f34a1ffb640 1 --2- 172.21.15.150:0/2355402762 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34bc0cf2a0 0x7f34bc151b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.943+0000 7f34a1ffb640 1 -- 172.21.15.150:0/2355402762 >> 172.21.15.150:0/2355402762 conn(0x7f34bc099a70 msgr2=0x7f34bc0ca430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:16.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.943+0000 7f34a1ffb640 1 -- 172.21.15.150:0/2355402762 shutdown_connections 2024-03-17T23:47:16.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.943+0000 7f34a1ffb640 1 -- 172.21.15.150:0/2355402762 wait complete. 2024-03-17T23:47:16.947 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.4 2024-03-17T23:47:16.947 DEBUG:teuthology.parallel:result is None 2024-03-17T23:47:16.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.948+0000 7f77c0777640 1 -- 172.21.15.150:0/246728261 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f77b80b5530 con 0x7f77b80670b0 2024-03-17T23:47:16.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.948+0000 7f77aeffd640 1 -- 172.21.15.150:0/246728261 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f77b006f070 con 0x7f77b80670b0 2024-03-17T23:47:16.949 INFO:teuthology.orchestra.run.smithi150.stdout:60129542154 2024-03-17T23:47:16.952 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.6 2024-03-17T23:47:16.952 DEBUG:teuthology.parallel:result is None 2024-03-17T23:47:16.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.951+0000 7f77c0777640 1 -- 172.21.15.150:0/246728261 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f779c06bdd0 msgr2=0x7f779c06e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.951+0000 7f77c0777640 1 --2- 172.21.15.150:0/246728261 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f779c06bdd0 0x7f779c06e290 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f77b8158920 tx=0x7f77a8009460 comp rx=0 tx=0).stop 2024-03-17T23:47:16.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.951+0000 7f77c0777640 1 -- 172.21.15.150:0/246728261 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77b80670b0 msgr2=0x7f77b8155740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:16.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.951+0000 7f77c0777640 1 --2- 172.21.15.150:0/246728261 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77b80670b0 0x7f77b8155740 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f77b0058040 tx=0x7f77b0004a40 comp rx=0 tx=0).stop 2024-03-17T23:47:16.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.951+0000 7f77c0777640 1 -- 172.21.15.150:0/246728261 shutdown_connections 2024-03-17T23:47:16.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.951+0000 7f77c0777640 1 --2- 172.21.15.150:0/246728261 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f779c06bdd0 0x7f779c06e290 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.951+0000 7f77c0777640 1 --2- 172.21.15.150:0/246728261 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77b8162530 0x7f77b81616e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.951+0000 7f77c0777640 1 --2- 172.21.15.150:0/246728261 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77b80ccfb0 0x7f77b8151b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.951+0000 7f77c0777640 1 --2- 172.21.15.150:0/246728261 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77b80670b0 0x7f77b8155740 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:16.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.951+0000 7f77c0777640 1 -- 172.21.15.150:0/246728261 >> 172.21.15.150:0/246728261 conn(0x7f77b8099a70 msgr2=0x7f77b806bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:16.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.951+0000 7f77c0777640 1 -- 172.21.15.150:0/246728261 shutdown_connections 2024-03-17T23:47:16.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:16.951+0000 7f77c0777640 1 -- 172.21.15.150:0/246728261 wait complete. 2024-03-17T23:47:16.957 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.1 2024-03-17T23:47:16.957 DEBUG:teuthology.parallel:result is None 2024-03-17T23:47:16.967 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542154 for osd.3 2024-03-17T23:47:16.967 DEBUG:teuthology.parallel:result is None 2024-03-17T23:47:17.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.013+0000 7f5390475640 1 -- 172.21.15.150:0/943539024 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f53881607f0 con 0x7f538815df40 2024-03-17T23:47:17.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.014+0000 7f537effd640 1 -- 172.21.15.150:0/943539024 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f537807b980 con 0x7f538815df40 2024-03-17T23:47:17.015 INFO:teuthology.orchestra.run.smithi150.stdout:60129542154 2024-03-17T23:47:17.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.016+0000 7f5390475640 1 -- 172.21.15.150:0/943539024 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f536c06bc20 msgr2=0x7f536c06e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:17.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.016+0000 7f5390475640 1 --2- 172.21.15.150:0/943539024 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f536c06bc20 0x7f536c06e0e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5388149ba0 tx=0x7f5384009040 comp rx=0 tx=0).stop 2024-03-17T23:47:17.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.016+0000 7f5390475640 1 -- 172.21.15.150:0/943539024 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f538815df40 msgr2=0x7f53880c9ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:17.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.016+0000 7f5390475640 1 --2- 172.21.15.150:0/943539024 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f538815df40 0x7f53880c9ea0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f53780098c0 tx=0x7f53780046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.016+0000 7f5390475640 1 -- 172.21.15.150:0/943539024 shutdown_connections 2024-03-17T23:47:17.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.016+0000 7f5390475640 1 --2- 172.21.15.150:0/943539024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f538815e7a0 0x7f53881f8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.016+0000 7f5390475640 1 --2- 172.21.15.150:0/943539024 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f536c06bc20 0x7f536c06e0e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.016+0000 7f5390475640 1 --2- 172.21.15.150:0/943539024 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f538815e320 0x7f53881f89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.016+0000 7f5390475640 1 --2- 172.21.15.150:0/943539024 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f538815df40 0x7f53880c9ea0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.016+0000 7f5390475640 1 -- 172.21.15.150:0/943539024 >> 172.21.15.150:0/943539024 conn(0x7f5388072ef0 msgr2=0x7f538806c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:17.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.016+0000 7f5390475640 1 -- 172.21.15.150:0/943539024 shutdown_connections 2024-03-17T23:47:17.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.016+0000 7f5390475640 1 -- 172.21.15.150:0/943539024 wait complete. 2024-03-17T23:47:17.027 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542154 for osd.0 2024-03-17T23:47:17.027 DEBUG:teuthology.parallel:result is None 2024-03-17T23:47:17.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.061+0000 7fb7f9c63640 1 -- 172.21.15.150:0/746292548 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fb7d4005850 con 0x7fb7ec049fa0 2024-03-17T23:47:17.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.061+0000 7fb7dbfff640 1 -- 172.21.15.150:0/746292548 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb7e405d300 con 0x7fb7ec049fa0 2024-03-17T23:47:17.062 INFO:teuthology.orchestra.run.smithi150.stdout:55834574857 2024-03-17T23:47:17.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.063+0000 7fb7f9c63640 1 -- 172.21.15.150:0/746292548 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb7e006bc20 msgr2=0x7fb7e006e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:17.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.063+0000 7fb7f9c63640 1 --2- 172.21.15.150:0/746292548 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb7e006bc20 0x7fb7e006e0e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8036000 tx=0x7fb7e803c040 comp rx=0 tx=0).stop 2024-03-17T23:47:17.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.063+0000 7fb7f9c63640 1 -- 172.21.15.150:0/746292548 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7ec049fa0 msgr2=0x7fb7ec04c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:17.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.063+0000 7fb7f9c63640 1 --2- 172.21.15.150:0/746292548 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7ec049fa0 0x7fb7ec04c180 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400d800 tx=0x7fb7e400dcc0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.063+0000 7fb7f9c63640 1 -- 172.21.15.150:0/746292548 shutdown_connections 2024-03-17T23:47:17.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.063+0000 7fb7f9c63640 1 --2- 172.21.15.150:0/746292548 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb7e006bc20 0x7fb7e006e0e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.063+0000 7fb7f9c63640 1 --2- 172.21.15.150:0/746292548 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7ec035ec0 0x7fb7ec036370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.063+0000 7fb7f9c63640 1 --2- 172.21.15.150:0/746292548 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7ec04a380 0x7fb7ec0339d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.063+0000 7fb7f9c63640 1 --2- 172.21.15.150:0/746292548 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb7ec049fa0 0x7fb7ec04c180 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.063+0000 7fb7f9c63640 1 -- 172.21.15.150:0/746292548 >> 172.21.15.150:0/746292548 conn(0x7fb7ec020030 msgr2=0x7fb7ec056280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:17.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.063+0000 7fb7f9c63640 1 -- 172.21.15.150:0/746292548 shutdown_connections 2024-03-17T23:47:17.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.063+0000 7fb7f9c63640 1 -- 172.21.15.150:0/746292548 wait complete. 2024-03-17T23:47:17.074 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.7 2024-03-17T23:47:17.075 DEBUG:teuthology.parallel:result is None 2024-03-17T23:47:17.075 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-17T23:47:17.075 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-17T23:47:17.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.216+0000 7f0369cd1640 1 Processor -- start 2024-03-17T23:47:17.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.216+0000 7f0369cd1640 1 -- start start 2024-03-17T23:47:17.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.217+0000 7f0369cd1640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f03641514f0 0x7f036415bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:17.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.217+0000 7f0369cd1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f036415ab90 0x7f036406f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:17.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.217+0000 7f0369cd1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03640ce790 con 0x7f036406e5c0 2024-03-17T23:47:17.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.217+0000 7f0369cd1640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f036406fba0 con 0x7f03641514f0 2024-03-17T23:47:17.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.217+0000 7f0369cd1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f036415b050 con 0x7f036415ab90 2024-03-17T23:47:17.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.217+0000 7f0363fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f036406e5c0 0x7f036406ea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34726/0 (socket says 172.21.15.150:34726) 2024-03-17T23:47:17.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.217+0000 7f0363fff640 1 -- 172.21.15.150:0/2024801581 learned_addr learned my addr 172.21.15.150:0/2024801581 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:17.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.217+0000 7f0362ffd640 1 --2- 172.21.15.150:0/2024801581 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f036415ab90 0x7f036406f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:17.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.217+0000 7f03637fe640 1 --2- 172.21.15.150:0/2024801581 >> v2:172.21.15.157:3300/0 conn(0x7f03641514f0 0x7f036415bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:17.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.217+0000 7f03627fc640 1 -- 172.21.15.150:0/2024801581 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 304500218 0 0) 0x7f035c001490 con 0x7f036406e5c0 2024-03-17T23:47:17.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.217+0000 7f03627fc640 1 -- 172.21.15.150:0/2024801581 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f034c003520 con 0x7f036406e5c0 2024-03-17T23:47:17.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.217+0000 7f0362ffd640 1 -- 172.21.15.150:0/2024801581 >> v2:172.21.15.157:3300/0 conn(0x7f03641514f0 msgr2=0x7f036415bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:17.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.217+0000 7f0362ffd640 1 --2- 172.21.15.150:0/2024801581 >> v2:172.21.15.157:3300/0 conn(0x7f03641514f0 0x7f036415bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.217+0000 7f0362ffd640 1 -- 172.21.15.150:0/2024801581 >> v1:172.21.15.150:6789/0 conn(0x7f036406e5c0 legacy=0x7f036406ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:17.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.217+0000 7f0362ffd640 1 -- 172.21.15.150:0/2024801581 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f036415b330 con 0x7f036415ab90 2024-03-17T23:47:17.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.217+0000 7f03637fe640 1 --2- 172.21.15.150:0/2024801581 >> v2:172.21.15.157:3300/0 conn(0x7f03641514f0 0x7f036415bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:17.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.218+0000 7f0362ffd640 1 --2- 172.21.15.150:0/2024801581 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f036415ab90 0x7f036406f660 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f03540097a0 tx=0x7f035402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe87b1a58324e18f server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:17.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.218+0000 7f03627fc640 1 -- 172.21.15.150:0/2024801581 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0354035020 con 0x7f036415ab90 2024-03-17T23:47:17.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.218+0000 7f03627fc640 1 -- 172.21.15.150:0/2024801581 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0354031830 con 0x7f036415ab90 2024-03-17T23:47:17.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.218+0000 7f03627fc640 1 -- 172.21.15.150:0/2024801581 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0354031be0 con 0x7f036415ab90 2024-03-17T23:47:17.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.219+0000 7f0369cd1640 1 -- 172.21.15.150:0/2024801581 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f036415ab90 msgr2=0x7f036406f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:17.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.219+0000 7f0369cd1640 1 --2- 172.21.15.150:0/2024801581 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f036415ab90 0x7f036406f660 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f03540097a0 tx=0x7f035402b8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.219+0000 7f0369cd1640 1 -- 172.21.15.150:0/2024801581 shutdown_connections 2024-03-17T23:47:17.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.219+0000 7f0369cd1640 1 --2- 172.21.15.150:0/2024801581 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f036415ab90 0x7f036406f660 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.219+0000 7f0369cd1640 1 --2- 172.21.15.150:0/2024801581 >> v2:172.21.15.157:3300/0 conn(0x7f03641514f0 0x7f036415bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.219+0000 7f0369cd1640 1 -- 172.21.15.150:0/2024801581 >> 172.21.15.150:0/2024801581 conn(0x7f036406b190 msgr2=0x7f036406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:17.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.219+0000 7f0369cd1640 1 -- 172.21.15.150:0/2024801581 shutdown_connections 2024-03-17T23:47:17.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.219+0000 7f0369cd1640 1 -- 172.21.15.150:0/2024801581 wait complete. 2024-03-17T23:47:17.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.220+0000 7f0369cd1640 1 Processor -- start 2024-03-17T23:47:17.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.220+0000 7f0369cd1640 1 -- start start 2024-03-17T23:47:17.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.221+0000 7f0369cd1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f036406e5c0 0x7f03641548a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:17.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.221+0000 7f0369cd1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03641514f0 0x7f03641f36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:17.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.221+0000 7f0369cd1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f036415ab90 0x7f036414b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:17.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.221+0000 7f0369cd1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f03640af510 con 0x7f03641514f0 2024-03-17T23:47:17.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.221+0000 7f0369cd1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f036415b8f0 con 0x7f036406e5c0 2024-03-17T23:47:17.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.221+0000 7f0369cd1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f03641518d0 con 0x7f036415ab90 2024-03-17T23:47:17.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.221+0000 7f0362ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03641514f0 0x7f03641f36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:17.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.221+0000 7f0362ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03641514f0 0x7f03641f36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40354/0 (socket says 172.21.15.150:40354) 2024-03-17T23:47:17.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.221+0000 7f0362ffd640 1 -- 172.21.15.150:0/903143933 learned_addr learned my addr 172.21.15.150:0/903143933 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:17.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.221+0000 7f0363fff640 1 --2- 172.21.15.150:0/903143933 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f036415ab90 0x7f036414b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:17.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.221+0000 7f03637fe640 1 --2- 172.21.15.150:0/903143933 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f036406e5c0 0x7f03641548a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:17.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.221+0000 7f0363fff640 1 -- 172.21.15.150:0/903143933 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f036406e5c0 msgr2=0x7f03641548a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:17.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.221+0000 7f0363fff640 1 --2- 172.21.15.150:0/903143933 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f036406e5c0 0x7f03641548a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.221+0000 7f0363fff640 1 -- 172.21.15.150:0/903143933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03641514f0 msgr2=0x7f03641f36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:17.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.221+0000 7f0363fff640 1 --2- 172.21.15.150:0/903143933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03641514f0 0x7f03641f36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.221+0000 7f0363fff640 1 -- 172.21.15.150:0/903143933 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03641521c0 con 0x7f036415ab90 2024-03-17T23:47:17.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.221+0000 7f0362ffd640 1 --2- 172.21.15.150:0/903143933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03641514f0 0x7f03641f36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:17.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.221+0000 7f03637fe640 1 --2- 172.21.15.150:0/903143933 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f036406e5c0 0x7f03641548a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:17.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.221+0000 7f0363fff640 1 --2- 172.21.15.150:0/903143933 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f036415ab90 0x7f036414b740 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f035c00caa0 tx=0x7f035c00cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:17.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.221+0000 7f034bfff640 1 -- 172.21.15.150:0/903143933 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f035c017070 con 0x7f036415ab90 2024-03-17T23:47:17.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.222+0000 7f034bfff640 1 -- 172.21.15.150:0/903143933 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f035c00ac00 con 0x7f036415ab90 2024-03-17T23:47:17.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.222+0000 7f0369cd1640 1 -- 172.21.15.150:0/903143933 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03641524a0 con 0x7f036415ab90 2024-03-17T23:47:17.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.222+0000 7f034bfff640 1 -- 172.21.15.150:0/903143933 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f035c012400 con 0x7f036415ab90 2024-03-17T23:47:17.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.222+0000 7f0369cd1640 1 -- 172.21.15.150:0/903143933 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0364152d40 con 0x7f036415ab90 2024-03-17T23:47:17.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.222+0000 7f0369cd1640 1 -- 172.21.15.150:0/903143933 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0364152890 con 0x7f036415ab90 2024-03-17T23:47:17.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.223+0000 7f034bfff640 1 -- 172.21.15.150:0/903143933 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f035c01a070 con 0x7f036415ab90 2024-03-17T23:47:17.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.223+0000 7f034bfff640 1 --2- 172.21.15.150:0/903143933 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f035006bbd0 0x7f035006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:17.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.223+0000 7f034bfff640 1 -- 172.21.15.150:0/903143933 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f035c08abc0 con 0x7f036415ab90 2024-03-17T23:47:17.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.223+0000 7f03637fe640 1 --2- 172.21.15.150:0/903143933 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f035006bbd0 0x7f035006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:17.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.224+0000 7f03637fe640 1 --2- 172.21.15.150:0/903143933 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f035006bbd0 0x7f035006e090 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0358009560 tx=0x7f0358009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:17.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.228+0000 7f034bfff640 1 -- 172.21.15.150:0/903143933 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f03640ce790 con 0x7f036415ab90 2024-03-17T23:47:17.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.396+0000 7f0369cd1640 1 -- 172.21.15.150:0/903143933 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f03640681a0 con 0x7f035006bbd0 2024-03-17T23:47:17.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.404+0000 7f034bfff640 1 -- 172.21.15.150:0/903143933 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7f03640681a0 con 0x7f035006bbd0 2024-03-17T23:47:17.409 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:17.410 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-03-17T23:47:17.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.408+0000 7f0369cd1640 1 -- 172.21.15.150:0/903143933 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f035006bbd0 msgr2=0x7f035006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:17.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.408+0000 7f0369cd1640 1 --2- 172.21.15.150:0/903143933 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f035006bbd0 0x7f035006e090 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0358009560 tx=0x7f0358009190 comp rx=0 tx=0).stop 2024-03-17T23:47:17.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.408+0000 7f0369cd1640 1 -- 172.21.15.150:0/903143933 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f036415ab90 msgr2=0x7f036414b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:17.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.408+0000 7f0369cd1640 1 --2- 172.21.15.150:0/903143933 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f036415ab90 0x7f036414b740 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f035c00caa0 tx=0x7f035c00cf60 comp rx=0 tx=0).stop 2024-03-17T23:47:17.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.408+0000 7f0369cd1640 1 -- 172.21.15.150:0/903143933 shutdown_connections 2024-03-17T23:47:17.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.408+0000 7f0369cd1640 1 --2- 172.21.15.150:0/903143933 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f036415ab90 0x7f036414b740 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.408+0000 7f0369cd1640 1 --2- 172.21.15.150:0/903143933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03641514f0 0x7f03641f36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.408+0000 7f0369cd1640 1 --2- 172.21.15.150:0/903143933 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f035006bbd0 0x7f035006e090 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.408+0000 7f0369cd1640 1 --2- 172.21.15.150:0/903143933 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f036406e5c0 0x7f03641548a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.408+0000 7f0369cd1640 1 -- 172.21.15.150:0/903143933 >> 172.21.15.150:0/903143933 conn(0x7f036406b190 msgr2=0x7f03640af960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:17.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.408+0000 7f0369cd1640 1 -- 172.21.15.150:0/903143933 shutdown_connections 2024-03-17T23:47:17.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.408+0000 7f0369cd1640 1 -- 172.21.15.150:0/903143933 wait complete. 2024-03-17T23:47:17.419 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-03-17T23:47:15.600162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":232000,"kb_used_data":3368,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754742720,"statfs":{"total":773094113280,"available":772856545280,"internally_reserved":0,"allocated":3448832,"data_stored":2079808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002654"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713953+0000","last_change":"2024-03-17T23:46:54.923706+0000","last_active":"2024-03-17T23:47:02.713953+0000","last_peered":"2024-03-17T23:47:02.713953+0000","last_clean":"2024-03-17T23:47:02.713953+0000","last_became_active":"2024-03-17T23:46:54.923462+0000","last_became_peered":"2024-03-17T23:46:54.923462+0000","last_unstale":"2024-03-17T23:47:02.713953+0000","last_undegraded":"2024-03-17T23:47:02.713953+0000","last_fullsized":"2024-03-17T23:47:02.713953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:34:48.054365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943580+0000","last_change":"2024-03-17T23:46:55.932962+0000","last_active":"2024-03-17T23:47:01.943580+0000","last_peered":"2024-03-17T23:47:01.943580+0000","last_clean":"2024-03-17T23:47:01.943580+0000","last_became_active":"2024-03-17T23:46:55.932316+0000","last_became_peered":"2024-03-17T23:46:55.932316+0000","last_unstale":"2024-03-17T23:47:01.943580+0000","last_undegraded":"2024-03-17T23:47:01.943580+0000","last_fullsized":"2024-03-17T23:47:01.943580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:45:20.327436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943046+0000","last_change":"2024-03-17T23:47:00.412319+0000","last_active":"2024-03-17T23:47:01.943046+0000","last_peered":"2024-03-17T23:47:01.943046+0000","last_clean":"2024-03-17T23:47:01.943046+0000","last_became_active":"2024-03-17T23:47:00.411544+0000","last_became_peered":"2024-03-17T23:47:00.411544+0000","last_unstale":"2024-03-17T23:47:01.943046+0000","last_undegraded":"2024-03-17T23:47:01.943046+0000","last_fullsized":"2024-03-17T23:47:01.943046+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:24:18.544213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943689+0000","last_change":"2024-03-17T23:46:54.921077+0000","last_active":"2024-03-17T23:47:01.943689+0000","last_peered":"2024-03-17T23:47:01.943689+0000","last_clean":"2024-03-17T23:47:01.943689+0000","last_became_active":"2024-03-17T23:46:54.920797+0000","last_became_peered":"2024-03-17T23:46:54.920797+0000","last_unstale":"2024-03-17T23:47:01.943689+0000","last_undegraded":"2024-03-17T23:47:01.943689+0000","last_fullsized":"2024-03-17T23:47:01.943689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:18.195935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713708+0000","last_change":"2024-03-17T23:46:55.943217+0000","last_active":"2024-03-17T23:47:02.713708+0000","last_peered":"2024-03-17T23:47:02.713708+0000","last_clean":"2024-03-17T23:47:02.713708+0000","last_became_active":"2024-03-17T23:46:55.942641+0000","last_became_peered":"2024-03-17T23:46:55.942641+0000","last_unstale":"2024-03-17T23:47:02.713708+0000","last_undegraded":"2024-03-17T23:47:02.713708+0000","last_fullsized":"2024-03-17T23:47:02.713708+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:45:20.327436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106123+0000","last_change":"2024-03-17T23:46:59.957747+0000","last_active":"2024-03-17T23:47:02.106123+0000","last_peered":"2024-03-17T23:47:02.106123+0000","last_clean":"2024-03-17T23:47:02.106123+0000","last_became_active":"2024-03-17T23:46:59.957329+0000","last_became_peered":"2024-03-17T23:46:59.957329+0000","last_unstale":"2024-03-17T23:47:02.106123+0000","last_undegraded":"2024-03-17T23:47:02.106123+0000","last_fullsized":"2024-03-17T23:47:02.106123+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:27:51.964285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713914+0000","last_change":"2024-03-17T23:46:54.926580+0000","last_active":"2024-03-17T23:47:02.713914+0000","last_peered":"2024-03-17T23:47:02.713914+0000","last_clean":"2024-03-17T23:47:02.713914+0000","last_became_active":"2024-03-17T23:46:54.926254+0000","last_became_peered":"2024-03-17T23:46:54.926254+0000","last_unstale":"2024-03-17T23:47:02.713914+0000","last_undegraded":"2024-03-17T23:47:02.713914+0000","last_fullsized":"2024-03-17T23:47:02.713914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:11:01.353944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943965+0000","last_change":"2024-03-17T23:46:55.937073+0000","last_active":"2024-03-17T23:47:01.943965+0000","last_peered":"2024-03-17T23:47:01.943965+0000","last_clean":"2024-03-17T23:47:01.943965+0000","last_became_active":"2024-03-17T23:46:55.936068+0000","last_became_peered":"2024-03-17T23:46:55.936068+0000","last_unstale":"2024-03-17T23:47:01.943965+0000","last_undegraded":"2024-03-17T23:47:01.943965+0000","last_fullsized":"2024-03-17T23:47:01.943965+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:45:29.447600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106472+0000","last_change":"2024-03-17T23:46:59.957945+0000","last_active":"2024-03-17T23:47:02.106472+0000","last_peered":"2024-03-17T23:47:02.106472+0000","last_clean":"2024-03-17T23:47:02.106472+0000","last_became_active":"2024-03-17T23:46:59.957324+0000","last_became_peered":"2024-03-17T23:46:59.957324+0000","last_unstale":"2024-03-17T23:47:02.106472+0000","last_undegraded":"2024-03-17T23:47:02.106472+0000","last_fullsized":"2024-03-17T23:47:02.106472+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:52:17.152097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713691+0000","last_change":"2024-03-17T23:46:54.924488+0000","last_active":"2024-03-17T23:47:02.713691+0000","last_peered":"2024-03-17T23:47:02.713691+0000","last_clean":"2024-03-17T23:47:02.713691+0000","last_became_active":"2024-03-17T23:46:54.924170+0000","last_became_peered":"2024-03-17T23:46:54.924170+0000","last_unstale":"2024-03-17T23:47:02.713691+0000","last_undegraded":"2024-03-17T23:47:02.713691+0000","last_fullsized":"2024-03-17T23:47:02.713691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:06:54.433520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944190+0000","last_change":"2024-03-17T23:46:55.938268+0000","last_active":"2024-03-17T23:47:01.944190+0000","last_peered":"2024-03-17T23:47:01.944190+0000","last_clean":"2024-03-17T23:47:01.944190+0000","last_became_active":"2024-03-17T23:46:55.937957+0000","last_became_peered":"2024-03-17T23:46:55.937957+0000","last_unstale":"2024-03-17T23:47:01.944190+0000","last_undegraded":"2024-03-17T23:47:01.944190+0000","last_fullsized":"2024-03-17T23:47:01.944190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T23:52:25.543746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106183+0000","last_change":"2024-03-17T23:46:59.959993+0000","last_active":"2024-03-17T23:47:02.106183+0000","last_peered":"2024-03-17T23:47:02.106183+0000","last_clean":"2024-03-17T23:47:02.106183+0000","last_became_active":"2024-03-17T23:46:59.959636+0000","last_became_peered":"2024-03-17T23:46:59.959636+0000","last_unstale":"2024-03-17T23:47:02.106183+0000","last_undegraded":"2024-03-17T23:47:02.106183+0000","last_fullsized":"2024-03-17T23:47:02.106183+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:39:36.583214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943413+0000","last_change":"2024-03-17T23:46:55.917068+0000","last_active":"2024-03-17T23:47:01.943413+0000","last_peered":"2024-03-17T23:47:01.943413+0000","last_clean":"2024-03-17T23:47:01.943413+0000","last_became_active":"2024-03-17T23:46:55.916759+0000","last_became_peered":"2024-03-17T23:46:55.916759+0000","last_unstale":"2024-03-17T23:47:01.943413+0000","last_undegraded":"2024-03-17T23:47:01.943413+0000","last_fullsized":"2024-03-17T23:47:01.943413+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:45:20.327436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713785+0000","last_change":"2024-03-17T23:46:54.925485+0000","last_active":"2024-03-17T23:47:02.713785+0000","last_peered":"2024-03-17T23:47:02.713785+0000","last_clean":"2024-03-17T23:47:02.713785+0000","last_became_active":"2024-03-17T23:46:54.925173+0000","last_became_peered":"2024-03-17T23:46:54.925173+0000","last_unstale":"2024-03-17T23:47:02.713785+0000","last_undegraded":"2024-03-17T23:47:02.713785+0000","last_fullsized":"2024-03-17T23:47:02.713785+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:51:18.545842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106704+0000","last_change":"2024-03-17T23:46:59.961930+0000","last_active":"2024-03-17T23:47:02.106704+0000","last_peered":"2024-03-17T23:47:02.106704+0000","last_clean":"2024-03-17T23:47:02.106704+0000","last_became_active":"2024-03-17T23:46:59.961495+0000","last_became_peered":"2024-03-17T23:46:59.961495+0000","last_unstale":"2024-03-17T23:47:02.106704+0000","last_undegraded":"2024-03-17T23:47:02.106704+0000","last_fullsized":"2024-03-17T23:47:02.106704+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:59:20.183441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943415+0000","last_change":"2024-03-17T23:46:55.927904+0000","last_active":"2024-03-17T23:47:01.943415+0000","last_peered":"2024-03-17T23:47:01.943415+0000","last_clean":"2024-03-17T23:47:01.943415+0000","last_became_active":"2024-03-17T23:46:55.927637+0000","last_became_peered":"2024-03-17T23:46:55.927637+0000","last_unstale":"2024-03-17T23:47:01.943415+0000","last_undegraded":"2024-03-17T23:47:01.943415+0000","last_fullsized":"2024-03-17T23:47:01.943415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:03.353861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714207+0000","last_change":"2024-03-17T23:46:54.920822+0000","last_active":"2024-03-17T23:47:02.714207+0000","last_peered":"2024-03-17T23:47:02.714207+0000","last_clean":"2024-03-17T23:47:02.714207+0000","last_became_active":"2024-03-17T23:46:54.920534+0000","last_became_peered":"2024-03-17T23:46:54.920534+0000","last_unstale":"2024-03-17T23:47:02.714207+0000","last_undegraded":"2024-03-17T23:47:02.714207+0000","last_fullsized":"2024-03-17T23:47:02.714207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:56.665566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106433+0000","last_change":"2024-03-17T23:46:59.961245+0000","last_active":"2024-03-17T23:47:02.106433+0000","last_peered":"2024-03-17T23:47:02.106433+0000","last_clean":"2024-03-17T23:47:02.106433+0000","last_became_active":"2024-03-17T23:46:59.960895+0000","last_became_peered":"2024-03-17T23:46:59.960895+0000","last_unstale":"2024-03-17T23:47:02.106433+0000","last_undegraded":"2024-03-17T23:47:02.106433+0000","last_fullsized":"2024-03-17T23:47:02.106433+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:57.296991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943288+0000","last_change":"2024-03-17T23:46:55.924013+0000","last_active":"2024-03-17T23:47:01.943288+0000","last_peered":"2024-03-17T23:47:01.943288+0000","last_clean":"2024-03-17T23:47:01.943288+0000","last_became_active":"2024-03-17T23:46:55.923741+0000","last_became_peered":"2024-03-17T23:46:55.923741+0000","last_unstale":"2024-03-17T23:47:01.943288+0000","last_undegraded":"2024-03-17T23:47:01.943288+0000","last_fullsized":"2024-03-17T23:47:01.943288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:00.571611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713487+0000","last_change":"2024-03-17T23:46:54.918712+0000","last_active":"2024-03-17T23:47:02.713487+0000","last_peered":"2024-03-17T23:47:02.713487+0000","last_clean":"2024-03-17T23:47:02.713487+0000","last_became_active":"2024-03-17T23:46:54.918470+0000","last_became_peered":"2024-03-17T23:46:54.918470+0000","last_unstale":"2024-03-17T23:47:02.713487+0000","last_undegraded":"2024-03-17T23:47:02.713487+0000","last_fullsized":"2024-03-17T23:47:02.713487+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:18.195935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.715105+0000","last_change":"2024-03-17T23:47:00.410049+0000","last_active":"2024-03-17T23:47:02.715105+0000","last_peered":"2024-03-17T23:47:02.715105+0000","last_clean":"2024-03-17T23:47:02.715105+0000","last_became_active":"2024-03-17T23:47:00.409688+0000","last_became_peered":"2024-03-17T23:47:00.409688+0000","last_unstale":"2024-03-17T23:47:02.715105+0000","last_undegraded":"2024-03-17T23:47:02.715105+0000","last_fullsized":"2024-03-17T23:47:02.715105+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:39:36.583214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106313+0000","last_change":"2024-03-17T23:46:55.925699+0000","last_active":"2024-03-17T23:47:02.106313+0000","last_peered":"2024-03-17T23:47:02.106313+0000","last_clean":"2024-03-17T23:47:02.106313+0000","last_became_active":"2024-03-17T23:46:55.925467+0000","last_became_peered":"2024-03-17T23:46:55.925467+0000","last_unstale":"2024-03-17T23:47:02.106313+0000","last_undegraded":"2024-03-17T23:47:02.106313+0000","last_fullsized":"2024-03-17T23:47:02.106313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:11:02.357691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713372+0000","last_change":"2024-03-17T23:46:54.923992+0000","last_active":"2024-03-17T23:47:02.713372+0000","last_peered":"2024-03-17T23:47:02.713372+0000","last_clean":"2024-03-17T23:47:02.713372+0000","last_became_active":"2024-03-17T23:46:54.923721+0000","last_became_peered":"2024-03-17T23:46:54.923721+0000","last_unstale":"2024-03-17T23:47:02.713372+0000","last_undegraded":"2024-03-17T23:47:02.713372+0000","last_fullsized":"2024-03-17T23:47:02.713372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:46:16.782215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106661+0000","last_change":"2024-03-17T23:46:59.960940+0000","last_active":"2024-03-17T23:47:02.106661+0000","last_peered":"2024-03-17T23:47:02.106661+0000","last_clean":"2024-03-17T23:47:02.106661+0000","last_became_active":"2024-03-17T23:46:59.960472+0000","last_became_peered":"2024-03-17T23:46:59.960472+0000","last_unstale":"2024-03-17T23:47:02.106661+0000","last_undegraded":"2024-03-17T23:47:02.106661+0000","last_fullsized":"2024-03-17T23:47:02.106661+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:21:41.104850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714557+0000","last_change":"2024-03-17T23:46:55.927164+0000","last_active":"2024-03-17T23:47:02.714557+0000","last_peered":"2024-03-17T23:47:02.714557+0000","last_clean":"2024-03-17T23:47:02.714557+0000","last_became_active":"2024-03-17T23:46:55.926883+0000","last_became_peered":"2024-03-17T23:46:55.926883+0000","last_unstale":"2024-03-17T23:47:02.714557+0000","last_undegraded":"2024-03-17T23:47:02.714557+0000","last_fullsized":"2024-03-17T23:47:02.714557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:03.353861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944736+0000","last_change":"2024-03-17T23:46:54.921997+0000","last_active":"2024-03-17T23:47:01.944736+0000","last_peered":"2024-03-17T23:47:01.944736+0000","last_clean":"2024-03-17T23:47:01.944736+0000","last_became_active":"2024-03-17T23:46:54.921701+0000","last_became_peered":"2024-03-17T23:46:54.921701+0000","last_unstale":"2024-03-17T23:47:01.944736+0000","last_undegraded":"2024-03-17T23:47:01.944736+0000","last_fullsized":"2024-03-17T23:47:01.944736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:11:01.353944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106094+0000","last_change":"2024-03-17T23:47:00.410238+0000","last_active":"2024-03-17T23:47:02.106094+0000","last_peered":"2024-03-17T23:47:02.106094+0000","last_clean":"2024-03-17T23:47:02.106094+0000","last_became_active":"2024-03-17T23:47:00.409845+0000","last_became_peered":"2024-03-17T23:47:00.409845+0000","last_unstale":"2024-03-17T23:47:02.106094+0000","last_undegraded":"2024-03-17T23:47:02.106094+0000","last_fullsized":"2024-03-17T23:47:02.106094+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:43:57.153828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943841+0000","last_change":"2024-03-17T23:46:55.929991+0000","last_active":"2024-03-17T23:47:01.943841+0000","last_peered":"2024-03-17T23:47:01.943841+0000","last_clean":"2024-03-17T23:47:01.943841+0000","last_became_active":"2024-03-17T23:46:55.929722+0000","last_became_peered":"2024-03-17T23:46:55.929722+0000","last_unstale":"2024-03-17T23:47:01.943841+0000","last_undegraded":"2024-03-17T23:47:01.943841+0000","last_fullsized":"2024-03-17T23:47:01.943841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:11:08.829506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713963+0000","last_change":"2024-03-17T23:46:54.913493+0000","last_active":"2024-03-17T23:47:02.713963+0000","last_peered":"2024-03-17T23:47:02.713963+0000","last_clean":"2024-03-17T23:47:02.713963+0000","last_became_active":"2024-03-17T23:46:54.913233+0000","last_became_peered":"2024-03-17T23:46:54.913233+0000","last_unstale":"2024-03-17T23:47:02.713963+0000","last_undegraded":"2024-03-17T23:47:02.713963+0000","last_fullsized":"2024-03-17T23:47:02.713963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:58.417421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944060+0000","last_change":"2024-03-17T23:47:00.408616+0000","last_active":"2024-03-17T23:47:01.944060+0000","last_peered":"2024-03-17T23:47:01.944060+0000","last_clean":"2024-03-17T23:47:01.944060+0000","last_became_active":"2024-03-17T23:47:00.408251+0000","last_became_peered":"2024-03-17T23:47:00.408251+0000","last_unstale":"2024-03-17T23:47:01.944060+0000","last_undegraded":"2024-03-17T23:47:01.944060+0000","last_fullsized":"2024-03-17T23:47:01.944060+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:16:52.426485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714810+0000","last_change":"2024-03-17T23:46:55.921416+0000","last_active":"2024-03-17T23:47:02.714810+0000","last_peered":"2024-03-17T23:47:02.714810+0000","last_clean":"2024-03-17T23:47:02.714810+0000","last_became_active":"2024-03-17T23:46:55.921128+0000","last_became_peered":"2024-03-17T23:46:55.921128+0000","last_unstale":"2024-03-17T23:47:02.714810+0000","last_undegraded":"2024-03-17T23:47:02.714810+0000","last_fullsized":"2024-03-17T23:47:02.714810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:49:47.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714025+0000","last_change":"2024-03-17T23:46:54.926748+0000","last_active":"2024-03-17T23:47:02.714025+0000","last_peered":"2024-03-17T23:47:02.714025+0000","last_clean":"2024-03-17T23:47:02.714025+0000","last_became_active":"2024-03-17T23:46:54.926467+0000","last_became_peered":"2024-03-17T23:46:54.926467+0000","last_unstale":"2024-03-17T23:47:02.714025+0000","last_undegraded":"2024-03-17T23:47:02.714025+0000","last_fullsized":"2024-03-17T23:47:02.714025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:58.417421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714085+0000","last_change":"2024-03-17T23:47:00.409306+0000","last_active":"2024-03-17T23:47:02.714085+0000","last_peered":"2024-03-17T23:47:02.714085+0000","last_clean":"2024-03-17T23:47:02.714085+0000","last_became_active":"2024-03-17T23:47:00.408934+0000","last_became_peered":"2024-03-17T23:47:00.408934+0000","last_unstale":"2024-03-17T23:47:02.714085+0000","last_undegraded":"2024-03-17T23:47:02.714085+0000","last_fullsized":"2024-03-17T23:47:02.714085+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:35:49.838832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944302+0000","last_change":"2024-03-17T23:46:55.933418+0000","last_active":"2024-03-17T23:47:01.944302+0000","last_peered":"2024-03-17T23:47:01.944302+0000","last_clean":"2024-03-17T23:47:01.944302+0000","last_became_active":"2024-03-17T23:46:55.931133+0000","last_became_peered":"2024-03-17T23:46:55.931133+0000","last_unstale":"2024-03-17T23:47:01.944302+0000","last_undegraded":"2024-03-17T23:47:01.944302+0000","last_fullsized":"2024-03-17T23:47:01.944302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:32:27.349125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714826+0000","last_change":"2024-03-17T23:46:54.914746+0000","last_active":"2024-03-17T23:47:02.714826+0000","last_peered":"2024-03-17T23:47:02.714826+0000","last_clean":"2024-03-17T23:47:02.714826+0000","last_became_active":"2024-03-17T23:46:54.914482+0000","last_became_peered":"2024-03-17T23:46:54.914482+0000","last_unstale":"2024-03-17T23:47:02.714826+0000","last_undegraded":"2024-03-17T23:47:02.714826+0000","last_fullsized":"2024-03-17T23:47:02.714826+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:42:33.325150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713844+0000","last_change":"2024-03-17T23:46:59.961836+0000","last_active":"2024-03-17T23:47:02.713844+0000","last_peered":"2024-03-17T23:47:02.713844+0000","last_clean":"2024-03-17T23:47:02.713844+0000","last_became_active":"2024-03-17T23:46:59.961458+0000","last_became_peered":"2024-03-17T23:46:59.961458+0000","last_unstale":"2024-03-17T23:47:02.713844+0000","last_undegraded":"2024-03-17T23:47:02.713844+0000","last_fullsized":"2024-03-17T23:47:02.713844+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:43:57.153828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106542+0000","last_change":"2024-03-17T23:46:55.926629+0000","last_active":"2024-03-17T23:47:02.106542+0000","last_peered":"2024-03-17T23:47:02.106542+0000","last_clean":"2024-03-17T23:47:02.106542+0000","last_became_active":"2024-03-17T23:46:55.926391+0000","last_became_peered":"2024-03-17T23:46:55.926391+0000","last_unstale":"2024-03-17T23:47:02.106542+0000","last_undegraded":"2024-03-17T23:47:02.106542+0000","last_fullsized":"2024-03-17T23:47:02.106542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:59.421169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943876+0000","last_change":"2024-03-17T23:46:54.921058+0000","last_active":"2024-03-17T23:47:01.943876+0000","last_peered":"2024-03-17T23:47:01.943876+0000","last_clean":"2024-03-17T23:47:01.943876+0000","last_became_active":"2024-03-17T23:46:54.919926+0000","last_became_peered":"2024-03-17T23:46:54.919926+0000","last_unstale":"2024-03-17T23:47:01.943876+0000","last_undegraded":"2024-03-17T23:47:01.943876+0000","last_fullsized":"2024-03-17T23:47:01.943876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:11:01.353944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714725+0000","last_change":"2024-03-17T23:46:59.957516+0000","last_active":"2024-03-17T23:47:02.714725+0000","last_peered":"2024-03-17T23:47:02.714725+0000","last_clean":"2024-03-17T23:47:02.714725+0000","last_became_active":"2024-03-17T23:46:59.957168+0000","last_became_peered":"2024-03-17T23:46:59.957168+0000","last_unstale":"2024-03-17T23:47:02.714725+0000","last_undegraded":"2024-03-17T23:47:02.714725+0000","last_fullsized":"2024-03-17T23:47:02.714725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:35:49.838832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713602+0000","last_change":"2024-03-17T23:46:55.921369+0000","last_active":"2024-03-17T23:47:02.713602+0000","last_peered":"2024-03-17T23:47:02.713602+0000","last_clean":"2024-03-17T23:47:02.713602+0000","last_became_active":"2024-03-17T23:46:55.921109+0000","last_became_peered":"2024-03-17T23:46:55.921109+0000","last_unstale":"2024-03-17T23:47:02.713602+0000","last_undegraded":"2024-03-17T23:47:02.713602+0000","last_fullsized":"2024-03-17T23:47:02.713602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:24.419077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713228+0000","last_change":"2024-03-17T23:46:54.918722+0000","last_active":"2024-03-17T23:47:02.713228+0000","last_peered":"2024-03-17T23:47:02.713228+0000","last_clean":"2024-03-17T23:47:02.713228+0000","last_became_active":"2024-03-17T23:46:54.918440+0000","last_became_peered":"2024-03-17T23:46:54.918440+0000","last_unstale":"2024-03-17T23:47:02.713228+0000","last_undegraded":"2024-03-17T23:47:02.713228+0000","last_fullsized":"2024-03-17T23:47:02.713228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:56.665566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.715106+0000","last_change":"2024-03-17T23:46:59.962185+0000","last_active":"2024-03-17T23:47:02.715106+0000","last_peered":"2024-03-17T23:47:02.715106+0000","last_clean":"2024-03-17T23:47:02.715106+0000","last_became_active":"2024-03-17T23:46:59.961682+0000","last_became_peered":"2024-03-17T23:46:59.961682+0000","last_unstale":"2024-03-17T23:47:02.715106+0000","last_undegraded":"2024-03-17T23:47:02.715106+0000","last_fullsized":"2024-03-17T23:47:02.715106+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:47:52.502020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713596+0000","last_change":"2024-03-17T23:46:55.924464+0000","last_active":"2024-03-17T23:47:02.713596+0000","last_peered":"2024-03-17T23:47:02.713596+0000","last_clean":"2024-03-17T23:47:02.713596+0000","last_became_active":"2024-03-17T23:46:55.924193+0000","last_became_peered":"2024-03-17T23:46:55.924193+0000","last_unstale":"2024-03-17T23:47:02.713596+0000","last_undegraded":"2024-03-17T23:47:02.713596+0000","last_fullsized":"2024-03-17T23:47:02.713596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:54.348184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713381+0000","last_change":"2024-03-17T23:46:54.921188+0000","last_active":"2024-03-17T23:47:02.713381+0000","last_peered":"2024-03-17T23:47:02.713381+0000","last_clean":"2024-03-17T23:47:02.713381+0000","last_became_active":"2024-03-17T23:46:54.920926+0000","last_became_peered":"2024-03-17T23:46:54.920926+0000","last_unstale":"2024-03-17T23:47:02.713381+0000","last_undegraded":"2024-03-17T23:47:02.713381+0000","last_fullsized":"2024-03-17T23:47:02.713381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:01:23.150557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714399+0000","last_change":"2024-03-17T23:47:00.410855+0000","last_active":"2024-03-17T23:47:02.714399+0000","last_peered":"2024-03-17T23:47:02.714399+0000","last_clean":"2024-03-17T23:47:02.714399+0000","last_became_active":"2024-03-17T23:47:00.410485+0000","last_became_peered":"2024-03-17T23:47:00.410485+0000","last_unstale":"2024-03-17T23:47:02.714399+0000","last_undegraded":"2024-03-17T23:47:02.714399+0000","last_fullsized":"2024-03-17T23:47:02.714399+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:21:41.104850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943282+0000","last_change":"2024-03-17T23:46:55.931906+0000","last_active":"2024-03-17T23:47:01.943282+0000","last_peered":"2024-03-17T23:47:01.943282+0000","last_clean":"2024-03-17T23:47:01.943282+0000","last_became_active":"2024-03-17T23:46:55.931643+0000","last_became_peered":"2024-03-17T23:46:55.931643+0000","last_unstale":"2024-03-17T23:47:01.943282+0000","last_undegraded":"2024-03-17T23:47:01.943282+0000","last_fullsized":"2024-03-17T23:47:01.943282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:24.419077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943618+0000","last_change":"2024-03-17T23:46:54.920477+0000","last_active":"2024-03-17T23:47:01.943618+0000","last_peered":"2024-03-17T23:47:01.943618+0000","last_clean":"2024-03-17T23:47:01.943618+0000","last_became_active":"2024-03-17T23:46:54.920116+0000","last_became_peered":"2024-03-17T23:46:54.920116+0000","last_unstale":"2024-03-17T23:47:01.943618+0000","last_undegraded":"2024-03-17T23:47:01.943618+0000","last_fullsized":"2024-03-17T23:47:01.943618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:56.665566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":180,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713494+0000","last_change":"2024-03-17T23:46:44.768668+0000","last_active":"2024-03-17T23:47:02.713494+0000","last_peered":"2024-03-17T23:47:02.713494+0000","last_clean":"2024-03-17T23:47:02.713494+0000","last_became_active":"2024-03-17T23:46:44.767826+0000","last_became_peered":"2024-03-17T23:46:44.767826+0000","last_unstale":"2024-03-17T23:47:02.713494+0000","last_undegraded":"2024-03-17T23:47:02.713494+0000","last_fullsized":"2024-03-17T23:47:02.713494+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_clean_scrub_stamp":"2024-03-17T23:46:43.744878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:54:22.286155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944382+0000","last_change":"2024-03-17T23:47:00.410345+0000","last_active":"2024-03-17T23:47:01.944382+0000","last_peered":"2024-03-17T23:47:01.944382+0000","last_clean":"2024-03-17T23:47:01.944382+0000","last_became_active":"2024-03-17T23:47:00.409948+0000","last_became_peered":"2024-03-17T23:47:00.409948+0000","last_unstale":"2024-03-17T23:47:01.944382+0000","last_undegraded":"2024-03-17T23:47:01.944382+0000","last_fullsized":"2024-03-17T23:47:01.944382+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:20:25.538662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713826+0000","last_change":"2024-03-17T23:46:55.943072+0000","last_active":"2024-03-17T23:47:02.713826+0000","last_peered":"2024-03-17T23:47:02.713826+0000","last_clean":"2024-03-17T23:47:02.713826+0000","last_became_active":"2024-03-17T23:46:55.942516+0000","last_became_peered":"2024-03-17T23:46:55.942516+0000","last_unstale":"2024-03-17T23:47:02.713826+0000","last_undegraded":"2024-03-17T23:47:02.713826+0000","last_fullsized":"2024-03-17T23:47:02.713826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:00.571611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713510+0000","last_change":"2024-03-17T23:46:54.916398+0000","last_active":"2024-03-17T23:47:02.713510+0000","last_peered":"2024-03-17T23:47:02.713510+0000","last_clean":"2024-03-17T23:47:02.713510+0000","last_became_active":"2024-03-17T23:46:54.916167+0000","last_became_peered":"2024-03-17T23:46:54.916167+0000","last_unstale":"2024-03-17T23:47:02.713510+0000","last_undegraded":"2024-03-17T23:47:02.713510+0000","last_fullsized":"2024-03-17T23:47:02.713510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:42:33.325150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943370+0000","last_change":"2024-03-17T23:47:00.408551+0000","last_active":"2024-03-17T23:47:01.943370+0000","last_peered":"2024-03-17T23:47:01.943370+0000","last_clean":"2024-03-17T23:47:01.943370+0000","last_became_active":"2024-03-17T23:47:00.408223+0000","last_became_peered":"2024-03-17T23:47:00.408223+0000","last_unstale":"2024-03-17T23:47:01.943370+0000","last_undegraded":"2024-03-17T23:47:01.943370+0000","last_fullsized":"2024-03-17T23:47:01.943370+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:52:49.194504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944055+0000","last_change":"2024-03-17T23:46:55.933223+0000","last_active":"2024-03-17T23:47:01.944055+0000","last_peered":"2024-03-17T23:47:01.944055+0000","last_clean":"2024-03-17T23:47:01.944055+0000","last_became_active":"2024-03-17T23:46:55.932957+0000","last_became_peered":"2024-03-17T23:46:55.932957+0000","last_unstale":"2024-03-17T23:47:01.944055+0000","last_undegraded":"2024-03-17T23:47:01.944055+0000","last_fullsized":"2024-03-17T23:47:01.944055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:52:45.163567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713856+0000","last_change":"2024-03-17T23:46:54.919441+0000","last_active":"2024-03-17T23:47:02.713856+0000","last_peered":"2024-03-17T23:47:02.713856+0000","last_clean":"2024-03-17T23:47:02.713856+0000","last_became_active":"2024-03-17T23:46:54.919151+0000","last_became_peered":"2024-03-17T23:46:54.919151+0000","last_unstale":"2024-03-17T23:47:02.713856+0000","last_undegraded":"2024-03-17T23:47:02.713856+0000","last_fullsized":"2024-03-17T23:47:02.713856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:11:01.353944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714221+0000","last_change":"2024-03-17T23:46:59.963445+0000","last_active":"2024-03-17T23:47:02.714221+0000","last_peered":"2024-03-17T23:47:02.714221+0000","last_clean":"2024-03-17T23:47:02.714221+0000","last_became_active":"2024-03-17T23:46:59.963078+0000","last_became_peered":"2024-03-17T23:46:59.963078+0000","last_unstale":"2024-03-17T23:47:02.714221+0000","last_undegraded":"2024-03-17T23:47:02.714221+0000","last_fullsized":"2024-03-17T23:47:02.714221+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:47:52.502020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713966+0000","last_change":"2024-03-17T23:46:55.930951+0000","last_active":"2024-03-17T23:47:02.713966+0000","last_peered":"2024-03-17T23:47:02.713966+0000","last_clean":"2024-03-17T23:47:02.713966+0000","last_became_active":"2024-03-17T23:46:55.930689+0000","last_became_peered":"2024-03-17T23:46:55.930689+0000","last_unstale":"2024-03-17T23:47:02.713966+0000","last_undegraded":"2024-03-17T23:47:02.713966+0000","last_fullsized":"2024-03-17T23:47:02.713966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:43:53.122891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713358+0000","last_change":"2024-03-17T23:46:54.921408+0000","last_active":"2024-03-17T23:47:02.713358+0000","last_peered":"2024-03-17T23:47:02.713358+0000","last_clean":"2024-03-17T23:47:02.713358+0000","last_became_active":"2024-03-17T23:46:54.920825+0000","last_became_peered":"2024-03-17T23:46:54.920825+0000","last_unstale":"2024-03-17T23:47:02.713358+0000","last_undegraded":"2024-03-17T23:47:02.713358+0000","last_fullsized":"2024-03-17T23:47:02.713358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:18.195935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.942947+0000","last_change":"2024-03-17T23:47:00.411371+0000","last_active":"2024-03-17T23:47:01.942947+0000","last_peered":"2024-03-17T23:47:01.942947+0000","last_clean":"2024-03-17T23:47:01.942947+0000","last_became_active":"2024-03-17T23:47:00.410726+0000","last_became_peered":"2024-03-17T23:47:00.410726+0000","last_unstale":"2024-03-17T23:47:01.942947+0000","last_undegraded":"2024-03-17T23:47:01.942947+0000","last_fullsized":"2024-03-17T23:47:01.942947+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:47:52.502020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713787+0000","last_change":"2024-03-17T23:46:55.933053+0000","last_active":"2024-03-17T23:47:02.713787+0000","last_peered":"2024-03-17T23:47:02.713787+0000","last_clean":"2024-03-17T23:47:02.713787+0000","last_became_active":"2024-03-17T23:46:55.932756+0000","last_became_peered":"2024-03-17T23:46:55.932756+0000","last_unstale":"2024-03-17T23:47:02.713787+0000","last_undegraded":"2024-03-17T23:47:02.713787+0000","last_fullsized":"2024-03-17T23:47:02.713787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:45:29.447600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106603+0000","last_change":"2024-03-17T23:46:54.916992+0000","last_active":"2024-03-17T23:47:02.106603+0000","last_peered":"2024-03-17T23:47:02.106603+0000","last_clean":"2024-03-17T23:47:02.106603+0000","last_became_active":"2024-03-17T23:46:54.916296+0000","last_became_peered":"2024-03-17T23:46:54.916296+0000","last_unstale":"2024-03-17T23:47:02.106603+0000","last_undegraded":"2024-03-17T23:47:02.106603+0000","last_fullsized":"2024-03-17T23:47:02.106603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:17:26.217921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106127+0000","last_change":"2024-03-17T23:46:59.957769+0000","last_active":"2024-03-17T23:47:02.106127+0000","last_peered":"2024-03-17T23:47:02.106127+0000","last_clean":"2024-03-17T23:47:02.106127+0000","last_became_active":"2024-03-17T23:46:59.957385+0000","last_became_peered":"2024-03-17T23:46:59.957385+0000","last_unstale":"2024-03-17T23:47:02.106127+0000","last_undegraded":"2024-03-17T23:47:02.106127+0000","last_fullsized":"2024-03-17T23:47:02.106127+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:35:49.838832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713581+0000","last_change":"2024-03-17T23:46:55.935175+0000","last_active":"2024-03-17T23:47:02.713581+0000","last_peered":"2024-03-17T23:47:02.713581+0000","last_clean":"2024-03-17T23:47:02.713581+0000","last_became_active":"2024-03-17T23:46:55.934338+0000","last_became_peered":"2024-03-17T23:46:55.934338+0000","last_unstale":"2024-03-17T23:47:02.713581+0000","last_undegraded":"2024-03-17T23:47:02.713581+0000","last_fullsized":"2024-03-17T23:47:02.713581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:59:32.027430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713064+0000","last_change":"2024-03-17T23:46:54.913663+0000","last_active":"2024-03-17T23:47:02.713064+0000","last_peered":"2024-03-17T23:47:02.713064+0000","last_clean":"2024-03-17T23:47:02.713064+0000","last_became_active":"2024-03-17T23:46:54.913410+0000","last_became_peered":"2024-03-17T23:46:54.913410+0000","last_unstale":"2024-03-17T23:47:02.713064+0000","last_undegraded":"2024-03-17T23:47:02.713064+0000","last_fullsized":"2024-03-17T23:47:02.713064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:45:17.571056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714251+0000","last_change":"2024-03-17T23:46:59.967164+0000","last_active":"2024-03-17T23:47:02.714251+0000","last_peered":"2024-03-17T23:47:02.714251+0000","last_clean":"2024-03-17T23:47:02.714251+0000","last_became_active":"2024-03-17T23:46:59.966653+0000","last_became_peered":"2024-03-17T23:46:59.966653+0000","last_unstale":"2024-03-17T23:47:02.714251+0000","last_undegraded":"2024-03-17T23:47:02.714251+0000","last_fullsized":"2024-03-17T23:47:02.714251+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:57.296991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943052+0000","last_change":"2024-03-17T23:46:55.925322+0000","last_active":"2024-03-17T23:47:01.943052+0000","last_peered":"2024-03-17T23:47:01.943052+0000","last_clean":"2024-03-17T23:47:01.943052+0000","last_became_active":"2024-03-17T23:46:55.925037+0000","last_became_peered":"2024-03-17T23:46:55.925037+0000","last_unstale":"2024-03-17T23:47:01.943052+0000","last_undegraded":"2024-03-17T23:47:01.943052+0000","last_fullsized":"2024-03-17T23:47:01.943052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:39.972607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943437+0000","last_change":"2024-03-17T23:46:54.918033+0000","last_active":"2024-03-17T23:47:01.943437+0000","last_peered":"2024-03-17T23:47:01.943437+0000","last_clean":"2024-03-17T23:47:01.943437+0000","last_became_active":"2024-03-17T23:46:54.917735+0000","last_became_peered":"2024-03-17T23:46:54.917735+0000","last_unstale":"2024-03-17T23:47:01.943437+0000","last_undegraded":"2024-03-17T23:47:01.943437+0000","last_fullsized":"2024-03-17T23:47:01.943437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:42:33.325150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943778+0000","last_change":"2024-03-17T23:47:00.412019+0000","last_active":"2024-03-17T23:47:01.943778+0000","last_peered":"2024-03-17T23:47:01.943778+0000","last_clean":"2024-03-17T23:47:01.943778+0000","last_became_active":"2024-03-17T23:47:00.411603+0000","last_became_peered":"2024-03-17T23:47:00.411603+0000","last_unstale":"2024-03-17T23:47:01.943778+0000","last_undegraded":"2024-03-17T23:47:01.943778+0000","last_fullsized":"2024-03-17T23:47:01.943778+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:18:26.998562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713747+0000","last_change":"2024-03-17T23:46:55.935249+0000","last_active":"2024-03-17T23:47:02.713747+0000","last_peered":"2024-03-17T23:47:02.713747+0000","last_clean":"2024-03-17T23:47:02.713747+0000","last_became_active":"2024-03-17T23:46:55.934833+0000","last_became_peered":"2024-03-17T23:46:55.934833+0000","last_unstale":"2024-03-17T23:47:02.713747+0000","last_undegraded":"2024-03-17T23:47:02.713747+0000","last_fullsized":"2024-03-17T23:47:02.713747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:32:27.349125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943797+0000","last_change":"2024-03-17T23:46:54.924340+0000","last_active":"2024-03-17T23:47:01.943797+0000","last_peered":"2024-03-17T23:47:01.943797+0000","last_clean":"2024-03-17T23:47:01.943797+0000","last_became_active":"2024-03-17T23:46:54.924080+0000","last_became_peered":"2024-03-17T23:46:54.924080+0000","last_unstale":"2024-03-17T23:47:01.943797+0000","last_undegraded":"2024-03-17T23:47:01.943797+0000","last_fullsized":"2024-03-17T23:47:01.943797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:06:54.433520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943804+0000","last_change":"2024-03-17T23:47:00.408517+0000","last_active":"2024-03-17T23:47:01.943804+0000","last_peered":"2024-03-17T23:47:01.943804+0000","last_clean":"2024-03-17T23:47:01.943804+0000","last_became_active":"2024-03-17T23:47:00.408125+0000","last_became_peered":"2024-03-17T23:47:00.408125+0000","last_unstale":"2024-03-17T23:47:01.943804+0000","last_undegraded":"2024-03-17T23:47:01.943804+0000","last_fullsized":"2024-03-17T23:47:01.943804+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:12:39.327858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714486+0000","last_change":"2024-03-17T23:46:55.928837+0000","last_active":"2024-03-17T23:47:02.714486+0000","last_peered":"2024-03-17T23:47:02.714486+0000","last_clean":"2024-03-17T23:47:02.714486+0000","last_became_active":"2024-03-17T23:46:55.928557+0000","last_became_peered":"2024-03-17T23:46:55.928557+0000","last_unstale":"2024-03-17T23:47:02.714486+0000","last_undegraded":"2024-03-17T23:47:02.714486+0000","last_fullsized":"2024-03-17T23:47:02.714486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:23:58.221114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714395+0000","last_change":"2024-03-17T23:46:54.912215+0000","last_active":"2024-03-17T23:47:02.714395+0000","last_peered":"2024-03-17T23:47:02.714395+0000","last_clean":"2024-03-17T23:47:02.714395+0000","last_became_active":"2024-03-17T23:46:54.911909+0000","last_became_peered":"2024-03-17T23:46:54.911909+0000","last_unstale":"2024-03-17T23:47:02.714395+0000","last_undegraded":"2024-03-17T23:47:02.714395+0000","last_fullsized":"2024-03-17T23:47:02.714395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:45:17.571056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944243+0000","last_change":"2024-03-17T23:47:00.408798+0000","last_active":"2024-03-17T23:47:01.944243+0000","last_peered":"2024-03-17T23:47:01.944243+0000","last_clean":"2024-03-17T23:47:01.944243+0000","last_became_active":"2024-03-17T23:47:00.408420+0000","last_became_peered":"2024-03-17T23:47:00.408420+0000","last_unstale":"2024-03-17T23:47:01.944243+0000","last_undegraded":"2024-03-17T23:47:01.944243+0000","last_fullsized":"2024-03-17T23:47:01.944243+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:57.296991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713721+0000","last_change":"2024-03-17T23:46:55.935456+0000","last_active":"2024-03-17T23:47:02.713721+0000","last_peered":"2024-03-17T23:47:02.713721+0000","last_clean":"2024-03-17T23:47:02.713721+0000","last_became_active":"2024-03-17T23:46:55.935187+0000","last_became_peered":"2024-03-17T23:46:55.935187+0000","last_unstale":"2024-03-17T23:47:02.713721+0000","last_undegraded":"2024-03-17T23:47:02.713721+0000","last_fullsized":"2024-03-17T23:47:02.713721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:11:08.829506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"26'9","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:05.348071+0000","last_change":"2024-03-17T23:46:54.918644+0000","last_active":"2024-03-17T23:47:05.348071+0000","last_peered":"2024-03-17T23:47:05.348071+0000","last_clean":"2024-03-17T23:47:05.348071+0000","last_became_active":"2024-03-17T23:46:54.918290+0000","last_became_peered":"2024-03-17T23:46:54.918290+0000","last_unstale":"2024-03-17T23:47:05.348071+0000","last_undegraded":"2024-03-17T23:47:05.348071+0000","last_fullsized":"2024-03-17T23:47:05.348071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:58.417421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714038+0000","last_change":"2024-03-17T23:46:59.961721+0000","last_active":"2024-03-17T23:47:02.714038+0000","last_peered":"2024-03-17T23:47:02.714038+0000","last_clean":"2024-03-17T23:47:02.714038+0000","last_became_active":"2024-03-17T23:46:59.960948+0000","last_became_peered":"2024-03-17T23:46:59.960948+0000","last_unstale":"2024-03-17T23:47:02.714038+0000","last_undegraded":"2024-03-17T23:47:02.714038+0000","last_fullsized":"2024-03-17T23:47:02.714038+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:27:51.964285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106592+0000","last_change":"2024-03-17T23:46:55.928225+0000","last_active":"2024-03-17T23:47:02.106592+0000","last_peered":"2024-03-17T23:47:02.106592+0000","last_clean":"2024-03-17T23:47:02.106592+0000","last_became_active":"2024-03-17T23:46:55.927359+0000","last_became_peered":"2024-03-17T23:46:55.927359+0000","last_unstale":"2024-03-17T23:47:02.106592+0000","last_undegraded":"2024-03-17T23:47:02.106592+0000","last_fullsized":"2024-03-17T23:47:02.106592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:46:17.785962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713797+0000","last_change":"2024-03-17T23:46:54.917416+0000","last_active":"2024-03-17T23:47:02.713797+0000","last_peered":"2024-03-17T23:47:02.713797+0000","last_clean":"2024-03-17T23:47:02.713797+0000","last_became_active":"2024-03-17T23:46:54.917136+0000","last_became_peered":"2024-03-17T23:46:54.917136+0000","last_unstale":"2024-03-17T23:47:02.713797+0000","last_undegraded":"2024-03-17T23:47:02.713797+0000","last_fullsized":"2024-03-17T23:47:02.713797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:06:54.433520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106414+0000","last_change":"2024-03-17T23:46:59.957651+0000","last_active":"2024-03-17T23:47:02.106414+0000","last_peered":"2024-03-17T23:47:02.106414+0000","last_clean":"2024-03-17T23:47:02.106414+0000","last_became_active":"2024-03-17T23:46:59.957261+0000","last_became_peered":"2024-03-17T23:46:59.957261+0000","last_unstale":"2024-03-17T23:47:02.106414+0000","last_undegraded":"2024-03-17T23:47:02.106414+0000","last_fullsized":"2024-03-17T23:47:02.106414+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:47:52.502020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713868+0000","last_change":"2024-03-17T23:46:55.937105+0000","last_active":"2024-03-17T23:47:02.713868+0000","last_peered":"2024-03-17T23:47:02.713868+0000","last_clean":"2024-03-17T23:47:02.713868+0000","last_became_active":"2024-03-17T23:46:55.936762+0000","last_became_peered":"2024-03-17T23:46:55.936762+0000","last_unstale":"2024-03-17T23:47:02.713868+0000","last_undegraded":"2024-03-17T23:47:02.713868+0000","last_fullsized":"2024-03-17T23:47:02.713868+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:39.972607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944371+0000","last_change":"2024-03-17T23:46:54.922118+0000","last_active":"2024-03-17T23:47:01.944371+0000","last_peered":"2024-03-17T23:47:01.944371+0000","last_clean":"2024-03-17T23:47:01.944371+0000","last_became_active":"2024-03-17T23:46:54.921834+0000","last_became_peered":"2024-03-17T23:46:54.921834+0000","last_unstale":"2024-03-17T23:47:01.944371+0000","last_undegraded":"2024-03-17T23:47:01.944371+0000","last_fullsized":"2024-03-17T23:47:01.944371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:42:33.325150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943156+0000","last_change":"2024-03-17T23:46:59.957652+0000","last_active":"2024-03-17T23:47:01.943156+0000","last_peered":"2024-03-17T23:47:01.943156+0000","last_clean":"2024-03-17T23:47:01.943156+0000","last_became_active":"2024-03-17T23:46:59.957242+0000","last_became_peered":"2024-03-17T23:46:59.957242+0000","last_unstale":"2024-03-17T23:47:01.943156+0000","last_undegraded":"2024-03-17T23:47:01.943156+0000","last_fullsized":"2024-03-17T23:47:01.943156+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:48:08.469403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944280+0000","last_change":"2024-03-17T23:46:55.930428+0000","last_active":"2024-03-17T23:47:01.944280+0000","last_peered":"2024-03-17T23:47:01.944280+0000","last_clean":"2024-03-17T23:47:01.944280+0000","last_became_active":"2024-03-17T23:46:55.930119+0000","last_became_peered":"2024-03-17T23:46:55.930119+0000","last_unstale":"2024-03-17T23:47:01.944280+0000","last_undegraded":"2024-03-17T23:47:01.944280+0000","last_fullsized":"2024-03-17T23:47:01.944280+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:00.571611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944550+0000","last_change":"2024-03-17T23:46:54.918186+0000","last_active":"2024-03-17T23:47:01.944550+0000","last_peered":"2024-03-17T23:47:01.944550+0000","last_clean":"2024-03-17T23:47:01.944550+0000","last_became_active":"2024-03-17T23:46:54.917839+0000","last_became_peered":"2024-03-17T23:46:54.917839+0000","last_unstale":"2024-03-17T23:47:01.944550+0000","last_undegraded":"2024-03-17T23:47:01.944550+0000","last_fullsized":"2024-03-17T23:47:01.944550+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:06:54.433520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943131+0000","last_change":"2024-03-17T23:47:00.411924+0000","last_active":"2024-03-17T23:47:01.943131+0000","last_peered":"2024-03-17T23:47:01.943131+0000","last_clean":"2024-03-17T23:47:01.943131+0000","last_became_active":"2024-03-17T23:47:00.411410+0000","last_became_peered":"2024-03-17T23:47:00.411410+0000","last_unstale":"2024-03-17T23:47:01.943131+0000","last_undegraded":"2024-03-17T23:47:01.943131+0000","last_fullsized":"2024-03-17T23:47:01.943131+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:56:09.808314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714107+0000","last_change":"2024-03-17T23:47:00.410837+0000","last_active":"2024-03-17T23:47:02.714107+0000","last_peered":"2024-03-17T23:47:02.714107+0000","last_clean":"2024-03-17T23:47:02.714107+0000","last_became_active":"2024-03-17T23:47:00.410459+0000","last_became_peered":"2024-03-17T23:47:00.410459+0000","last_unstale":"2024-03-17T23:47:02.714107+0000","last_undegraded":"2024-03-17T23:47:02.714107+0000","last_fullsized":"2024-03-17T23:47:02.714107+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:39:36.583214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713305+0000","last_change":"2024-03-17T23:46:55.928213+0000","last_active":"2024-03-17T23:47:02.713305+0000","last_peered":"2024-03-17T23:47:02.713305+0000","last_clean":"2024-03-17T23:47:02.713305+0000","last_became_active":"2024-03-17T23:46:55.927957+0000","last_became_peered":"2024-03-17T23:46:55.927957+0000","last_unstale":"2024-03-17T23:47:02.713305+0000","last_undegraded":"2024-03-17T23:47:02.713305+0000","last_fullsized":"2024-03-17T23:47:02.713305+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:03.353861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943171+0000","last_change":"2024-03-17T23:46:54.919392+0000","last_active":"2024-03-17T23:47:01.943171+0000","last_peered":"2024-03-17T23:47:01.943171+0000","last_clean":"2024-03-17T23:47:01.943171+0000","last_became_active":"2024-03-17T23:46:54.918594+0000","last_became_peered":"2024-03-17T23:46:54.918594+0000","last_unstale":"2024-03-17T23:47:01.943171+0000","last_undegraded":"2024-03-17T23:47:01.943171+0000","last_fullsized":"2024-03-17T23:47:01.943171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:18.195935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944720+0000","last_change":"2024-03-17T23:46:59.959748+0000","last_active":"2024-03-17T23:47:01.944720+0000","last_peered":"2024-03-17T23:47:01.944720+0000","last_clean":"2024-03-17T23:47:01.944720+0000","last_became_active":"2024-03-17T23:46:59.959383+0000","last_became_peered":"2024-03-17T23:46:59.959383+0000","last_unstale":"2024-03-17T23:47:01.944720+0000","last_undegraded":"2024-03-17T23:47:01.944720+0000","last_fullsized":"2024-03-17T23:47:01.944720+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:43:57.153828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943202+0000","last_change":"2024-03-17T23:46:55.929140+0000","last_active":"2024-03-17T23:47:01.943202+0000","last_peered":"2024-03-17T23:47:01.943202+0000","last_clean":"2024-03-17T23:47:01.943202+0000","last_became_active":"2024-03-17T23:46:55.928874+0000","last_became_peered":"2024-03-17T23:46:55.928874+0000","last_unstale":"2024-03-17T23:47:01.943202+0000","last_undegraded":"2024-03-17T23:47:01.943202+0000","last_fullsized":"2024-03-17T23:47:01.943202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:59:32.027430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943228+0000","last_change":"2024-03-17T23:46:54.923319+0000","last_active":"2024-03-17T23:47:01.943228+0000","last_peered":"2024-03-17T23:47:01.943228+0000","last_clean":"2024-03-17T23:47:01.943228+0000","last_became_active":"2024-03-17T23:46:54.923060+0000","last_became_peered":"2024-03-17T23:46:54.923060+0000","last_unstale":"2024-03-17T23:47:01.943228+0000","last_undegraded":"2024-03-17T23:47:01.943228+0000","last_fullsized":"2024-03-17T23:47:01.943228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:45:17.571056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943597+0000","last_change":"2024-03-17T23:46:54.917589+0000","last_active":"2024-03-17T23:47:01.943597+0000","last_peered":"2024-03-17T23:47:01.943597+0000","last_clean":"2024-03-17T23:47:01.943597+0000","last_became_active":"2024-03-17T23:46:54.917183+0000","last_became_peered":"2024-03-17T23:46:54.917183+0000","last_unstale":"2024-03-17T23:47:01.943597+0000","last_undegraded":"2024-03-17T23:47:01.943597+0000","last_fullsized":"2024-03-17T23:47:01.943597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:18.195935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714330+0000","last_change":"2024-03-17T23:46:55.924339+0000","last_active":"2024-03-17T23:47:02.714330+0000","last_peered":"2024-03-17T23:47:02.714330+0000","last_clean":"2024-03-17T23:47:02.714330+0000","last_became_active":"2024-03-17T23:46:55.923891+0000","last_became_peered":"2024-03-17T23:46:55.923891+0000","last_unstale":"2024-03-17T23:47:02.714330+0000","last_undegraded":"2024-03-17T23:47:02.714330+0000","last_fullsized":"2024-03-17T23:47:02.714330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:11:08.829506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713322+0000","last_change":"2024-03-17T23:46:59.959425+0000","last_active":"2024-03-17T23:47:02.713322+0000","last_peered":"2024-03-17T23:47:02.713322+0000","last_clean":"2024-03-17T23:47:02.713322+0000","last_became_active":"2024-03-17T23:46:59.958851+0000","last_became_peered":"2024-03-17T23:46:59.958851+0000","last_unstale":"2024-03-17T23:47:02.713322+0000","last_undegraded":"2024-03-17T23:47:02.713322+0000","last_fullsized":"2024-03-17T23:47:02.713322+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:58.379120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943490+0000","last_change":"2024-03-17T23:46:54.921325+0000","last_active":"2024-03-17T23:47:01.943490+0000","last_peered":"2024-03-17T23:47:01.943490+0000","last_clean":"2024-03-17T23:47:01.943490+0000","last_became_active":"2024-03-17T23:46:54.921093+0000","last_became_peered":"2024-03-17T23:46:54.921093+0000","last_unstale":"2024-03-17T23:47:01.943490+0000","last_undegraded":"2024-03-17T23:47:01.943490+0000","last_fullsized":"2024-03-17T23:47:01.943490+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:56.665566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106239+0000","last_change":"2024-03-17T23:46:55.922093+0000","last_active":"2024-03-17T23:47:02.106239+0000","last_peered":"2024-03-17T23:47:02.106239+0000","last_clean":"2024-03-17T23:47:02.106239+0000","last_became_active":"2024-03-17T23:46:55.919778+0000","last_became_peered":"2024-03-17T23:46:55.919778+0000","last_unstale":"2024-03-17T23:47:02.106239+0000","last_undegraded":"2024-03-17T23:47:02.106239+0000","last_fullsized":"2024-03-17T23:47:02.106239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:06:55.437268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943978+0000","last_change":"2024-03-17T23:46:59.964222+0000","last_active":"2024-03-17T23:47:01.943978+0000","last_peered":"2024-03-17T23:47:01.943978+0000","last_clean":"2024-03-17T23:47:01.943978+0000","last_became_active":"2024-03-17T23:46:59.963807+0000","last_became_peered":"2024-03-17T23:46:59.963807+0000","last_unstale":"2024-03-17T23:47:01.943978+0000","last_undegraded":"2024-03-17T23:47:01.943978+0000","last_fullsized":"2024-03-17T23:47:01.943978+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:25:11.791005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.942962+0000","last_change":"2024-03-17T23:46:55.926785+0000","last_active":"2024-03-17T23:47:01.942962+0000","last_peered":"2024-03-17T23:47:01.942962+0000","last_clean":"2024-03-17T23:47:01.942962+0000","last_became_active":"2024-03-17T23:46:55.926494+0000","last_became_peered":"2024-03-17T23:46:55.926494+0000","last_unstale":"2024-03-17T23:47:01.942962+0000","last_undegraded":"2024-03-17T23:47:01.942962+0000","last_fullsized":"2024-03-17T23:47:01.942962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:24.419077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713141+0000","last_change":"2024-03-17T23:46:55.930094+0000","last_active":"2024-03-17T23:47:02.713141+0000","last_peered":"2024-03-17T23:47:02.713141+0000","last_clean":"2024-03-17T23:47:02.713141+0000","last_became_active":"2024-03-17T23:46:55.929834+0000","last_became_peered":"2024-03-17T23:46:55.929834+0000","last_unstale":"2024-03-17T23:47:02.713141+0000","last_undegraded":"2024-03-17T23:47:02.713141+0000","last_fullsized":"2024-03-17T23:47:02.713141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:59.421169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943791+0000","last_change":"2024-03-17T23:46:55.934265+0000","last_active":"2024-03-17T23:47:01.943791+0000","last_peered":"2024-03-17T23:47:01.943791+0000","last_clean":"2024-03-17T23:47:01.943791+0000","last_became_active":"2024-03-17T23:46:55.933735+0000","last_became_peered":"2024-03-17T23:46:55.933735+0000","last_unstale":"2024-03-17T23:47:01.943791+0000","last_undegraded":"2024-03-17T23:47:01.943791+0000","last_fullsized":"2024-03-17T23:47:01.943791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:59:32.027430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714250+0000","last_change":"2024-03-17T23:46:55.930627+0000","last_active":"2024-03-17T23:47:02.714250+0000","last_peered":"2024-03-17T23:47:02.714250+0000","last_clean":"2024-03-17T23:47:02.714250+0000","last_became_active":"2024-03-17T23:46:55.930368+0000","last_became_peered":"2024-03-17T23:46:55.930368+0000","last_unstale":"2024-03-17T23:47:02.714250+0000","last_undegraded":"2024-03-17T23:47:02.714250+0000","last_fullsized":"2024-03-17T23:47:02.714250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:47:30.023063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943914+0000","last_change":"2024-03-17T23:46:55.930544+0000","last_active":"2024-03-17T23:47:01.943914+0000","last_peered":"2024-03-17T23:47:01.943914+0000","last_clean":"2024-03-17T23:47:01.943914+0000","last_became_active":"2024-03-17T23:46:55.930277+0000","last_became_peered":"2024-03-17T23:46:55.930277+0000","last_unstale":"2024-03-17T23:47:01.943914+0000","last_undegraded":"2024-03-17T23:47:01.943914+0000","last_fullsized":"2024-03-17T23:47:01.943914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:23:58.221114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713632+0000","last_change":"2024-03-17T23:46:55.927054+0000","last_active":"2024-03-17T23:47:02.713632+0000","last_peered":"2024-03-17T23:47:02.713632+0000","last_clean":"2024-03-17T23:47:02.713632+0000","last_became_active":"2024-03-17T23:46:55.926762+0000","last_became_peered":"2024-03-17T23:46:55.926762+0000","last_unstale":"2024-03-17T23:47:02.713632+0000","last_undegraded":"2024-03-17T23:47:02.713632+0000","last_fullsized":"2024-03-17T23:47:02.713632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:39.972607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944787+0000","last_change":"2024-03-17T23:46:55.929740+0000","last_active":"2024-03-17T23:47:01.944787+0000","last_peered":"2024-03-17T23:47:01.944787+0000","last_clean":"2024-03-17T23:47:01.944787+0000","last_became_active":"2024-03-17T23:46:55.929433+0000","last_became_peered":"2024-03-17T23:46:55.929433+0000","last_unstale":"2024-03-17T23:47:01.944787+0000","last_undegraded":"2024-03-17T23:47:01.944787+0000","last_fullsized":"2024-03-17T23:47:01.944787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:11:08.829506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943753+0000","last_change":"2024-03-17T23:46:55.930851+0000","last_active":"2024-03-17T23:47:01.943753+0000","last_peered":"2024-03-17T23:47:01.943753+0000","last_clean":"2024-03-17T23:47:01.943753+0000","last_became_active":"2024-03-17T23:46:55.930305+0000","last_became_peered":"2024-03-17T23:46:55.930305+0000","last_unstale":"2024-03-17T23:47:01.943753+0000","last_undegraded":"2024-03-17T23:47:01.943753+0000","last_fullsized":"2024-03-17T23:47:01.943753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:47:30.023063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943134+0000","last_change":"2024-03-17T23:46:55.933139+0000","last_active":"2024-03-17T23:47:01.943134+0000","last_peered":"2024-03-17T23:47:01.943134+0000","last_clean":"2024-03-17T23:47:01.943134+0000","last_became_active":"2024-03-17T23:46:55.932843+0000","last_became_peered":"2024-03-17T23:46:55.932843+0000","last_unstale":"2024-03-17T23:47:01.943134+0000","last_undegraded":"2024-03-17T23:47:01.943134+0000","last_fullsized":"2024-03-17T23:47:01.943134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:44:47.002489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943147+0000","last_change":"2024-03-17T23:46:55.930881+0000","last_active":"2024-03-17T23:47:01.943147+0000","last_peered":"2024-03-17T23:47:01.943147+0000","last_clean":"2024-03-17T23:47:01.943147+0000","last_became_active":"2024-03-17T23:46:55.930629+0000","last_became_peered":"2024-03-17T23:46:55.930629+0000","last_unstale":"2024-03-17T23:47:01.943147+0000","last_undegraded":"2024-03-17T23:47:01.943147+0000","last_fullsized":"2024-03-17T23:47:01.943147+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:49:47.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943696+0000","last_change":"2024-03-17T23:46:55.936236+0000","last_active":"2024-03-17T23:47:01.943696+0000","last_peered":"2024-03-17T23:47:01.943696+0000","last_clean":"2024-03-17T23:47:01.943696+0000","last_became_active":"2024-03-17T23:46:55.933485+0000","last_became_peered":"2024-03-17T23:46:55.933485+0000","last_unstale":"2024-03-17T23:47:01.943696+0000","last_undegraded":"2024-03-17T23:47:01.943696+0000","last_fullsized":"2024-03-17T23:47:01.943696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:11:08.829506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714118+0000","last_change":"2024-03-17T23:46:55.930067+0000","last_active":"2024-03-17T23:47:02.714118+0000","last_peered":"2024-03-17T23:47:02.714118+0000","last_clean":"2024-03-17T23:47:02.714118+0000","last_became_active":"2024-03-17T23:46:55.929808+0000","last_became_peered":"2024-03-17T23:46:55.929808+0000","last_unstale":"2024-03-17T23:47:02.714118+0000","last_undegraded":"2024-03-17T23:47:02.714118+0000","last_fullsized":"2024-03-17T23:47:02.714118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:49:47.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943993+0000","last_change":"2024-03-17T23:46:55.917074+0000","last_active":"2024-03-17T23:47:01.943993+0000","last_peered":"2024-03-17T23:47:01.943993+0000","last_clean":"2024-03-17T23:47:01.943993+0000","last_became_active":"2024-03-17T23:46:55.916784+0000","last_became_peered":"2024-03-17T23:46:55.916784+0000","last_unstale":"2024-03-17T23:47:01.943993+0000","last_undegraded":"2024-03-17T23:47:01.943993+0000","last_fullsized":"2024-03-17T23:47:01.943993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:49:47.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713939+0000","last_change":"2024-03-17T23:46:55.928539+0000","last_active":"2024-03-17T23:47:02.713939+0000","last_peered":"2024-03-17T23:47:02.713939+0000","last_clean":"2024-03-17T23:47:02.713939+0000","last_became_active":"2024-03-17T23:46:55.928275+0000","last_became_peered":"2024-03-17T23:46:55.928275+0000","last_unstale":"2024-03-17T23:47:02.713939+0000","last_undegraded":"2024-03-17T23:47:02.713939+0000","last_fullsized":"2024-03-17T23:47:02.713939+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T23:52:25.543746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713273+0000","last_change":"2024-03-17T23:46:55.924366+0000","last_active":"2024-03-17T23:47:02.713273+0000","last_peered":"2024-03-17T23:47:02.713273+0000","last_clean":"2024-03-17T23:47:02.713273+0000","last_became_active":"2024-03-17T23:46:55.924100+0000","last_became_peered":"2024-03-17T23:46:55.924100+0000","last_unstale":"2024-03-17T23:47:02.713273+0000","last_undegraded":"2024-03-17T23:47:02.713273+0000","last_fullsized":"2024-03-17T23:47:02.713273+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:51:19.549589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944530+0000","last_change":"2024-03-17T23:46:55.932514+0000","last_active":"2024-03-17T23:47:01.944530+0000","last_peered":"2024-03-17T23:47:01.944530+0000","last_clean":"2024-03-17T23:47:01.944530+0000","last_became_active":"2024-03-17T23:46:55.930899+0000","last_became_peered":"2024-03-17T23:46:55.930899+0000","last_unstale":"2024-03-17T23:47:01.944530+0000","last_undegraded":"2024-03-17T23:47:01.944530+0000","last_fullsized":"2024-03-17T23:47:01.944530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:47:30.023063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943088+0000","last_change":"2024-03-17T23:46:55.940176+0000","last_active":"2024-03-17T23:47:01.943088+0000","last_peered":"2024-03-17T23:47:01.943088+0000","last_clean":"2024-03-17T23:47:01.943088+0000","last_became_active":"2024-03-17T23:46:55.939853+0000","last_became_peered":"2024-03-17T23:46:55.939853+0000","last_unstale":"2024-03-17T23:47:01.943088+0000","last_undegraded":"2024-03-17T23:47:01.943088+0000","last_fullsized":"2024-03-17T23:47:01.943088+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:49:47.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713396+0000","last_change":"2024-03-17T23:46:55.917430+0000","last_active":"2024-03-17T23:47:02.713396+0000","last_peered":"2024-03-17T23:47:02.713396+0000","last_clean":"2024-03-17T23:47:02.713396+0000","last_became_active":"2024-03-17T23:46:55.917173+0000","last_became_peered":"2024-03-17T23:46:55.917173+0000","last_unstale":"2024-03-17T23:47:02.713396+0000","last_undegraded":"2024-03-17T23:47:02.713396+0000","last_fullsized":"2024-03-17T23:47:02.713396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:23:58.221114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714399+0000","last_change":"2024-03-17T23:46:55.927198+0000","last_active":"2024-03-17T23:47:02.714399+0000","last_peered":"2024-03-17T23:47:02.714399+0000","last_clean":"2024-03-17T23:47:02.714399+0000","last_became_active":"2024-03-17T23:46:55.926919+0000","last_became_peered":"2024-03-17T23:46:55.926919+0000","last_unstale":"2024-03-17T23:47:02.714399+0000","last_undegraded":"2024-03-17T23:47:02.714399+0000","last_fullsized":"2024-03-17T23:47:02.714399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:24.419077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943395+0000","last_change":"2024-03-17T23:46:55.932835+0000","last_active":"2024-03-17T23:47:01.943395+0000","last_peered":"2024-03-17T23:47:01.943395+0000","last_clean":"2024-03-17T23:47:01.943395+0000","last_became_active":"2024-03-17T23:46:55.932339+0000","last_became_peered":"2024-03-17T23:46:55.932339+0000","last_unstale":"2024-03-17T23:47:01.943395+0000","last_undegraded":"2024-03-17T23:47:01.943395+0000","last_fullsized":"2024-03-17T23:47:01.943395+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:39.972607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106535+0000","last_change":"2024-03-17T23:46:55.916730+0000","last_active":"2024-03-17T23:47:02.106535+0000","last_peered":"2024-03-17T23:47:02.106535+0000","last_clean":"2024-03-17T23:47:02.106535+0000","last_became_active":"2024-03-17T23:46:55.916438+0000","last_became_peered":"2024-03-17T23:46:55.916438+0000","last_unstale":"2024-03-17T23:47:02.106535+0000","last_undegraded":"2024-03-17T23:47:02.106535+0000","last_fullsized":"2024-03-17T23:47:02.106535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:51:19.549589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944384+0000","last_change":"2024-03-17T23:46:55.932468+0000","last_active":"2024-03-17T23:47:01.944384+0000","last_peered":"2024-03-17T23:47:01.944384+0000","last_clean":"2024-03-17T23:47:01.944384+0000","last_became_active":"2024-03-17T23:46:55.932210+0000","last_became_peered":"2024-03-17T23:46:55.932210+0000","last_unstale":"2024-03-17T23:47:01.944384+0000","last_undegraded":"2024-03-17T23:47:01.944384+0000","last_fullsized":"2024-03-17T23:47:01.944384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:23:58.221114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944784+0000","last_change":"2024-03-17T23:46:55.934093+0000","last_active":"2024-03-17T23:47:01.944784+0000","last_peered":"2024-03-17T23:47:01.944784+0000","last_clean":"2024-03-17T23:47:01.944784+0000","last_became_active":"2024-03-17T23:46:55.933516+0000","last_became_peered":"2024-03-17T23:46:55.933516+0000","last_unstale":"2024-03-17T23:47:01.944784+0000","last_undegraded":"2024-03-17T23:47:01.944784+0000","last_fullsized":"2024-03-17T23:47:01.944784+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:47:30.023063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713902+0000","last_change":"2024-03-17T23:46:55.932426+0000","last_active":"2024-03-17T23:47:02.713902+0000","last_peered":"2024-03-17T23:47:02.713902+0000","last_clean":"2024-03-17T23:47:02.713902+0000","last_became_active":"2024-03-17T23:46:55.932045+0000","last_became_peered":"2024-03-17T23:46:55.932045+0000","last_unstale":"2024-03-17T23:47:02.713902+0000","last_undegraded":"2024-03-17T23:47:02.713902+0000","last_fullsized":"2024-03-17T23:47:02.713902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:20:21.507725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.942978+0000","last_change":"2024-03-17T23:46:55.932245+0000","last_active":"2024-03-17T23:47:01.942978+0000","last_peered":"2024-03-17T23:47:01.942978+0000","last_clean":"2024-03-17T23:47:01.942978+0000","last_became_active":"2024-03-17T23:46:55.932001+0000","last_became_peered":"2024-03-17T23:46:55.932001+0000","last_unstale":"2024-03-17T23:47:01.942978+0000","last_undegraded":"2024-03-17T23:47:01.942978+0000","last_fullsized":"2024-03-17T23:47:01.942978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:44:47.002489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713629+0000","last_change":"2024-03-17T23:46:55.935868+0000","last_active":"2024-03-17T23:47:02.713629+0000","last_peered":"2024-03-17T23:47:02.713629+0000","last_clean":"2024-03-17T23:47:02.713629+0000","last_became_active":"2024-03-17T23:46:55.935180+0000","last_became_peered":"2024-03-17T23:46:55.935180+0000","last_unstale":"2024-03-17T23:47:02.713629+0000","last_undegraded":"2024-03-17T23:47:02.713629+0000","last_fullsized":"2024-03-17T23:47:02.713629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:44:47.002489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943696+0000","last_change":"2024-03-17T23:46:55.936947+0000","last_active":"2024-03-17T23:47:01.943696+0000","last_peered":"2024-03-17T23:47:01.943696+0000","last_clean":"2024-03-17T23:47:01.943696+0000","last_became_active":"2024-03-17T23:46:55.936694+0000","last_became_peered":"2024-03-17T23:46:55.936694+0000","last_unstale":"2024-03-17T23:47:01.943696+0000","last_undegraded":"2024-03-17T23:47:01.943696+0000","last_fullsized":"2024-03-17T23:47:01.943696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:39.972607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713525+0000","last_change":"2024-03-17T23:46:55.927487+0000","last_active":"2024-03-17T23:47:02.713525+0000","last_peered":"2024-03-17T23:47:02.713525+0000","last_clean":"2024-03-17T23:47:02.713525+0000","last_became_active":"2024-03-17T23:46:55.927225+0000","last_became_peered":"2024-03-17T23:46:55.927225+0000","last_unstale":"2024-03-17T23:47:02.713525+0000","last_undegraded":"2024-03-17T23:47:02.713525+0000","last_fullsized":"2024-03-17T23:47:02.713525+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:22:12.841390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943865+0000","last_change":"2024-03-17T23:46:55.933146+0000","last_active":"2024-03-17T23:47:01.943865+0000","last_peered":"2024-03-17T23:47:01.943865+0000","last_clean":"2024-03-17T23:47:01.943865+0000","last_became_active":"2024-03-17T23:46:55.932861+0000","last_became_peered":"2024-03-17T23:46:55.932861+0000","last_unstale":"2024-03-17T23:47:01.943865+0000","last_undegraded":"2024-03-17T23:47:01.943865+0000","last_fullsized":"2024-03-17T23:47:01.943865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:22:12.841390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713455+0000","last_change":"2024-03-17T23:46:55.924118+0000","last_active":"2024-03-17T23:47:02.713455+0000","last_peered":"2024-03-17T23:47:02.713455+0000","last_clean":"2024-03-17T23:47:02.713455+0000","last_became_active":"2024-03-17T23:46:55.923844+0000","last_became_peered":"2024-03-17T23:46:55.923844+0000","last_unstale":"2024-03-17T23:47:02.713455+0000","last_undegraded":"2024-03-17T23:47:02.713455+0000","last_fullsized":"2024-03-17T23:47:02.713455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:30:49.518932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713741+0000","last_change":"2024-03-17T23:46:55.925191+0000","last_active":"2024-03-17T23:47:02.713741+0000","last_peered":"2024-03-17T23:47:02.713741+0000","last_clean":"2024-03-17T23:47:02.713741+0000","last_became_active":"2024-03-17T23:46:55.924822+0000","last_became_peered":"2024-03-17T23:46:55.924822+0000","last_unstale":"2024-03-17T23:47:02.713741+0000","last_undegraded":"2024-03-17T23:47:02.713741+0000","last_fullsized":"2024-03-17T23:47:02.713741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:45:20.327436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713837+0000","last_change":"2024-03-17T23:46:55.928943+0000","last_active":"2024-03-17T23:47:02.713837+0000","last_peered":"2024-03-17T23:47:02.713837+0000","last_clean":"2024-03-17T23:47:02.713837+0000","last_became_active":"2024-03-17T23:46:55.928669+0000","last_became_peered":"2024-03-17T23:46:55.928669+0000","last_unstale":"2024-03-17T23:47:02.713837+0000","last_undegraded":"2024-03-17T23:47:02.713837+0000","last_fullsized":"2024-03-17T23:47:02.713837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:52:45.163567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":14,"seq":60129542154,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":286720,"data_stored":115066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542154,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":111224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542154,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":880640,"data_stored":702230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":872448,"data_stored":702174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31456,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340384,"statfs":{"total":96636764160,"available":96604553216,"internally_reserved":0,"allocated":286720,"data_stored":115066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":278528,"data_stored":111314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":286720,"data_stored":111350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":111384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-17T23:47:17.425 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-17T23:47:17.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.560+0000 7fd826cd1640 1 Processor -- start 2024-03-17T23:47:17.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.560+0000 7fd826cd1640 1 -- start start 2024-03-17T23:47:17.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.560+0000 7fd826cd1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd8200cd420 0x7fd820165d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:17.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.560+0000 7fd826cd1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8200cf3b0 con 0x7fd820155320 2024-03-17T23:47:17.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.560+0000 7fd826cd1640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8200cf690 con 0x7fd82015bfd0 2024-03-17T23:47:17.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.561+0000 7fd826cd1640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fd82015c550 con 0x7fd8200cd420 2024-03-17T23:47:17.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.561+0000 7fd82526e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd8200cd420 0x7fd820165d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:17.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.561+0000 7fd82526e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd8200cd420 0x7fd820165d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40362/0 (socket says 172.21.15.150:40362) 2024-03-17T23:47:17.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.561+0000 7fd824a6d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd820155320 0x7fd82006dd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34730/0 (socket says 172.21.15.150:34730) 2024-03-17T23:47:17.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.561+0000 7fd82526e640 1 -- 172.21.15.150:0/3747992630 learned_addr learned my addr 172.21.15.150:0/3747992630 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:17.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.561+0000 7fd82526e640 1 -- 172.21.15.150:0/3747992630 >> v1:172.21.15.157:6789/0 conn(0x7fd82015bfd0 legacy=0x7fd8200ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:17.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.561+0000 7fd81f7fe640 1 -- 172.21.15.150:0/3747992630 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3428571792 0 0) 0x7fd8200cf690 con 0x7fd82015bfd0 2024-03-17T23:47:17.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.561+0000 7fd82526e640 1 -- 172.21.15.150:0/3747992630 >> v1:172.21.15.150:6789/0 conn(0x7fd820155320 legacy=0x7fd82006dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:17.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.561+0000 7fd82526e640 1 -- 172.21.15.150:0/3747992630 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd82015c7a0 con 0x7fd8200cd420 2024-03-17T23:47:17.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.562+0000 7fd82526e640 1 --2- 172.21.15.150:0/3747992630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8200cd420 0x7fd820165d10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd810009870 tx=0x7fd81002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb2875342daf4061 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:17.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.562+0000 7fd81f7fe640 1 -- 172.21.15.150:0/3747992630 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd810038070 con 0x7fd8200cd420 2024-03-17T23:47:17.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.562+0000 7fd81f7fe640 1 -- 172.21.15.150:0/3747992630 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd810031740 con 0x7fd8200cd420 2024-03-17T23:47:17.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.562+0000 7fd81f7fe640 1 -- 172.21.15.150:0/3747992630 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81002f050 con 0x7fd8200cd420 2024-03-17T23:47:17.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.562+0000 7fd826cd1640 1 -- 172.21.15.150:0/3747992630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8200cd420 msgr2=0x7fd820165d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:17.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.562+0000 7fd826cd1640 1 --2- 172.21.15.150:0/3747992630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8200cd420 0x7fd820165d10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd810009870 tx=0x7fd81002b860 comp rx=0 tx=0).stop 2024-03-17T23:47:17.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.562+0000 7fd826cd1640 1 -- 172.21.15.150:0/3747992630 shutdown_connections 2024-03-17T23:47:17.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.562+0000 7fd826cd1640 1 --2- 172.21.15.150:0/3747992630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8200cd420 0x7fd820165d10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.562+0000 7fd826cd1640 1 -- 172.21.15.150:0/3747992630 >> 172.21.15.150:0/3747992630 conn(0x7fd82006b190 msgr2=0x7fd82006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:17.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.562+0000 7fd826cd1640 1 -- 172.21.15.150:0/3747992630 shutdown_connections 2024-03-17T23:47:17.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.562+0000 7fd826cd1640 1 -- 172.21.15.150:0/3747992630 wait complete. 2024-03-17T23:47:17.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.564+0000 7fd826cd1640 1 Processor -- start 2024-03-17T23:47:17.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.564+0000 7fd826cd1640 1 -- start start 2024-03-17T23:47:17.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.564+0000 7fd826cd1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8200cd420 0x7fd82014a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:17.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.564+0000 7fd826cd1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd820155320 0x7fd8200b3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:17.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.564+0000 7fd826cd1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd82015bfd0 0x7fd820158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:17.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.564+0000 7fd826cd1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd820155880 con 0x7fd820155320 2024-03-17T23:47:17.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.564+0000 7fd826cd1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd820155700 con 0x7fd8200cd420 2024-03-17T23:47:17.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.564+0000 7fd826cd1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd82015dcd0 con 0x7fd82015bfd0 2024-03-17T23:47:17.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.565+0000 7fd81ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd820155320 0x7fd8200b3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:17.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.565+0000 7fd81ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd820155320 0x7fd8200b3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40378/0 (socket says 172.21.15.150:40378) 2024-03-17T23:47:17.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.565+0000 7fd81ffff640 1 -- 172.21.15.150:0/3429316319 learned_addr learned my addr 172.21.15.150:0/3429316319 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:17.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.565+0000 7fd824a6d640 1 --2- 172.21.15.150:0/3429316319 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8200cd420 0x7fd82014a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:17.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.565+0000 7fd82526e640 1 --2- 172.21.15.150:0/3429316319 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd82015bfd0 0x7fd820158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:17.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.565+0000 7fd81ffff640 1 -- 172.21.15.150:0/3429316319 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd82015bfd0 msgr2=0x7fd820158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:17.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.565+0000 7fd81ffff640 1 --2- 172.21.15.150:0/3429316319 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd82015bfd0 0x7fd820158020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.565+0000 7fd81ffff640 1 -- 172.21.15.150:0/3429316319 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8200cd420 msgr2=0x7fd82014a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:17.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.565+0000 7fd81ffff640 1 --2- 172.21.15.150:0/3429316319 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8200cd420 0x7fd82014a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.565+0000 7fd81ffff640 1 -- 172.21.15.150:0/3429316319 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd82014b210 con 0x7fd820155320 2024-03-17T23:47:17.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.565+0000 7fd81ffff640 1 --2- 172.21.15.150:0/3429316319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd820155320 0x7fd8200b3be0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd80800c410 tx=0x7fd80800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:17.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.565+0000 7fd81d7fa640 1 -- 172.21.15.150:0/3429316319 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd808017070 con 0x7fd820155320 2024-03-17T23:47:17.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.565+0000 7fd81d7fa640 1 -- 172.21.15.150:0/3429316319 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8080035f0 con 0x7fd820155320 2024-03-17T23:47:17.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.565+0000 7fd826cd1640 1 -- 172.21.15.150:0/3429316319 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8201f0c00 con 0x7fd820155320 2024-03-17T23:47:17.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.565+0000 7fd81d7fa640 1 -- 172.21.15.150:0/3429316319 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8080039e0 con 0x7fd820155320 2024-03-17T23:47:17.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.565+0000 7fd826cd1640 1 -- 172.21.15.150:0/3429316319 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8201f08c0 con 0x7fd820155320 2024-03-17T23:47:17.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.566+0000 7fd826cd1640 1 -- 172.21.15.150:0/3429316319 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8201f0dd0 con 0x7fd820155320 2024-03-17T23:47:17.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.567+0000 7fd81d7fa640 1 -- 172.21.15.150:0/3429316319 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd80801a070 con 0x7fd820155320 2024-03-17T23:47:17.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.567+0000 7fd81d7fa640 1 --2- 172.21.15.150:0/3429316319 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd80006bbd0 0x7fd80006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:17.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.567+0000 7fd81d7fa640 1 -- 172.21.15.150:0/3429316319 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fd80808a160 con 0x7fd820155320 2024-03-17T23:47:17.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.567+0000 7fd824a6d640 1 --2- 172.21.15.150:0/3429316319 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd80006bbd0 0x7fd80006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:17.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.567+0000 7fd824a6d640 1 --2- 172.21.15.150:0/3429316319 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd80006bbd0 0x7fd80006e090 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd8180012e0 tx=0x7fd818001840 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:17.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.571+0000 7fd81d7fa640 1 -- 172.21.15.150:0/3429316319 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd8080582c0 con 0x7fd820155320 2024-03-17T23:47:17.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.740+0000 7fd826cd1640 1 -- 172.21.15.150:0/3429316319 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd8200681a0 con 0x7fd80006bbd0 2024-03-17T23:47:17.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.747+0000 7fd81d7fa640 1 -- 172.21.15.150:0/3429316319 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7fd8200681a0 con 0x7fd80006bbd0 2024-03-17T23:47:17.752 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:17.754 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-03-17T23:47:17.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.751+0000 7fd826cd1640 1 -- 172.21.15.150:0/3429316319 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd80006bbd0 msgr2=0x7fd80006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:17.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.751+0000 7fd826cd1640 1 --2- 172.21.15.150:0/3429316319 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd80006bbd0 0x7fd80006e090 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd8180012e0 tx=0x7fd818001840 comp rx=0 tx=0).stop 2024-03-17T23:47:17.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.751+0000 7fd826cd1640 1 -- 172.21.15.150:0/3429316319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd820155320 msgr2=0x7fd8200b3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:17.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.751+0000 7fd826cd1640 1 --2- 172.21.15.150:0/3429316319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd820155320 0x7fd8200b3be0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd80800c410 tx=0x7fd80800c8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.751+0000 7fd826cd1640 1 -- 172.21.15.150:0/3429316319 shutdown_connections 2024-03-17T23:47:17.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.751+0000 7fd826cd1640 1 --2- 172.21.15.150:0/3429316319 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd82015bfd0 0x7fd820158020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.751+0000 7fd826cd1640 1 --2- 172.21.15.150:0/3429316319 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd820155320 0x7fd8200b3be0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.751+0000 7fd826cd1640 1 --2- 172.21.15.150:0/3429316319 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd80006bbd0 0x7fd80006e090 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.751+0000 7fd826cd1640 1 --2- 172.21.15.150:0/3429316319 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8200cd420 0x7fd82014a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.751+0000 7fd826cd1640 1 -- 172.21.15.150:0/3429316319 >> 172.21.15.150:0/3429316319 conn(0x7fd82006b190 msgr2=0x7fd8201591e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:17.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.751+0000 7fd826cd1640 1 -- 172.21.15.150:0/3429316319 shutdown_connections 2024-03-17T23:47:17.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.751+0000 7fd826cd1640 1 -- 172.21.15.150:0/3429316319 wait complete. 2024-03-17T23:47:17.762 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-03-17T23:47:17.600928+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":232000,"kb_used_data":3368,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":754742720,"statfs":{"total":773094113280,"available":772856545280,"internally_reserved":0,"allocated":3448832,"data_stored":2079808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":234016304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002935"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713953+0000","last_change":"2024-03-17T23:46:54.923706+0000","last_active":"2024-03-17T23:47:02.713953+0000","last_peered":"2024-03-17T23:47:02.713953+0000","last_clean":"2024-03-17T23:47:02.713953+0000","last_became_active":"2024-03-17T23:46:54.923462+0000","last_became_peered":"2024-03-17T23:46:54.923462+0000","last_unstale":"2024-03-17T23:47:02.713953+0000","last_undegraded":"2024-03-17T23:47:02.713953+0000","last_fullsized":"2024-03-17T23:47:02.713953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:34:48.054365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943580+0000","last_change":"2024-03-17T23:46:55.932962+0000","last_active":"2024-03-17T23:47:01.943580+0000","last_peered":"2024-03-17T23:47:01.943580+0000","last_clean":"2024-03-17T23:47:01.943580+0000","last_became_active":"2024-03-17T23:46:55.932316+0000","last_became_peered":"2024-03-17T23:46:55.932316+0000","last_unstale":"2024-03-17T23:47:01.943580+0000","last_undegraded":"2024-03-17T23:47:01.943580+0000","last_fullsized":"2024-03-17T23:47:01.943580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:45:20.327436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943046+0000","last_change":"2024-03-17T23:47:00.412319+0000","last_active":"2024-03-17T23:47:01.943046+0000","last_peered":"2024-03-17T23:47:01.943046+0000","last_clean":"2024-03-17T23:47:01.943046+0000","last_became_active":"2024-03-17T23:47:00.411544+0000","last_became_peered":"2024-03-17T23:47:00.411544+0000","last_unstale":"2024-03-17T23:47:01.943046+0000","last_undegraded":"2024-03-17T23:47:01.943046+0000","last_fullsized":"2024-03-17T23:47:01.943046+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:24:18.544213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943689+0000","last_change":"2024-03-17T23:46:54.921077+0000","last_active":"2024-03-17T23:47:01.943689+0000","last_peered":"2024-03-17T23:47:01.943689+0000","last_clean":"2024-03-17T23:47:01.943689+0000","last_became_active":"2024-03-17T23:46:54.920797+0000","last_became_peered":"2024-03-17T23:46:54.920797+0000","last_unstale":"2024-03-17T23:47:01.943689+0000","last_undegraded":"2024-03-17T23:47:01.943689+0000","last_fullsized":"2024-03-17T23:47:01.943689+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:18.195935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713708+0000","last_change":"2024-03-17T23:46:55.943217+0000","last_active":"2024-03-17T23:47:02.713708+0000","last_peered":"2024-03-17T23:47:02.713708+0000","last_clean":"2024-03-17T23:47:02.713708+0000","last_became_active":"2024-03-17T23:46:55.942641+0000","last_became_peered":"2024-03-17T23:46:55.942641+0000","last_unstale":"2024-03-17T23:47:02.713708+0000","last_undegraded":"2024-03-17T23:47:02.713708+0000","last_fullsized":"2024-03-17T23:47:02.713708+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:45:20.327436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106123+0000","last_change":"2024-03-17T23:46:59.957747+0000","last_active":"2024-03-17T23:47:02.106123+0000","last_peered":"2024-03-17T23:47:02.106123+0000","last_clean":"2024-03-17T23:47:02.106123+0000","last_became_active":"2024-03-17T23:46:59.957329+0000","last_became_peered":"2024-03-17T23:46:59.957329+0000","last_unstale":"2024-03-17T23:47:02.106123+0000","last_undegraded":"2024-03-17T23:47:02.106123+0000","last_fullsized":"2024-03-17T23:47:02.106123+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:27:51.964285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713914+0000","last_change":"2024-03-17T23:46:54.926580+0000","last_active":"2024-03-17T23:47:02.713914+0000","last_peered":"2024-03-17T23:47:02.713914+0000","last_clean":"2024-03-17T23:47:02.713914+0000","last_became_active":"2024-03-17T23:46:54.926254+0000","last_became_peered":"2024-03-17T23:46:54.926254+0000","last_unstale":"2024-03-17T23:47:02.713914+0000","last_undegraded":"2024-03-17T23:47:02.713914+0000","last_fullsized":"2024-03-17T23:47:02.713914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:11:01.353944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943965+0000","last_change":"2024-03-17T23:46:55.937073+0000","last_active":"2024-03-17T23:47:01.943965+0000","last_peered":"2024-03-17T23:47:01.943965+0000","last_clean":"2024-03-17T23:47:01.943965+0000","last_became_active":"2024-03-17T23:46:55.936068+0000","last_became_peered":"2024-03-17T23:46:55.936068+0000","last_unstale":"2024-03-17T23:47:01.943965+0000","last_undegraded":"2024-03-17T23:47:01.943965+0000","last_fullsized":"2024-03-17T23:47:01.943965+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:45:29.447600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106472+0000","last_change":"2024-03-17T23:46:59.957945+0000","last_active":"2024-03-17T23:47:02.106472+0000","last_peered":"2024-03-17T23:47:02.106472+0000","last_clean":"2024-03-17T23:47:02.106472+0000","last_became_active":"2024-03-17T23:46:59.957324+0000","last_became_peered":"2024-03-17T23:46:59.957324+0000","last_unstale":"2024-03-17T23:47:02.106472+0000","last_undegraded":"2024-03-17T23:47:02.106472+0000","last_fullsized":"2024-03-17T23:47:02.106472+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:52:17.152097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713691+0000","last_change":"2024-03-17T23:46:54.924488+0000","last_active":"2024-03-17T23:47:02.713691+0000","last_peered":"2024-03-17T23:47:02.713691+0000","last_clean":"2024-03-17T23:47:02.713691+0000","last_became_active":"2024-03-17T23:46:54.924170+0000","last_became_peered":"2024-03-17T23:46:54.924170+0000","last_unstale":"2024-03-17T23:47:02.713691+0000","last_undegraded":"2024-03-17T23:47:02.713691+0000","last_fullsized":"2024-03-17T23:47:02.713691+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:06:54.433520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944190+0000","last_change":"2024-03-17T23:46:55.938268+0000","last_active":"2024-03-17T23:47:01.944190+0000","last_peered":"2024-03-17T23:47:01.944190+0000","last_clean":"2024-03-17T23:47:01.944190+0000","last_became_active":"2024-03-17T23:46:55.937957+0000","last_became_peered":"2024-03-17T23:46:55.937957+0000","last_unstale":"2024-03-17T23:47:01.944190+0000","last_undegraded":"2024-03-17T23:47:01.944190+0000","last_fullsized":"2024-03-17T23:47:01.944190+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T23:52:25.543746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106183+0000","last_change":"2024-03-17T23:46:59.959993+0000","last_active":"2024-03-17T23:47:02.106183+0000","last_peered":"2024-03-17T23:47:02.106183+0000","last_clean":"2024-03-17T23:47:02.106183+0000","last_became_active":"2024-03-17T23:46:59.959636+0000","last_became_peered":"2024-03-17T23:46:59.959636+0000","last_unstale":"2024-03-17T23:47:02.106183+0000","last_undegraded":"2024-03-17T23:47:02.106183+0000","last_fullsized":"2024-03-17T23:47:02.106183+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:39:36.583214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943413+0000","last_change":"2024-03-17T23:46:55.917068+0000","last_active":"2024-03-17T23:47:01.943413+0000","last_peered":"2024-03-17T23:47:01.943413+0000","last_clean":"2024-03-17T23:47:01.943413+0000","last_became_active":"2024-03-17T23:46:55.916759+0000","last_became_peered":"2024-03-17T23:46:55.916759+0000","last_unstale":"2024-03-17T23:47:01.943413+0000","last_undegraded":"2024-03-17T23:47:01.943413+0000","last_fullsized":"2024-03-17T23:47:01.943413+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:45:20.327436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713785+0000","last_change":"2024-03-17T23:46:54.925485+0000","last_active":"2024-03-17T23:47:02.713785+0000","last_peered":"2024-03-17T23:47:02.713785+0000","last_clean":"2024-03-17T23:47:02.713785+0000","last_became_active":"2024-03-17T23:46:54.925173+0000","last_became_peered":"2024-03-17T23:46:54.925173+0000","last_unstale":"2024-03-17T23:47:02.713785+0000","last_undegraded":"2024-03-17T23:47:02.713785+0000","last_fullsized":"2024-03-17T23:47:02.713785+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:51:18.545842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106704+0000","last_change":"2024-03-17T23:46:59.961930+0000","last_active":"2024-03-17T23:47:02.106704+0000","last_peered":"2024-03-17T23:47:02.106704+0000","last_clean":"2024-03-17T23:47:02.106704+0000","last_became_active":"2024-03-17T23:46:59.961495+0000","last_became_peered":"2024-03-17T23:46:59.961495+0000","last_unstale":"2024-03-17T23:47:02.106704+0000","last_undegraded":"2024-03-17T23:47:02.106704+0000","last_fullsized":"2024-03-17T23:47:02.106704+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:59:20.183441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943415+0000","last_change":"2024-03-17T23:46:55.927904+0000","last_active":"2024-03-17T23:47:01.943415+0000","last_peered":"2024-03-17T23:47:01.943415+0000","last_clean":"2024-03-17T23:47:01.943415+0000","last_became_active":"2024-03-17T23:46:55.927637+0000","last_became_peered":"2024-03-17T23:46:55.927637+0000","last_unstale":"2024-03-17T23:47:01.943415+0000","last_undegraded":"2024-03-17T23:47:01.943415+0000","last_fullsized":"2024-03-17T23:47:01.943415+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:03.353861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714207+0000","last_change":"2024-03-17T23:46:54.920822+0000","last_active":"2024-03-17T23:47:02.714207+0000","last_peered":"2024-03-17T23:47:02.714207+0000","last_clean":"2024-03-17T23:47:02.714207+0000","last_became_active":"2024-03-17T23:46:54.920534+0000","last_became_peered":"2024-03-17T23:46:54.920534+0000","last_unstale":"2024-03-17T23:47:02.714207+0000","last_undegraded":"2024-03-17T23:47:02.714207+0000","last_fullsized":"2024-03-17T23:47:02.714207+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:56.665566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106433+0000","last_change":"2024-03-17T23:46:59.961245+0000","last_active":"2024-03-17T23:47:02.106433+0000","last_peered":"2024-03-17T23:47:02.106433+0000","last_clean":"2024-03-17T23:47:02.106433+0000","last_became_active":"2024-03-17T23:46:59.960895+0000","last_became_peered":"2024-03-17T23:46:59.960895+0000","last_unstale":"2024-03-17T23:47:02.106433+0000","last_undegraded":"2024-03-17T23:47:02.106433+0000","last_fullsized":"2024-03-17T23:47:02.106433+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:57.296991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943288+0000","last_change":"2024-03-17T23:46:55.924013+0000","last_active":"2024-03-17T23:47:01.943288+0000","last_peered":"2024-03-17T23:47:01.943288+0000","last_clean":"2024-03-17T23:47:01.943288+0000","last_became_active":"2024-03-17T23:46:55.923741+0000","last_became_peered":"2024-03-17T23:46:55.923741+0000","last_unstale":"2024-03-17T23:47:01.943288+0000","last_undegraded":"2024-03-17T23:47:01.943288+0000","last_fullsized":"2024-03-17T23:47:01.943288+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:00.571611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713487+0000","last_change":"2024-03-17T23:46:54.918712+0000","last_active":"2024-03-17T23:47:02.713487+0000","last_peered":"2024-03-17T23:47:02.713487+0000","last_clean":"2024-03-17T23:47:02.713487+0000","last_became_active":"2024-03-17T23:46:54.918470+0000","last_became_peered":"2024-03-17T23:46:54.918470+0000","last_unstale":"2024-03-17T23:47:02.713487+0000","last_undegraded":"2024-03-17T23:47:02.713487+0000","last_fullsized":"2024-03-17T23:47:02.713487+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:18.195935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.715105+0000","last_change":"2024-03-17T23:47:00.410049+0000","last_active":"2024-03-17T23:47:02.715105+0000","last_peered":"2024-03-17T23:47:02.715105+0000","last_clean":"2024-03-17T23:47:02.715105+0000","last_became_active":"2024-03-17T23:47:00.409688+0000","last_became_peered":"2024-03-17T23:47:00.409688+0000","last_unstale":"2024-03-17T23:47:02.715105+0000","last_undegraded":"2024-03-17T23:47:02.715105+0000","last_fullsized":"2024-03-17T23:47:02.715105+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:39:36.583214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106313+0000","last_change":"2024-03-17T23:46:55.925699+0000","last_active":"2024-03-17T23:47:02.106313+0000","last_peered":"2024-03-17T23:47:02.106313+0000","last_clean":"2024-03-17T23:47:02.106313+0000","last_became_active":"2024-03-17T23:46:55.925467+0000","last_became_peered":"2024-03-17T23:46:55.925467+0000","last_unstale":"2024-03-17T23:47:02.106313+0000","last_undegraded":"2024-03-17T23:47:02.106313+0000","last_fullsized":"2024-03-17T23:47:02.106313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:11:02.357691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713372+0000","last_change":"2024-03-17T23:46:54.923992+0000","last_active":"2024-03-17T23:47:02.713372+0000","last_peered":"2024-03-17T23:47:02.713372+0000","last_clean":"2024-03-17T23:47:02.713372+0000","last_became_active":"2024-03-17T23:46:54.923721+0000","last_became_peered":"2024-03-17T23:46:54.923721+0000","last_unstale":"2024-03-17T23:47:02.713372+0000","last_undegraded":"2024-03-17T23:47:02.713372+0000","last_fullsized":"2024-03-17T23:47:02.713372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:46:16.782215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106661+0000","last_change":"2024-03-17T23:46:59.960940+0000","last_active":"2024-03-17T23:47:02.106661+0000","last_peered":"2024-03-17T23:47:02.106661+0000","last_clean":"2024-03-17T23:47:02.106661+0000","last_became_active":"2024-03-17T23:46:59.960472+0000","last_became_peered":"2024-03-17T23:46:59.960472+0000","last_unstale":"2024-03-17T23:47:02.106661+0000","last_undegraded":"2024-03-17T23:47:02.106661+0000","last_fullsized":"2024-03-17T23:47:02.106661+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:21:41.104850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714557+0000","last_change":"2024-03-17T23:46:55.927164+0000","last_active":"2024-03-17T23:47:02.714557+0000","last_peered":"2024-03-17T23:47:02.714557+0000","last_clean":"2024-03-17T23:47:02.714557+0000","last_became_active":"2024-03-17T23:46:55.926883+0000","last_became_peered":"2024-03-17T23:46:55.926883+0000","last_unstale":"2024-03-17T23:47:02.714557+0000","last_undegraded":"2024-03-17T23:47:02.714557+0000","last_fullsized":"2024-03-17T23:47:02.714557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:03.353861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944736+0000","last_change":"2024-03-17T23:46:54.921997+0000","last_active":"2024-03-17T23:47:01.944736+0000","last_peered":"2024-03-17T23:47:01.944736+0000","last_clean":"2024-03-17T23:47:01.944736+0000","last_became_active":"2024-03-17T23:46:54.921701+0000","last_became_peered":"2024-03-17T23:46:54.921701+0000","last_unstale":"2024-03-17T23:47:01.944736+0000","last_undegraded":"2024-03-17T23:47:01.944736+0000","last_fullsized":"2024-03-17T23:47:01.944736+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:11:01.353944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106094+0000","last_change":"2024-03-17T23:47:00.410238+0000","last_active":"2024-03-17T23:47:02.106094+0000","last_peered":"2024-03-17T23:47:02.106094+0000","last_clean":"2024-03-17T23:47:02.106094+0000","last_became_active":"2024-03-17T23:47:00.409845+0000","last_became_peered":"2024-03-17T23:47:00.409845+0000","last_unstale":"2024-03-17T23:47:02.106094+0000","last_undegraded":"2024-03-17T23:47:02.106094+0000","last_fullsized":"2024-03-17T23:47:02.106094+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:43:57.153828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943841+0000","last_change":"2024-03-17T23:46:55.929991+0000","last_active":"2024-03-17T23:47:01.943841+0000","last_peered":"2024-03-17T23:47:01.943841+0000","last_clean":"2024-03-17T23:47:01.943841+0000","last_became_active":"2024-03-17T23:46:55.929722+0000","last_became_peered":"2024-03-17T23:46:55.929722+0000","last_unstale":"2024-03-17T23:47:01.943841+0000","last_undegraded":"2024-03-17T23:47:01.943841+0000","last_fullsized":"2024-03-17T23:47:01.943841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:11:08.829506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713963+0000","last_change":"2024-03-17T23:46:54.913493+0000","last_active":"2024-03-17T23:47:02.713963+0000","last_peered":"2024-03-17T23:47:02.713963+0000","last_clean":"2024-03-17T23:47:02.713963+0000","last_became_active":"2024-03-17T23:46:54.913233+0000","last_became_peered":"2024-03-17T23:46:54.913233+0000","last_unstale":"2024-03-17T23:47:02.713963+0000","last_undegraded":"2024-03-17T23:47:02.713963+0000","last_fullsized":"2024-03-17T23:47:02.713963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:58.417421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944060+0000","last_change":"2024-03-17T23:47:00.408616+0000","last_active":"2024-03-17T23:47:01.944060+0000","last_peered":"2024-03-17T23:47:01.944060+0000","last_clean":"2024-03-17T23:47:01.944060+0000","last_became_active":"2024-03-17T23:47:00.408251+0000","last_became_peered":"2024-03-17T23:47:00.408251+0000","last_unstale":"2024-03-17T23:47:01.944060+0000","last_undegraded":"2024-03-17T23:47:01.944060+0000","last_fullsized":"2024-03-17T23:47:01.944060+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:16:52.426485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714810+0000","last_change":"2024-03-17T23:46:55.921416+0000","last_active":"2024-03-17T23:47:02.714810+0000","last_peered":"2024-03-17T23:47:02.714810+0000","last_clean":"2024-03-17T23:47:02.714810+0000","last_became_active":"2024-03-17T23:46:55.921128+0000","last_became_peered":"2024-03-17T23:46:55.921128+0000","last_unstale":"2024-03-17T23:47:02.714810+0000","last_undegraded":"2024-03-17T23:47:02.714810+0000","last_fullsized":"2024-03-17T23:47:02.714810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:49:47.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714025+0000","last_change":"2024-03-17T23:46:54.926748+0000","last_active":"2024-03-17T23:47:02.714025+0000","last_peered":"2024-03-17T23:47:02.714025+0000","last_clean":"2024-03-17T23:47:02.714025+0000","last_became_active":"2024-03-17T23:46:54.926467+0000","last_became_peered":"2024-03-17T23:46:54.926467+0000","last_unstale":"2024-03-17T23:47:02.714025+0000","last_undegraded":"2024-03-17T23:47:02.714025+0000","last_fullsized":"2024-03-17T23:47:02.714025+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:58.417421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714085+0000","last_change":"2024-03-17T23:47:00.409306+0000","last_active":"2024-03-17T23:47:02.714085+0000","last_peered":"2024-03-17T23:47:02.714085+0000","last_clean":"2024-03-17T23:47:02.714085+0000","last_became_active":"2024-03-17T23:47:00.408934+0000","last_became_peered":"2024-03-17T23:47:00.408934+0000","last_unstale":"2024-03-17T23:47:02.714085+0000","last_undegraded":"2024-03-17T23:47:02.714085+0000","last_fullsized":"2024-03-17T23:47:02.714085+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:35:49.838832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944302+0000","last_change":"2024-03-17T23:46:55.933418+0000","last_active":"2024-03-17T23:47:01.944302+0000","last_peered":"2024-03-17T23:47:01.944302+0000","last_clean":"2024-03-17T23:47:01.944302+0000","last_became_active":"2024-03-17T23:46:55.931133+0000","last_became_peered":"2024-03-17T23:46:55.931133+0000","last_unstale":"2024-03-17T23:47:01.944302+0000","last_undegraded":"2024-03-17T23:47:01.944302+0000","last_fullsized":"2024-03-17T23:47:01.944302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:32:27.349125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714826+0000","last_change":"2024-03-17T23:46:54.914746+0000","last_active":"2024-03-17T23:47:02.714826+0000","last_peered":"2024-03-17T23:47:02.714826+0000","last_clean":"2024-03-17T23:47:02.714826+0000","last_became_active":"2024-03-17T23:46:54.914482+0000","last_became_peered":"2024-03-17T23:46:54.914482+0000","last_unstale":"2024-03-17T23:47:02.714826+0000","last_undegraded":"2024-03-17T23:47:02.714826+0000","last_fullsized":"2024-03-17T23:47:02.714826+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:42:33.325150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713844+0000","last_change":"2024-03-17T23:46:59.961836+0000","last_active":"2024-03-17T23:47:02.713844+0000","last_peered":"2024-03-17T23:47:02.713844+0000","last_clean":"2024-03-17T23:47:02.713844+0000","last_became_active":"2024-03-17T23:46:59.961458+0000","last_became_peered":"2024-03-17T23:46:59.961458+0000","last_unstale":"2024-03-17T23:47:02.713844+0000","last_undegraded":"2024-03-17T23:47:02.713844+0000","last_fullsized":"2024-03-17T23:47:02.713844+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:43:57.153828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106542+0000","last_change":"2024-03-17T23:46:55.926629+0000","last_active":"2024-03-17T23:47:02.106542+0000","last_peered":"2024-03-17T23:47:02.106542+0000","last_clean":"2024-03-17T23:47:02.106542+0000","last_became_active":"2024-03-17T23:46:55.926391+0000","last_became_peered":"2024-03-17T23:46:55.926391+0000","last_unstale":"2024-03-17T23:47:02.106542+0000","last_undegraded":"2024-03-17T23:47:02.106542+0000","last_fullsized":"2024-03-17T23:47:02.106542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:59.421169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943876+0000","last_change":"2024-03-17T23:46:54.921058+0000","last_active":"2024-03-17T23:47:01.943876+0000","last_peered":"2024-03-17T23:47:01.943876+0000","last_clean":"2024-03-17T23:47:01.943876+0000","last_became_active":"2024-03-17T23:46:54.919926+0000","last_became_peered":"2024-03-17T23:46:54.919926+0000","last_unstale":"2024-03-17T23:47:01.943876+0000","last_undegraded":"2024-03-17T23:47:01.943876+0000","last_fullsized":"2024-03-17T23:47:01.943876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:11:01.353944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714725+0000","last_change":"2024-03-17T23:46:59.957516+0000","last_active":"2024-03-17T23:47:02.714725+0000","last_peered":"2024-03-17T23:47:02.714725+0000","last_clean":"2024-03-17T23:47:02.714725+0000","last_became_active":"2024-03-17T23:46:59.957168+0000","last_became_peered":"2024-03-17T23:46:59.957168+0000","last_unstale":"2024-03-17T23:47:02.714725+0000","last_undegraded":"2024-03-17T23:47:02.714725+0000","last_fullsized":"2024-03-17T23:47:02.714725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:35:49.838832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713602+0000","last_change":"2024-03-17T23:46:55.921369+0000","last_active":"2024-03-17T23:47:02.713602+0000","last_peered":"2024-03-17T23:47:02.713602+0000","last_clean":"2024-03-17T23:47:02.713602+0000","last_became_active":"2024-03-17T23:46:55.921109+0000","last_became_peered":"2024-03-17T23:46:55.921109+0000","last_unstale":"2024-03-17T23:47:02.713602+0000","last_undegraded":"2024-03-17T23:47:02.713602+0000","last_fullsized":"2024-03-17T23:47:02.713602+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:24.419077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713228+0000","last_change":"2024-03-17T23:46:54.918722+0000","last_active":"2024-03-17T23:47:02.713228+0000","last_peered":"2024-03-17T23:47:02.713228+0000","last_clean":"2024-03-17T23:47:02.713228+0000","last_became_active":"2024-03-17T23:46:54.918440+0000","last_became_peered":"2024-03-17T23:46:54.918440+0000","last_unstale":"2024-03-17T23:47:02.713228+0000","last_undegraded":"2024-03-17T23:47:02.713228+0000","last_fullsized":"2024-03-17T23:47:02.713228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:56.665566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.715106+0000","last_change":"2024-03-17T23:46:59.962185+0000","last_active":"2024-03-17T23:47:02.715106+0000","last_peered":"2024-03-17T23:47:02.715106+0000","last_clean":"2024-03-17T23:47:02.715106+0000","last_became_active":"2024-03-17T23:46:59.961682+0000","last_became_peered":"2024-03-17T23:46:59.961682+0000","last_unstale":"2024-03-17T23:47:02.715106+0000","last_undegraded":"2024-03-17T23:47:02.715106+0000","last_fullsized":"2024-03-17T23:47:02.715106+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:47:52.502020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713596+0000","last_change":"2024-03-17T23:46:55.924464+0000","last_active":"2024-03-17T23:47:02.713596+0000","last_peered":"2024-03-17T23:47:02.713596+0000","last_clean":"2024-03-17T23:47:02.713596+0000","last_became_active":"2024-03-17T23:46:55.924193+0000","last_became_peered":"2024-03-17T23:46:55.924193+0000","last_unstale":"2024-03-17T23:47:02.713596+0000","last_undegraded":"2024-03-17T23:47:02.713596+0000","last_fullsized":"2024-03-17T23:47:02.713596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:54.348184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713381+0000","last_change":"2024-03-17T23:46:54.921188+0000","last_active":"2024-03-17T23:47:02.713381+0000","last_peered":"2024-03-17T23:47:02.713381+0000","last_clean":"2024-03-17T23:47:02.713381+0000","last_became_active":"2024-03-17T23:46:54.920926+0000","last_became_peered":"2024-03-17T23:46:54.920926+0000","last_unstale":"2024-03-17T23:47:02.713381+0000","last_undegraded":"2024-03-17T23:47:02.713381+0000","last_fullsized":"2024-03-17T23:47:02.713381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:01:23.150557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714399+0000","last_change":"2024-03-17T23:47:00.410855+0000","last_active":"2024-03-17T23:47:02.714399+0000","last_peered":"2024-03-17T23:47:02.714399+0000","last_clean":"2024-03-17T23:47:02.714399+0000","last_became_active":"2024-03-17T23:47:00.410485+0000","last_became_peered":"2024-03-17T23:47:00.410485+0000","last_unstale":"2024-03-17T23:47:02.714399+0000","last_undegraded":"2024-03-17T23:47:02.714399+0000","last_fullsized":"2024-03-17T23:47:02.714399+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:21:41.104850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943282+0000","last_change":"2024-03-17T23:46:55.931906+0000","last_active":"2024-03-17T23:47:01.943282+0000","last_peered":"2024-03-17T23:47:01.943282+0000","last_clean":"2024-03-17T23:47:01.943282+0000","last_became_active":"2024-03-17T23:46:55.931643+0000","last_became_peered":"2024-03-17T23:46:55.931643+0000","last_unstale":"2024-03-17T23:47:01.943282+0000","last_undegraded":"2024-03-17T23:47:01.943282+0000","last_fullsized":"2024-03-17T23:47:01.943282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:24.419077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943618+0000","last_change":"2024-03-17T23:46:54.920477+0000","last_active":"2024-03-17T23:47:01.943618+0000","last_peered":"2024-03-17T23:47:01.943618+0000","last_clean":"2024-03-17T23:47:01.943618+0000","last_became_active":"2024-03-17T23:46:54.920116+0000","last_became_peered":"2024-03-17T23:46:54.920116+0000","last_unstale":"2024-03-17T23:47:01.943618+0000","last_undegraded":"2024-03-17T23:47:01.943618+0000","last_fullsized":"2024-03-17T23:47:01.943618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:56.665566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":180,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713494+0000","last_change":"2024-03-17T23:46:44.768668+0000","last_active":"2024-03-17T23:47:02.713494+0000","last_peered":"2024-03-17T23:47:02.713494+0000","last_clean":"2024-03-17T23:47:02.713494+0000","last_became_active":"2024-03-17T23:46:44.767826+0000","last_became_peered":"2024-03-17T23:46:44.767826+0000","last_unstale":"2024-03-17T23:47:02.713494+0000","last_undegraded":"2024-03-17T23:47:02.713494+0000","last_fullsized":"2024-03-17T23:47:02.713494+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_clean_scrub_stamp":"2024-03-17T23:46:43.744878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:54:22.286155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944382+0000","last_change":"2024-03-17T23:47:00.410345+0000","last_active":"2024-03-17T23:47:01.944382+0000","last_peered":"2024-03-17T23:47:01.944382+0000","last_clean":"2024-03-17T23:47:01.944382+0000","last_became_active":"2024-03-17T23:47:00.409948+0000","last_became_peered":"2024-03-17T23:47:00.409948+0000","last_unstale":"2024-03-17T23:47:01.944382+0000","last_undegraded":"2024-03-17T23:47:01.944382+0000","last_fullsized":"2024-03-17T23:47:01.944382+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:20:25.538662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713826+0000","last_change":"2024-03-17T23:46:55.943072+0000","last_active":"2024-03-17T23:47:02.713826+0000","last_peered":"2024-03-17T23:47:02.713826+0000","last_clean":"2024-03-17T23:47:02.713826+0000","last_became_active":"2024-03-17T23:46:55.942516+0000","last_became_peered":"2024-03-17T23:46:55.942516+0000","last_unstale":"2024-03-17T23:47:02.713826+0000","last_undegraded":"2024-03-17T23:47:02.713826+0000","last_fullsized":"2024-03-17T23:47:02.713826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:00.571611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713510+0000","last_change":"2024-03-17T23:46:54.916398+0000","last_active":"2024-03-17T23:47:02.713510+0000","last_peered":"2024-03-17T23:47:02.713510+0000","last_clean":"2024-03-17T23:47:02.713510+0000","last_became_active":"2024-03-17T23:46:54.916167+0000","last_became_peered":"2024-03-17T23:46:54.916167+0000","last_unstale":"2024-03-17T23:47:02.713510+0000","last_undegraded":"2024-03-17T23:47:02.713510+0000","last_fullsized":"2024-03-17T23:47:02.713510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:42:33.325150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943370+0000","last_change":"2024-03-17T23:47:00.408551+0000","last_active":"2024-03-17T23:47:01.943370+0000","last_peered":"2024-03-17T23:47:01.943370+0000","last_clean":"2024-03-17T23:47:01.943370+0000","last_became_active":"2024-03-17T23:47:00.408223+0000","last_became_peered":"2024-03-17T23:47:00.408223+0000","last_unstale":"2024-03-17T23:47:01.943370+0000","last_undegraded":"2024-03-17T23:47:01.943370+0000","last_fullsized":"2024-03-17T23:47:01.943370+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:52:49.194504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944055+0000","last_change":"2024-03-17T23:46:55.933223+0000","last_active":"2024-03-17T23:47:01.944055+0000","last_peered":"2024-03-17T23:47:01.944055+0000","last_clean":"2024-03-17T23:47:01.944055+0000","last_became_active":"2024-03-17T23:46:55.932957+0000","last_became_peered":"2024-03-17T23:46:55.932957+0000","last_unstale":"2024-03-17T23:47:01.944055+0000","last_undegraded":"2024-03-17T23:47:01.944055+0000","last_fullsized":"2024-03-17T23:47:01.944055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:52:45.163567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713856+0000","last_change":"2024-03-17T23:46:54.919441+0000","last_active":"2024-03-17T23:47:02.713856+0000","last_peered":"2024-03-17T23:47:02.713856+0000","last_clean":"2024-03-17T23:47:02.713856+0000","last_became_active":"2024-03-17T23:46:54.919151+0000","last_became_peered":"2024-03-17T23:46:54.919151+0000","last_unstale":"2024-03-17T23:47:02.713856+0000","last_undegraded":"2024-03-17T23:47:02.713856+0000","last_fullsized":"2024-03-17T23:47:02.713856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:11:01.353944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714221+0000","last_change":"2024-03-17T23:46:59.963445+0000","last_active":"2024-03-17T23:47:02.714221+0000","last_peered":"2024-03-17T23:47:02.714221+0000","last_clean":"2024-03-17T23:47:02.714221+0000","last_became_active":"2024-03-17T23:46:59.963078+0000","last_became_peered":"2024-03-17T23:46:59.963078+0000","last_unstale":"2024-03-17T23:47:02.714221+0000","last_undegraded":"2024-03-17T23:47:02.714221+0000","last_fullsized":"2024-03-17T23:47:02.714221+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:47:52.502020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713966+0000","last_change":"2024-03-17T23:46:55.930951+0000","last_active":"2024-03-17T23:47:02.713966+0000","last_peered":"2024-03-17T23:47:02.713966+0000","last_clean":"2024-03-17T23:47:02.713966+0000","last_became_active":"2024-03-17T23:46:55.930689+0000","last_became_peered":"2024-03-17T23:46:55.930689+0000","last_unstale":"2024-03-17T23:47:02.713966+0000","last_undegraded":"2024-03-17T23:47:02.713966+0000","last_fullsized":"2024-03-17T23:47:02.713966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:43:53.122891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713358+0000","last_change":"2024-03-17T23:46:54.921408+0000","last_active":"2024-03-17T23:47:02.713358+0000","last_peered":"2024-03-17T23:47:02.713358+0000","last_clean":"2024-03-17T23:47:02.713358+0000","last_became_active":"2024-03-17T23:46:54.920825+0000","last_became_peered":"2024-03-17T23:46:54.920825+0000","last_unstale":"2024-03-17T23:47:02.713358+0000","last_undegraded":"2024-03-17T23:47:02.713358+0000","last_fullsized":"2024-03-17T23:47:02.713358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:18.195935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.942947+0000","last_change":"2024-03-17T23:47:00.411371+0000","last_active":"2024-03-17T23:47:01.942947+0000","last_peered":"2024-03-17T23:47:01.942947+0000","last_clean":"2024-03-17T23:47:01.942947+0000","last_became_active":"2024-03-17T23:47:00.410726+0000","last_became_peered":"2024-03-17T23:47:00.410726+0000","last_unstale":"2024-03-17T23:47:01.942947+0000","last_undegraded":"2024-03-17T23:47:01.942947+0000","last_fullsized":"2024-03-17T23:47:01.942947+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:47:52.502020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713787+0000","last_change":"2024-03-17T23:46:55.933053+0000","last_active":"2024-03-17T23:47:02.713787+0000","last_peered":"2024-03-17T23:47:02.713787+0000","last_clean":"2024-03-17T23:47:02.713787+0000","last_became_active":"2024-03-17T23:46:55.932756+0000","last_became_peered":"2024-03-17T23:46:55.932756+0000","last_unstale":"2024-03-17T23:47:02.713787+0000","last_undegraded":"2024-03-17T23:47:02.713787+0000","last_fullsized":"2024-03-17T23:47:02.713787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:45:29.447600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106603+0000","last_change":"2024-03-17T23:46:54.916992+0000","last_active":"2024-03-17T23:47:02.106603+0000","last_peered":"2024-03-17T23:47:02.106603+0000","last_clean":"2024-03-17T23:47:02.106603+0000","last_became_active":"2024-03-17T23:46:54.916296+0000","last_became_peered":"2024-03-17T23:46:54.916296+0000","last_unstale":"2024-03-17T23:47:02.106603+0000","last_undegraded":"2024-03-17T23:47:02.106603+0000","last_fullsized":"2024-03-17T23:47:02.106603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:17:26.217921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106127+0000","last_change":"2024-03-17T23:46:59.957769+0000","last_active":"2024-03-17T23:47:02.106127+0000","last_peered":"2024-03-17T23:47:02.106127+0000","last_clean":"2024-03-17T23:47:02.106127+0000","last_became_active":"2024-03-17T23:46:59.957385+0000","last_became_peered":"2024-03-17T23:46:59.957385+0000","last_unstale":"2024-03-17T23:47:02.106127+0000","last_undegraded":"2024-03-17T23:47:02.106127+0000","last_fullsized":"2024-03-17T23:47:02.106127+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:35:49.838832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713581+0000","last_change":"2024-03-17T23:46:55.935175+0000","last_active":"2024-03-17T23:47:02.713581+0000","last_peered":"2024-03-17T23:47:02.713581+0000","last_clean":"2024-03-17T23:47:02.713581+0000","last_became_active":"2024-03-17T23:46:55.934338+0000","last_became_peered":"2024-03-17T23:46:55.934338+0000","last_unstale":"2024-03-17T23:47:02.713581+0000","last_undegraded":"2024-03-17T23:47:02.713581+0000","last_fullsized":"2024-03-17T23:47:02.713581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:59:32.027430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713064+0000","last_change":"2024-03-17T23:46:54.913663+0000","last_active":"2024-03-17T23:47:02.713064+0000","last_peered":"2024-03-17T23:47:02.713064+0000","last_clean":"2024-03-17T23:47:02.713064+0000","last_became_active":"2024-03-17T23:46:54.913410+0000","last_became_peered":"2024-03-17T23:46:54.913410+0000","last_unstale":"2024-03-17T23:47:02.713064+0000","last_undegraded":"2024-03-17T23:47:02.713064+0000","last_fullsized":"2024-03-17T23:47:02.713064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:45:17.571056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714251+0000","last_change":"2024-03-17T23:46:59.967164+0000","last_active":"2024-03-17T23:47:02.714251+0000","last_peered":"2024-03-17T23:47:02.714251+0000","last_clean":"2024-03-17T23:47:02.714251+0000","last_became_active":"2024-03-17T23:46:59.966653+0000","last_became_peered":"2024-03-17T23:46:59.966653+0000","last_unstale":"2024-03-17T23:47:02.714251+0000","last_undegraded":"2024-03-17T23:47:02.714251+0000","last_fullsized":"2024-03-17T23:47:02.714251+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:57.296991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943052+0000","last_change":"2024-03-17T23:46:55.925322+0000","last_active":"2024-03-17T23:47:01.943052+0000","last_peered":"2024-03-17T23:47:01.943052+0000","last_clean":"2024-03-17T23:47:01.943052+0000","last_became_active":"2024-03-17T23:46:55.925037+0000","last_became_peered":"2024-03-17T23:46:55.925037+0000","last_unstale":"2024-03-17T23:47:01.943052+0000","last_undegraded":"2024-03-17T23:47:01.943052+0000","last_fullsized":"2024-03-17T23:47:01.943052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:39.972607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943437+0000","last_change":"2024-03-17T23:46:54.918033+0000","last_active":"2024-03-17T23:47:01.943437+0000","last_peered":"2024-03-17T23:47:01.943437+0000","last_clean":"2024-03-17T23:47:01.943437+0000","last_became_active":"2024-03-17T23:46:54.917735+0000","last_became_peered":"2024-03-17T23:46:54.917735+0000","last_unstale":"2024-03-17T23:47:01.943437+0000","last_undegraded":"2024-03-17T23:47:01.943437+0000","last_fullsized":"2024-03-17T23:47:01.943437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:42:33.325150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943778+0000","last_change":"2024-03-17T23:47:00.412019+0000","last_active":"2024-03-17T23:47:01.943778+0000","last_peered":"2024-03-17T23:47:01.943778+0000","last_clean":"2024-03-17T23:47:01.943778+0000","last_became_active":"2024-03-17T23:47:00.411603+0000","last_became_peered":"2024-03-17T23:47:00.411603+0000","last_unstale":"2024-03-17T23:47:01.943778+0000","last_undegraded":"2024-03-17T23:47:01.943778+0000","last_fullsized":"2024-03-17T23:47:01.943778+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:18:26.998562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713747+0000","last_change":"2024-03-17T23:46:55.935249+0000","last_active":"2024-03-17T23:47:02.713747+0000","last_peered":"2024-03-17T23:47:02.713747+0000","last_clean":"2024-03-17T23:47:02.713747+0000","last_became_active":"2024-03-17T23:46:55.934833+0000","last_became_peered":"2024-03-17T23:46:55.934833+0000","last_unstale":"2024-03-17T23:47:02.713747+0000","last_undegraded":"2024-03-17T23:47:02.713747+0000","last_fullsized":"2024-03-17T23:47:02.713747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:32:27.349125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943797+0000","last_change":"2024-03-17T23:46:54.924340+0000","last_active":"2024-03-17T23:47:01.943797+0000","last_peered":"2024-03-17T23:47:01.943797+0000","last_clean":"2024-03-17T23:47:01.943797+0000","last_became_active":"2024-03-17T23:46:54.924080+0000","last_became_peered":"2024-03-17T23:46:54.924080+0000","last_unstale":"2024-03-17T23:47:01.943797+0000","last_undegraded":"2024-03-17T23:47:01.943797+0000","last_fullsized":"2024-03-17T23:47:01.943797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:06:54.433520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943804+0000","last_change":"2024-03-17T23:47:00.408517+0000","last_active":"2024-03-17T23:47:01.943804+0000","last_peered":"2024-03-17T23:47:01.943804+0000","last_clean":"2024-03-17T23:47:01.943804+0000","last_became_active":"2024-03-17T23:47:00.408125+0000","last_became_peered":"2024-03-17T23:47:00.408125+0000","last_unstale":"2024-03-17T23:47:01.943804+0000","last_undegraded":"2024-03-17T23:47:01.943804+0000","last_fullsized":"2024-03-17T23:47:01.943804+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:12:39.327858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714486+0000","last_change":"2024-03-17T23:46:55.928837+0000","last_active":"2024-03-17T23:47:02.714486+0000","last_peered":"2024-03-17T23:47:02.714486+0000","last_clean":"2024-03-17T23:47:02.714486+0000","last_became_active":"2024-03-17T23:46:55.928557+0000","last_became_peered":"2024-03-17T23:46:55.928557+0000","last_unstale":"2024-03-17T23:47:02.714486+0000","last_undegraded":"2024-03-17T23:47:02.714486+0000","last_fullsized":"2024-03-17T23:47:02.714486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:23:58.221114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714395+0000","last_change":"2024-03-17T23:46:54.912215+0000","last_active":"2024-03-17T23:47:02.714395+0000","last_peered":"2024-03-17T23:47:02.714395+0000","last_clean":"2024-03-17T23:47:02.714395+0000","last_became_active":"2024-03-17T23:46:54.911909+0000","last_became_peered":"2024-03-17T23:46:54.911909+0000","last_unstale":"2024-03-17T23:47:02.714395+0000","last_undegraded":"2024-03-17T23:47:02.714395+0000","last_fullsized":"2024-03-17T23:47:02.714395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:45:17.571056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944243+0000","last_change":"2024-03-17T23:47:00.408798+0000","last_active":"2024-03-17T23:47:01.944243+0000","last_peered":"2024-03-17T23:47:01.944243+0000","last_clean":"2024-03-17T23:47:01.944243+0000","last_became_active":"2024-03-17T23:47:00.408420+0000","last_became_peered":"2024-03-17T23:47:00.408420+0000","last_unstale":"2024-03-17T23:47:01.944243+0000","last_undegraded":"2024-03-17T23:47:01.944243+0000","last_fullsized":"2024-03-17T23:47:01.944243+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:57.296991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713721+0000","last_change":"2024-03-17T23:46:55.935456+0000","last_active":"2024-03-17T23:47:02.713721+0000","last_peered":"2024-03-17T23:47:02.713721+0000","last_clean":"2024-03-17T23:47:02.713721+0000","last_became_active":"2024-03-17T23:46:55.935187+0000","last_became_peered":"2024-03-17T23:46:55.935187+0000","last_unstale":"2024-03-17T23:47:02.713721+0000","last_undegraded":"2024-03-17T23:47:02.713721+0000","last_fullsized":"2024-03-17T23:47:02.713721+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:11:08.829506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"26'9","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:05.348071+0000","last_change":"2024-03-17T23:46:54.918644+0000","last_active":"2024-03-17T23:47:05.348071+0000","last_peered":"2024-03-17T23:47:05.348071+0000","last_clean":"2024-03-17T23:47:05.348071+0000","last_became_active":"2024-03-17T23:46:54.918290+0000","last_became_peered":"2024-03-17T23:46:54.918290+0000","last_unstale":"2024-03-17T23:47:05.348071+0000","last_undegraded":"2024-03-17T23:47:05.348071+0000","last_fullsized":"2024-03-17T23:47:05.348071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:58.417421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714038+0000","last_change":"2024-03-17T23:46:59.961721+0000","last_active":"2024-03-17T23:47:02.714038+0000","last_peered":"2024-03-17T23:47:02.714038+0000","last_clean":"2024-03-17T23:47:02.714038+0000","last_became_active":"2024-03-17T23:46:59.960948+0000","last_became_peered":"2024-03-17T23:46:59.960948+0000","last_unstale":"2024-03-17T23:47:02.714038+0000","last_undegraded":"2024-03-17T23:47:02.714038+0000","last_fullsized":"2024-03-17T23:47:02.714038+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:27:51.964285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106592+0000","last_change":"2024-03-17T23:46:55.928225+0000","last_active":"2024-03-17T23:47:02.106592+0000","last_peered":"2024-03-17T23:47:02.106592+0000","last_clean":"2024-03-17T23:47:02.106592+0000","last_became_active":"2024-03-17T23:46:55.927359+0000","last_became_peered":"2024-03-17T23:46:55.927359+0000","last_unstale":"2024-03-17T23:47:02.106592+0000","last_undegraded":"2024-03-17T23:47:02.106592+0000","last_fullsized":"2024-03-17T23:47:02.106592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:46:17.785962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":26,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713797+0000","last_change":"2024-03-17T23:46:54.917416+0000","last_active":"2024-03-17T23:47:02.713797+0000","last_peered":"2024-03-17T23:47:02.713797+0000","last_clean":"2024-03-17T23:47:02.713797+0000","last_became_active":"2024-03-17T23:46:54.917136+0000","last_became_peered":"2024-03-17T23:46:54.917136+0000","last_unstale":"2024-03-17T23:47:02.713797+0000","last_undegraded":"2024-03-17T23:47:02.713797+0000","last_fullsized":"2024-03-17T23:47:02.713797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:06:54.433520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106414+0000","last_change":"2024-03-17T23:46:59.957651+0000","last_active":"2024-03-17T23:47:02.106414+0000","last_peered":"2024-03-17T23:47:02.106414+0000","last_clean":"2024-03-17T23:47:02.106414+0000","last_became_active":"2024-03-17T23:46:59.957261+0000","last_became_peered":"2024-03-17T23:46:59.957261+0000","last_unstale":"2024-03-17T23:47:02.106414+0000","last_undegraded":"2024-03-17T23:47:02.106414+0000","last_fullsized":"2024-03-17T23:47:02.106414+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:47:52.502020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713868+0000","last_change":"2024-03-17T23:46:55.937105+0000","last_active":"2024-03-17T23:47:02.713868+0000","last_peered":"2024-03-17T23:47:02.713868+0000","last_clean":"2024-03-17T23:47:02.713868+0000","last_became_active":"2024-03-17T23:46:55.936762+0000","last_became_peered":"2024-03-17T23:46:55.936762+0000","last_unstale":"2024-03-17T23:47:02.713868+0000","last_undegraded":"2024-03-17T23:47:02.713868+0000","last_fullsized":"2024-03-17T23:47:02.713868+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:39.972607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944371+0000","last_change":"2024-03-17T23:46:54.922118+0000","last_active":"2024-03-17T23:47:01.944371+0000","last_peered":"2024-03-17T23:47:01.944371+0000","last_clean":"2024-03-17T23:47:01.944371+0000","last_became_active":"2024-03-17T23:46:54.921834+0000","last_became_peered":"2024-03-17T23:46:54.921834+0000","last_unstale":"2024-03-17T23:47:01.944371+0000","last_undegraded":"2024-03-17T23:47:01.944371+0000","last_fullsized":"2024-03-17T23:47:01.944371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:42:33.325150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943156+0000","last_change":"2024-03-17T23:46:59.957652+0000","last_active":"2024-03-17T23:47:01.943156+0000","last_peered":"2024-03-17T23:47:01.943156+0000","last_clean":"2024-03-17T23:47:01.943156+0000","last_became_active":"2024-03-17T23:46:59.957242+0000","last_became_peered":"2024-03-17T23:46:59.957242+0000","last_unstale":"2024-03-17T23:47:01.943156+0000","last_undegraded":"2024-03-17T23:47:01.943156+0000","last_fullsized":"2024-03-17T23:47:01.943156+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:48:08.469403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944280+0000","last_change":"2024-03-17T23:46:55.930428+0000","last_active":"2024-03-17T23:47:01.944280+0000","last_peered":"2024-03-17T23:47:01.944280+0000","last_clean":"2024-03-17T23:47:01.944280+0000","last_became_active":"2024-03-17T23:46:55.930119+0000","last_became_peered":"2024-03-17T23:46:55.930119+0000","last_unstale":"2024-03-17T23:47:01.944280+0000","last_undegraded":"2024-03-17T23:47:01.944280+0000","last_fullsized":"2024-03-17T23:47:01.944280+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:59:00.571611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944550+0000","last_change":"2024-03-17T23:46:54.918186+0000","last_active":"2024-03-17T23:47:01.944550+0000","last_peered":"2024-03-17T23:47:01.944550+0000","last_clean":"2024-03-17T23:47:01.944550+0000","last_became_active":"2024-03-17T23:46:54.917839+0000","last_became_peered":"2024-03-17T23:46:54.917839+0000","last_unstale":"2024-03-17T23:47:01.944550+0000","last_undegraded":"2024-03-17T23:47:01.944550+0000","last_fullsized":"2024-03-17T23:47:01.944550+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:06:54.433520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943131+0000","last_change":"2024-03-17T23:47:00.411924+0000","last_active":"2024-03-17T23:47:01.943131+0000","last_peered":"2024-03-17T23:47:01.943131+0000","last_clean":"2024-03-17T23:47:01.943131+0000","last_became_active":"2024-03-17T23:47:00.411410+0000","last_became_peered":"2024-03-17T23:47:00.411410+0000","last_unstale":"2024-03-17T23:47:01.943131+0000","last_undegraded":"2024-03-17T23:47:01.943131+0000","last_fullsized":"2024-03-17T23:47:01.943131+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:56:09.808314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714107+0000","last_change":"2024-03-17T23:47:00.410837+0000","last_active":"2024-03-17T23:47:02.714107+0000","last_peered":"2024-03-17T23:47:02.714107+0000","last_clean":"2024-03-17T23:47:02.714107+0000","last_became_active":"2024-03-17T23:47:00.410459+0000","last_became_peered":"2024-03-17T23:47:00.410459+0000","last_unstale":"2024-03-17T23:47:02.714107+0000","last_undegraded":"2024-03-17T23:47:02.714107+0000","last_fullsized":"2024-03-17T23:47:02.714107+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:39:36.583214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713305+0000","last_change":"2024-03-17T23:46:55.928213+0000","last_active":"2024-03-17T23:47:02.713305+0000","last_peered":"2024-03-17T23:47:02.713305+0000","last_clean":"2024-03-17T23:47:02.713305+0000","last_became_active":"2024-03-17T23:46:55.927957+0000","last_became_peered":"2024-03-17T23:46:55.927957+0000","last_unstale":"2024-03-17T23:47:02.713305+0000","last_undegraded":"2024-03-17T23:47:02.713305+0000","last_fullsized":"2024-03-17T23:47:02.713305+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:03.353861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943171+0000","last_change":"2024-03-17T23:46:54.919392+0000","last_active":"2024-03-17T23:47:01.943171+0000","last_peered":"2024-03-17T23:47:01.943171+0000","last_clean":"2024-03-17T23:47:01.943171+0000","last_became_active":"2024-03-17T23:46:54.918594+0000","last_became_peered":"2024-03-17T23:46:54.918594+0000","last_unstale":"2024-03-17T23:47:01.943171+0000","last_undegraded":"2024-03-17T23:47:01.943171+0000","last_fullsized":"2024-03-17T23:47:01.943171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:18.195935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944720+0000","last_change":"2024-03-17T23:46:59.959748+0000","last_active":"2024-03-17T23:47:01.944720+0000","last_peered":"2024-03-17T23:47:01.944720+0000","last_clean":"2024-03-17T23:47:01.944720+0000","last_became_active":"2024-03-17T23:46:59.959383+0000","last_became_peered":"2024-03-17T23:46:59.959383+0000","last_unstale":"2024-03-17T23:47:01.944720+0000","last_undegraded":"2024-03-17T23:47:01.944720+0000","last_fullsized":"2024-03-17T23:47:01.944720+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:43:57.153828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943202+0000","last_change":"2024-03-17T23:46:55.929140+0000","last_active":"2024-03-17T23:47:01.943202+0000","last_peered":"2024-03-17T23:47:01.943202+0000","last_clean":"2024-03-17T23:47:01.943202+0000","last_became_active":"2024-03-17T23:46:55.928874+0000","last_became_peered":"2024-03-17T23:46:55.928874+0000","last_unstale":"2024-03-17T23:47:01.943202+0000","last_undegraded":"2024-03-17T23:47:01.943202+0000","last_fullsized":"2024-03-17T23:47:01.943202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:59:32.027430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943228+0000","last_change":"2024-03-17T23:46:54.923319+0000","last_active":"2024-03-17T23:47:01.943228+0000","last_peered":"2024-03-17T23:47:01.943228+0000","last_clean":"2024-03-17T23:47:01.943228+0000","last_became_active":"2024-03-17T23:46:54.923060+0000","last_became_peered":"2024-03-17T23:46:54.923060+0000","last_unstale":"2024-03-17T23:47:01.943228+0000","last_undegraded":"2024-03-17T23:47:01.943228+0000","last_fullsized":"2024-03-17T23:47:01.943228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:45:17.571056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943597+0000","last_change":"2024-03-17T23:46:54.917589+0000","last_active":"2024-03-17T23:47:01.943597+0000","last_peered":"2024-03-17T23:47:01.943597+0000","last_clean":"2024-03-17T23:47:01.943597+0000","last_became_active":"2024-03-17T23:46:54.917183+0000","last_became_peered":"2024-03-17T23:46:54.917183+0000","last_unstale":"2024-03-17T23:47:01.943597+0000","last_undegraded":"2024-03-17T23:47:01.943597+0000","last_fullsized":"2024-03-17T23:47:01.943597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:02:18.195935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714330+0000","last_change":"2024-03-17T23:46:55.924339+0000","last_active":"2024-03-17T23:47:02.714330+0000","last_peered":"2024-03-17T23:47:02.714330+0000","last_clean":"2024-03-17T23:47:02.714330+0000","last_became_active":"2024-03-17T23:46:55.923891+0000","last_became_peered":"2024-03-17T23:46:55.923891+0000","last_unstale":"2024-03-17T23:47:02.714330+0000","last_undegraded":"2024-03-17T23:47:02.714330+0000","last_fullsized":"2024-03-17T23:47:02.714330+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:11:08.829506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713322+0000","last_change":"2024-03-17T23:46:59.959425+0000","last_active":"2024-03-17T23:47:02.713322+0000","last_peered":"2024-03-17T23:47:02.713322+0000","last_clean":"2024-03-17T23:47:02.713322+0000","last_became_active":"2024-03-17T23:46:59.958851+0000","last_became_peered":"2024-03-17T23:46:59.958851+0000","last_unstale":"2024-03-17T23:47:02.713322+0000","last_undegraded":"2024-03-17T23:47:02.713322+0000","last_fullsized":"2024-03-17T23:47:02.713322+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:58.379120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943490+0000","last_change":"2024-03-17T23:46:54.921325+0000","last_active":"2024-03-17T23:47:01.943490+0000","last_peered":"2024-03-17T23:47:01.943490+0000","last_clean":"2024-03-17T23:47:01.943490+0000","last_became_active":"2024-03-17T23:46:54.921093+0000","last_became_peered":"2024-03-17T23:46:54.921093+0000","last_unstale":"2024-03-17T23:47:01.943490+0000","last_undegraded":"2024-03-17T23:47:01.943490+0000","last_fullsized":"2024-03-17T23:47:01.943490+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:53.895765+0000","last_clean_scrub_stamp":"2024-03-17T23:46:53.895765+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:56.665566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106239+0000","last_change":"2024-03-17T23:46:55.922093+0000","last_active":"2024-03-17T23:47:02.106239+0000","last_peered":"2024-03-17T23:47:02.106239+0000","last_clean":"2024-03-17T23:47:02.106239+0000","last_became_active":"2024-03-17T23:46:55.919778+0000","last_became_peered":"2024-03-17T23:46:55.919778+0000","last_unstale":"2024-03-17T23:47:02.106239+0000","last_undegraded":"2024-03-17T23:47:02.106239+0000","last_fullsized":"2024-03-17T23:47:02.106239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:06:55.437268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943978+0000","last_change":"2024-03-17T23:46:59.964222+0000","last_active":"2024-03-17T23:47:01.943978+0000","last_peered":"2024-03-17T23:47:01.943978+0000","last_clean":"2024-03-17T23:47:01.943978+0000","last_became_active":"2024-03-17T23:46:59.963807+0000","last_became_peered":"2024-03-17T23:46:59.963807+0000","last_unstale":"2024-03-17T23:47:01.943978+0000","last_undegraded":"2024-03-17T23:47:01.943978+0000","last_fullsized":"2024-03-17T23:47:01.943978+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:58.930449+0000","last_clean_scrub_stamp":"2024-03-17T23:46:58.930449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:25:11.791005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.942962+0000","last_change":"2024-03-17T23:46:55.926785+0000","last_active":"2024-03-17T23:47:01.942962+0000","last_peered":"2024-03-17T23:47:01.942962+0000","last_clean":"2024-03-17T23:47:01.942962+0000","last_became_active":"2024-03-17T23:46:55.926494+0000","last_became_peered":"2024-03-17T23:46:55.926494+0000","last_unstale":"2024-03-17T23:47:01.942962+0000","last_undegraded":"2024-03-17T23:47:01.942962+0000","last_fullsized":"2024-03-17T23:47:01.942962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:24.419077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713141+0000","last_change":"2024-03-17T23:46:55.930094+0000","last_active":"2024-03-17T23:47:02.713141+0000","last_peered":"2024-03-17T23:47:02.713141+0000","last_clean":"2024-03-17T23:47:02.713141+0000","last_became_active":"2024-03-17T23:46:55.929834+0000","last_became_peered":"2024-03-17T23:46:55.929834+0000","last_unstale":"2024-03-17T23:47:02.713141+0000","last_undegraded":"2024-03-17T23:47:02.713141+0000","last_fullsized":"2024-03-17T23:47:02.713141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:28:59.421169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943791+0000","last_change":"2024-03-17T23:46:55.934265+0000","last_active":"2024-03-17T23:47:01.943791+0000","last_peered":"2024-03-17T23:47:01.943791+0000","last_clean":"2024-03-17T23:47:01.943791+0000","last_became_active":"2024-03-17T23:46:55.933735+0000","last_became_peered":"2024-03-17T23:46:55.933735+0000","last_unstale":"2024-03-17T23:47:01.943791+0000","last_undegraded":"2024-03-17T23:47:01.943791+0000","last_fullsized":"2024-03-17T23:47:01.943791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:59:32.027430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714250+0000","last_change":"2024-03-17T23:46:55.930627+0000","last_active":"2024-03-17T23:47:02.714250+0000","last_peered":"2024-03-17T23:47:02.714250+0000","last_clean":"2024-03-17T23:47:02.714250+0000","last_became_active":"2024-03-17T23:46:55.930368+0000","last_became_peered":"2024-03-17T23:46:55.930368+0000","last_unstale":"2024-03-17T23:47:02.714250+0000","last_undegraded":"2024-03-17T23:47:02.714250+0000","last_fullsized":"2024-03-17T23:47:02.714250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:47:30.023063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943914+0000","last_change":"2024-03-17T23:46:55.930544+0000","last_active":"2024-03-17T23:47:01.943914+0000","last_peered":"2024-03-17T23:47:01.943914+0000","last_clean":"2024-03-17T23:47:01.943914+0000","last_became_active":"2024-03-17T23:46:55.930277+0000","last_became_peered":"2024-03-17T23:46:55.930277+0000","last_unstale":"2024-03-17T23:47:01.943914+0000","last_undegraded":"2024-03-17T23:47:01.943914+0000","last_fullsized":"2024-03-17T23:47:01.943914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:23:58.221114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713632+0000","last_change":"2024-03-17T23:46:55.927054+0000","last_active":"2024-03-17T23:47:02.713632+0000","last_peered":"2024-03-17T23:47:02.713632+0000","last_clean":"2024-03-17T23:47:02.713632+0000","last_became_active":"2024-03-17T23:46:55.926762+0000","last_became_peered":"2024-03-17T23:46:55.926762+0000","last_unstale":"2024-03-17T23:47:02.713632+0000","last_undegraded":"2024-03-17T23:47:02.713632+0000","last_fullsized":"2024-03-17T23:47:02.713632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:39.972607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944787+0000","last_change":"2024-03-17T23:46:55.929740+0000","last_active":"2024-03-17T23:47:01.944787+0000","last_peered":"2024-03-17T23:47:01.944787+0000","last_clean":"2024-03-17T23:47:01.944787+0000","last_became_active":"2024-03-17T23:46:55.929433+0000","last_became_peered":"2024-03-17T23:46:55.929433+0000","last_unstale":"2024-03-17T23:47:01.944787+0000","last_undegraded":"2024-03-17T23:47:01.944787+0000","last_fullsized":"2024-03-17T23:47:01.944787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:11:08.829506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943753+0000","last_change":"2024-03-17T23:46:55.930851+0000","last_active":"2024-03-17T23:47:01.943753+0000","last_peered":"2024-03-17T23:47:01.943753+0000","last_clean":"2024-03-17T23:47:01.943753+0000","last_became_active":"2024-03-17T23:46:55.930305+0000","last_became_peered":"2024-03-17T23:46:55.930305+0000","last_unstale":"2024-03-17T23:47:01.943753+0000","last_undegraded":"2024-03-17T23:47:01.943753+0000","last_fullsized":"2024-03-17T23:47:01.943753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:47:30.023063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943134+0000","last_change":"2024-03-17T23:46:55.933139+0000","last_active":"2024-03-17T23:47:01.943134+0000","last_peered":"2024-03-17T23:47:01.943134+0000","last_clean":"2024-03-17T23:47:01.943134+0000","last_became_active":"2024-03-17T23:46:55.932843+0000","last_became_peered":"2024-03-17T23:46:55.932843+0000","last_unstale":"2024-03-17T23:47:01.943134+0000","last_undegraded":"2024-03-17T23:47:01.943134+0000","last_fullsized":"2024-03-17T23:47:01.943134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:44:47.002489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943147+0000","last_change":"2024-03-17T23:46:55.930881+0000","last_active":"2024-03-17T23:47:01.943147+0000","last_peered":"2024-03-17T23:47:01.943147+0000","last_clean":"2024-03-17T23:47:01.943147+0000","last_became_active":"2024-03-17T23:46:55.930629+0000","last_became_peered":"2024-03-17T23:46:55.930629+0000","last_unstale":"2024-03-17T23:47:01.943147+0000","last_undegraded":"2024-03-17T23:47:01.943147+0000","last_fullsized":"2024-03-17T23:47:01.943147+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:49:47.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943696+0000","last_change":"2024-03-17T23:46:55.936236+0000","last_active":"2024-03-17T23:47:01.943696+0000","last_peered":"2024-03-17T23:47:01.943696+0000","last_clean":"2024-03-17T23:47:01.943696+0000","last_became_active":"2024-03-17T23:46:55.933485+0000","last_became_peered":"2024-03-17T23:46:55.933485+0000","last_unstale":"2024-03-17T23:47:01.943696+0000","last_undegraded":"2024-03-17T23:47:01.943696+0000","last_fullsized":"2024-03-17T23:47:01.943696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:11:08.829506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714118+0000","last_change":"2024-03-17T23:46:55.930067+0000","last_active":"2024-03-17T23:47:02.714118+0000","last_peered":"2024-03-17T23:47:02.714118+0000","last_clean":"2024-03-17T23:47:02.714118+0000","last_became_active":"2024-03-17T23:46:55.929808+0000","last_became_peered":"2024-03-17T23:46:55.929808+0000","last_unstale":"2024-03-17T23:47:02.714118+0000","last_undegraded":"2024-03-17T23:47:02.714118+0000","last_fullsized":"2024-03-17T23:47:02.714118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:49:47.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943993+0000","last_change":"2024-03-17T23:46:55.917074+0000","last_active":"2024-03-17T23:47:01.943993+0000","last_peered":"2024-03-17T23:47:01.943993+0000","last_clean":"2024-03-17T23:47:01.943993+0000","last_became_active":"2024-03-17T23:46:55.916784+0000","last_became_peered":"2024-03-17T23:46:55.916784+0000","last_unstale":"2024-03-17T23:47:01.943993+0000","last_undegraded":"2024-03-17T23:47:01.943993+0000","last_fullsized":"2024-03-17T23:47:01.943993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:49:47.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713939+0000","last_change":"2024-03-17T23:46:55.928539+0000","last_active":"2024-03-17T23:47:02.713939+0000","last_peered":"2024-03-17T23:47:02.713939+0000","last_clean":"2024-03-17T23:47:02.713939+0000","last_became_active":"2024-03-17T23:46:55.928275+0000","last_became_peered":"2024-03-17T23:46:55.928275+0000","last_unstale":"2024-03-17T23:47:02.713939+0000","last_undegraded":"2024-03-17T23:47:02.713939+0000","last_fullsized":"2024-03-17T23:47:02.713939+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T23:52:25.543746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713273+0000","last_change":"2024-03-17T23:46:55.924366+0000","last_active":"2024-03-17T23:47:02.713273+0000","last_peered":"2024-03-17T23:47:02.713273+0000","last_clean":"2024-03-17T23:47:02.713273+0000","last_became_active":"2024-03-17T23:46:55.924100+0000","last_became_peered":"2024-03-17T23:46:55.924100+0000","last_unstale":"2024-03-17T23:47:02.713273+0000","last_undegraded":"2024-03-17T23:47:02.713273+0000","last_fullsized":"2024-03-17T23:47:02.713273+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:51:19.549589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944530+0000","last_change":"2024-03-17T23:46:55.932514+0000","last_active":"2024-03-17T23:47:01.944530+0000","last_peered":"2024-03-17T23:47:01.944530+0000","last_clean":"2024-03-17T23:47:01.944530+0000","last_became_active":"2024-03-17T23:46:55.930899+0000","last_became_peered":"2024-03-17T23:46:55.930899+0000","last_unstale":"2024-03-17T23:47:01.944530+0000","last_undegraded":"2024-03-17T23:47:01.944530+0000","last_fullsized":"2024-03-17T23:47:01.944530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:47:30.023063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943088+0000","last_change":"2024-03-17T23:46:55.940176+0000","last_active":"2024-03-17T23:47:01.943088+0000","last_peered":"2024-03-17T23:47:01.943088+0000","last_clean":"2024-03-17T23:47:01.943088+0000","last_became_active":"2024-03-17T23:46:55.939853+0000","last_became_peered":"2024-03-17T23:46:55.939853+0000","last_unstale":"2024-03-17T23:47:01.943088+0000","last_undegraded":"2024-03-17T23:47:01.943088+0000","last_fullsized":"2024-03-17T23:47:01.943088+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:49:47.423449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713396+0000","last_change":"2024-03-17T23:46:55.917430+0000","last_active":"2024-03-17T23:47:02.713396+0000","last_peered":"2024-03-17T23:47:02.713396+0000","last_clean":"2024-03-17T23:47:02.713396+0000","last_became_active":"2024-03-17T23:46:55.917173+0000","last_became_peered":"2024-03-17T23:46:55.917173+0000","last_unstale":"2024-03-17T23:47:02.713396+0000","last_undegraded":"2024-03-17T23:47:02.713396+0000","last_fullsized":"2024-03-17T23:47:02.713396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:23:58.221114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.714399+0000","last_change":"2024-03-17T23:46:55.927198+0000","last_active":"2024-03-17T23:47:02.714399+0000","last_peered":"2024-03-17T23:47:02.714399+0000","last_clean":"2024-03-17T23:47:02.714399+0000","last_became_active":"2024-03-17T23:46:55.926919+0000","last_became_peered":"2024-03-17T23:46:55.926919+0000","last_unstale":"2024-03-17T23:47:02.714399+0000","last_undegraded":"2024-03-17T23:47:02.714399+0000","last_fullsized":"2024-03-17T23:47:02.714399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:24.419077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943395+0000","last_change":"2024-03-17T23:46:55.932835+0000","last_active":"2024-03-17T23:47:01.943395+0000","last_peered":"2024-03-17T23:47:01.943395+0000","last_clean":"2024-03-17T23:47:01.943395+0000","last_became_active":"2024-03-17T23:46:55.932339+0000","last_became_peered":"2024-03-17T23:46:55.932339+0000","last_unstale":"2024-03-17T23:47:01.943395+0000","last_undegraded":"2024-03-17T23:47:01.943395+0000","last_fullsized":"2024-03-17T23:47:01.943395+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:39.972607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.106535+0000","last_change":"2024-03-17T23:46:55.916730+0000","last_active":"2024-03-17T23:47:02.106535+0000","last_peered":"2024-03-17T23:47:02.106535+0000","last_clean":"2024-03-17T23:47:02.106535+0000","last_became_active":"2024-03-17T23:46:55.916438+0000","last_became_peered":"2024-03-17T23:46:55.916438+0000","last_unstale":"2024-03-17T23:47:02.106535+0000","last_undegraded":"2024-03-17T23:47:02.106535+0000","last_fullsized":"2024-03-17T23:47:02.106535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:51:19.549589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944384+0000","last_change":"2024-03-17T23:46:55.932468+0000","last_active":"2024-03-17T23:47:01.944384+0000","last_peered":"2024-03-17T23:47:01.944384+0000","last_clean":"2024-03-17T23:47:01.944384+0000","last_became_active":"2024-03-17T23:46:55.932210+0000","last_became_peered":"2024-03-17T23:46:55.932210+0000","last_unstale":"2024-03-17T23:47:01.944384+0000","last_undegraded":"2024-03-17T23:47:01.944384+0000","last_fullsized":"2024-03-17T23:47:01.944384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:23:58.221114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.944784+0000","last_change":"2024-03-17T23:46:55.934093+0000","last_active":"2024-03-17T23:47:01.944784+0000","last_peered":"2024-03-17T23:47:01.944784+0000","last_clean":"2024-03-17T23:47:01.944784+0000","last_became_active":"2024-03-17T23:46:55.933516+0000","last_became_peered":"2024-03-17T23:46:55.933516+0000","last_unstale":"2024-03-17T23:47:01.944784+0000","last_undegraded":"2024-03-17T23:47:01.944784+0000","last_fullsized":"2024-03-17T23:47:01.944784+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:47:30.023063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713902+0000","last_change":"2024-03-17T23:46:55.932426+0000","last_active":"2024-03-17T23:47:02.713902+0000","last_peered":"2024-03-17T23:47:02.713902+0000","last_clean":"2024-03-17T23:47:02.713902+0000","last_became_active":"2024-03-17T23:46:55.932045+0000","last_became_peered":"2024-03-17T23:46:55.932045+0000","last_unstale":"2024-03-17T23:47:02.713902+0000","last_undegraded":"2024-03-17T23:47:02.713902+0000","last_fullsized":"2024-03-17T23:47:02.713902+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:20:21.507725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.942978+0000","last_change":"2024-03-17T23:46:55.932245+0000","last_active":"2024-03-17T23:47:01.942978+0000","last_peered":"2024-03-17T23:47:01.942978+0000","last_clean":"2024-03-17T23:47:01.942978+0000","last_became_active":"2024-03-17T23:46:55.932001+0000","last_became_peered":"2024-03-17T23:46:55.932001+0000","last_unstale":"2024-03-17T23:47:01.942978+0000","last_undegraded":"2024-03-17T23:47:01.942978+0000","last_fullsized":"2024-03-17T23:47:01.942978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:44:47.002489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713629+0000","last_change":"2024-03-17T23:46:55.935868+0000","last_active":"2024-03-17T23:47:02.713629+0000","last_peered":"2024-03-17T23:47:02.713629+0000","last_clean":"2024-03-17T23:47:02.713629+0000","last_became_active":"2024-03-17T23:46:55.935180+0000","last_became_peered":"2024-03-17T23:46:55.935180+0000","last_unstale":"2024-03-17T23:47:02.713629+0000","last_undegraded":"2024-03-17T23:47:02.713629+0000","last_fullsized":"2024-03-17T23:47:02.713629+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:44:47.002489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943696+0000","last_change":"2024-03-17T23:46:55.936947+0000","last_active":"2024-03-17T23:47:01.943696+0000","last_peered":"2024-03-17T23:47:01.943696+0000","last_clean":"2024-03-17T23:47:01.943696+0000","last_became_active":"2024-03-17T23:46:55.936694+0000","last_became_peered":"2024-03-17T23:46:55.936694+0000","last_unstale":"2024-03-17T23:47:01.943696+0000","last_undegraded":"2024-03-17T23:47:01.943696+0000","last_fullsized":"2024-03-17T23:47:01.943696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:39.972607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713525+0000","last_change":"2024-03-17T23:46:55.927487+0000","last_active":"2024-03-17T23:47:02.713525+0000","last_peered":"2024-03-17T23:47:02.713525+0000","last_clean":"2024-03-17T23:47:02.713525+0000","last_became_active":"2024-03-17T23:46:55.927225+0000","last_became_peered":"2024-03-17T23:46:55.927225+0000","last_unstale":"2024-03-17T23:47:02.713525+0000","last_undegraded":"2024-03-17T23:47:02.713525+0000","last_fullsized":"2024-03-17T23:47:02.713525+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:22:12.841390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:01.943865+0000","last_change":"2024-03-17T23:46:55.933146+0000","last_active":"2024-03-17T23:47:01.943865+0000","last_peered":"2024-03-17T23:47:01.943865+0000","last_clean":"2024-03-17T23:47:01.943865+0000","last_became_active":"2024-03-17T23:46:55.932861+0000","last_became_peered":"2024-03-17T23:46:55.932861+0000","last_unstale":"2024-03-17T23:47:01.943865+0000","last_undegraded":"2024-03-17T23:47:01.943865+0000","last_fullsized":"2024-03-17T23:47:01.943865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:22:12.841390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713455+0000","last_change":"2024-03-17T23:46:55.924118+0000","last_active":"2024-03-17T23:47:02.713455+0000","last_peered":"2024-03-17T23:47:02.713455+0000","last_clean":"2024-03-17T23:47:02.713455+0000","last_became_active":"2024-03-17T23:46:55.923844+0000","last_became_peered":"2024-03-17T23:46:55.923844+0000","last_unstale":"2024-03-17T23:47:02.713455+0000","last_undegraded":"2024-03-17T23:47:02.713455+0000","last_fullsized":"2024-03-17T23:47:02.713455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:30:49.518932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713741+0000","last_change":"2024-03-17T23:46:55.925191+0000","last_active":"2024-03-17T23:47:02.713741+0000","last_peered":"2024-03-17T23:47:02.713741+0000","last_clean":"2024-03-17T23:47:02.713741+0000","last_became_active":"2024-03-17T23:46:55.924822+0000","last_became_peered":"2024-03-17T23:46:55.924822+0000","last_unstale":"2024-03-17T23:47:02.713741+0000","last_undegraded":"2024-03-17T23:47:02.713741+0000","last_fullsized":"2024-03-17T23:47:02.713741+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:45:20.327436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-17T23:47:02.713837+0000","last_change":"2024-03-17T23:46:55.928943+0000","last_active":"2024-03-17T23:47:02.713837+0000","last_peered":"2024-03-17T23:47:02.713837+0000","last_clean":"2024-03-17T23:47:02.713837+0000","last_became_active":"2024-03-17T23:46:55.928669+0000","last_became_peered":"2024-03-17T23:46:55.928669+0000","last_unstale":"2024-03-17T23:47:02.713837+0000","last_undegraded":"2024-03-17T23:47:02.713837+0000","last_fullsized":"2024-03-17T23:47:02.713837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:54.899512+0000","last_clean_scrub_stamp":"2024-03-17T23:46:54.899512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:52:45.163567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":14,"seq":60129542154,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":286720,"data_stored":115066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542154,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":274432,"data_stored":111224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542154,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":880640,"data_stored":702230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":872448,"data_stored":702174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31456,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340384,"statfs":{"total":96636764160,"available":96604553216,"internally_reserved":0,"allocated":286720,"data_stored":115066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":278528,"data_stored":111314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":286720,"data_stored":111350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":111384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-17T23:47:17.767 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-17T23:47:17.767 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-17T23:47:17.767 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-17T23:47:17.767 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-17T23:47:17.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.902+0000 7fe31c2c6640 1 Processor -- start 2024-03-17T23:47:17.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.902+0000 7fe31c2c6640 1 -- start start 2024-03-17T23:47:17.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.903+0000 7fe31c2c6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe31409a470 0x7fe31409a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:17.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.903+0000 7fe31c2c6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe31409ad90 0x7fe31415f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:17.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.903+0000 7fe31c2c6640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe31415fcd0 0x7fe31415c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:17.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.903+0000 7fe31c2c6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fe3140cfad0 con 0x7fe31409ad90 2024-03-17T23:47:17.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.903+0000 7fe31c2c6640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fe3140cfdb0 con 0x7fe31415fcd0 2024-03-17T23:47:17.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.903+0000 7fe31c2c6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe3140d0090 con 0x7fe31409a470 2024-03-17T23:47:17.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.903+0000 7fe319861640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe31409ad90 0x7fe31415f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:17.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.903+0000 7fe319861640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe31409ad90 0x7fe31415f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40394/0 (socket says 172.21.15.150:40394) 2024-03-17T23:47:17.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.903+0000 7fe319861640 1 -- 172.21.15.150:0/4142868584 learned_addr learned my addr 172.21.15.150:0/4142868584 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:17.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.903+0000 7fe31a062640 1 --2- 172.21.15.150:0/4142868584 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe31409a470 0x7fe31409a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:17.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.903+0000 7fe31a863640 1 --2- 172.21.15.150:0/4142868584 >> v2:172.21.15.157:3300/0 conn(0x7fe31415fcd0 0x7fe31415c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:17.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.903+0000 7fe319861640 1 -- 172.21.15.150:0/4142868584 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe31409a470 msgr2=0x7fe31409a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:17.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.903+0000 7fe319861640 1 --2- 172.21.15.150:0/4142868584 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe31409a470 0x7fe31409a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.903+0000 7fe319861640 1 -- 172.21.15.150:0/4142868584 >> v2:172.21.15.157:3300/0 conn(0x7fe31415fcd0 msgr2=0x7fe31415c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:17.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.903+0000 7fe319861640 1 --2- 172.21.15.150:0/4142868584 >> v2:172.21.15.157:3300/0 conn(0x7fe31415fcd0 0x7fe31415c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.903+0000 7fe319861640 1 -- 172.21.15.150:0/4142868584 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3140d0370 con 0x7fe31409ad90 2024-03-17T23:47:17.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.904+0000 7fe319861640 1 --2- 172.21.15.150:0/4142868584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe31409ad90 0x7fe31415f790 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe3140cc330 tx=0x7fe30402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25a827c9eef88851 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:17.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.904+0000 7fe319060640 1 -- 172.21.15.150:0/4142868584 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304035020 con 0x7fe31409ad90 2024-03-17T23:47:17.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.904+0000 7fe319060640 1 -- 172.21.15.150:0/4142868584 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe304031870 con 0x7fe31409ad90 2024-03-17T23:47:17.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.904+0000 7fe319060640 1 -- 172.21.15.150:0/4142868584 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304031c40 con 0x7fe31409ad90 2024-03-17T23:47:17.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.904+0000 7fe31c2c6640 1 -- 172.21.15.150:0/4142868584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe31409ad90 msgr2=0x7fe31415f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:17.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.904+0000 7fe31c2c6640 1 --2- 172.21.15.150:0/4142868584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe31409ad90 0x7fe31415f790 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe3140cc330 tx=0x7fe30402b910 comp rx=0 tx=0).stop 2024-03-17T23:47:17.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.905+0000 7fe31c2c6640 1 -- 172.21.15.150:0/4142868584 shutdown_connections 2024-03-17T23:47:17.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.905+0000 7fe31c2c6640 1 --2- 172.21.15.150:0/4142868584 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe31409ad90 0x7fe31415f790 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.905+0000 7fe31c2c6640 1 --2- 172.21.15.150:0/4142868584 >> v2:172.21.15.157:3300/0 conn(0x7fe31415fcd0 0x7fe31415c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.905+0000 7fe31c2c6640 1 --2- 172.21.15.150:0/4142868584 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe31409a470 0x7fe31409a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.905+0000 7fe31c2c6640 1 -- 172.21.15.150:0/4142868584 >> 172.21.15.150:0/4142868584 conn(0x7fe3140b3630 msgr2=0x7fe3140b3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:17.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.905+0000 7fe31c2c6640 1 -- 172.21.15.150:0/4142868584 shutdown_connections 2024-03-17T23:47:17.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.905+0000 7fe31c2c6640 1 -- 172.21.15.150:0/4142868584 wait complete. 2024-03-17T23:47:17.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.906+0000 7fe31c2c6640 1 Processor -- start 2024-03-17T23:47:17.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.906+0000 7fe31c2c6640 1 -- start start 2024-03-17T23:47:17.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.906+0000 7fe31c2c6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe31409a470 0x7fe3140b70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:17.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.906+0000 7fe31c2c6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe31409ad90 0x7fe3140c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:17.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.907+0000 7fe31c2c6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe31415fcd0 0x7fe3140c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:17.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.907+0000 7fe31c2c6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe3140cf890 con 0x7fe31409ad90 2024-03-17T23:47:17.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.907+0000 7fe31c2c6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe3140cf710 con 0x7fe31409a470 2024-03-17T23:47:17.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.907+0000 7fe31c2c6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe31415c690 con 0x7fe31415fcd0 2024-03-17T23:47:17.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.907+0000 7fe319861640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe31409ad90 0x7fe3140c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:17.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.907+0000 7fe319861640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe31409ad90 0x7fe3140c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40408/0 (socket says 172.21.15.150:40408) 2024-03-17T23:47:17.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.907+0000 7fe319861640 1 -- 172.21.15.150:0/4108871814 learned_addr learned my addr 172.21.15.150:0/4108871814 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:17.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.907+0000 7fe31a863640 1 --2- 172.21.15.150:0/4108871814 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe31415fcd0 0x7fe3140c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:17.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.907+0000 7fe31a062640 1 --2- 172.21.15.150:0/4108871814 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe31409a470 0x7fe3140b70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:17.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.907+0000 7fe319861640 1 -- 172.21.15.150:0/4108871814 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe31415fcd0 msgr2=0x7fe3140c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:17.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.907+0000 7fe319861640 1 --2- 172.21.15.150:0/4108871814 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe31415fcd0 0x7fe3140c9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.907+0000 7fe319861640 1 -- 172.21.15.150:0/4108871814 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe31409a470 msgr2=0x7fe3140b70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:17.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.907+0000 7fe319861640 1 --2- 172.21.15.150:0/4108871814 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe31409a470 0x7fe3140b70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:17.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.907+0000 7fe319861640 1 -- 172.21.15.150:0/4108871814 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe314159dd0 con 0x7fe31409ad90 2024-03-17T23:47:17.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.907+0000 7fe319861640 1 --2- 172.21.15.150:0/4108871814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe31409ad90 0x7fe3140c8c10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe30402b950 tx=0x7fe3040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:17.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.907+0000 7fe30affd640 1 -- 172.21.15.150:0/4108871814 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304040070 con 0x7fe31409ad90 2024-03-17T23:47:17.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.907+0000 7fe30affd640 1 -- 172.21.15.150:0/4108871814 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe304031e10 con 0x7fe31409ad90 2024-03-17T23:47:17.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.907+0000 7fe30affd640 1 -- 172.21.15.150:0/4108871814 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304035040 con 0x7fe31409ad90 2024-03-17T23:47:17.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.907+0000 7fe31c2c6640 1 -- 172.21.15.150:0/4108871814 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe31415a0b0 con 0x7fe31409ad90 2024-03-17T23:47:17.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.907+0000 7fe31c2c6640 1 -- 172.21.15.150:0/4108871814 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe31415a670 con 0x7fe31409ad90 2024-03-17T23:47:17.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.908+0000 7fe31c2c6640 1 -- 172.21.15.150:0/4108871814 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3140681a0 con 0x7fe31409ad90 2024-03-17T23:47:17.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.909+0000 7fe30affd640 1 -- 172.21.15.150:0/4108871814 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe30404b060 con 0x7fe31409ad90 2024-03-17T23:47:17.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.909+0000 7fe30affd640 1 --2- 172.21.15.150:0/4108871814 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe2f806bc00 0x7fe2f806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:17.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.909+0000 7fe31a062640 1 --2- 172.21.15.150:0/4108871814 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe2f806bc00 0x7fe2f806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:17.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.909+0000 7fe30affd640 1 -- 172.21.15.150:0/4108871814 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fe3040ac8d0 con 0x7fe31409ad90 2024-03-17T23:47:17.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.910+0000 7fe31a062640 1 --2- 172.21.15.150:0/4108871814 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe2f806bc00 0x7fe2f806e0c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe310009ac0 tx=0x7fe310009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:17.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:17.914+0000 7fe30affd640 1 -- 172.21.15.150:0/4108871814 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe30407aac0 con 0x7fe31409ad90 2024-03-17T23:47:18.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.124+0000 7fe31c2c6640 1 -- 172.21.15.150:0/4108871814 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fe31409b3c0 con 0x7fe31409ad90 2024-03-17T23:47:18.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.124+0000 7fe30affd640 1 -- 172.21.15.150:0/4108871814 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fe30407eb60 con 0x7fe31409ad90 2024-03-17T23:47:18.125 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:18.125 INFO:teuthology.orchestra.run.smithi150.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-17T23:47:18.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.126+0000 7fe31c2c6640 1 -- 172.21.15.150:0/4108871814 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe2f806bc00 msgr2=0x7fe2f806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:18.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.126+0000 7fe31c2c6640 1 --2- 172.21.15.150:0/4108871814 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe2f806bc00 0x7fe2f806e0c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe310009ac0 tx=0x7fe310009190 comp rx=0 tx=0).stop 2024-03-17T23:47:18.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.126+0000 7fe31c2c6640 1 -- 172.21.15.150:0/4108871814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe31409ad90 msgr2=0x7fe3140c8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:18.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.126+0000 7fe31c2c6640 1 --2- 172.21.15.150:0/4108871814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe31409ad90 0x7fe3140c8c10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe30402b950 tx=0x7fe3040046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:18.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.126+0000 7fe31c2c6640 1 -- 172.21.15.150:0/4108871814 shutdown_connections 2024-03-17T23:47:18.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.126+0000 7fe31c2c6640 1 --2- 172.21.15.150:0/4108871814 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe31415fcd0 0x7fe3140c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:18.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.126+0000 7fe31c2c6640 1 --2- 172.21.15.150:0/4108871814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe31409ad90 0x7fe3140c8c10 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:18.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.126+0000 7fe31c2c6640 1 --2- 172.21.15.150:0/4108871814 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe2f806bc00 0x7fe2f806e0c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:18.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.126+0000 7fe31c2c6640 1 --2- 172.21.15.150:0/4108871814 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe31409a470 0x7fe3140b70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:18.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.126+0000 7fe31c2c6640 1 -- 172.21.15.150:0/4108871814 >> 172.21.15.150:0/4108871814 conn(0x7fe3140b3630 msgr2=0x7fe31415e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:18.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.126+0000 7fe31c2c6640 1 -- 172.21.15.150:0/4108871814 shutdown_connections 2024-03-17T23:47:18.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.126+0000 7fe31c2c6640 1 -- 172.21.15.150:0/4108871814 wait complete. 2024-03-17T23:47:18.137 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-17T23:47:18.138 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:18.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.238+0000 7f5e8916f640 1 Processor -- start 2024-03-17T23:47:18.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.238+0000 7f5e8916f640 1 -- start start 2024-03-17T23:47:18.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.239+0000 7f5e8916f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5e84144bb0 0x7f5e8415ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:18.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.239+0000 7f5e8916f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e84145170 con 0x7f5e8415d390 2024-03-17T23:47:18.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.239+0000 7f5e8916f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e84145450 con 0x7f5e840ce6a0 2024-03-17T23:47:18.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.239+0000 7f5e8916f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f5e84145730 con 0x7f5e84144bb0 2024-03-17T23:47:18.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.239+0000 7f5e8259c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f5e8415d390 0x7f5e840cdf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34732/0 (socket says 172.21.15.150:34732) 2024-03-17T23:47:18.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.239+0000 7f5e8259c640 1 -- 172.21.15.150:0/1573028686 learned_addr learned my addr 172.21.15.150:0/1573028686 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:18.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.239+0000 7f5e82d9d640 1 --2- 172.21.15.150:0/1573028686 >> v2:172.21.15.157:3300/0 conn(0x7f5e84144bb0 0x7f5e8415ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:18.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.239+0000 7f5e81d9b640 1 -- 172.21.15.150:0/1573028686 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3254116989 0 0) 0x7f5e84145170 con 0x7f5e8415d390 2024-03-17T23:47:18.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.239+0000 7f5e81d9b640 1 -- 172.21.15.150:0/1573028686 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e6c003520 con 0x7f5e8415d390 2024-03-17T23:47:18.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.239+0000 7f5e82d9d640 1 -- 172.21.15.150:0/1573028686 >> v1:172.21.15.157:6789/0 conn(0x7f5e840ce6a0 legacy=0x7f5e84165a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:18.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.239+0000 7f5e81d9b640 1 -- 172.21.15.150:0/1573028686 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3694119179 0 0) 0x7f5e84145450 con 0x7f5e840ce6a0 2024-03-17T23:47:18.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.239+0000 7f5e82d9d640 1 -- 172.21.15.150:0/1573028686 >> v1:172.21.15.150:6789/0 conn(0x7f5e8415d390 legacy=0x7f5e840cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:18.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.239+0000 7f5e82d9d640 1 -- 172.21.15.150:0/1573028686 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e8406bd60 con 0x7f5e84144bb0 2024-03-17T23:47:18.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.240+0000 7f5e82d9d640 1 --2- 172.21.15.150:0/1573028686 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e84144bb0 0x7f5e8415ce50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5e74009d90 tx=0x7f5e7402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2eb04c9c362aac48 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:18.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.240+0000 7f5e81d9b640 1 -- 172.21.15.150:0/1573028686 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e74039070 con 0x7f5e84144bb0 2024-03-17T23:47:18.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.240+0000 7f5e81d9b640 1 -- 172.21.15.150:0/1573028686 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e7402cc60 con 0x7f5e84144bb0 2024-03-17T23:47:18.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.240+0000 7f5e81d9b640 1 -- 172.21.15.150:0/1573028686 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e74034660 con 0x7f5e84144bb0 2024-03-17T23:47:18.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.241+0000 7f5e8916f640 1 -- 172.21.15.150:0/1573028686 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e84144bb0 msgr2=0x7f5e8415ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:18.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.241+0000 7f5e8916f640 1 --2- 172.21.15.150:0/1573028686 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e84144bb0 0x7f5e8415ce50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5e74009d90 tx=0x7f5e7402beb0 comp rx=0 tx=0).stop 2024-03-17T23:47:18.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.241+0000 7f5e8916f640 1 -- 172.21.15.150:0/1573028686 shutdown_connections 2024-03-17T23:47:18.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.241+0000 7f5e8916f640 1 --2- 172.21.15.150:0/1573028686 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e84144bb0 0x7f5e8415ce50 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:18.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.241+0000 7f5e8916f640 1 -- 172.21.15.150:0/1573028686 >> 172.21.15.150:0/1573028686 conn(0x7f5e8406b190 msgr2=0x7f5e8406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:18.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.241+0000 7f5e8916f640 1 -- 172.21.15.150:0/1573028686 shutdown_connections 2024-03-17T23:47:18.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.241+0000 7f5e8916f640 1 -- 172.21.15.150:0/1573028686 wait complete. 2024-03-17T23:47:18.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.242+0000 7f5e8916f640 1 Processor -- start 2024-03-17T23:47:18.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.242+0000 7f5e8916f640 1 -- start start 2024-03-17T23:47:18.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.243+0000 7f5e8916f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e840ce6a0 0x7f5e84157a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:18.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.243+0000 7f5e8916f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e84144bb0 0x7f5e8413b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:18.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.243+0000 7f5e8916f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e8415d390 0x7f5e840b02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:18.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.243+0000 7f5e8916f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5e84167150 con 0x7f5e840ce6a0 2024-03-17T23:47:18.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.243+0000 7f5e8916f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5e8406f650 con 0x7f5e8415d390 2024-03-17T23:47:18.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.243+0000 7f5e8916f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5e841672d0 con 0x7f5e84144bb0 2024-03-17T23:47:18.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.243+0000 7f5e82d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e840ce6a0 0x7f5e84157a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:18.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.243+0000 7f5e82d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e840ce6a0 0x7f5e84157a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40410/0 (socket says 172.21.15.150:40410) 2024-03-17T23:47:18.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.243+0000 7f5e82d9d640 1 -- 172.21.15.150:0/458941242 learned_addr learned my addr 172.21.15.150:0/458941242 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:18.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.243+0000 7f5e8259c640 1 --2- 172.21.15.150:0/458941242 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e84144bb0 0x7f5e8413b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:18.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.243+0000 7f5e8359e640 1 --2- 172.21.15.150:0/458941242 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e8415d390 0x7f5e840b02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:18.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.243+0000 7f5e82d9d640 1 -- 172.21.15.150:0/458941242 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e84144bb0 msgr2=0x7f5e8413b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:18.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.243+0000 7f5e82d9d640 1 --2- 172.21.15.150:0/458941242 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e84144bb0 0x7f5e8413b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:18.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.243+0000 7f5e82d9d640 1 -- 172.21.15.150:0/458941242 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e8415d390 msgr2=0x7f5e840b02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:18.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.243+0000 7f5e82d9d640 1 --2- 172.21.15.150:0/458941242 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e8415d390 0x7f5e840b02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:18.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.243+0000 7f5e82d9d640 1 -- 172.21.15.150:0/458941242 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e84157320 con 0x7f5e840ce6a0 2024-03-17T23:47:18.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.244+0000 7f5e82d9d640 1 --2- 172.21.15.150:0/458941242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e840ce6a0 0x7f5e84157a40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5e74037040 tx=0x7f5e74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:18.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.244+0000 7f5e6b7fe640 1 -- 172.21.15.150:0/458941242 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e74039050 con 0x7f5e840ce6a0 2024-03-17T23:47:18.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.244+0000 7f5e6b7fe640 1 -- 172.21.15.150:0/458941242 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e7403a040 con 0x7f5e840ce6a0 2024-03-17T23:47:18.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.244+0000 7f5e8916f640 1 -- 172.21.15.150:0/458941242 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e8415b5d0 con 0x7f5e840ce6a0 2024-03-17T23:47:18.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.244+0000 7f5e6b7fe640 1 -- 172.21.15.150:0/458941242 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e74003bd0 con 0x7f5e840ce6a0 2024-03-17T23:47:18.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.244+0000 7f5e8916f640 1 -- 172.21.15.150:0/458941242 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e8415bb90 con 0x7f5e840ce6a0 2024-03-17T23:47:18.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.245+0000 7f5e8916f640 1 -- 172.21.15.150:0/458941242 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e840681a0 con 0x7f5e840ce6a0 2024-03-17T23:47:18.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.245+0000 7f5e6b7fe640 1 -- 172.21.15.150:0/458941242 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5e74004030 con 0x7f5e840ce6a0 2024-03-17T23:47:18.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.245+0000 7f5e6b7fe640 1 --2- 172.21.15.150:0/458941242 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5e7006bc00 0x7f5e7006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:18.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.246+0000 7f5e6b7fe640 1 -- 172.21.15.150:0/458941242 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f5e740ad630 con 0x7f5e840ce6a0 2024-03-17T23:47:18.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.246+0000 7f5e8259c640 1 --2- 172.21.15.150:0/458941242 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5e7006bc00 0x7f5e7006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:18.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.246+0000 7f5e8259c640 1 --2- 172.21.15.150:0/458941242 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5e7006bc00 0x7f5e7006e0c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5e84166c20 tx=0x7f5e78009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:18.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.250+0000 7f5e6b7fe640 1 -- 172.21.15.150:0/458941242 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5e7407b790 con 0x7f5e840ce6a0 2024-03-17T23:47:18.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.422+0000 7f5e8916f640 1 -- 172.21.15.150:0/458941242 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5e8415bdd0 con 0x7f5e840ce6a0 2024-03-17T23:47:18.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.423+0000 7f5e6b7fe640 1 -- 172.21.15.150:0/458941242 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18550 (secure 0 0 0) 0x7f5e7407f830 con 0x7f5e840ce6a0 2024-03-17T23:47:18.424 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:18.424 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:01.940019+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:18.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.425+0000 7f5e8916f640 1 -- 172.21.15.150:0/458941242 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5e7006bc00 msgr2=0x7f5e7006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:18.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.425+0000 7f5e8916f640 1 --2- 172.21.15.150:0/458941242 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5e7006bc00 0x7f5e7006e0c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5e84166c20 tx=0x7f5e78009040 comp rx=0 tx=0).stop 2024-03-17T23:47:18.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.425+0000 7f5e8916f640 1 -- 172.21.15.150:0/458941242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e840ce6a0 msgr2=0x7f5e84157a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:18.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.425+0000 7f5e8916f640 1 --2- 172.21.15.150:0/458941242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e840ce6a0 0x7f5e84157a40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5e74037040 tx=0x7f5e74004a40 comp rx=0 tx=0).stop 2024-03-17T23:47:18.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.425+0000 7f5e8916f640 1 -- 172.21.15.150:0/458941242 shutdown_connections 2024-03-17T23:47:18.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.425+0000 7f5e8916f640 1 --2- 172.21.15.150:0/458941242 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5e7006bc00 0x7f5e7006e0c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:18.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.425+0000 7f5e8916f640 1 --2- 172.21.15.150:0/458941242 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e8415d390 0x7f5e840b02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:18.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.425+0000 7f5e8916f640 1 --2- 172.21.15.150:0/458941242 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e84144bb0 0x7f5e8413b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:18.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.425+0000 7f5e8916f640 1 --2- 172.21.15.150:0/458941242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e840ce6a0 0x7f5e84157a40 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:18.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.425+0000 7f5e8916f640 1 -- 172.21.15.150:0/458941242 >> 172.21.15.150:0/458941242 conn(0x7f5e8406b190 msgr2=0x7f5e840b1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:18.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.425+0000 7f5e8916f640 1 -- 172.21.15.150:0/458941242 shutdown_connections 2024-03-17T23:47:18.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.425+0000 7f5e8916f640 1 -- 172.21.15.150:0/458941242 wait complete. 2024-03-17T23:47:18.437 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:46:53.610678+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:46:54.227335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-17T23:46:57.949235+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-17T23:47:18.438 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:47:18.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.540+0000 7f495cc59640 1 Processor -- start 2024-03-17T23:47:18.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.541+0000 7f495cc59640 1 -- start start 2024-03-17T23:47:18.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.541+0000 7f495cc59640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f495815e0a0 0x7f495815e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:18.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.541+0000 7f495cc59640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f495806c2a0 0x7f4958099db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:18.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.541+0000 7f495cc59640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49580d0040 con 0x7f495809a2f0 2024-03-17T23:47:18.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.541+0000 7f495cc59640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f495809a740 con 0x7f495815e0a0 2024-03-17T23:47:18.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.541+0000 7f495cc59640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f495815edb0 con 0x7f495806c2a0 2024-03-17T23:47:18.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.542+0000 7f495659c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f495815e0a0 0x7f495815e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:18.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.542+0000 7f495659c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f495815e0a0 0x7f495815e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40424/0 (socket says 172.21.15.150:40424) 2024-03-17T23:47:18.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.542+0000 7f495659c640 1 -- 172.21.15.150:0/3186936886 learned_addr learned my addr 172.21.15.150:0/3186936886 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:18.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.542+0000 7f4955d9b640 1 --2- 172.21.15.150:0/3186936886 >> v2:172.21.15.157:3300/0 conn(0x7f495806c2a0 0x7f4958099db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:18.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.542+0000 7f495659c640 1 -- 172.21.15.150:0/3186936886 >> v2:172.21.15.157:3300/0 conn(0x7f495806c2a0 msgr2=0x7f4958099db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:18.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.542+0000 7f495659c640 1 --2- 172.21.15.150:0/3186936886 >> v2:172.21.15.157:3300/0 conn(0x7f495806c2a0 0x7f4958099db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:18.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.542+0000 7f495659c640 1 -- 172.21.15.150:0/3186936886 >> v1:172.21.15.157:6789/0 conn(0x7f495809a2f0 legacy=0x7f4958161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:18.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.542+0000 7f495659c640 1 -- 172.21.15.150:0/3186936886 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f495815f090 con 0x7f495815e0a0 2024-03-17T23:47:18.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.543+0000 7f495659c640 1 --2- 172.21.15.150:0/3186936886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f495815e0a0 0x7f495815e480 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4958072b70 tx=0x7f494802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6320dbc2781e402 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:18.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.543+0000 7f495559a640 1 -- 172.21.15.150:0/3186936886 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4948034070 con 0x7f495815e0a0 2024-03-17T23:47:18.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.543+0000 7f495559a640 1 -- 172.21.15.150:0/3186936886 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4948030860 con 0x7f495815e0a0 2024-03-17T23:47:18.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.543+0000 7f495559a640 1 -- 172.21.15.150:0/3186936886 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4948030c30 con 0x7f495815e0a0 2024-03-17T23:47:18.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.543+0000 7f495cc59640 1 -- 172.21.15.150:0/3186936886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f495815e0a0 msgr2=0x7f495815e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:18.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.543+0000 7f495cc59640 1 --2- 172.21.15.150:0/3186936886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f495815e0a0 0x7f495815e480 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4958072b70 tx=0x7f494802b9a0 comp rx=0 tx=0).stop 2024-03-17T23:47:18.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.543+0000 7f495cc59640 1 -- 172.21.15.150:0/3186936886 shutdown_connections 2024-03-17T23:47:18.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.543+0000 7f495cc59640 1 --2- 172.21.15.150:0/3186936886 >> v2:172.21.15.157:3300/0 conn(0x7f495806c2a0 0x7f4958099db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:18.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.543+0000 7f495cc59640 1 --2- 172.21.15.150:0/3186936886 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f495815e0a0 0x7f495815e480 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:18.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.543+0000 7f495cc59640 1 -- 172.21.15.150:0/3186936886 >> 172.21.15.150:0/3186936886 conn(0x7f49580b36c0 msgr2=0x7f49580b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:18.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.543+0000 7f495cc59640 1 -- 172.21.15.150:0/3186936886 shutdown_connections 2024-03-17T23:47:18.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.543+0000 7f495cc59640 1 -- 172.21.15.150:0/3186936886 wait complete. 2024-03-17T23:47:18.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.545+0000 7f495cc59640 1 Processor -- start 2024-03-17T23:47:18.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.545+0000 7f495cc59640 1 -- start start 2024-03-17T23:47:18.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.545+0000 7f495cc59640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f495806c2a0 0x7f495814fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:18.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.546+0000 7f495cc59640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f495809a2f0 0x7f49580b6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:18.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.546+0000 7f495cc59640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f495815e0a0 0x7f49580b6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:18.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.546+0000 7f495cc59640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f495815e9c0 con 0x7f495809a2f0 2024-03-17T23:47:18.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.546+0000 7f495cc59640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f495815f650 con 0x7f495806c2a0 2024-03-17T23:47:18.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.546+0000 7f495cc59640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f495815eb40 con 0x7f495815e0a0 2024-03-17T23:47:18.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.546+0000 7f4955d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f495809a2f0 0x7f49580b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:18.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.546+0000 7f4955d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f495809a2f0 0x7f49580b6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40430/0 (socket says 172.21.15.150:40430) 2024-03-17T23:47:18.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.546+0000 7f4955d9b640 1 -- 172.21.15.150:0/2683511325 learned_addr learned my addr 172.21.15.150:0/2683511325 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:18.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.546+0000 7f495659c640 1 --2- 172.21.15.150:0/2683511325 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f495806c2a0 0x7f495814fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:18.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.546+0000 7f4956d9d640 1 --2- 172.21.15.150:0/2683511325 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f495815e0a0 0x7f49580b6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:18.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.546+0000 7f4955d9b640 1 -- 172.21.15.150:0/2683511325 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f495815e0a0 msgr2=0x7f49580b6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:18.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.546+0000 7f4955d9b640 1 --2- 172.21.15.150:0/2683511325 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f495815e0a0 0x7f49580b6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:18.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.546+0000 7f4955d9b640 1 -- 172.21.15.150:0/2683511325 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f495806c2a0 msgr2=0x7f495814fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:18.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.546+0000 7f4955d9b640 1 --2- 172.21.15.150:0/2683511325 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f495806c2a0 0x7f495814fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:18.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.546+0000 7f4955d9b640 1 -- 172.21.15.150:0/2683511325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49580b70f0 con 0x7f495809a2f0 2024-03-17T23:47:18.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.546+0000 7f4955d9b640 1 --2- 172.21.15.150:0/2683511325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f495809a2f0 0x7f49580b6580 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f494c00e9b0 tx=0x7f494c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:18.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.546+0000 7f4946ffd640 1 -- 172.21.15.150:0/2683511325 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f494c017070 con 0x7f495809a2f0 2024-03-17T23:47:18.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.546+0000 7f4946ffd640 1 -- 172.21.15.150:0/2683511325 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f494c00c9a0 con 0x7f495809a2f0 2024-03-17T23:47:18.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.546+0000 7f495cc59640 1 -- 172.21.15.150:0/2683511325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49580b73d0 con 0x7f495809a2f0 2024-03-17T23:47:18.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.546+0000 7f4946ffd640 1 -- 172.21.15.150:0/2683511325 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f494c00cd90 con 0x7f495809a2f0 2024-03-17T23:47:18.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.546+0000 7f495cc59640 1 -- 172.21.15.150:0/2683511325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49580b0e90 con 0x7f495809a2f0 2024-03-17T23:47:18.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.547+0000 7f495cc59640 1 -- 172.21.15.150:0/2683511325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49580688b0 con 0x7f495809a2f0 2024-03-17T23:47:18.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.548+0000 7f4946ffd640 1 -- 172.21.15.150:0/2683511325 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f494c01a070 con 0x7f495809a2f0 2024-03-17T23:47:18.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.548+0000 7f4946ffd640 1 --2- 172.21.15.150:0/2683511325 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f493806bc20 0x7f493806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:18.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.548+0000 7f495659c640 1 --2- 172.21.15.150:0/2683511325 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f493806bc20 0x7f493806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:18.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.548+0000 7f4946ffd640 1 -- 172.21.15.150:0/2683511325 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f494c08af60 con 0x7f495809a2f0 2024-03-17T23:47:18.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.549+0000 7f495659c640 1 --2- 172.21.15.150:0/2683511325 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f493806bc20 0x7f493806e0e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4948036000 tx=0x7f494803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:18.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.553+0000 7f4946ffd640 1 -- 172.21.15.150:0/2683511325 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f494c059230 con 0x7f495809a2f0 2024-03-17T23:47:18.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.722+0000 7f495cc59640 1 -- 172.21.15.150:0/2683511325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f49580c8a10 con 0x7f495809a2f0 2024-03-17T23:47:18.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.723+0000 7f4946ffd640 1 -- 172.21.15.150:0/2683511325 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f494c05d2d0 con 0x7f495809a2f0 2024-03-17T23:47:18.723 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:47:18.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.725+0000 7f495cc59640 1 -- 172.21.15.150:0/2683511325 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f493806bc20 msgr2=0x7f493806e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:18.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.725+0000 7f495cc59640 1 --2- 172.21.15.150:0/2683511325 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f493806bc20 0x7f493806e0e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4948036000 tx=0x7f494803b040 comp rx=0 tx=0).stop 2024-03-17T23:47:18.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.725+0000 7f495cc59640 1 -- 172.21.15.150:0/2683511325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f495809a2f0 msgr2=0x7f49580b6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:18.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.725+0000 7f495cc59640 1 --2- 172.21.15.150:0/2683511325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f495809a2f0 0x7f49580b6580 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f494c00e9b0 tx=0x7f494c00ee70 comp rx=0 tx=0).stop 2024-03-17T23:47:18.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.725+0000 7f495cc59640 1 -- 172.21.15.150:0/2683511325 shutdown_connections 2024-03-17T23:47:18.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.725+0000 7f495cc59640 1 --2- 172.21.15.150:0/2683511325 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f495815e0a0 0x7f49580b6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:18.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.725+0000 7f495cc59640 1 --2- 172.21.15.150:0/2683511325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f495809a2f0 0x7f49580b6580 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:18.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.725+0000 7f495cc59640 1 --2- 172.21.15.150:0/2683511325 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f493806bc20 0x7f493806e0e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:18.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.725+0000 7f495cc59640 1 --2- 172.21.15.150:0/2683511325 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f495806c2a0 0x7f495814fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:18.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.725+0000 7f495cc59640 1 -- 172.21.15.150:0/2683511325 >> 172.21.15.150:0/2683511325 conn(0x7f49580b36c0 msgr2=0x7f495809af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:18.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.725+0000 7f495cc59640 1 -- 172.21.15.150:0/2683511325 shutdown_connections 2024-03-17T23:47:18.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.725+0000 7f495cc59640 1 -- 172.21.15.150:0/2683511325 wait complete. 2024-03-17T23:47:18.736 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:47:18.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.839+0000 7f5b8e4bd640 1 Processor -- start 2024-03-17T23:47:18.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.839+0000 7f5b8e4bd640 1 -- start start 2024-03-17T23:47:18.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.840+0000 7f5b8e4bd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b881457d0 0x7f5b8809a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:18.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.840+0000 7f5b8e4bd640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b880b1e20 con 0x7f5b880d10f0 2024-03-17T23:47:18.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.840+0000 7f5b8e4bd640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b880b2100 con 0x7f5b8806b8a0 2024-03-17T23:47:18.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.840+0000 7f5b8e4bd640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5b880d0980 con 0x7f5b881457d0 2024-03-17T23:47:18.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.840+0000 7f5b877fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f5b880d10f0 0x7f5b8806b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34734/0 (socket says 172.21.15.150:34734) 2024-03-17T23:47:18.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.840+0000 7f5b877fe640 1 -- 172.21.15.150:0/810897481 learned_addr learned my addr 172.21.15.150:0/810897481 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:18.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.840+0000 7f5b87fff640 1 --2- 172.21.15.150:0/810897481 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b881457d0 0x7f5b8809a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:18.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.840+0000 7f5b86ffd640 1 -- 172.21.15.150:0/810897481 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1925966367 0 0) 0x7f5b880b2100 con 0x7f5b8806b8a0 2024-03-17T23:47:18.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.840+0000 7f5b87fff640 1 -- 172.21.15.150:0/810897481 >> v1:172.21.15.157:6789/0 conn(0x7f5b8806b8a0 legacy=0x7f5b8815cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:18.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.840+0000 7f5b87fff640 1 -- 172.21.15.150:0/810897481 >> v1:172.21.15.150:6789/0 conn(0x7f5b880d10f0 legacy=0x7f5b8806b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:18.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.840+0000 7f5b87fff640 1 -- 172.21.15.150:0/810897481 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b880b23e0 con 0x7f5b881457d0 2024-03-17T23:47:18.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.841+0000 7f5b87fff640 1 --2- 172.21.15.150:0/810897481 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b881457d0 0x7f5b8809a180 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5b780097d0 tx=0x7f5b7802b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ace2a1c3f272c13 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:18.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.841+0000 7f5b86ffd640 1 -- 172.21.15.150:0/810897481 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b78037070 con 0x7f5b881457d0 2024-03-17T23:47:18.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.841+0000 7f5b86ffd640 1 -- 172.21.15.150:0/810897481 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b78030710 con 0x7f5b881457d0 2024-03-17T23:47:18.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.841+0000 7f5b86ffd640 1 -- 172.21.15.150:0/810897481 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b78030ae0 con 0x7f5b881457d0 2024-03-17T23:47:18.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.841+0000 7f5b8e4bd640 1 -- 172.21.15.150:0/810897481 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b881457d0 msgr2=0x7f5b8809a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:18.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.841+0000 7f5b8e4bd640 1 --2- 172.21.15.150:0/810897481 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b881457d0 0x7f5b8809a180 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5b780097d0 tx=0x7f5b7802b770 comp rx=0 tx=0).stop 2024-03-17T23:47:18.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.842+0000 7f5b8e4bd640 1 -- 172.21.15.150:0/810897481 shutdown_connections 2024-03-17T23:47:18.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.842+0000 7f5b8e4bd640 1 --2- 172.21.15.150:0/810897481 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b881457d0 0x7f5b8809a180 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:18.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.842+0000 7f5b8e4bd640 1 -- 172.21.15.150:0/810897481 >> 172.21.15.150:0/810897481 conn(0x7f5b880b6170 msgr2=0x7f5b88099a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:18.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.842+0000 7f5b8e4bd640 1 -- 172.21.15.150:0/810897481 shutdown_connections 2024-03-17T23:47:18.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.842+0000 7f5b8e4bd640 1 -- 172.21.15.150:0/810897481 wait complete. 2024-03-17T23:47:18.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.843+0000 7f5b8e4bd640 1 Processor -- start 2024-03-17T23:47:18.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.843+0000 7f5b8e4bd640 1 -- start start 2024-03-17T23:47:18.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.844+0000 7f5b8e4bd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b8806b8a0 0x7f5b8815b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:18.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.844+0000 7f5b8e4bd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b880d10f0 0x7f5b880cf960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:18.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.844+0000 7f5b8e4bd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b881457d0 0x7f5b880b4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:18.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.844+0000 7f5b8e4bd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5b880d0c60 con 0x7f5b880d10f0 2024-03-17T23:47:18.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.844+0000 7f5b8e4bd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5b880d1760 con 0x7f5b8806b8a0 2024-03-17T23:47:18.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.844+0000 7f5b8e4bd640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5b8815d4a0 con 0x7f5b881457d0 2024-03-17T23:47:18.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.844+0000 7f5b877fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b880d10f0 0x7f5b880cf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:18.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.844+0000 7f5b87fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b8806b8a0 0x7f5b8815b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:18.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.844+0000 7f5b877fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b880d10f0 0x7f5b880cf960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40434/0 (socket says 172.21.15.150:40434) 2024-03-17T23:47:18.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.844+0000 7f5b877fe640 1 -- 172.21.15.150:0/3527328893 learned_addr learned my addr 172.21.15.150:0/3527328893 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:18.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.844+0000 7f5b8ca5a640 1 --2- 172.21.15.150:0/3527328893 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b881457d0 0x7f5b880b4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:18.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.844+0000 7f5b87fff640 1 -- 172.21.15.150:0/3527328893 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b881457d0 msgr2=0x7f5b880b4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:18.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.844+0000 7f5b87fff640 1 --2- 172.21.15.150:0/3527328893 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b881457d0 0x7f5b880b4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:18.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.844+0000 7f5b87fff640 1 -- 172.21.15.150:0/3527328893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b880d10f0 msgr2=0x7f5b880cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:18.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.844+0000 7f5b87fff640 1 --2- 172.21.15.150:0/3527328893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b880d10f0 0x7f5b880cf960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:18.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.844+0000 7f5b87fff640 1 -- 172.21.15.150:0/3527328893 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b880cd940 con 0x7f5b8806b8a0 2024-03-17T23:47:18.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.844+0000 7f5b87fff640 1 --2- 172.21.15.150:0/3527328893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b8806b8a0 0x7f5b8815b4c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5b78032000 tx=0x7f5b780046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:18.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.844+0000 7f5b84ff9640 1 -- 172.21.15.150:0/3527328893 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b78037040 con 0x7f5b8806b8a0 2024-03-17T23:47:18.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.844+0000 7f5b84ff9640 1 -- 172.21.15.150:0/3527328893 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b78004250 con 0x7f5b8806b8a0 2024-03-17T23:47:18.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.844+0000 7f5b8e4bd640 1 -- 172.21.15.150:0/3527328893 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b880cd380 con 0x7f5b8806b8a0 2024-03-17T23:47:18.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.844+0000 7f5b84ff9640 1 -- 172.21.15.150:0/3527328893 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b78032730 con 0x7f5b8806b8a0 2024-03-17T23:47:18.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.844+0000 7f5b8e4bd640 1 -- 172.21.15.150:0/3527328893 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b880b4180 con 0x7f5b8806b8a0 2024-03-17T23:47:18.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.845+0000 7f5b8e4bd640 1 -- 172.21.15.150:0/3527328893 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b880681a0 con 0x7f5b8806b8a0 2024-03-17T23:47:18.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.846+0000 7f5b84ff9640 1 -- 172.21.15.150:0/3527328893 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5b780305d0 con 0x7f5b8806b8a0 2024-03-17T23:47:18.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.846+0000 7f5b84ff9640 1 --2- 172.21.15.150:0/3527328893 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5b7406bbd0 0x7f5b7406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:18.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.846+0000 7f5b84ff9640 1 -- 172.21.15.150:0/3527328893 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f5b780ad050 con 0x7f5b8806b8a0 2024-03-17T23:47:18.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.846+0000 7f5b877fe640 1 --2- 172.21.15.150:0/3527328893 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5b7406bbd0 0x7f5b7406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:18.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.847+0000 7f5b877fe640 1 --2- 172.21.15.150:0/3527328893 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5b7406bbd0 0x7f5b7406e090 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5b8813f120 tx=0x7f5b7c000bc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:18.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:18.851+0000 7f5b84ff9640 1 -- 172.21.15.150:0/3527328893 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5b7807b190 con 0x7f5b8806b8a0 2024-03-17T23:47:19.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.024+0000 7f5b8e4bd640 1 -- 172.21.15.150:0/3527328893 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5b880ccf40 con 0x7f5b8806b8a0 2024-03-17T23:47:19.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.024+0000 7f5b84ff9640 1 -- 172.21.15.150:0/3527328893 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f5b7807f230 con 0x7f5b8806b8a0 2024-03-17T23:47:19.025 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:19.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.026+0000 7f5b8e4bd640 1 -- 172.21.15.150:0/3527328893 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5b7406bbd0 msgr2=0x7f5b7406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:19.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.026+0000 7f5b8e4bd640 1 --2- 172.21.15.150:0/3527328893 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5b7406bbd0 0x7f5b7406e090 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5b8813f120 tx=0x7f5b7c000bc0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.026+0000 7f5b8e4bd640 1 -- 172.21.15.150:0/3527328893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b8806b8a0 msgr2=0x7f5b8815b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:19.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.026+0000 7f5b8e4bd640 1 --2- 172.21.15.150:0/3527328893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b8806b8a0 0x7f5b8815b4c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5b78032000 tx=0x7f5b780046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.026+0000 7f5b8e4bd640 1 -- 172.21.15.150:0/3527328893 shutdown_connections 2024-03-17T23:47:19.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.026+0000 7f5b8e4bd640 1 --2- 172.21.15.150:0/3527328893 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b881457d0 0x7f5b880b4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.026+0000 7f5b8e4bd640 1 --2- 172.21.15.150:0/3527328893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b880d10f0 0x7f5b880cf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.026+0000 7f5b8e4bd640 1 --2- 172.21.15.150:0/3527328893 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5b7406bbd0 0x7f5b7406e090 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.026+0000 7f5b8e4bd640 1 --2- 172.21.15.150:0/3527328893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b8806b8a0 0x7f5b8815b4c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.026+0000 7f5b8e4bd640 1 -- 172.21.15.150:0/3527328893 >> 172.21.15.150:0/3527328893 conn(0x7f5b880b6170 msgr2=0x7f5b8806ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:19.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.027+0000 7f5b8e4bd640 1 -- 172.21.15.150:0/3527328893 shutdown_connections 2024-03-17T23:47:19.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.027+0000 7f5b8e4bd640 1 -- 172.21.15.150:0/3527328893 wait complete. 2024-03-17T23:47:19.038 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:47:19.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.140+0000 7fa1c76ff640 1 Processor -- start 2024-03-17T23:47:19.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.141+0000 7fa1c76ff640 1 -- start start 2024-03-17T23:47:19.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.141+0000 7fa1c76ff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa1c01615d0 0x7fa1c00d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:19.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.141+0000 7fa1c76ff640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1c00d2870 con 0x7fa1c00d3910 2024-03-17T23:47:19.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.141+0000 7fa1c76ff640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1c00d2b50 con 0x7fa1c00d0e30 2024-03-17T23:47:19.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.141+0000 7fa1c76ff640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa1c01619b0 con 0x7fa1c01615d0 2024-03-17T23:47:19.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.142+0000 7fa1c4c9a640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa1c00d3910 0x7fa1c00d0720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34750/0 (socket says 172.21.15.150:34750) 2024-03-17T23:47:19.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.142+0000 7fa1c4c9a640 1 -- 172.21.15.150:0/4186193710 learned_addr learned my addr 172.21.15.150:0/4186193710 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:19.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.142+0000 7fa1c549b640 1 --2- 172.21.15.150:0/4186193710 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa1c01615d0 0x7fa1c00d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:19.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.142+0000 7fa1b7fff640 1 -- 172.21.15.150:0/4186193710 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2900560152 0 0) 0x7fa1c00d2870 con 0x7fa1c00d3910 2024-03-17T23:47:19.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.142+0000 7fa1b7fff640 1 -- 172.21.15.150:0/4186193710 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1ac003520 con 0x7fa1c00d3910 2024-03-17T23:47:19.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.142+0000 7fa1b7fff640 1 -- 172.21.15.150:0/4186193710 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1250240943 0 0) 0x7fa1c00d2b50 con 0x7fa1c00d0e30 2024-03-17T23:47:19.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.142+0000 7fa1b7fff640 1 -- 172.21.15.150:0/4186193710 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1c00d2870 con 0x7fa1c00d0e30 2024-03-17T23:47:19.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.142+0000 7fa1c549b640 1 -- 172.21.15.150:0/4186193710 >> v1:172.21.15.157:6789/0 conn(0x7fa1c00d0e30 legacy=0x7fa1c00b6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:19.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.142+0000 7fa1c549b640 1 -- 172.21.15.150:0/4186193710 >> v1:172.21.15.150:6789/0 conn(0x7fa1c00d3910 legacy=0x7fa1c00d0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:19.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.142+0000 7fa1c549b640 1 -- 172.21.15.150:0/4186193710 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1c00d2e30 con 0x7fa1c01615d0 2024-03-17T23:47:19.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.143+0000 7fa1c549b640 1 --2- 172.21.15.150:0/4186193710 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa1c01615d0 0x7fa1c00d2020 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc0097d0 tx=0x7fa1bc02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26d08158ea7699d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:19.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.143+0000 7fa1b7fff640 1 -- 172.21.15.150:0/4186193710 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc035020 con 0x7fa1c01615d0 2024-03-17T23:47:19.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.143+0000 7fa1b7fff640 1 -- 172.21.15.150:0/4186193710 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1bc031890 con 0x7fa1c01615d0 2024-03-17T23:47:19.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.143+0000 7fa1b7fff640 1 -- 172.21.15.150:0/4186193710 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc031c60 con 0x7fa1c01615d0 2024-03-17T23:47:19.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.143+0000 7fa1c76ff640 1 -- 172.21.15.150:0/4186193710 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa1c01615d0 msgr2=0x7fa1c00d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:19.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.143+0000 7fa1c76ff640 1 --2- 172.21.15.150:0/4186193710 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa1c01615d0 0x7fa1c00d2020 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc0097d0 tx=0x7fa1bc02b930 comp rx=0 tx=0).stop 2024-03-17T23:47:19.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.143+0000 7fa1c76ff640 1 -- 172.21.15.150:0/4186193710 shutdown_connections 2024-03-17T23:47:19.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.143+0000 7fa1c76ff640 1 --2- 172.21.15.150:0/4186193710 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa1c01615d0 0x7fa1c00d2020 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.143+0000 7fa1c76ff640 1 -- 172.21.15.150:0/4186193710 >> 172.21.15.150:0/4186193710 conn(0x7fa1c00730b0 msgr2=0x7fa1c00b5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:19.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.144+0000 7fa1c76ff640 1 -- 172.21.15.150:0/4186193710 shutdown_connections 2024-03-17T23:47:19.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.144+0000 7fa1c76ff640 1 -- 172.21.15.150:0/4186193710 wait complete. 2024-03-17T23:47:19.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.145+0000 7fa1c76ff640 1 Processor -- start 2024-03-17T23:47:19.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.145+0000 7fa1c76ff640 1 -- start start 2024-03-17T23:47:19.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.145+0000 7fa1c76ff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa1c00d0e30 0x7fa1c01f3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:19.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.145+0000 7fa1c76ff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa1c00d3910 0x7fa1c01531d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:19.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.145+0000 7fa1c76ff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1c01615d0 0x7fa1c00ad380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:19.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.145+0000 7fa1c76ff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c00d2560 con 0x7fa1c01615d0 2024-03-17T23:47:19.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.145+0000 7fa1c76ff640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c00d33f0 con 0x7fa1c00d0e30 2024-03-17T23:47:19.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.145+0000 7fa1c76ff640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa1c00b9660 con 0x7fa1c00d3910 2024-03-17T23:47:19.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.145+0000 7fa1c5c9c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1c01615d0 0x7fa1c00ad380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:19.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.145+0000 7fa1c5c9c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1c01615d0 0x7fa1c00ad380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40436/0 (socket says 172.21.15.150:40436) 2024-03-17T23:47:19.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.145+0000 7fa1c5c9c640 1 -- 172.21.15.150:0/3658695695 learned_addr learned my addr 172.21.15.150:0/3658695695 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:19.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.145+0000 7fa1c549b640 1 --2- 172.21.15.150:0/3658695695 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa1c00d0e30 0x7fa1c01f3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:19.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.145+0000 7fa1c4c9a640 1 --2- 172.21.15.150:0/3658695695 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa1c00d3910 0x7fa1c01531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:19.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.146+0000 7fa1c5c9c640 1 -- 172.21.15.150:0/3658695695 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa1c00d3910 msgr2=0x7fa1c01531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:19.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.146+0000 7fa1c5c9c640 1 --2- 172.21.15.150:0/3658695695 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa1c00d3910 0x7fa1c01531d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.146+0000 7fa1c5c9c640 1 -- 172.21.15.150:0/3658695695 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa1c00d0e30 msgr2=0x7fa1c01f3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:19.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.146+0000 7fa1c5c9c640 1 --2- 172.21.15.150:0/3658695695 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa1c00d0e30 0x7fa1c01f3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.146+0000 7fa1c5c9c640 1 -- 172.21.15.150:0/3658695695 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1c0153710 con 0x7fa1c01615d0 2024-03-17T23:47:19.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.146+0000 7fa1c5c9c640 1 --2- 172.21.15.150:0/3658695695 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1c01615d0 0x7fa1c00ad380 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa1b800c7a0 tx=0x7fa1b800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:19.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.146+0000 7fa1b5ffb640 1 -- 172.21.15.150:0/3658695695 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b8003b70 con 0x7fa1c01615d0 2024-03-17T23:47:19.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.146+0000 7fa1b5ffb640 1 -- 172.21.15.150:0/3658695695 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1b8003d10 con 0x7fa1c01615d0 2024-03-17T23:47:19.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.146+0000 7fa1c76ff640 1 -- 172.21.15.150:0/3658695695 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1c00aade0 con 0x7fa1c01615d0 2024-03-17T23:47:19.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.146+0000 7fa1b5ffb640 1 -- 172.21.15.150:0/3658695695 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b800aa90 con 0x7fa1c01615d0 2024-03-17T23:47:19.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.146+0000 7fa1c76ff640 1 -- 172.21.15.150:0/3658695695 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1c00ab3a0 con 0x7fa1c01615d0 2024-03-17T23:47:19.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.147+0000 7fa1c76ff640 1 -- 172.21.15.150:0/3658695695 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1c00688b0 con 0x7fa1c01615d0 2024-03-17T23:47:19.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.147+0000 7fa1b5ffb640 1 -- 172.21.15.150:0/3658695695 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa1b800ac30 con 0x7fa1c01615d0 2024-03-17T23:47:19.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.148+0000 7fa1b5ffb640 1 --2- 172.21.15.150:0/3658695695 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa1a406bbd0 0x7fa1a406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:19.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.148+0000 7fa1b5ffb640 1 -- 172.21.15.150:0/3658695695 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fa1b808ab50 con 0x7fa1c01615d0 2024-03-17T23:47:19.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.148+0000 7fa1c549b640 1 --2- 172.21.15.150:0/3658695695 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa1a406bbd0 0x7fa1a406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:19.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.148+0000 7fa1c549b640 1 --2- 172.21.15.150:0/3658695695 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa1a406bbd0 0x7fa1a406e090 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc036000 tx=0x7fa1bc008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:19.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.152+0000 7fa1b5ffb640 1 -- 172.21.15.150:0/3658695695 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa1c0153710 con 0x7fa1c01615d0 2024-03-17T23:47:19.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.324+0000 7fa1c76ff640 1 -- 172.21.15.150:0/3658695695 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa1c00d4430 con 0x7fa1c01615d0 2024-03-17T23:47:19.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.324+0000 7fa1b5ffb640 1 -- 172.21.15.150:0/3658695695 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fa1b8058d40 con 0x7fa1c01615d0 2024-03-17T23:47:19.325 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:47:19.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.326+0000 7fa1c76ff640 1 -- 172.21.15.150:0/3658695695 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa1a406bbd0 msgr2=0x7fa1a406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:19.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.326+0000 7fa1c76ff640 1 --2- 172.21.15.150:0/3658695695 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa1a406bbd0 0x7fa1a406e090 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc036000 tx=0x7fa1bc008030 comp rx=0 tx=0).stop 2024-03-17T23:47:19.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.326+0000 7fa1c76ff640 1 -- 172.21.15.150:0/3658695695 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1c01615d0 msgr2=0x7fa1c00ad380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:19.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.326+0000 7fa1c76ff640 1 --2- 172.21.15.150:0/3658695695 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1c01615d0 0x7fa1c00ad380 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa1b800c7a0 tx=0x7fa1b800cc60 comp rx=0 tx=0).stop 2024-03-17T23:47:19.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.326+0000 7fa1c76ff640 1 -- 172.21.15.150:0/3658695695 shutdown_connections 2024-03-17T23:47:19.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.326+0000 7fa1c76ff640 1 --2- 172.21.15.150:0/3658695695 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1c01615d0 0x7fa1c00ad380 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.326+0000 7fa1c76ff640 1 --2- 172.21.15.150:0/3658695695 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa1c00d3910 0x7fa1c01531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.326+0000 7fa1c76ff640 1 --2- 172.21.15.150:0/3658695695 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa1a406bbd0 0x7fa1a406e090 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.326+0000 7fa1c76ff640 1 --2- 172.21.15.150:0/3658695695 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa1c00d0e30 0x7fa1c01f3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.326+0000 7fa1c76ff640 1 -- 172.21.15.150:0/3658695695 >> 172.21.15.150:0/3658695695 conn(0x7fa1c00730b0 msgr2=0x7fa1c00ce5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:19.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.326+0000 7fa1c76ff640 1 -- 172.21.15.150:0/3658695695 shutdown_connections 2024-03-17T23:47:19.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.326+0000 7fa1c76ff640 1 -- 172.21.15.150:0/3658695695 wait complete. 2024-03-17T23:47:19.338 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-17T23:47:19.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.440+0000 7f601a67e640 1 Processor -- start 2024-03-17T23:47:19.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.440+0000 7f601a67e640 1 -- start start 2024-03-17T23:47:19.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.441+0000 7f601a67e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f601406f400 0x7f60140cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:19.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.441+0000 7f601a67e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f60140ce4d0 0x7f6014165a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:19.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.441+0000 7f601a67e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6014145170 con 0x7f6014144bb0 2024-03-17T23:47:19.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.441+0000 7f601a67e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f60140ce920 con 0x7f60140ce4d0 2024-03-17T23:47:19.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.441+0000 7f601a67e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6014145450 con 0x7f601406f400 2024-03-17T23:47:19.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.441+0000 7f6013fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f6014144bb0 0x7f601415ce50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34760/0 (socket says 172.21.15.150:34760) 2024-03-17T23:47:19.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.441+0000 7f6013fff640 1 -- 172.21.15.150:0/2710994601 learned_addr learned my addr 172.21.15.150:0/2710994601 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:19.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.441+0000 7f60137fe640 1 --2- 172.21.15.150:0/2710994601 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f601406f400 0x7f60140cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:19.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.441+0000 7f6018c1b640 1 --2- 172.21.15.150:0/2710994601 >> v2:172.21.15.157:3300/0 conn(0x7f60140ce4d0 0x7f6014165a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:19.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.441+0000 7f6012ffd640 1 -- 172.21.15.150:0/2710994601 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3983891345 0 0) 0x7f6014145170 con 0x7f6014144bb0 2024-03-17T23:47:19.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.441+0000 7f6012ffd640 1 -- 172.21.15.150:0/2710994601 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ffc003520 con 0x7f6014144bb0 2024-03-17T23:47:19.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.441+0000 7f60137fe640 1 -- 172.21.15.150:0/2710994601 >> v2:172.21.15.157:3300/0 conn(0x7f60140ce4d0 msgr2=0x7f6014165a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:19.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.441+0000 7f60137fe640 1 --2- 172.21.15.150:0/2710994601 >> v2:172.21.15.157:3300/0 conn(0x7f60140ce4d0 0x7f6014165a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.441+0000 7f60137fe640 1 -- 172.21.15.150:0/2710994601 >> v1:172.21.15.150:6789/0 conn(0x7f6014144bb0 legacy=0x7f601415ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:19.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.441+0000 7f60137fe640 1 -- 172.21.15.150:0/2710994601 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6014145730 con 0x7f601406f400 2024-03-17T23:47:19.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.441+0000 7f6018c1b640 1 --2- 172.21.15.150:0/2710994601 >> v2:172.21.15.157:3300/0 conn(0x7f60140ce4d0 0x7f6014165a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:19.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.442+0000 7f60137fe640 1 --2- 172.21.15.150:0/2710994601 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f601406f400 0x7f60140cdf90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f60080097a0 tx=0x7f600802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d9418165522d9ff server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:19.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.442+0000 7f6012ffd640 1 -- 172.21.15.150:0/2710994601 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60080315a0 con 0x7f601406f400 2024-03-17T23:47:19.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.442+0000 7f6012ffd640 1 -- 172.21.15.150:0/2710994601 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6008037070 con 0x7f601406f400 2024-03-17T23:47:19.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.442+0000 7f6012ffd640 1 -- 172.21.15.150:0/2710994601 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6008031960 con 0x7f601406f400 2024-03-17T23:47:19.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.443+0000 7f601a67e640 1 -- 172.21.15.150:0/2710994601 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f601406f400 msgr2=0x7f60140cdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:19.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.443+0000 7f601a67e640 1 --2- 172.21.15.150:0/2710994601 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f601406f400 0x7f60140cdf90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f60080097a0 tx=0x7f600802b840 comp rx=0 tx=0).stop 2024-03-17T23:47:19.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.443+0000 7f601a67e640 1 -- 172.21.15.150:0/2710994601 shutdown_connections 2024-03-17T23:47:19.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.443+0000 7f601a67e640 1 --2- 172.21.15.150:0/2710994601 >> v2:172.21.15.157:3300/0 conn(0x7f60140ce4d0 0x7f6014165a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.443+0000 7f601a67e640 1 --2- 172.21.15.150:0/2710994601 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f601406f400 0x7f60140cdf90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.443+0000 7f601a67e640 1 -- 172.21.15.150:0/2710994601 >> 172.21.15.150:0/2710994601 conn(0x7f601406b190 msgr2=0x7f601406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:19.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.443+0000 7f601a67e640 1 -- 172.21.15.150:0/2710994601 shutdown_connections 2024-03-17T23:47:19.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.443+0000 7f601a67e640 1 -- 172.21.15.150:0/2710994601 wait complete. 2024-03-17T23:47:19.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.444+0000 7f601a67e640 1 Processor -- start 2024-03-17T23:47:19.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.444+0000 7f601a67e640 1 -- start start 2024-03-17T23:47:19.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.445+0000 7f601a67e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f601406f400 0x7f60141382d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:19.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.445+0000 7f601a67e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f60140ce4d0 0x7f6014153f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:19.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.445+0000 7f601a67e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6014144bb0 0x7f60141f4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:19.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.445+0000 7f601a67e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f601406bee0 con 0x7f601406f400 2024-03-17T23:47:19.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.445+0000 7f601a67e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f601406bd60 con 0x7f6014144bb0 2024-03-17T23:47:19.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.445+0000 7f601a67e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6014165f70 con 0x7f60140ce4d0 2024-03-17T23:47:19.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.445+0000 7f6013fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f601406f400 0x7f60141382d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:19.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.445+0000 7f6013fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f601406f400 0x7f60141382d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40444/0 (socket says 172.21.15.150:40444) 2024-03-17T23:47:19.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.445+0000 7f6013fff640 1 -- 172.21.15.150:0/3769552448 learned_addr learned my addr 172.21.15.150:0/3769552448 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:19.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.445+0000 7f60137fe640 1 --2- 172.21.15.150:0/3769552448 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f60140ce4d0 0x7f6014153f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:19.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.445+0000 7f6018c1b640 1 --2- 172.21.15.150:0/3769552448 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6014144bb0 0x7f60141f4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:19.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.445+0000 7f6013fff640 1 -- 172.21.15.150:0/3769552448 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f60140ce4d0 msgr2=0x7f6014153f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:19.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.445+0000 7f6013fff640 1 --2- 172.21.15.150:0/3769552448 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f60140ce4d0 0x7f6014153f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.445+0000 7f6013fff640 1 -- 172.21.15.150:0/3769552448 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6014144bb0 msgr2=0x7f60141f4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:19.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.445+0000 7f6013fff640 1 --2- 172.21.15.150:0/3769552448 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6014144bb0 0x7f60141f4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.445+0000 7f6013fff640 1 -- 172.21.15.150:0/3769552448 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60141549b0 con 0x7f601406f400 2024-03-17T23:47:19.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.445+0000 7f6013fff640 1 --2- 172.21.15.150:0/3769552448 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f601406f400 0x7f60141382d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f600400c390 tx=0x7f600400c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:19.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.445+0000 7f6010ff9640 1 -- 172.21.15.150:0/3769552448 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6004017070 con 0x7f601406f400 2024-03-17T23:47:19.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.445+0000 7f6010ff9640 1 -- 172.21.15.150:0/3769552448 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6004005ce0 con 0x7f601406f400 2024-03-17T23:47:19.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.445+0000 7f601a67e640 1 -- 172.21.15.150:0/3769552448 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6014154c90 con 0x7f601406f400 2024-03-17T23:47:19.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.445+0000 7f6010ff9640 1 -- 172.21.15.150:0/3769552448 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60040072f0 con 0x7f601406f400 2024-03-17T23:47:19.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.446+0000 7f601a67e640 1 -- 172.21.15.150:0/3769552448 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f601413be90 con 0x7f601406f400 2024-03-17T23:47:19.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.446+0000 7f601a67e640 1 -- 172.21.15.150:0/3769552448 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60140681a0 con 0x7f601406f400 2024-03-17T23:47:19.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.447+0000 7f6010ff9640 1 -- 172.21.15.150:0/3769552448 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f600401a070 con 0x7f601406f400 2024-03-17T23:47:19.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.447+0000 7f6010ff9640 1 --2- 172.21.15.150:0/3769552448 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f600006bbd0 0x7f600006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:19.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.447+0000 7f6010ff9640 1 -- 172.21.15.150:0/3769552448 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f600408a000 con 0x7f601406f400 2024-03-17T23:47:19.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.447+0000 7f60137fe640 1 --2- 172.21.15.150:0/3769552448 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f600006bbd0 0x7f600006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:19.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.448+0000 7f60137fe640 1 --2- 172.21.15.150:0/3769552448 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f600006bbd0 0x7f600006e090 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6008036000 tx=0x7f600803d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:19.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.452+0000 7f6010ff9640 1 -- 172.21.15.150:0/3769552448 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f60040581f0 con 0x7f601406f400 2024-03-17T23:47:19.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.623+0000 7f601a67e640 1 -- 172.21.15.150:0/3769552448 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f60140b0a30 con 0x7f601406f400 2024-03-17T23:47:19.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.624+0000 7f6010ff9640 1 -- 172.21.15.150:0/3769552448 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f600405e030 con 0x7f601406f400 2024-03-17T23:47:19.625 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:19.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.626+0000 7f601a67e640 1 -- 172.21.15.150:0/3769552448 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f600006bbd0 msgr2=0x7f600006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:19.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.626+0000 7f601a67e640 1 --2- 172.21.15.150:0/3769552448 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f600006bbd0 0x7f600006e090 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6008036000 tx=0x7f600803d040 comp rx=0 tx=0).stop 2024-03-17T23:47:19.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.626+0000 7f601a67e640 1 -- 172.21.15.150:0/3769552448 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f601406f400 msgr2=0x7f60141382d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:19.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.626+0000 7f601a67e640 1 --2- 172.21.15.150:0/3769552448 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f601406f400 0x7f60141382d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f600400c390 tx=0x7f600400c790 comp rx=0 tx=0).stop 2024-03-17T23:47:19.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.626+0000 7f601a67e640 1 -- 172.21.15.150:0/3769552448 shutdown_connections 2024-03-17T23:47:19.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.626+0000 7f601a67e640 1 --2- 172.21.15.150:0/3769552448 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f600006bbd0 0x7f600006e090 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.626+0000 7f601a67e640 1 --2- 172.21.15.150:0/3769552448 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6014144bb0 0x7f60141f4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.626+0000 7f601a67e640 1 --2- 172.21.15.150:0/3769552448 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f60140ce4d0 0x7f6014153f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.626+0000 7f601a67e640 1 --2- 172.21.15.150:0/3769552448 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f601406f400 0x7f60141382d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.626+0000 7f601a67e640 1 -- 172.21.15.150:0/3769552448 >> 172.21.15.150:0/3769552448 conn(0x7f601406b190 msgr2=0x7f601413b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:19.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.626+0000 7f601a67e640 1 -- 172.21.15.150:0/3769552448 shutdown_connections 2024-03-17T23:47:19.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.626+0000 7f601a67e640 1 -- 172.21.15.150:0/3769552448 wait complete. 2024-03-17T23:47:19.637 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:47:19.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.737+0000 7f12d85c4640 1 Processor -- start 2024-03-17T23:47:19.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.737+0000 7f12d85c4640 1 -- start start 2024-03-17T23:47:19.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.738+0000 7f12d85c4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12d009a470 0x7f12d009a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:19.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.738+0000 7f12d85c4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12d00cfad0 con 0x7f12d009ad90 2024-03-17T23:47:19.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.738+0000 7f12d85c4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12d00cfdb0 con 0x7f12d015fea0 2024-03-17T23:47:19.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.738+0000 7f12d85c4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f12d00d0090 con 0x7f12d009a470 2024-03-17T23:47:19.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.738+0000 7f12d5b5f640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f12d009ad90 0x7f12d015f790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34764/0 (socket says 172.21.15.150:34764) 2024-03-17T23:47:19.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.738+0000 7f12d5b5f640 1 -- 172.21.15.150:0/2192971108 learned_addr learned my addr 172.21.15.150:0/2192971108 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:19.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.738+0000 7f12d6360640 1 --2- 172.21.15.150:0/2192971108 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12d009a470 0x7f12d009a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:19.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.738+0000 7f12d535e640 1 -- 172.21.15.150:0/2192971108 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1686389073 0 0) 0x7f12d00cfdb0 con 0x7f12d015fea0 2024-03-17T23:47:19.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.738+0000 7f12d535e640 1 -- 172.21.15.150:0/2192971108 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12bc003520 con 0x7f12d015fea0 2024-03-17T23:47:19.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.738+0000 7f12d535e640 1 -- 172.21.15.150:0/2192971108 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2164888587 0 0) 0x7f12d00cfad0 con 0x7f12d009ad90 2024-03-17T23:47:19.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.738+0000 7f12d6360640 1 -- 172.21.15.150:0/2192971108 >> v1:172.21.15.157:6789/0 conn(0x7f12d015fea0 legacy=0x7f12d00cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:19.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.738+0000 7f12d6360640 1 -- 172.21.15.150:0/2192971108 >> v1:172.21.15.150:6789/0 conn(0x7f12d009ad90 legacy=0x7f12d015f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:19.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.738+0000 7f12d6360640 1 -- 172.21.15.150:0/2192971108 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12d00d0370 con 0x7f12d009a470 2024-03-17T23:47:19.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.739+0000 7f12d6360640 1 --2- 172.21.15.150:0/2192971108 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12d009a470 0x7f12d009a850 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f12cc0097d0 tx=0x7f12cc02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f9d3326549e412e server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:19.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.739+0000 7f12d535e640 1 -- 172.21.15.150:0/2192971108 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12cc035020 con 0x7f12d009a470 2024-03-17T23:47:19.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.739+0000 7f12d535e640 1 -- 172.21.15.150:0/2192971108 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12cc031890 con 0x7f12d009a470 2024-03-17T23:47:19.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.739+0000 7f12d535e640 1 -- 172.21.15.150:0/2192971108 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12cc031c60 con 0x7f12d009a470 2024-03-17T23:47:19.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.740+0000 7f12d85c4640 1 -- 172.21.15.150:0/2192971108 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12d009a470 msgr2=0x7f12d009a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:19.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.740+0000 7f12d85c4640 1 --2- 172.21.15.150:0/2192971108 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12d009a470 0x7f12d009a850 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f12cc0097d0 tx=0x7f12cc02b930 comp rx=0 tx=0).stop 2024-03-17T23:47:19.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.740+0000 7f12d85c4640 1 -- 172.21.15.150:0/2192971108 shutdown_connections 2024-03-17T23:47:19.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.740+0000 7f12d85c4640 1 --2- 172.21.15.150:0/2192971108 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12d009a470 0x7f12d009a850 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.740+0000 7f12d85c4640 1 -- 172.21.15.150:0/2192971108 >> 172.21.15.150:0/2192971108 conn(0x7f12d00b3630 msgr2=0x7f12d00b3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:19.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.740+0000 7f12d85c4640 1 -- 172.21.15.150:0/2192971108 shutdown_connections 2024-03-17T23:47:19.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.740+0000 7f12d85c4640 1 -- 172.21.15.150:0/2192971108 wait complete. 2024-03-17T23:47:19.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.741+0000 7f12d85c4640 1 Processor -- start 2024-03-17T23:47:19.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.741+0000 7f12d85c4640 1 -- start start 2024-03-17T23:47:19.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.742+0000 7f12d85c4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12d009a470 0x7f12d0152ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:19.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.742+0000 7f12d85c4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12d009ad90 0x7f12d00c6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:19.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.742+0000 7f12d85c4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f12d015fea0 0x7f12d00c80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:19.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.742+0000 7f12d85c4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f12d00cf890 con 0x7f12d009a470 2024-03-17T23:47:19.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.742+0000 7f12d85c4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f12d00cf710 con 0x7f12d015fea0 2024-03-17T23:47:19.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.742+0000 7f12d85c4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f12d00cccf0 con 0x7f12d009ad90 2024-03-17T23:47:19.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.742+0000 7f12d6360640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12d009a470 0x7f12d0152ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:19.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.742+0000 7f12d6360640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12d009a470 0x7f12d0152ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40454/0 (socket says 172.21.15.150:40454) 2024-03-17T23:47:19.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.742+0000 7f12d6360640 1 -- 172.21.15.150:0/147501616 learned_addr learned my addr 172.21.15.150:0/147501616 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:19.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.742+0000 7f12d5b5f640 1 --2- 172.21.15.150:0/147501616 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12d009ad90 0x7f12d00c6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:19.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.742+0000 7f12d6b61640 1 --2- 172.21.15.150:0/147501616 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f12d015fea0 0x7f12d00c80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:19.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.742+0000 7f12d6360640 1 -- 172.21.15.150:0/147501616 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12d009ad90 msgr2=0x7f12d00c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:19.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.742+0000 7f12d6360640 1 --2- 172.21.15.150:0/147501616 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12d009ad90 0x7f12d00c6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.742+0000 7f12d6360640 1 -- 172.21.15.150:0/147501616 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f12d015fea0 msgr2=0x7f12d00c80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:19.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.742+0000 7f12d6360640 1 --2- 172.21.15.150:0/147501616 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f12d015fea0 0x7f12d00c80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.742+0000 7f12d6360640 1 -- 172.21.15.150:0/147501616 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12d00c85f0 con 0x7f12d009a470 2024-03-17T23:47:19.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.742+0000 7f12d5b5f640 1 --2- 172.21.15.150:0/147501616 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12d009ad90 0x7f12d00c6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:19.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.742+0000 7f12d6360640 1 --2- 172.21.15.150:0/147501616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12d009a470 0x7f12d0152ba0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f12cc0098f0 tx=0x7f12cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:19.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.742+0000 7f12c6ffd640 1 -- 172.21.15.150:0/147501616 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12cc04a070 con 0x7f12d009a470 2024-03-17T23:47:19.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.742+0000 7f12c6ffd640 1 -- 172.21.15.150:0/147501616 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12cc03bae0 con 0x7f12d009a470 2024-03-17T23:47:19.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.742+0000 7f12c6ffd640 1 -- 172.21.15.150:0/147501616 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12cc045420 con 0x7f12d009a470 2024-03-17T23:47:19.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.742+0000 7f12d85c4640 1 -- 172.21.15.150:0/147501616 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12d00ca0b0 con 0x7f12d009a470 2024-03-17T23:47:19.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.743+0000 7f12d85c4640 1 -- 172.21.15.150:0/147501616 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12d00ca670 con 0x7f12d009a470 2024-03-17T23:47:19.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.743+0000 7f12d85c4640 1 -- 172.21.15.150:0/147501616 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12d00d1260 con 0x7f12d009a470 2024-03-17T23:47:19.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.744+0000 7f12c6ffd640 1 -- 172.21.15.150:0/147501616 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f12cc04d070 con 0x7f12d009a470 2024-03-17T23:47:19.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.744+0000 7f12c6ffd640 1 --2- 172.21.15.150:0/147501616 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f12b406bbd0 0x7f12b406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:19.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.744+0000 7f12c6ffd640 1 -- 172.21.15.150:0/147501616 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f12cc0ace10 con 0x7f12d009a470 2024-03-17T23:47:19.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.744+0000 7f12d5b5f640 1 --2- 172.21.15.150:0/147501616 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f12b406bbd0 0x7f12b406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:19.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.745+0000 7f12d5b5f640 1 --2- 172.21.15.150:0/147501616 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f12b406bbd0 0x7f12b406e090 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f12c0007f70 tx=0x7f12c0007f00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:19.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.749+0000 7f12c6ffd640 1 -- 172.21.15.150:0/147501616 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f12d00d1260 con 0x7f12d009a470 2024-03-17T23:47:19.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.960+0000 7f12d85c4640 1 -- 172.21.15.150:0/147501616 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f12d00681a0 con 0x7f12d009a470 2024-03-17T23:47:19.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.961+0000 7f12c6ffd640 1 -- 172.21.15.150:0/147501616 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5186 (secure 0 0 0) 0x7f12cc040070 con 0x7f12d009a470 2024-03-17T23:47:19.962 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:19.962 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/1949609985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6833","nonce":1949609985},{"type":"v1","addr":"172.21.15.157:6835","nonce":1949609985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/758264728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":758264728},{"type":"v1","addr":"172.21.15.150:6835","nonce":758264728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3655589010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3655589010},{"type":"v1","addr":"172.21.15.150:6837","nonce":3655589010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-17T23:46:55.253739+0000","modified":"2024-03-17T23:47:11.316249+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6834/1625058792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1625058792},{"type":"v1","addr":"172.21.15.157:6834","nonce":1625058792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4297,"qdb_cluster":[4297]},"id":1}]} 2024-03-17T23:47:19.962 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 8 2024-03-17T23:47:19.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.963+0000 7f12d85c4640 1 -- 172.21.15.150:0/147501616 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f12b406bbd0 msgr2=0x7f12b406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:19.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.963+0000 7f12d85c4640 1 --2- 172.21.15.150:0/147501616 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f12b406bbd0 0x7f12b406e090 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f12c0007f70 tx=0x7f12c0007f00 comp rx=0 tx=0).stop 2024-03-17T23:47:19.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.963+0000 7f12d85c4640 1 -- 172.21.15.150:0/147501616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12d009a470 msgr2=0x7f12d0152ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:19.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.963+0000 7f12d85c4640 1 --2- 172.21.15.150:0/147501616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12d009a470 0x7f12d0152ba0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f12cc0098f0 tx=0x7f12cc0046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.963+0000 7f12d85c4640 1 -- 172.21.15.150:0/147501616 shutdown_connections 2024-03-17T23:47:19.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.963+0000 7f12d85c4640 1 --2- 172.21.15.150:0/147501616 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f12b406bbd0 0x7f12b406e090 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.963+0000 7f12d85c4640 1 --2- 172.21.15.150:0/147501616 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f12d015fea0 0x7f12d00c80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.963+0000 7f12d85c4640 1 --2- 172.21.15.150:0/147501616 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12d009ad90 0x7f12d00c6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.963+0000 7f12d85c4640 1 --2- 172.21.15.150:0/147501616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12d009a470 0x7f12d0152ba0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:19.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.963+0000 7f12d85c4640 1 -- 172.21.15.150:0/147501616 >> 172.21.15.150:0/147501616 conn(0x7f12d00b3630 msgr2=0x7f12d00b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:19.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.963+0000 7f12d85c4640 1 -- 172.21.15.150:0/147501616 shutdown_connections 2024-03-17T23:47:19.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:19.963+0000 7f12d85c4640 1 -- 172.21.15.150:0/147501616 wait complete. 2024-03-17T23:47:19.975 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:20.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.076+0000 7f8bed897640 1 Processor -- start 2024-03-17T23:47:20.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.076+0000 7f8bed897640 1 -- start start 2024-03-17T23:47:20.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.077+0000 7f8bed897640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8be81616b0 0x7f8be8099e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:20.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.077+0000 7f8bed897640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8be809b320 0x7f8be809a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:20.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.077+0000 7f8bed897640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8be80b0ab0 0x7f8be806c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:20.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.077+0000 7f8bed897640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f8be80cc220 con 0x7f8be809b320 2024-03-17T23:47:20.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.077+0000 7f8bed897640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f8be80d0770 con 0x7f8be80b0ab0 2024-03-17T23:47:20.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.077+0000 7f8bed897640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8be80d0a10 con 0x7f8be81616b0 2024-03-17T23:47:20.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.077+0000 7f8be67fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8be809b320 0x7f8be809a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:20.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.077+0000 7f8be67fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8be809b320 0x7f8be809a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40464/0 (socket says 172.21.15.150:40464) 2024-03-17T23:47:20.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.077+0000 7f8be6ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8be81616b0 0x7f8be8099e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:20.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.077+0000 7f8be67fc640 1 -- 172.21.15.150:0/4058229039 learned_addr learned my addr 172.21.15.150:0/4058229039 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:20.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.077+0000 7f8be77fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8be80b0ab0 0x7f8be806c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:20.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.077+0000 7f8be6ffd640 1 -- 172.21.15.150:0/4058229039 >> v2:172.21.15.157:3300/0 conn(0x7f8be80b0ab0 msgr2=0x7f8be806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:20.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.077+0000 7f8be6ffd640 1 --2- 172.21.15.150:0/4058229039 >> v2:172.21.15.157:3300/0 conn(0x7f8be80b0ab0 0x7f8be806c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.077+0000 7f8be6ffd640 1 -- 172.21.15.150:0/4058229039 >> v2:172.21.15.150:3300/0 conn(0x7f8be809b320 msgr2=0x7f8be809a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:20.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.077+0000 7f8be6ffd640 1 --2- 172.21.15.150:0/4058229039 >> v2:172.21.15.150:3300/0 conn(0x7f8be809b320 0x7f8be809a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.077+0000 7f8be6ffd640 1 -- 172.21.15.150:0/4058229039 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8be806d520 con 0x7f8be81616b0 2024-03-17T23:47:20.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.077+0000 7f8be67fc640 1 --2- 172.21.15.150:0/4058229039 >> v2:172.21.15.150:3300/0 conn(0x7f8be809b320 0x7f8be809a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:20.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.078+0000 7f8be6ffd640 1 --2- 172.21.15.150:0/4058229039 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8be81616b0 0x7f8be8099e20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8be80b7b50 tx=0x7f8be002bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebef8effc9aeef server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:20.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.078+0000 7f8be5ffb640 1 -- 172.21.15.150:0/4058229039 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be0039070 con 0x7f8be81616b0 2024-03-17T23:47:20.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.078+0000 7f8be5ffb640 1 -- 172.21.15.150:0/4058229039 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8be002cb40 con 0x7f8be81616b0 2024-03-17T23:47:20.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.078+0000 7f8be5ffb640 1 -- 172.21.15.150:0/4058229039 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be0034440 con 0x7f8be81616b0 2024-03-17T23:47:20.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.079+0000 7f8bed897640 1 -- 172.21.15.150:0/4058229039 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8be81616b0 msgr2=0x7f8be8099e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:20.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.079+0000 7f8bed897640 1 --2- 172.21.15.150:0/4058229039 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8be81616b0 0x7f8be8099e20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8be80b7b50 tx=0x7f8be002bd90 comp rx=0 tx=0).stop 2024-03-17T23:47:20.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.079+0000 7f8bed897640 1 -- 172.21.15.150:0/4058229039 shutdown_connections 2024-03-17T23:47:20.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.079+0000 7f8bed897640 1 --2- 172.21.15.150:0/4058229039 >> v2:172.21.15.150:3300/0 conn(0x7f8be809b320 0x7f8be809a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.079+0000 7f8bed897640 1 --2- 172.21.15.150:0/4058229039 >> v2:172.21.15.157:3300/0 conn(0x7f8be80b0ab0 0x7f8be806c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.079+0000 7f8bed897640 1 --2- 172.21.15.150:0/4058229039 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8be81616b0 0x7f8be8099e20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.079+0000 7f8bed897640 1 -- 172.21.15.150:0/4058229039 >> 172.21.15.150:0/4058229039 conn(0x7f8be80b3630 msgr2=0x7f8be80b3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:20.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.079+0000 7f8bed897640 1 -- 172.21.15.150:0/4058229039 shutdown_connections 2024-03-17T23:47:20.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.079+0000 7f8bed897640 1 -- 172.21.15.150:0/4058229039 wait complete. 2024-03-17T23:47:20.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.080+0000 7f8bed897640 1 Processor -- start 2024-03-17T23:47:20.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.080+0000 7f8bed897640 1 -- start start 2024-03-17T23:47:20.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.081+0000 7f8bed897640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8be809b320 0x7f8be8152ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:20.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.081+0000 7f8bed897640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be80b0ab0 0x7f8be80ca150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:20.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.081+0000 7f8bed897640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8be81616b0 0x7f8be80ca690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:20.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.081+0000 7f8bed897640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8be80d1290 con 0x7f8be80b0ab0 2024-03-17T23:47:20.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.081+0000 7f8bed897640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8be80d0fb0 con 0x7f8be81616b0 2024-03-17T23:47:20.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.081+0000 7f8bed897640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8be806c7e0 con 0x7f8be809b320 2024-03-17T23:47:20.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.081+0000 7f8be67fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be80b0ab0 0x7f8be80ca150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:20.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.081+0000 7f8be6ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8be809b320 0x7f8be8152ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:20.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.081+0000 7f8be67fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be80b0ab0 0x7f8be80ca150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40470/0 (socket says 172.21.15.150:40470) 2024-03-17T23:47:20.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.081+0000 7f8be6ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8be809b320 0x7f8be8152ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:55532/0 (socket says 172.21.15.150:55532) 2024-03-17T23:47:20.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.081+0000 7f8be67fc640 1 -- 172.21.15.150:0/998901828 learned_addr learned my addr 172.21.15.150:0/998901828 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:20.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.081+0000 7f8be77fe640 1 --2- 172.21.15.150:0/998901828 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8be81616b0 0x7f8be80ca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:20.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.081+0000 7f8be6ffd640 1 -- 172.21.15.150:0/998901828 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8be81616b0 msgr2=0x7f8be80ca690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:20.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.081+0000 7f8be6ffd640 1 --2- 172.21.15.150:0/998901828 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8be81616b0 0x7f8be80ca690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.081+0000 7f8be6ffd640 1 -- 172.21.15.150:0/998901828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be80b0ab0 msgr2=0x7f8be80ca150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:20.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.081+0000 7f8be6ffd640 1 --2- 172.21.15.150:0/998901828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be80b0ab0 0x7f8be80ca150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.081+0000 7f8be6ffd640 1 -- 172.21.15.150:0/998901828 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8be80cbd70 con 0x7f8be809b320 2024-03-17T23:47:20.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.081+0000 7f8be67fc640 1 --2- 172.21.15.150:0/998901828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be80b0ab0 0x7f8be80ca150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:20.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.081+0000 7f8be6ffd640 1 --2- 172.21.15.150:0/998901828 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8be809b320 0x7f8be8152ff0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8be0033000 tx=0x7f8be0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:20.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.081+0000 7f8bdffff640 1 -- 172.21.15.150:0/998901828 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be0039050 con 0x7f8be809b320 2024-03-17T23:47:20.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.082+0000 7f8bdffff640 1 -- 172.21.15.150:0/998901828 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8be0045070 con 0x7f8be809b320 2024-03-17T23:47:20.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.082+0000 7f8bdffff640 1 -- 172.21.15.150:0/998901828 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be0009450 con 0x7f8be809b320 2024-03-17T23:47:20.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.082+0000 7f8bed897640 1 -- 172.21.15.150:0/998901828 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8be80cacd0 con 0x7f8be809b320 2024-03-17T23:47:20.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.082+0000 7f8bed897640 1 -- 172.21.15.150:0/998901828 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8be80cb290 con 0x7f8be809b320 2024-03-17T23:47:20.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.082+0000 7f8bed897640 1 -- 172.21.15.150:0/998901828 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8be80681a0 con 0x7f8be809b320 2024-03-17T23:47:20.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.083+0000 7f8bdffff640 1 -- 172.21.15.150:0/998901828 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8be003e850 con 0x7f8be809b320 2024-03-17T23:47:20.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.083+0000 7f8bdffff640 1 --2- 172.21.15.150:0/998901828 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8bc806bbd0 0x7f8bc806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:20.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.083+0000 7f8be67fc640 1 --2- 172.21.15.150:0/998901828 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8bc806bbd0 0x7f8bc806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:20.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.083+0000 7f8bdffff640 1 -- 172.21.15.150:0/998901828 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f8be00ada50 con 0x7f8be809b320 2024-03-17T23:47:20.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.084+0000 7f8be67fc640 1 --2- 172.21.15.150:0/998901828 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8bc806bbd0 0x7f8bc806e090 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0009630 tx=0x7f8bd0009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:20.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.088+0000 7f8bdffff640 1 -- 172.21.15.150:0/998901828 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8be007bcc0 con 0x7f8be809b320 2024-03-17T23:47:20.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.262+0000 7f8bed897640 1 -- 172.21.15.150:0/998901828 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8be809ac00 con 0x7f8be809b320 2024-03-17T23:47:20.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.264+0000 7f8bdffff640 1 -- 172.21.15.150:0/998901828 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18550 (secure 0 0 0) 0x7f8be007fd60 con 0x7f8be809b320 2024-03-17T23:47:20.265 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:20.265 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:01.940019+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:20.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.266+0000 7f8bed897640 1 -- 172.21.15.150:0/998901828 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8bc806bbd0 msgr2=0x7f8bc806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:20.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.266+0000 7f8bed897640 1 --2- 172.21.15.150:0/998901828 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8bc806bbd0 0x7f8bc806e090 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0009630 tx=0x7f8bd0009240 comp rx=0 tx=0).stop 2024-03-17T23:47:20.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.266+0000 7f8bed897640 1 -- 172.21.15.150:0/998901828 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8be809b320 msgr2=0x7f8be8152ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:20.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.266+0000 7f8bed897640 1 --2- 172.21.15.150:0/998901828 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8be809b320 0x7f8be8152ff0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8be0033000 tx=0x7f8be0004a40 comp rx=0 tx=0).stop 2024-03-17T23:47:20.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.266+0000 7f8bed897640 1 -- 172.21.15.150:0/998901828 shutdown_connections 2024-03-17T23:47:20.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.266+0000 7f8bed897640 1 --2- 172.21.15.150:0/998901828 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8bc806bbd0 0x7f8bc806e090 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.266+0000 7f8bed897640 1 --2- 172.21.15.150:0/998901828 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8be81616b0 0x7f8be80ca690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.266+0000 7f8bed897640 1 --2- 172.21.15.150:0/998901828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8be80b0ab0 0x7f8be80ca150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.266+0000 7f8bed897640 1 --2- 172.21.15.150:0/998901828 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8be809b320 0x7f8be8152ff0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.266+0000 7f8bed897640 1 -- 172.21.15.150:0/998901828 >> 172.21.15.150:0/998901828 conn(0x7f8be80b3630 msgr2=0x7f8be8099e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:20.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.266+0000 7f8bed897640 1 -- 172.21.15.150:0/998901828 shutdown_connections 2024-03-17T23:47:20.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.266+0000 7f8bed897640 1 -- 172.21.15.150:0/998901828 wait complete. 2024-03-17T23:47:20.277 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:46:53.610678+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:46:54.227335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-17T23:46:57.949235+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-17T23:47:20.278 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:47:20.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.381+0000 7f3b0484b640 1 Processor -- start 2024-03-17T23:47:20.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.381+0000 7f3b0484b640 1 -- start start 2024-03-17T23:47:20.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.382+0000 7f3b0484b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3b0006ecd0 0x7f3b0015bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:20.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.382+0000 7f3b0484b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b000ce790 con 0x7f3b0015ab90 2024-03-17T23:47:20.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.382+0000 7f3b0484b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b0015b050 con 0x7f3b001514f0 2024-03-17T23:47:20.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.382+0000 7f3b0484b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f3b0015b330 con 0x7f3b0006ecd0 2024-03-17T23:47:20.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.382+0000 7f3afed9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3b0006ecd0 0x7f3b0015bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:20.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.382+0000 7f3afdd9b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f3b0015ab90 0x7f3b000cec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34778/0 (socket says 172.21.15.150:34778) 2024-03-17T23:47:20.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.382+0000 7f3afdd9b640 1 -- 172.21.15.150:0/1996731271 learned_addr learned my addr 172.21.15.150:0/1996731271 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:20.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.382+0000 7f3afe59c640 1 --1- 172.21.15.150:0/1996731271 >> v1:172.21.15.157:6789/0 conn(0x7f3b001514f0 0x7f3b0006e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:52366/0 (socket says 172.21.15.150:52366) 2024-03-17T23:47:20.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.382+0000 7f3afd59a640 1 -- 172.21.15.150:0/1996731271 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1876280639 0 0) 0x7f3b0015b050 con 0x7f3b001514f0 2024-03-17T23:47:20.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.382+0000 7f3afed9d640 1 -- 172.21.15.150:0/1996731271 >> v1:172.21.15.157:6789/0 conn(0x7f3b001514f0 legacy=0x7f3b0006e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:20.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.382+0000 7f3afed9d640 1 -- 172.21.15.150:0/1996731271 >> v1:172.21.15.150:6789/0 conn(0x7f3b0015ab90 legacy=0x7f3b000cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:20.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.382+0000 7f3afed9d640 1 -- 172.21.15.150:0/1996731271 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b0015b610 con 0x7f3b0006ecd0 2024-03-17T23:47:20.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.383+0000 7f3afed9d640 1 --2- 172.21.15.150:0/1996731271 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b0006ecd0 0x7f3b0015bd10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3af800ba40 tx=0x7f3af802df60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c37652608c5e8a61 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:20.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.383+0000 7f3afd59a640 1 -- 172.21.15.150:0/1996731271 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af803b070 con 0x7f3b0006ecd0 2024-03-17T23:47:20.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.383+0000 7f3afd59a640 1 -- 172.21.15.150:0/1996731271 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3af802ec90 con 0x7f3b0006ecd0 2024-03-17T23:47:20.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.383+0000 7f3afd59a640 1 -- 172.21.15.150:0/1996731271 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af8036660 con 0x7f3b0006ecd0 2024-03-17T23:47:20.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.384+0000 7f3b0484b640 1 -- 172.21.15.150:0/1996731271 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b0006ecd0 msgr2=0x7f3b0015bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:20.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.384+0000 7f3b0484b640 1 --2- 172.21.15.150:0/1996731271 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b0006ecd0 0x7f3b0015bd10 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3af800ba40 tx=0x7f3af802df60 comp rx=0 tx=0).stop 2024-03-17T23:47:20.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.384+0000 7f3b0484b640 1 -- 172.21.15.150:0/1996731271 shutdown_connections 2024-03-17T23:47:20.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.384+0000 7f3b0484b640 1 --2- 172.21.15.150:0/1996731271 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b0006ecd0 0x7f3b0015bd10 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.384+0000 7f3b0484b640 1 -- 172.21.15.150:0/1996731271 >> 172.21.15.150:0/1996731271 conn(0x7f3b0006b190 msgr2=0x7f3b0006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:20.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.384+0000 7f3b0484b640 1 -- 172.21.15.150:0/1996731271 shutdown_connections 2024-03-17T23:47:20.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.384+0000 7f3b0484b640 1 -- 172.21.15.150:0/1996731271 wait complete. 2024-03-17T23:47:20.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.385+0000 7f3b0484b640 1 Processor -- start 2024-03-17T23:47:20.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.385+0000 7f3b0484b640 1 -- start start 2024-03-17T23:47:20.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.386+0000 7f3b0484b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3b0006ecd0 0x7f3b00154120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:20.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.386+0000 7f3b0484b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b001514f0 0x7f3b00156430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:20.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.386+0000 7f3b0484b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3b0015ab90 0x7f3b00156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:20.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.386+0000 7f3b0484b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3b001518d0 con 0x7f3b001514f0 2024-03-17T23:47:20.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.386+0000 7f3b0484b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3b000af510 con 0x7f3b0015ab90 2024-03-17T23:47:20.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.386+0000 7f3b0484b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3b00151a50 con 0x7f3b0006ecd0 2024-03-17T23:47:20.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.386+0000 7f3afdd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b001514f0 0x7f3b00156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:20.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.386+0000 7f3afe59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3b0006ecd0 0x7f3b00154120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:20.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.386+0000 7f3afdd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b001514f0 0x7f3b00156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40482/0 (socket says 172.21.15.150:40482) 2024-03-17T23:47:20.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.386+0000 7f3afe59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3b0006ecd0 0x7f3b00154120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:55534/0 (socket says 172.21.15.150:55534) 2024-03-17T23:47:20.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.386+0000 7f3afed9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3b0015ab90 0x7f3b00156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:20.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.386+0000 7f3afdd9b640 1 -- 172.21.15.150:0/3535655903 learned_addr learned my addr 172.21.15.150:0/3535655903 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:20.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.386+0000 7f3afdd9b640 1 -- 172.21.15.150:0/3535655903 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3b0006ecd0 msgr2=0x7f3b00154120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:20.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.387+0000 7f3afdd9b640 1 --2- 172.21.15.150:0/3535655903 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3b0006ecd0 0x7f3b00154120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.387+0000 7f3afdd9b640 1 -- 172.21.15.150:0/3535655903 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3b0015ab90 msgr2=0x7f3b00156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:20.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.387+0000 7f3afdd9b640 1 --2- 172.21.15.150:0/3535655903 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3b0015ab90 0x7f3b00156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.387+0000 7f3afdd9b640 1 -- 172.21.15.150:0/3535655903 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b00153080 con 0x7f3b001514f0 2024-03-17T23:47:20.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.387+0000 7f3afe59c640 1 --2- 172.21.15.150:0/3535655903 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3b0006ecd0 0x7f3b00154120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:20.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.387+0000 7f3afed9d640 1 --2- 172.21.15.150:0/3535655903 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3b0015ab90 0x7f3b00156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:20.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.387+0000 7f3afdd9b640 1 --2- 172.21.15.150:0/3535655903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b001514f0 0x7f3b00156430 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3af400c7d0 tx=0x7f3af400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:20.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.387+0000 7f3aeeffd640 1 -- 172.21.15.150:0/3535655903 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af4018070 con 0x7f3b001514f0 2024-03-17T23:47:20.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.387+0000 7f3aeeffd640 1 -- 172.21.15.150:0/3535655903 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3af40074e0 con 0x7f3b001514f0 2024-03-17T23:47:20.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.387+0000 7f3b0484b640 1 -- 172.21.15.150:0/3535655903 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b00153360 con 0x7f3b001514f0 2024-03-17T23:47:20.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.387+0000 7f3aeeffd640 1 -- 172.21.15.150:0/3535655903 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af4004a20 con 0x7f3b001514f0 2024-03-17T23:47:20.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.387+0000 7f3b0484b640 1 -- 172.21.15.150:0/3535655903 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b00153920 con 0x7f3b001514f0 2024-03-17T23:47:20.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.388+0000 7f3b0484b640 1 -- 172.21.15.150:0/3535655903 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b000681a0 con 0x7f3b001514f0 2024-03-17T23:47:20.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.389+0000 7f3aeeffd640 1 -- 172.21.15.150:0/3535655903 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3af401b070 con 0x7f3b001514f0 2024-03-17T23:47:20.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.389+0000 7f3aeeffd640 1 --2- 172.21.15.150:0/3535655903 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3ae006bbd0 0x7f3ae006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:20.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.389+0000 7f3afe59c640 1 --2- 172.21.15.150:0/3535655903 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3ae006bbd0 0x7f3ae006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:20.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.389+0000 7f3aeeffd640 1 -- 172.21.15.150:0/3535655903 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f3af408a310 con 0x7f3b001514f0 2024-03-17T23:47:20.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.389+0000 7f3afe59c640 1 --2- 172.21.15.150:0/3535655903 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3ae006bbd0 0x7f3ae006e090 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3af00012e0 tx=0x7f3af0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:20.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.393+0000 7f3aeeffd640 1 -- 172.21.15.150:0/3535655903 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3b000ce790 con 0x7f3b001514f0 2024-03-17T23:47:20.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.566+0000 7f3b0484b640 1 -- 172.21.15.150:0/3535655903 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3b000b07c0 con 0x7f3b001514f0 2024-03-17T23:47:20.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.566+0000 7f3aeeffd640 1 -- 172.21.15.150:0/3535655903 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f3af4001480 con 0x7f3b001514f0 2024-03-17T23:47:20.567 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:47:20.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.568+0000 7f3b0484b640 1 -- 172.21.15.150:0/3535655903 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3ae006bbd0 msgr2=0x7f3ae006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:20.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.568+0000 7f3b0484b640 1 --2- 172.21.15.150:0/3535655903 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3ae006bbd0 0x7f3ae006e090 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3af00012e0 tx=0x7f3af0009040 comp rx=0 tx=0).stop 2024-03-17T23:47:20.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.568+0000 7f3b0484b640 1 -- 172.21.15.150:0/3535655903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b001514f0 msgr2=0x7f3b00156430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:20.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.568+0000 7f3b0484b640 1 --2- 172.21.15.150:0/3535655903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b001514f0 0x7f3b00156430 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3af400c7d0 tx=0x7f3af400cc90 comp rx=0 tx=0).stop 2024-03-17T23:47:20.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.568+0000 7f3b0484b640 1 -- 172.21.15.150:0/3535655903 shutdown_connections 2024-03-17T23:47:20.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.568+0000 7f3b0484b640 1 --2- 172.21.15.150:0/3535655903 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3ae006bbd0 0x7f3ae006e090 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.568+0000 7f3b0484b640 1 --2- 172.21.15.150:0/3535655903 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3b0015ab90 0x7f3b00156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.568+0000 7f3b0484b640 1 --2- 172.21.15.150:0/3535655903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b001514f0 0x7f3b00156430 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.568+0000 7f3b0484b640 1 --2- 172.21.15.150:0/3535655903 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3b0006ecd0 0x7f3b00154120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.568+0000 7f3b0484b640 1 -- 172.21.15.150:0/3535655903 >> 172.21.15.150:0/3535655903 conn(0x7f3b0006b190 msgr2=0x7f3b000b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:20.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.569+0000 7f3b0484b640 1 -- 172.21.15.150:0/3535655903 shutdown_connections 2024-03-17T23:47:20.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.569+0000 7f3b0484b640 1 -- 172.21.15.150:0/3535655903 wait complete. 2024-03-17T23:47:20.580 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:47:20.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.682+0000 7f40d371b640 1 Processor -- start 2024-03-17T23:47:20.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.682+0000 7f40d371b640 1 -- start start 2024-03-17T23:47:20.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.683+0000 7f40d371b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f40cc06f400 0x7f40cc0cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:20.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.683+0000 7f40d371b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40cc0ce4d0 0x7f40cc165a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:20.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.683+0000 7f40d371b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40cc145170 con 0x7f40cc144bb0 2024-03-17T23:47:20.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.683+0000 7f40d371b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f40cc0ce940 con 0x7f40cc06f400 2024-03-17T23:47:20.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.683+0000 7f40d371b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f40cc145450 con 0x7f40cc0ce4d0 2024-03-17T23:47:20.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.683+0000 7f40d0cb6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f40cc06f400 0x7f40cc0cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:20.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.683+0000 7f40d0cb6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f40cc06f400 0x7f40cc0cdf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40488/0 (socket says 172.21.15.150:40488) 2024-03-17T23:47:20.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.683+0000 7f40d14b7640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f40cc144bb0 0x7f40cc15ce50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:52370/0 (socket says 172.21.15.150:52370) 2024-03-17T23:47:20.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.683+0000 7f40d0cb6640 1 -- 172.21.15.150:0/640872329 learned_addr learned my addr 172.21.15.150:0/640872329 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:20.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.683+0000 7f40d1cb8640 1 --2- 172.21.15.150:0/640872329 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40cc0ce4d0 0x7f40cc165a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:20.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.684+0000 7f40d0cb6640 1 -- 172.21.15.150:0/640872329 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40cc0ce4d0 msgr2=0x7f40cc165a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:20.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.684+0000 7f40d0cb6640 1 --2- 172.21.15.150:0/640872329 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40cc0ce4d0 0x7f40cc165a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.684+0000 7f40d0cb6640 1 -- 172.21.15.150:0/640872329 >> v1:172.21.15.157:6789/0 conn(0x7f40cc144bb0 legacy=0x7f40cc15ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:20.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.684+0000 7f40c3fff640 1 -- 172.21.15.150:0/640872329 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1020274344 0 0) 0x7f40cc145170 con 0x7f40cc144bb0 2024-03-17T23:47:20.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.684+0000 7f40d0cb6640 1 -- 172.21.15.150:0/640872329 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40cc145730 con 0x7f40cc06f400 2024-03-17T23:47:20.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.684+0000 7f40d1cb8640 1 --2- 172.21.15.150:0/640872329 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40cc0ce4d0 0x7f40cc165a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:20.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.684+0000 7f40d0cb6640 1 --2- 172.21.15.150:0/640872329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc06f400 0x7f40cc0cdf90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f40bc009870 tx=0x7f40bc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18fb2c9b406ed2ba server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:20.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.684+0000 7f40c3fff640 1 -- 172.21.15.150:0/640872329 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40bc035020 con 0x7f40cc06f400 2024-03-17T23:47:20.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.684+0000 7f40c3fff640 1 -- 172.21.15.150:0/640872329 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40bc031640 con 0x7f40cc06f400 2024-03-17T23:47:20.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.685+0000 7f40c3fff640 1 -- 172.21.15.150:0/640872329 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40bc02d040 con 0x7f40cc06f400 2024-03-17T23:47:20.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.685+0000 7f40d371b640 1 -- 172.21.15.150:0/640872329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc06f400 msgr2=0x7f40cc0cdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:20.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.685+0000 7f40d371b640 1 --2- 172.21.15.150:0/640872329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc06f400 0x7f40cc0cdf90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f40bc009870 tx=0x7f40bc02b8f0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.685+0000 7f40d371b640 1 -- 172.21.15.150:0/640872329 shutdown_connections 2024-03-17T23:47:20.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.685+0000 7f40d371b640 1 --2- 172.21.15.150:0/640872329 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40cc0ce4d0 0x7f40cc165a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.685+0000 7f40d371b640 1 --2- 172.21.15.150:0/640872329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc06f400 0x7f40cc0cdf90 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.685+0000 7f40d371b640 1 -- 172.21.15.150:0/640872329 >> 172.21.15.150:0/640872329 conn(0x7f40cc06b190 msgr2=0x7f40cc06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:20.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.685+0000 7f40d371b640 1 -- 172.21.15.150:0/640872329 shutdown_connections 2024-03-17T23:47:20.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.685+0000 7f40d371b640 1 -- 172.21.15.150:0/640872329 wait complete. 2024-03-17T23:47:20.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.686+0000 7f40d371b640 1 Processor -- start 2024-03-17T23:47:20.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.686+0000 7f40d371b640 1 -- start start 2024-03-17T23:47:20.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.687+0000 7f40d371b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc06f400 0x7f40cc0ad1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:20.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.687+0000 7f40d371b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40cc0ce4d0 0x7f40cc0ae1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:20.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.687+0000 7f40d371b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40cc144bb0 0x7f40cc139d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:20.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.687+0000 7f40d371b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f40cc06bee0 con 0x7f40cc06f400 2024-03-17T23:47:20.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.687+0000 7f40d371b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f40cc06bd60 con 0x7f40cc0ce4d0 2024-03-17T23:47:20.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.687+0000 7f40d371b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f40cc165f70 con 0x7f40cc144bb0 2024-03-17T23:47:20.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.687+0000 7f40d14b7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc06f400 0x7f40cc0ad1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:20.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.687+0000 7f40d14b7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc06f400 0x7f40cc0ad1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40494/0 (socket says 172.21.15.150:40494) 2024-03-17T23:47:20.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.687+0000 7f40d14b7640 1 -- 172.21.15.150:0/180116151 learned_addr learned my addr 172.21.15.150:0/180116151 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:20.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.687+0000 7f40d0cb6640 1 --2- 172.21.15.150:0/180116151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40cc0ce4d0 0x7f40cc0ae1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:20.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.687+0000 7f40d1cb8640 1 --2- 172.21.15.150:0/180116151 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40cc144bb0 0x7f40cc139d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:20.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.687+0000 7f40d14b7640 1 -- 172.21.15.150:0/180116151 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40cc144bb0 msgr2=0x7f40cc139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:20.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.687+0000 7f40d14b7640 1 --2- 172.21.15.150:0/180116151 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40cc144bb0 0x7f40cc139d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.687+0000 7f40d14b7640 1 -- 172.21.15.150:0/180116151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40cc0ce4d0 msgr2=0x7f40cc0ae1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:20.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.687+0000 7f40d14b7640 1 --2- 172.21.15.150:0/180116151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40cc0ce4d0 0x7f40cc0ae1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.687+0000 7f40d14b7640 1 -- 172.21.15.150:0/180116151 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40cc0a8f00 con 0x7f40cc06f400 2024-03-17T23:47:20.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.688+0000 7f40d14b7640 1 --2- 172.21.15.150:0/180116151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc06f400 0x7f40cc0ad1d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f40c800c8c0 tx=0x7f40c800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:20.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.688+0000 7f40c1ffb640 1 -- 172.21.15.150:0/180116151 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c8018070 con 0x7f40cc06f400 2024-03-17T23:47:20.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.688+0000 7f40c1ffb640 1 -- 172.21.15.150:0/180116151 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40c800a640 con 0x7f40cc06f400 2024-03-17T23:47:20.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.688+0000 7f40c1ffb640 1 -- 172.21.15.150:0/180116151 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c800aa30 con 0x7f40cc06f400 2024-03-17T23:47:20.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.688+0000 7f40d371b640 1 -- 172.21.15.150:0/180116151 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40cc0a91e0 con 0x7f40cc06f400 2024-03-17T23:47:20.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.688+0000 7f40d371b640 1 -- 172.21.15.150:0/180116151 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40cc0a97a0 con 0x7f40cc06f400 2024-03-17T23:47:20.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.689+0000 7f40d371b640 1 -- 172.21.15.150:0/180116151 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40cc0681a0 con 0x7f40cc06f400 2024-03-17T23:47:20.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.689+0000 7f40c1ffb640 1 -- 172.21.15.150:0/180116151 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f40c801b070 con 0x7f40cc06f400 2024-03-17T23:47:20.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.690+0000 7f40c1ffb640 1 --2- 172.21.15.150:0/180116151 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f40b006bbd0 0x7f40b006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:20.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.690+0000 7f40d0cb6640 1 --2- 172.21.15.150:0/180116151 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f40b006bbd0 0x7f40b006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:20.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.690+0000 7f40c1ffb640 1 -- 172.21.15.150:0/180116151 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f40c808a3c0 con 0x7f40cc06f400 2024-03-17T23:47:20.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.690+0000 7f40d0cb6640 1 --2- 172.21.15.150:0/180116151 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f40b006bbd0 0x7f40b006e090 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f40bc036000 tx=0x7f40bc03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:20.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.694+0000 7f40c1ffb640 1 -- 172.21.15.150:0/180116151 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f40c8058490 con 0x7f40cc06f400 2024-03-17T23:47:20.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.866+0000 7f40d371b640 1 -- 172.21.15.150:0/180116151 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f40cc141520 con 0x7f40cc06f400 2024-03-17T23:47:20.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.866+0000 7f40c1ffb640 1 -- 172.21.15.150:0/180116151 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f40c805c530 con 0x7f40cc06f400 2024-03-17T23:47:20.867 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:20.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.868+0000 7f40d371b640 1 -- 172.21.15.150:0/180116151 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f40b006bbd0 msgr2=0x7f40b006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:20.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.868+0000 7f40d371b640 1 --2- 172.21.15.150:0/180116151 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f40b006bbd0 0x7f40b006e090 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f40bc036000 tx=0x7f40bc03c040 comp rx=0 tx=0).stop 2024-03-17T23:47:20.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.868+0000 7f40d371b640 1 -- 172.21.15.150:0/180116151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc06f400 msgr2=0x7f40cc0ad1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:20.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.868+0000 7f40d371b640 1 --2- 172.21.15.150:0/180116151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc06f400 0x7f40cc0ad1d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f40c800c8c0 tx=0x7f40c800ccc0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.868+0000 7f40d371b640 1 -- 172.21.15.150:0/180116151 shutdown_connections 2024-03-17T23:47:20.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.868+0000 7f40d371b640 1 --2- 172.21.15.150:0/180116151 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f40cc144bb0 0x7f40cc139d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.868+0000 7f40d371b640 1 --2- 172.21.15.150:0/180116151 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f40b006bbd0 0x7f40b006e090 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.868+0000 7f40d371b640 1 --2- 172.21.15.150:0/180116151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40cc0ce4d0 0x7f40cc0ae1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.868+0000 7f40d371b640 1 --2- 172.21.15.150:0/180116151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40cc06f400 0x7f40cc0ad1d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.868+0000 7f40d371b640 1 -- 172.21.15.150:0/180116151 >> 172.21.15.150:0/180116151 conn(0x7f40cc06b190 msgr2=0x7f40cc1421a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:20.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.868+0000 7f40d371b640 1 -- 172.21.15.150:0/180116151 shutdown_connections 2024-03-17T23:47:20.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.868+0000 7f40d371b640 1 -- 172.21.15.150:0/180116151 wait complete. 2024-03-17T23:47:20.879 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:47:20.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.982+0000 7f495a3c6640 1 Processor -- start 2024-03-17T23:47:20.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.982+0000 7f495a3c6640 1 -- start start 2024-03-17T23:47:20.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.983+0000 7f495a3c6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f49540ce6a0 0x7f4954166a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:20.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.983+0000 7f495a3c6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4954145170 con 0x7f4954144bb0 2024-03-17T23:47:20.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.983+0000 7f495a3c6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4954145450 con 0x7f495406f400 2024-03-17T23:47:20.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.983+0000 7f495a3c6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4954145730 con 0x7f49540ce6a0 2024-03-17T23:47:20.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.983+0000 7f4953fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f4954144bb0 0x7f495415ce50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34794/0 (socket says 172.21.15.150:34794) 2024-03-17T23:47:20.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.983+0000 7f4953fff640 1 -- 172.21.15.150:0/3099310970 learned_addr learned my addr 172.21.15.150:0/3099310970 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:20.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.983+0000 7f4958963640 1 --2- 172.21.15.150:0/3099310970 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f49540ce6a0 0x7f4954166a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:20.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.983+0000 7f4952ffd640 1 -- 172.21.15.150:0/3099310970 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3371365838 0 0) 0x7f4954145170 con 0x7f4954144bb0 2024-03-17T23:47:20.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.983+0000 7f4952ffd640 1 -- 172.21.15.150:0/3099310970 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f493c003520 con 0x7f4954144bb0 2024-03-17T23:47:20.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.983+0000 7f4952ffd640 1 -- 172.21.15.150:0/3099310970 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1997103738 0 0) 0x7f4954145450 con 0x7f495406f400 2024-03-17T23:47:20.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.983+0000 7f4952ffd640 1 -- 172.21.15.150:0/3099310970 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4954145170 con 0x7f495406f400 2024-03-17T23:47:20.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.983+0000 7f4958963640 1 -- 172.21.15.150:0/3099310970 >> v1:172.21.15.157:6789/0 conn(0x7f495406f400 legacy=0x7f49540cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:20.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.983+0000 7f4958963640 1 -- 172.21.15.150:0/3099310970 >> v1:172.21.15.150:6789/0 conn(0x7f4954144bb0 legacy=0x7f495415ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:20.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.983+0000 7f4958963640 1 -- 172.21.15.150:0/3099310970 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f495415d560 con 0x7f49540ce6a0 2024-03-17T23:47:20.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.984+0000 7f4958963640 1 --2- 172.21.15.150:0/3099310970 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f49540ce6a0 0x7f4954166a40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f494c009dc0 tx=0x7f494c02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2da7788a651869aa server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:20.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.984+0000 7f4952ffd640 1 -- 172.21.15.150:0/3099310970 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f494c039070 con 0x7f49540ce6a0 2024-03-17T23:47:20.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.984+0000 7f4952ffd640 1 -- 172.21.15.150:0/3099310970 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f494c02cc40 con 0x7f49540ce6a0 2024-03-17T23:47:20.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.984+0000 7f4952ffd640 1 -- 172.21.15.150:0/3099310970 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f494c034440 con 0x7f49540ce6a0 2024-03-17T23:47:20.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.985+0000 7f495a3c6640 1 -- 172.21.15.150:0/3099310970 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f49540ce6a0 msgr2=0x7f4954166a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:20.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.985+0000 7f495a3c6640 1 --2- 172.21.15.150:0/3099310970 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f49540ce6a0 0x7f4954166a40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f494c009dc0 tx=0x7f494c02be90 comp rx=0 tx=0).stop 2024-03-17T23:47:20.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.985+0000 7f495a3c6640 1 -- 172.21.15.150:0/3099310970 shutdown_connections 2024-03-17T23:47:20.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.985+0000 7f495a3c6640 1 --2- 172.21.15.150:0/3099310970 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f49540ce6a0 0x7f4954166a40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.985+0000 7f495a3c6640 1 -- 172.21.15.150:0/3099310970 >> 172.21.15.150:0/3099310970 conn(0x7f495406b190 msgr2=0x7f495406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:20.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.985+0000 7f495a3c6640 1 -- 172.21.15.150:0/3099310970 shutdown_connections 2024-03-17T23:47:20.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.985+0000 7f495a3c6640 1 -- 172.21.15.150:0/3099310970 wait complete. 2024-03-17T23:47:20.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.986+0000 7f495a3c6640 1 Processor -- start 2024-03-17T23:47:20.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.986+0000 7f495a3c6640 1 -- start start 2024-03-17T23:47:20.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.987+0000 7f495a3c6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f495406f400 0x7f495414c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:20.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.987+0000 7f495a3c6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49540ce6a0 0x7f495413f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:20.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.987+0000 7f495a3c6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4954144bb0 0x7f4954140c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:20.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.987+0000 7f495a3c6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4954167100 con 0x7f49540ce6a0 2024-03-17T23:47:20.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.987+0000 7f495a3c6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4954166f80 con 0x7f4954144bb0 2024-03-17T23:47:20.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.987+0000 7f495a3c6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4954167280 con 0x7f495406f400 2024-03-17T23:47:20.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.987+0000 7f49537fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49540ce6a0 0x7f495413f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:20.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.987+0000 7f49537fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49540ce6a0 0x7f495413f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40504/0 (socket says 172.21.15.150:40504) 2024-03-17T23:47:20.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.987+0000 7f49537fe640 1 -- 172.21.15.150:0/2200329758 learned_addr learned my addr 172.21.15.150:0/2200329758 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:20.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.987+0000 7f4953fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f495406f400 0x7f495414c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:20.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.987+0000 7f4958963640 1 --2- 172.21.15.150:0/2200329758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4954144bb0 0x7f4954140c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:20.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.987+0000 7f4953fff640 1 -- 172.21.15.150:0/2200329758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4954144bb0 msgr2=0x7f4954140c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:20.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.987+0000 7f4953fff640 1 --2- 172.21.15.150:0/2200329758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4954144bb0 0x7f4954140c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.987+0000 7f4953fff640 1 -- 172.21.15.150:0/2200329758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49540ce6a0 msgr2=0x7f495413f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:20.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.987+0000 7f4953fff640 1 --2- 172.21.15.150:0/2200329758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49540ce6a0 0x7f495413f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:20.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.987+0000 7f4953fff640 1 -- 172.21.15.150:0/2200329758 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f495413fe80 con 0x7f495406f400 2024-03-17T23:47:20.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.988+0000 7f49537fe640 1 --2- 172.21.15.150:0/2200329758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49540ce6a0 0x7f495413f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:20.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.988+0000 7f4958963640 1 --2- 172.21.15.150:0/2200329758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4954144bb0 0x7f4954140c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:20.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.988+0000 7f4953fff640 1 --2- 172.21.15.150:0/2200329758 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f495406f400 0x7f495414c380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f494400c390 tx=0x7f494400c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:20.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.988+0000 7f4950ff9640 1 -- 172.21.15.150:0/2200329758 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4944017070 con 0x7f495406f400 2024-03-17T23:47:20.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.988+0000 7f4950ff9640 1 -- 172.21.15.150:0/2200329758 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4944005ce0 con 0x7f495406f400 2024-03-17T23:47:20.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.988+0000 7f4950ff9640 1 -- 172.21.15.150:0/2200329758 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49440072d0 con 0x7f495406f400 2024-03-17T23:47:20.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.988+0000 7f495a3c6640 1 -- 172.21.15.150:0/2200329758 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4954140160 con 0x7f495406f400 2024-03-17T23:47:20.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.988+0000 7f495a3c6640 1 -- 172.21.15.150:0/2200329758 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f495414c8c0 con 0x7f495406f400 2024-03-17T23:47:20.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.989+0000 7f495a3c6640 1 -- 172.21.15.150:0/2200329758 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49540681a0 con 0x7f495406f400 2024-03-17T23:47:20.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.989+0000 7f4950ff9640 1 -- 172.21.15.150:0/2200329758 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f494401a070 con 0x7f495406f400 2024-03-17T23:47:20.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.990+0000 7f4950ff9640 1 --2- 172.21.15.150:0/2200329758 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f494006bbd0 0x7f494006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:20.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.990+0000 7f4950ff9640 1 -- 172.21.15.150:0/2200329758 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f4944089f70 con 0x7f495406f400 2024-03-17T23:47:20.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.990+0000 7f49537fe640 1 --2- 172.21.15.150:0/2200329758 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f494006bbd0 0x7f494006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:20.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.990+0000 7f49537fe640 1 --2- 172.21.15.150:0/2200329758 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f494006bbd0 0x7f494006e090 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f495415bb10 tx=0x7f4948009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:20.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:20.994+0000 7f4950ff9640 1 -- 172.21.15.150:0/2200329758 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4944058160 con 0x7f495406f400 2024-03-17T23:47:21.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.165+0000 7f495a3c6640 1 -- 172.21.15.150:0/2200329758 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f495413fb70 con 0x7f495406f400 2024-03-17T23:47:21.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.165+0000 7f4950ff9640 1 -- 172.21.15.150:0/2200329758 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f494405c200 con 0x7f495406f400 2024-03-17T23:47:21.166 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:47:21.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.167+0000 7f495a3c6640 1 -- 172.21.15.150:0/2200329758 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f494006bbd0 msgr2=0x7f494006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:21.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.167+0000 7f495a3c6640 1 --2- 172.21.15.150:0/2200329758 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f494006bbd0 0x7f494006e090 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f495415bb10 tx=0x7f4948009040 comp rx=0 tx=0).stop 2024-03-17T23:47:21.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.167+0000 7f495a3c6640 1 -- 172.21.15.150:0/2200329758 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f495406f400 msgr2=0x7f495414c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:21.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.167+0000 7f495a3c6640 1 --2- 172.21.15.150:0/2200329758 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f495406f400 0x7f495414c380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f494400c390 tx=0x7f494400c850 comp rx=0 tx=0).stop 2024-03-17T23:47:21.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.167+0000 7f495a3c6640 1 -- 172.21.15.150:0/2200329758 shutdown_connections 2024-03-17T23:47:21.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.167+0000 7f495a3c6640 1 --2- 172.21.15.150:0/2200329758 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f494006bbd0 0x7f494006e090 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.167+0000 7f495a3c6640 1 --2- 172.21.15.150:0/2200329758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4954144bb0 0x7f4954140c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.167+0000 7f495a3c6640 1 --2- 172.21.15.150:0/2200329758 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49540ce6a0 0x7f495413f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.167+0000 7f495a3c6640 1 --2- 172.21.15.150:0/2200329758 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f495406f400 0x7f495414c380 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.167+0000 7f495a3c6640 1 -- 172.21.15.150:0/2200329758 >> 172.21.15.150:0/2200329758 conn(0x7f495406b190 msgr2=0x7f4954138150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:21.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.167+0000 7f495a3c6640 1 -- 172.21.15.150:0/2200329758 shutdown_connections 2024-03-17T23:47:21.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.167+0000 7f495a3c6640 1 -- 172.21.15.150:0/2200329758 wait complete. 2024-03-17T23:47:21.178 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-17T23:47:21.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.281+0000 7fa3c66dc640 1 Processor -- start 2024-03-17T23:47:21.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.281+0000 7fa3c66dc640 1 -- start start 2024-03-17T23:47:21.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.282+0000 7fa3c66dc640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa3c006b190 0x7fa3c00b2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:21.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.282+0000 7fa3c66dc640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3c00d1230 con 0x7fa3c00b3c30 2024-03-17T23:47:21.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.282+0000 7fa3c66dc640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3c00b5c10 con 0x7fa3c00b5730 2024-03-17T23:47:21.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.282+0000 7fa3c66dc640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fa3c00b2560 con 0x7fa3c006b190 2024-03-17T23:47:21.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.282+0000 7fa3c4c79640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa3c00b3c30 0x7fa3c00b01a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34804/0 (socket says 172.21.15.150:34804) 2024-03-17T23:47:21.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.282+0000 7fa3c4c79640 1 -- 172.21.15.150:0/1720196040 learned_addr learned my addr 172.21.15.150:0/1720196040 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:21.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.282+0000 7fa3bffff640 1 --2- 172.21.15.150:0/1720196040 >> v2:172.21.15.157:3300/0 conn(0x7fa3c006b190 0x7fa3c00b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:21.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.282+0000 7fa3beffd640 1 -- 172.21.15.150:0/1720196040 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 548820515 0 0) 0x7fa3c00d1230 con 0x7fa3c00b3c30 2024-03-17T23:47:21.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.282+0000 7fa3beffd640 1 -- 172.21.15.150:0/1720196040 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3a8003520 con 0x7fa3c00b3c30 2024-03-17T23:47:21.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.283+0000 7fa3beffd640 1 -- 172.21.15.150:0/1720196040 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2697527018 0 0) 0x7fa3c00b5c10 con 0x7fa3c00b5730 2024-03-17T23:47:21.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.283+0000 7fa3bffff640 1 -- 172.21.15.150:0/1720196040 >> v1:172.21.15.157:6789/0 conn(0x7fa3c00b5730 legacy=0x7fa3c006b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:21.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.283+0000 7fa3bffff640 1 -- 172.21.15.150:0/1720196040 >> v1:172.21.15.150:6789/0 conn(0x7fa3c00b3c30 legacy=0x7fa3c00b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:21.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.283+0000 7fa3bffff640 1 -- 172.21.15.150:0/1720196040 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3c00d14d0 con 0x7fa3c006b190 2024-03-17T23:47:21.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.283+0000 7fa3bffff640 1 --2- 172.21.15.150:0/1720196040 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3c006b190 0x7fa3c00b2020 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa3b00097a0 tx=0x7fa3b002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eee46d3cf1f191d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:21.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.283+0000 7fa3beffd640 1 -- 172.21.15.150:0/1720196040 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b0035020 con 0x7fa3c006b190 2024-03-17T23:47:21.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.283+0000 7fa3beffd640 1 -- 172.21.15.150:0/1720196040 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3b0031830 con 0x7fa3c006b190 2024-03-17T23:47:21.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.283+0000 7fa3beffd640 1 -- 172.21.15.150:0/1720196040 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b0031c00 con 0x7fa3c006b190 2024-03-17T23:47:21.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.284+0000 7fa3c66dc640 1 -- 172.21.15.150:0/1720196040 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3c006b190 msgr2=0x7fa3c00b2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:21.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.284+0000 7fa3c66dc640 1 --2- 172.21.15.150:0/1720196040 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3c006b190 0x7fa3c00b2020 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa3b00097a0 tx=0x7fa3b002b8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.284+0000 7fa3c66dc640 1 -- 172.21.15.150:0/1720196040 shutdown_connections 2024-03-17T23:47:21.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.284+0000 7fa3c66dc640 1 --2- 172.21.15.150:0/1720196040 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3c006b190 0x7fa3c00b2020 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.284+0000 7fa3c66dc640 1 -- 172.21.15.150:0/1720196040 >> 172.21.15.150:0/1720196040 conn(0x7fa3c00b6170 msgr2=0x7fa3c0099a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:21.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.284+0000 7fa3c66dc640 1 -- 172.21.15.150:0/1720196040 shutdown_connections 2024-03-17T23:47:21.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.284+0000 7fa3c66dc640 1 -- 172.21.15.150:0/1720196040 wait complete. 2024-03-17T23:47:21.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.285+0000 7fa3c66dc640 1 Processor -- start 2024-03-17T23:47:21.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.285+0000 7fa3c66dc640 1 -- start start 2024-03-17T23:47:21.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.286+0000 7fa3c66dc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3c006b190 0x7fa3c014eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:21.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.286+0000 7fa3c66dc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3c00b3c30 0x7fa3c0158c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:21.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.286+0000 7fa3c66dc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3c00b5730 0x7fa3c01452f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:21.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.286+0000 7fa3c66dc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c009a380 con 0x7fa3c006b190 2024-03-17T23:47:21.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.286+0000 7fa3c66dc640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c00b5ef0 con 0x7fa3c00b5730 2024-03-17T23:47:21.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.286+0000 7fa3c66dc640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa3c015e260 con 0x7fa3c00b3c30 2024-03-17T23:47:21.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.286+0000 7fa3bffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3c006b190 0x7fa3c014eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:21.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.286+0000 7fa3bffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3c006b190 0x7fa3c014eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40516/0 (socket says 172.21.15.150:40516) 2024-03-17T23:47:21.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.286+0000 7fa3bffff640 1 -- 172.21.15.150:0/1141622973 learned_addr learned my addr 172.21.15.150:0/1141622973 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:21.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.286+0000 7fa3c4c79640 1 --2- 172.21.15.150:0/1141622973 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3c00b5730 0x7fa3c01452f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:21.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.286+0000 7fa3bf7fe640 1 --2- 172.21.15.150:0/1141622973 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3c00b3c30 0x7fa3c0158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:21.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.286+0000 7fa3bffff640 1 -- 172.21.15.150:0/1141622973 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3c00b3c30 msgr2=0x7fa3c0158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:21.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.286+0000 7fa3bffff640 1 --2- 172.21.15.150:0/1141622973 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3c00b3c30 0x7fa3c0158c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.287+0000 7fa3bffff640 1 -- 172.21.15.150:0/1141622973 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3c00b5730 msgr2=0x7fa3c01452f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:21.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.287+0000 7fa3bffff640 1 --2- 172.21.15.150:0/1141622973 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3c00b5730 0x7fa3c01452f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.287+0000 7fa3bffff640 1 -- 172.21.15.150:0/1141622973 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3c0157b80 con 0x7fa3c006b190 2024-03-17T23:47:21.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.287+0000 7fa3bffff640 1 --2- 172.21.15.150:0/1141622973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3c006b190 0x7fa3c014eb30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa3b00098c0 tx=0x7fa3b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:21.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.287+0000 7fa3bcff9640 1 -- 172.21.15.150:0/1141622973 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b0035040 con 0x7fa3c006b190 2024-03-17T23:47:21.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.287+0000 7fa3bcff9640 1 -- 172.21.15.150:0/1141622973 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3b003a840 con 0x7fa3c006b190 2024-03-17T23:47:21.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.287+0000 7fa3c66dc640 1 -- 172.21.15.150:0/1141622973 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3c0157e60 con 0x7fa3c006b190 2024-03-17T23:47:21.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.287+0000 7fa3bcff9640 1 -- 172.21.15.150:0/1141622973 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b003ac10 con 0x7fa3c006b190 2024-03-17T23:47:21.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.287+0000 7fa3c66dc640 1 -- 172.21.15.150:0/1141622973 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3c0158420 con 0x7fa3c006b190 2024-03-17T23:47:21.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.288+0000 7fa3c66dc640 1 -- 172.21.15.150:0/1141622973 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3c00681a0 con 0x7fa3c006b190 2024-03-17T23:47:21.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.288+0000 7fa3bcff9640 1 -- 172.21.15.150:0/1141622973 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa3b0038020 con 0x7fa3c006b190 2024-03-17T23:47:21.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.289+0000 7fa3bcff9640 1 --2- 172.21.15.150:0/1141622973 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa3ac06bbd0 0x7fa3ac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:21.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.289+0000 7fa3bcff9640 1 -- 172.21.15.150:0/1141622973 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fa3b00ace10 con 0x7fa3c006b190 2024-03-17T23:47:21.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.289+0000 7fa3bf7fe640 1 --2- 172.21.15.150:0/1141622973 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa3ac06bbd0 0x7fa3ac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:21.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.289+0000 7fa3bf7fe640 1 --2- 172.21.15.150:0/1141622973 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa3ac06bbd0 0x7fa3ac06e090 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa3c009ecf0 tx=0x7fa3b4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:21.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.293+0000 7fa3bcff9640 1 -- 172.21.15.150:0/1141622973 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa3b007b000 con 0x7fa3c006b190 2024-03-17T23:47:21.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.466+0000 7fa3c66dc640 1 -- 172.21.15.150:0/1141622973 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa3c006b6a0 con 0x7fa3c006b190 2024-03-17T23:47:21.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.466+0000 7fa3bcff9640 1 -- 172.21.15.150:0/1141622973 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fa3b007f0a0 con 0x7fa3c006b190 2024-03-17T23:47:21.467 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:21.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.468+0000 7fa3c66dc640 1 -- 172.21.15.150:0/1141622973 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa3ac06bbd0 msgr2=0x7fa3ac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:21.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.468+0000 7fa3c66dc640 1 --2- 172.21.15.150:0/1141622973 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa3ac06bbd0 0x7fa3ac06e090 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa3c009ecf0 tx=0x7fa3b4009040 comp rx=0 tx=0).stop 2024-03-17T23:47:21.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.468+0000 7fa3c66dc640 1 -- 172.21.15.150:0/1141622973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3c006b190 msgr2=0x7fa3c014eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:21.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.468+0000 7fa3c66dc640 1 --2- 172.21.15.150:0/1141622973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3c006b190 0x7fa3c014eb30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa3b00098c0 tx=0x7fa3b00046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.468+0000 7fa3c66dc640 1 -- 172.21.15.150:0/1141622973 shutdown_connections 2024-03-17T23:47:21.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.468+0000 7fa3c66dc640 1 --2- 172.21.15.150:0/1141622973 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa3ac06bbd0 0x7fa3ac06e090 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.468+0000 7fa3c66dc640 1 --2- 172.21.15.150:0/1141622973 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3c00b5730 0x7fa3c01452f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.468+0000 7fa3c66dc640 1 --2- 172.21.15.150:0/1141622973 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3c00b3c30 0x7fa3c0158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.468+0000 7fa3c66dc640 1 --2- 172.21.15.150:0/1141622973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3c006b190 0x7fa3c014eb30 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.468+0000 7fa3c66dc640 1 -- 172.21.15.150:0/1141622973 >> 172.21.15.150:0/1141622973 conn(0x7fa3c00b6170 msgr2=0x7fa3c006ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:21.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.469+0000 7fa3c66dc640 1 -- 172.21.15.150:0/1141622973 shutdown_connections 2024-03-17T23:47:21.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.469+0000 7fa3c66dc640 1 -- 172.21.15.150:0/1141622973 wait complete. 2024-03-17T23:47:21.480 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:47:21.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.580+0000 7f3ca91d4640 1 Processor -- start 2024-03-17T23:47:21.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.580+0000 7f3ca91d4640 1 -- start start 2024-03-17T23:47:21.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.581+0000 7f3ca91d4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3ca415bc50 0x7f3ca40cf910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:21.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.581+0000 7f3ca91d4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ca415c030 0x7f3ca40ad180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:21.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.581+0000 7f3ca91d4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ca40cfff0 con 0x7f3ca4099a70 2024-03-17T23:47:21.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.581+0000 7f3ca91d4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f3ca4067340 con 0x7f3ca415bc50 2024-03-17T23:47:21.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.581+0000 7f3ca91d4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3ca40d0340 con 0x7f3ca415c030 2024-03-17T23:47:21.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.581+0000 7f3ca2d9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3ca415bc50 0x7f3ca40cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:21.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.581+0000 7f3ca2d9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3ca415bc50 0x7f3ca40cf910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40526/0 (socket says 172.21.15.150:40526) 2024-03-17T23:47:21.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.581+0000 7f3ca2d9d640 1 -- 172.21.15.150:0/202875318 learned_addr learned my addr 172.21.15.150:0/202875318 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:21.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.581+0000 7f3ca259c640 1 --2- 172.21.15.150:0/202875318 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ca415c030 0x7f3ca40ad180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:21.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.581+0000 7f3ca2d9d640 1 -- 172.21.15.150:0/202875318 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ca415c030 msgr2=0x7f3ca40ad180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:21.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.581+0000 7f3ca2d9d640 1 --2- 172.21.15.150:0/202875318 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ca415c030 0x7f3ca40ad180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.581+0000 7f3ca2d9d640 1 -- 172.21.15.150:0/202875318 >> v1:172.21.15.157:6789/0 conn(0x7f3ca4099a70 legacy=0x7f3ca406b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:21.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.581+0000 7f3ca2d9d640 1 -- 172.21.15.150:0/202875318 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ca4067590 con 0x7f3ca415bc50 2024-03-17T23:47:21.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.582+0000 7f3ca2d9d640 1 --2- 172.21.15.150:0/202875318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ca415bc50 0x7f3ca40cf910 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3c94009870 tx=0x7f3c9402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5535d43cad2beea3 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:21.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.582+0000 7f3ca1d9b640 1 -- 172.21.15.150:0/202875318 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c940317c0 con 0x7f3ca415bc50 2024-03-17T23:47:21.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.582+0000 7f3ca1d9b640 1 -- 172.21.15.150:0/202875318 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c94037070 con 0x7f3ca415bc50 2024-03-17T23:47:21.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.582+0000 7f3ca1d9b640 1 -- 172.21.15.150:0/202875318 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c94031b80 con 0x7f3ca415bc50 2024-03-17T23:47:21.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.583+0000 7f3ca91d4640 1 -- 172.21.15.150:0/202875318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ca415bc50 msgr2=0x7f3ca40cf910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:21.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.583+0000 7f3ca91d4640 1 --2- 172.21.15.150:0/202875318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ca415bc50 0x7f3ca40cf910 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3c94009870 tx=0x7f3c9402b860 comp rx=0 tx=0).stop 2024-03-17T23:47:21.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.583+0000 7f3ca91d4640 1 -- 172.21.15.150:0/202875318 shutdown_connections 2024-03-17T23:47:21.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.583+0000 7f3ca91d4640 1 --2- 172.21.15.150:0/202875318 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ca415c030 0x7f3ca40ad180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.583+0000 7f3ca91d4640 1 --2- 172.21.15.150:0/202875318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ca415bc50 0x7f3ca40cf910 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.583+0000 7f3ca91d4640 1 -- 172.21.15.150:0/202875318 >> 172.21.15.150:0/202875318 conn(0x7f3ca40b3da0 msgr2=0x7f3ca40b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:21.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.583+0000 7f3ca91d4640 1 -- 172.21.15.150:0/202875318 shutdown_connections 2024-03-17T23:47:21.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.583+0000 7f3ca91d4640 1 -- 172.21.15.150:0/202875318 wait complete. 2024-03-17T23:47:21.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.584+0000 7f3ca91d4640 1 Processor -- start 2024-03-17T23:47:21.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.584+0000 7f3ca91d4640 1 -- start start 2024-03-17T23:47:21.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.585+0000 7f3ca91d4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ca4099a70 0x7f3ca414feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:21.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.585+0000 7f3ca91d4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ca415bc50 0x7f3ca409acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:21.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.585+0000 7f3ca91d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ca409b200 0x7f3ca4151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:21.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.585+0000 7f3ca91d4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca40d07d0 con 0x7f3ca409b200 2024-03-17T23:47:21.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.585+0000 7f3ca91d4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca4072d30 con 0x7f3ca4099a70 2024-03-17T23:47:21.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.585+0000 7f3ca91d4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3ca40d0950 con 0x7f3ca415bc50 2024-03-17T23:47:21.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.585+0000 7f3ca359e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ca409b200 0x7f3ca4151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:21.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.585+0000 7f3ca2d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ca4099a70 0x7f3ca414feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:21.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.585+0000 7f3ca259c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ca415bc50 0x7f3ca409acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:21.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.585+0000 7f3ca359e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ca409b200 0x7f3ca4151d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40528/0 (socket says 172.21.15.150:40528) 2024-03-17T23:47:21.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.585+0000 7f3ca359e640 1 -- 172.21.15.150:0/1098650874 learned_addr learned my addr 172.21.15.150:0/1098650874 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:21.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.585+0000 7f3ca259c640 1 -- 172.21.15.150:0/1098650874 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ca4099a70 msgr2=0x7f3ca414feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:21.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.585+0000 7f3ca259c640 1 --2- 172.21.15.150:0/1098650874 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ca4099a70 0x7f3ca414feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.585+0000 7f3ca259c640 1 -- 172.21.15.150:0/1098650874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ca409b200 msgr2=0x7f3ca4151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:21.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.585+0000 7f3ca259c640 1 --2- 172.21.15.150:0/1098650874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ca409b200 0x7f3ca4151d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.585+0000 7f3ca259c640 1 -- 172.21.15.150:0/1098650874 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ca41503f0 con 0x7f3ca415bc50 2024-03-17T23:47:21.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.585+0000 7f3ca359e640 1 --2- 172.21.15.150:0/1098650874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ca409b200 0x7f3ca4151d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:21.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.585+0000 7f3ca2d9d640 1 --2- 172.21.15.150:0/1098650874 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ca4099a70 0x7f3ca414feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:21.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.586+0000 7f3ca259c640 1 --2- 172.21.15.150:0/1098650874 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ca415bc50 0x7f3ca409acc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3c9800e850 tx=0x7f3c9800ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:21.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.586+0000 7f3c8b7fe640 1 -- 172.21.15.150:0/1098650874 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c98017070 con 0x7f3ca415bc50 2024-03-17T23:47:21.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.586+0000 7f3c8b7fe640 1 -- 172.21.15.150:0/1098650874 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c9800c940 con 0x7f3ca415bc50 2024-03-17T23:47:21.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.586+0000 7f3ca91d4640 1 -- 172.21.15.150:0/1098650874 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ca4150cf0 con 0x7f3ca415bc50 2024-03-17T23:47:21.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.586+0000 7f3c8b7fe640 1 -- 172.21.15.150:0/1098650874 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c9800cd10 con 0x7f3ca415bc50 2024-03-17T23:47:21.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.586+0000 7f3ca91d4640 1 -- 172.21.15.150:0/1098650874 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ca41512b0 con 0x7f3ca415bc50 2024-03-17T23:47:21.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.587+0000 7f3ca91d4640 1 -- 172.21.15.150:0/1098650874 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ca40681a0 con 0x7f3ca415bc50 2024-03-17T23:47:21.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.587+0000 7f3c8b7fe640 1 -- 172.21.15.150:0/1098650874 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3c9801a070 con 0x7f3ca415bc50 2024-03-17T23:47:21.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.587+0000 7f3c8b7fe640 1 --2- 172.21.15.150:0/1098650874 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3c9006bbd0 0x7f3c9006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:21.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.588+0000 7f3c8b7fe640 1 -- 172.21.15.150:0/1098650874 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f3c9808acb0 con 0x7f3ca415bc50 2024-03-17T23:47:21.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.588+0000 7f3ca2d9d640 1 --2- 172.21.15.150:0/1098650874 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3c9006bbd0 0x7f3c9006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:21.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.588+0000 7f3ca2d9d640 1 --2- 172.21.15.150:0/1098650874 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3c9006bbd0 0x7f3c9006e090 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3c94036000 tx=0x7f3c9403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:21.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.592+0000 7f3c8b7fe640 1 -- 172.21.15.150:0/1098650874 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3c98058e00 con 0x7f3ca415bc50 2024-03-17T23:47:21.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.800+0000 7f3ca91d4640 1 -- 172.21.15.150:0/1098650874 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ca40b0f40 con 0x7f3ca415bc50 2024-03-17T23:47:21.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.801+0000 7f3c8b7fe640 1 -- 172.21.15.150:0/1098650874 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5186 (secure 0 0 0) 0x7f3c9805cea0 con 0x7f3ca415bc50 2024-03-17T23:47:21.802 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:21.802 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/1949609985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6833","nonce":1949609985},{"type":"v1","addr":"172.21.15.157:6835","nonce":1949609985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/758264728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":758264728},{"type":"v1","addr":"172.21.15.150:6835","nonce":758264728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3655589010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3655589010},{"type":"v1","addr":"172.21.15.150:6837","nonce":3655589010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-17T23:46:55.253739+0000","modified":"2024-03-17T23:47:11.316249+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6834/1625058792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1625058792},{"type":"v1","addr":"172.21.15.157:6834","nonce":1625058792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4297,"qdb_cluster":[4297]},"id":1}]} 2024-03-17T23:47:21.802 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 8 2024-03-17T23:47:21.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.803+0000 7f3ca91d4640 1 -- 172.21.15.150:0/1098650874 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3c9006bbd0 msgr2=0x7f3c9006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:21.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.803+0000 7f3ca91d4640 1 --2- 172.21.15.150:0/1098650874 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3c9006bbd0 0x7f3c9006e090 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3c94036000 tx=0x7f3c9403d040 comp rx=0 tx=0).stop 2024-03-17T23:47:21.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.803+0000 7f3ca91d4640 1 -- 172.21.15.150:0/1098650874 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ca415bc50 msgr2=0x7f3ca409acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:21.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.803+0000 7f3ca91d4640 1 --2- 172.21.15.150:0/1098650874 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ca415bc50 0x7f3ca409acc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3c9800e850 tx=0x7f3c9800ed10 comp rx=0 tx=0).stop 2024-03-17T23:47:21.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.803+0000 7f3ca91d4640 1 -- 172.21.15.150:0/1098650874 shutdown_connections 2024-03-17T23:47:21.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.803+0000 7f3ca91d4640 1 --2- 172.21.15.150:0/1098650874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ca409b200 0x7f3ca4151d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.803+0000 7f3ca91d4640 1 --2- 172.21.15.150:0/1098650874 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ca415bc50 0x7f3ca409acc0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.803+0000 7f3ca91d4640 1 --2- 172.21.15.150:0/1098650874 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3c9006bbd0 0x7f3c9006e090 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.803+0000 7f3ca91d4640 1 --2- 172.21.15.150:0/1098650874 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ca4099a70 0x7f3ca414feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.803+0000 7f3ca91d4640 1 -- 172.21.15.150:0/1098650874 >> 172.21.15.150:0/1098650874 conn(0x7f3ca40b3da0 msgr2=0x7f3ca40ad180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:21.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.803+0000 7f3ca91d4640 1 -- 172.21.15.150:0/1098650874 shutdown_connections 2024-03-17T23:47:21.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.803+0000 7f3ca91d4640 1 -- 172.21.15.150:0/1098650874 wait complete. 2024-03-17T23:47:21.815 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:21.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.915+0000 7fb2fa67e640 1 Processor -- start 2024-03-17T23:47:21.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.915+0000 7fb2fa67e640 1 -- start start 2024-03-17T23:47:21.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.916+0000 7fb2fa67e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb2f415f790 0x7fb2f415fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:21.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.916+0000 7fb2fa67e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2f40cfad0 con 0x7fb2f40cc5e0 2024-03-17T23:47:21.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.916+0000 7fb2fa67e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2f40cfdb0 con 0x7fb2f409a470 2024-03-17T23:47:21.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.916+0000 7fb2fa67e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fb2f4160140 con 0x7fb2f415f790 2024-03-17T23:47:21.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.916+0000 7fb2f37fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb2f415f790 0x7fb2f415fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:21.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.916+0000 7fb2f8c1b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb2f40cc5e0 0x7fb2f40cc9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34810/0 (socket says 172.21.15.150:34810) 2024-03-17T23:47:21.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.916+0000 7fb2f8c1b640 1 -- 172.21.15.150:0/2860031412 learned_addr learned my addr 172.21.15.150:0/2860031412 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:21.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.916+0000 7fb2f2ffd640 1 -- 172.21.15.150:0/2860031412 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3872150495 0 0) 0x7fb2f40cfdb0 con 0x7fb2f409a470 2024-03-17T23:47:21.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.916+0000 7fb2f37fe640 1 -- 172.21.15.150:0/2860031412 >> v1:172.21.15.157:6789/0 conn(0x7fb2f409a470 legacy=0x7fb2f409a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:21.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.916+0000 7fb2f37fe640 1 -- 172.21.15.150:0/2860031412 >> v1:172.21.15.150:6789/0 conn(0x7fb2f40cc5e0 legacy=0x7fb2f40cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:21.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.916+0000 7fb2f37fe640 1 -- 172.21.15.150:0/2860031412 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2f40d0090 con 0x7fb2f415f790 2024-03-17T23:47:21.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.917+0000 7fb2f37fe640 1 --2- 172.21.15.150:0/2860031412 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2f415f790 0x7fb2f415fc00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc0097a0 tx=0x7fb2dc02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c51147b24eb22a15 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:21.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.917+0000 7fb2f2ffd640 1 -- 172.21.15.150:0/2860031412 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc037070 con 0x7fb2f415f790 2024-03-17T23:47:21.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.917+0000 7fb2f2ffd640 1 -- 172.21.15.150:0/2860031412 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2dc030710 con 0x7fb2f415f790 2024-03-17T23:47:21.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.917+0000 7fb2f2ffd640 1 -- 172.21.15.150:0/2860031412 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc030ae0 con 0x7fb2f415f790 2024-03-17T23:47:21.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.918+0000 7fb2fa67e640 1 -- 172.21.15.150:0/2860031412 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2f415f790 msgr2=0x7fb2f415fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:21.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.918+0000 7fb2fa67e640 1 --2- 172.21.15.150:0/2860031412 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2f415f790 0x7fb2f415fc00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc0097a0 tx=0x7fb2dc02b790 comp rx=0 tx=0).stop 2024-03-17T23:47:21.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.918+0000 7fb2fa67e640 1 -- 172.21.15.150:0/2860031412 shutdown_connections 2024-03-17T23:47:21.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.918+0000 7fb2fa67e640 1 --2- 172.21.15.150:0/2860031412 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2f415f790 0x7fb2f415fc00 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.918+0000 7fb2fa67e640 1 -- 172.21.15.150:0/2860031412 >> 172.21.15.150:0/2860031412 conn(0x7fb2f40b3630 msgr2=0x7fb2f40b3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:21.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.918+0000 7fb2fa67e640 1 -- 172.21.15.150:0/2860031412 shutdown_connections 2024-03-17T23:47:21.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.918+0000 7fb2fa67e640 1 -- 172.21.15.150:0/2860031412 wait complete. 2024-03-17T23:47:21.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.919+0000 7fb2fa67e640 1 Processor -- start 2024-03-17T23:47:21.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.920+0000 7fb2fa67e640 1 -- start start 2024-03-17T23:47:21.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.920+0000 7fb2fa67e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2f409a470 0x7fb2f40b7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:21.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.920+0000 7fb2fa67e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2f40cc5e0 0x7fb2f4151f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:21.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.920+0000 7fb2fa67e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2f415f790 0x7fb2f40c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:21.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.920+0000 7fb2fa67e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f409af60 con 0x7fb2f40cc5e0 2024-03-17T23:47:21.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.920+0000 7fb2fa67e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f40d0650 con 0x7fb2f415f790 2024-03-17T23:47:21.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.920+0000 7fb2fa67e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb2f40cf710 con 0x7fb2f409a470 2024-03-17T23:47:21.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.920+0000 7fb2f37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2f40cc5e0 0x7fb2f4151f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:21.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.920+0000 7fb2f37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2f40cc5e0 0x7fb2f4151f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40548/0 (socket says 172.21.15.150:40548) 2024-03-17T23:47:21.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.920+0000 7fb2f37fe640 1 -- 172.21.15.150:0/2663990567 learned_addr learned my addr 172.21.15.150:0/2663990567 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:21.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.920+0000 7fb2f3fff640 1 --2- 172.21.15.150:0/2663990567 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2f409a470 0x7fb2f40b7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:21.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.920+0000 7fb2f8c1b640 1 --2- 172.21.15.150:0/2663990567 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2f415f790 0x7fb2f40c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:21.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.921+0000 7fb2f37fe640 1 -- 172.21.15.150:0/2663990567 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2f409a470 msgr2=0x7fb2f40b7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:21.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.921+0000 7fb2f37fe640 1 --2- 172.21.15.150:0/2663990567 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2f409a470 0x7fb2f40b7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.921+0000 7fb2f37fe640 1 -- 172.21.15.150:0/2663990567 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2f415f790 msgr2=0x7fb2f40c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:21.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.921+0000 7fb2f37fe640 1 --2- 172.21.15.150:0/2663990567 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2f415f790 0x7fb2f40c6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:21.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.921+0000 7fb2f37fe640 1 -- 172.21.15.150:0/2663990567 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2f40b0fc0 con 0x7fb2f40cc5e0 2024-03-17T23:47:21.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.921+0000 7fb2f37fe640 1 --2- 172.21.15.150:0/2663990567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2f40cc5e0 0x7fb2f4151f70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc0098c0 tx=0x7fb2dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:21.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.921+0000 7fb2f0ff9640 1 -- 172.21.15.150:0/2663990567 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc042070 con 0x7fb2f40cc5e0 2024-03-17T23:47:21.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.921+0000 7fb2f0ff9640 1 -- 172.21.15.150:0/2663990567 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2dc032a40 con 0x7fb2f40cc5e0 2024-03-17T23:47:21.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.921+0000 7fb2fa67e640 1 -- 172.21.15.150:0/2663990567 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2f40b6150 con 0x7fb2f40cc5e0 2024-03-17T23:47:21.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.921+0000 7fb2f0ff9640 1 -- 172.21.15.150:0/2663990567 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc037040 con 0x7fb2f40cc5e0 2024-03-17T23:47:21.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.921+0000 7fb2fa67e640 1 -- 172.21.15.150:0/2663990567 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2f40b6710 con 0x7fb2f40cc5e0 2024-03-17T23:47:21.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.922+0000 7fb2fa67e640 1 -- 172.21.15.150:0/2663990567 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2f40b1190 con 0x7fb2f40cc5e0 2024-03-17T23:47:21.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.922+0000 7fb2f0ff9640 1 -- 172.21.15.150:0/2663990567 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb2dc045070 con 0x7fb2f40cc5e0 2024-03-17T23:47:21.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.923+0000 7fb2f0ff9640 1 --2- 172.21.15.150:0/2663990567 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb2d406bbd0 0x7fb2d406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:21.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.923+0000 7fb2f0ff9640 1 -- 172.21.15.150:0/2663990567 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fb2dc0aca30 con 0x7fb2f40cc5e0 2024-03-17T23:47:21.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.923+0000 7fb2f3fff640 1 --2- 172.21.15.150:0/2663990567 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb2d406bbd0 0x7fb2d406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:21.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.923+0000 7fb2f3fff640 1 --2- 172.21.15.150:0/2663990567 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb2d406bbd0 0x7fb2d406e090 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4000f60 tx=0x7fb2e4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:21.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:21.927+0000 7fb2f0ff9640 1 -- 172.21.15.150:0/2663990567 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb2dc07acb0 con 0x7fb2f40cc5e0 2024-03-17T23:47:22.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.103+0000 7fb2fa67e640 1 -- 172.21.15.150:0/2663990567 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb2f40681a0 con 0x7fb2f40cc5e0 2024-03-17T23:47:22.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.104+0000 7fb2f0ff9640 1 -- 172.21.15.150:0/2663990567 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18550 (secure 0 0 0) 0x7fb2dc07ed50 con 0x7fb2f40cc5e0 2024-03-17T23:47:22.105 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:22.106 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:01.940019+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:22.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.106+0000 7fb2fa67e640 1 -- 172.21.15.150:0/2663990567 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb2d406bbd0 msgr2=0x7fb2d406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:22.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.106+0000 7fb2fa67e640 1 --2- 172.21.15.150:0/2663990567 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb2d406bbd0 0x7fb2d406e090 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4000f60 tx=0x7fb2e4009040 comp rx=0 tx=0).stop 2024-03-17T23:47:22.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.106+0000 7fb2fa67e640 1 -- 172.21.15.150:0/2663990567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2f40cc5e0 msgr2=0x7fb2f4151f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:22.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.106+0000 7fb2fa67e640 1 --2- 172.21.15.150:0/2663990567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2f40cc5e0 0x7fb2f4151f70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc0098c0 tx=0x7fb2dc0046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.106+0000 7fb2fa67e640 1 -- 172.21.15.150:0/2663990567 shutdown_connections 2024-03-17T23:47:22.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.106+0000 7fb2fa67e640 1 --2- 172.21.15.150:0/2663990567 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb2d406bbd0 0x7fb2d406e090 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.106+0000 7fb2fa67e640 1 --2- 172.21.15.150:0/2663990567 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2f415f790 0x7fb2f40c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.106+0000 7fb2fa67e640 1 --2- 172.21.15.150:0/2663990567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2f40cc5e0 0x7fb2f4151f70 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.106+0000 7fb2fa67e640 1 --2- 172.21.15.150:0/2663990567 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2f409a470 0x7fb2f40b7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.106+0000 7fb2fa67e640 1 -- 172.21.15.150:0/2663990567 >> 172.21.15.150:0/2663990567 conn(0x7fb2f40b3630 msgr2=0x7fb2f409a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:22.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.106+0000 7fb2fa67e640 1 -- 172.21.15.150:0/2663990567 shutdown_connections 2024-03-17T23:47:22.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.106+0000 7fb2fa67e640 1 -- 172.21.15.150:0/2663990567 wait complete. 2024-03-17T23:47:22.118 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:47:22.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.219+0000 7f20d1b22640 1 Processor -- start 2024-03-17T23:47:22.220 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.219+0000 7f20d1b22640 1 -- start start 2024-03-17T23:47:22.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.220+0000 7f20d1b22640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f20cc099e20 0x7f20cc09a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:22.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.220+0000 7f20d1b22640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f20cc15e460 0x7f20cc06c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:22.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.220+0000 7f20d1b22640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20cc09a740 con 0x7f20cc06c8c0 2024-03-17T23:47:22.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.220+0000 7f20d1b22640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f20cc15e990 con 0x7f20cc15e460 2024-03-17T23:47:22.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.220+0000 7f20d1b22640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f20cc15ec70 con 0x7f20cc099e20 2024-03-17T23:47:22.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.220+0000 7f20cbfff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f20cc06c8c0 0x7f20cc0d0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34814/0 (socket says 172.21.15.150:34814) 2024-03-17T23:47:22.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.220+0000 7f20cbfff640 1 -- 172.21.15.150:0/787663156 learned_addr learned my addr 172.21.15.150:0/787663156 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:22.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.220+0000 7f20caffd640 1 --2- 172.21.15.150:0/787663156 >> v2:172.21.15.157:3300/0 conn(0x7f20cc15e460 0x7f20cc06c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:22.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.220+0000 7f20cb7fe640 1 --2- 172.21.15.150:0/787663156 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f20cc099e20 0x7f20cc09a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:22.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.220+0000 7f20ca7fc640 1 -- 172.21.15.150:0/787663156 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3053738629 0 0) 0x7f20cc09a740 con 0x7f20cc06c8c0 2024-03-17T23:47:22.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.220+0000 7f20ca7fc640 1 -- 172.21.15.150:0/787663156 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20b4003520 con 0x7f20cc06c8c0 2024-03-17T23:47:22.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.220+0000 7f20cb7fe640 1 -- 172.21.15.150:0/787663156 >> v2:172.21.15.157:3300/0 conn(0x7f20cc15e460 msgr2=0x7f20cc06c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:22.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.220+0000 7f20cb7fe640 1 --2- 172.21.15.150:0/787663156 >> v2:172.21.15.157:3300/0 conn(0x7f20cc15e460 0x7f20cc06c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.220+0000 7f20cb7fe640 1 -- 172.21.15.150:0/787663156 >> v1:172.21.15.150:6789/0 conn(0x7f20cc06c8c0 legacy=0x7f20cc0d0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:22.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.220+0000 7f20cb7fe640 1 -- 172.21.15.150:0/787663156 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20cc15ef50 con 0x7f20cc099e20 2024-03-17T23:47:22.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.220+0000 7f20caffd640 1 --2- 172.21.15.150:0/787663156 >> v2:172.21.15.157:3300/0 conn(0x7f20cc15e460 0x7f20cc06c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:22.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.221+0000 7f20cb7fe640 1 --2- 172.21.15.150:0/787663156 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f20cc099e20 0x7f20cc09a200 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f20bc0097a0 tx=0x7f20bc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a8b1e2c7ca82d4d server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:22.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.221+0000 7f20ca7fc640 1 -- 172.21.15.150:0/787663156 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20bc035020 con 0x7f20cc099e20 2024-03-17T23:47:22.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.221+0000 7f20ca7fc640 1 -- 172.21.15.150:0/787663156 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20bc031830 con 0x7f20cc099e20 2024-03-17T23:47:22.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.221+0000 7f20ca7fc640 1 -- 172.21.15.150:0/787663156 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20bc031be0 con 0x7f20cc099e20 2024-03-17T23:47:22.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.222+0000 7f20d1b22640 1 -- 172.21.15.150:0/787663156 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f20cc099e20 msgr2=0x7f20cc09a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:22.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.222+0000 7f20d1b22640 1 --2- 172.21.15.150:0/787663156 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f20cc099e20 0x7f20cc09a200 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f20bc0097a0 tx=0x7f20bc02b8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.222+0000 7f20d1b22640 1 -- 172.21.15.150:0/787663156 shutdown_connections 2024-03-17T23:47:22.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.222+0000 7f20d1b22640 1 --2- 172.21.15.150:0/787663156 >> v2:172.21.15.157:3300/0 conn(0x7f20cc15e460 0x7f20cc06c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.222+0000 7f20d1b22640 1 --2- 172.21.15.150:0/787663156 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f20cc099e20 0x7f20cc09a200 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.222+0000 7f20d1b22640 1 -- 172.21.15.150:0/787663156 >> 172.21.15.150:0/787663156 conn(0x7f20cc0b3630 msgr2=0x7f20cc0b3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:22.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.222+0000 7f20d1b22640 1 -- 172.21.15.150:0/787663156 shutdown_connections 2024-03-17T23:47:22.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.222+0000 7f20d1b22640 1 -- 172.21.15.150:0/787663156 wait complete. 2024-03-17T23:47:22.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.223+0000 7f20d1b22640 1 Processor -- start 2024-03-17T23:47:22.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.223+0000 7f20d1b22640 1 -- start start 2024-03-17T23:47:22.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.224+0000 7f20d1b22640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20cc099e20 0x7f20cc14eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:22.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.224+0000 7f20d1b22640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f20cc15e460 0x7f20cc0c5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:22.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.224+0000 7f20d1b22640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f20cc0b5590 0x7f20cc0c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:22.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.224+0000 7f20d1b22640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f20cc06d570 con 0x7f20cc099e20 2024-03-17T23:47:22.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.224+0000 7f20d1b22640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f20cc0cfcd0 con 0x7f20cc0b5590 2024-03-17T23:47:22.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.224+0000 7f20d1b22640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f20cc0d0910 con 0x7f20cc15e460 2024-03-17T23:47:22.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.224+0000 7f20cb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20cc099e20 0x7f20cc14eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:22.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.224+0000 7f20caffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f20cc15e460 0x7f20cc0c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:22.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.224+0000 7f20cb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20cc099e20 0x7f20cc14eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40560/0 (socket says 172.21.15.150:40560) 2024-03-17T23:47:22.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.224+0000 7f20cb7fe640 1 -- 172.21.15.150:0/2656232816 learned_addr learned my addr 172.21.15.150:0/2656232816 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:22.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.224+0000 7f20cbfff640 1 --2- 172.21.15.150:0/2656232816 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f20cc0b5590 0x7f20cc0c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:22.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.224+0000 7f20caffd640 1 -- 172.21.15.150:0/2656232816 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f20cc0b5590 msgr2=0x7f20cc0c95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:22.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.224+0000 7f20caffd640 1 --2- 172.21.15.150:0/2656232816 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f20cc0b5590 0x7f20cc0c95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.224+0000 7f20caffd640 1 -- 172.21.15.150:0/2656232816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20cc099e20 msgr2=0x7f20cc14eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:22.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.224+0000 7f20caffd640 1 --2- 172.21.15.150:0/2656232816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20cc099e20 0x7f20cc14eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.224+0000 7f20caffd640 1 -- 172.21.15.150:0/2656232816 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20cc1f3f90 con 0x7f20cc15e460 2024-03-17T23:47:22.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.224+0000 7f20cb7fe640 1 --2- 172.21.15.150:0/2656232816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20cc099e20 0x7f20cc14eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:22.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.224+0000 7f20caffd640 1 --2- 172.21.15.150:0/2656232816 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f20cc15e460 0x7f20cc0c5670 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f20c000e840 tx=0x7f20c000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:22.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.225+0000 7f20b3fff640 1 -- 172.21.15.150:0/2656232816 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c0017070 con 0x7f20cc15e460 2024-03-17T23:47:22.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.225+0000 7f20b3fff640 1 -- 172.21.15.150:0/2656232816 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20c000c940 con 0x7f20cc15e460 2024-03-17T23:47:22.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.225+0000 7f20d1b22640 1 -- 172.21.15.150:0/2656232816 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20cc0b5ae0 con 0x7f20cc15e460 2024-03-17T23:47:22.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.225+0000 7f20b3fff640 1 -- 172.21.15.150:0/2656232816 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c000cd10 con 0x7f20cc15e460 2024-03-17T23:47:22.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.225+0000 7f20d1b22640 1 -- 172.21.15.150:0/2656232816 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f20cc0b60a0 con 0x7f20cc15e460 2024-03-17T23:47:22.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.226+0000 7f20d1b22640 1 -- 172.21.15.150:0/2656232816 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20cc0681a0 con 0x7f20cc15e460 2024-03-17T23:47:22.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.226+0000 7f20b3fff640 1 -- 172.21.15.150:0/2656232816 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f20c001a070 con 0x7f20cc15e460 2024-03-17T23:47:22.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.226+0000 7f20b3fff640 1 --2- 172.21.15.150:0/2656232816 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f20b806bc00 0x7f20b806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:22.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.226+0000 7f20b3fff640 1 -- 172.21.15.150:0/2656232816 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f20c008aea0 con 0x7f20cc15e460 2024-03-17T23:47:22.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.226+0000 7f20cb7fe640 1 --2- 172.21.15.150:0/2656232816 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f20b806bc00 0x7f20b806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:22.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.227+0000 7f20cb7fe640 1 --2- 172.21.15.150:0/2656232816 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f20b806bc00 0x7f20b806e0c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f20bc02b910 tx=0x7f20bc008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:22.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.235+0000 7f20b3fff640 1 -- 172.21.15.150:0/2656232816 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f20cc1f3f90 con 0x7f20cc15e460 2024-03-17T23:47:22.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.444+0000 7f20d1b22640 1 -- 172.21.15.150:0/2656232816 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20cc06ce40 con 0x7f20cc15e460 2024-03-17T23:47:22.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.444+0000 7f20b3fff640 1 -- 172.21.15.150:0/2656232816 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5186 (secure 0 0 0) 0x7f20c0058f80 con 0x7f20cc15e460 2024-03-17T23:47:22.445 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:22.445 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/1949609985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6833","nonce":1949609985},{"type":"v1","addr":"172.21.15.157:6835","nonce":1949609985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/758264728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":758264728},{"type":"v1","addr":"172.21.15.150:6835","nonce":758264728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3655589010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3655589010},{"type":"v1","addr":"172.21.15.150:6837","nonce":3655589010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-17T23:46:55.253739+0000","modified":"2024-03-17T23:47:11.316249+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6834/1625058792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1625058792},{"type":"v1","addr":"172.21.15.157:6834","nonce":1625058792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4297,"qdb_cluster":[4297]},"id":1}]} 2024-03-17T23:47:22.445 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 8 2024-03-17T23:47:22.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.446+0000 7f20d1b22640 1 -- 172.21.15.150:0/2656232816 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f20b806bc00 msgr2=0x7f20b806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:22.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.446+0000 7f20d1b22640 1 --2- 172.21.15.150:0/2656232816 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f20b806bc00 0x7f20b806e0c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f20bc02b910 tx=0x7f20bc008030 comp rx=0 tx=0).stop 2024-03-17T23:47:22.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.446+0000 7f20d1b22640 1 -- 172.21.15.150:0/2656232816 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f20cc15e460 msgr2=0x7f20cc0c5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:22.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.446+0000 7f20d1b22640 1 --2- 172.21.15.150:0/2656232816 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f20cc15e460 0x7f20cc0c5670 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f20c000e840 tx=0x7f20c000ed00 comp rx=0 tx=0).stop 2024-03-17T23:47:22.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.446+0000 7f20d1b22640 1 -- 172.21.15.150:0/2656232816 shutdown_connections 2024-03-17T23:47:22.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.446+0000 7f20d1b22640 1 --2- 172.21.15.150:0/2656232816 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f20b806bc00 0x7f20b806e0c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.446+0000 7f20d1b22640 1 --2- 172.21.15.150:0/2656232816 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f20cc0b5590 0x7f20cc0c95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.446+0000 7f20d1b22640 1 --2- 172.21.15.150:0/2656232816 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f20cc15e460 0x7f20cc0c5670 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.446+0000 7f20d1b22640 1 --2- 172.21.15.150:0/2656232816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20cc099e20 0x7f20cc14eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.446+0000 7f20d1b22640 1 -- 172.21.15.150:0/2656232816 >> 172.21.15.150:0/2656232816 conn(0x7f20cc0b3630 msgr2=0x7f20cc0d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:22.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.446+0000 7f20d1b22640 1 -- 172.21.15.150:0/2656232816 shutdown_connections 2024-03-17T23:47:22.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.447+0000 7f20d1b22640 1 -- 172.21.15.150:0/2656232816 wait complete. 2024-03-17T23:47:22.861 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-17T23:46:55.253739+0000', 'modified': '2024-03-17T23:47:11.316249+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4297}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4297': {'gid': 4297, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.157:6834/1625058792', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6832', 'nonce': 1625058792}, {'type': 'v1', 'addr': '172.21.15.157:6834', 'nonce': 1625058792}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4297, 'qdb_cluster': [4297]} 2024-03-17T23:47:22.861 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-17T23:47:22.861 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-17T23:47:22.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.962+0000 7fe0bd110640 1 Processor -- start 2024-03-17T23:47:22.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.962+0000 7fe0bd110640 1 -- start start 2024-03-17T23:47:22.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.963+0000 7fe0bd110640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe0b815df40 0x7fe0b80b97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:22.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.963+0000 7fe0bd110640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe0b815e7d0 0x7fe0b80d2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:22.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.963+0000 7fe0bd110640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0b815ec90 con 0x7fe0b815e320 2024-03-17T23:47:22.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.963+0000 7fe0bd110640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fe0b80b6d40 con 0x7fe0b815df40 2024-03-17T23:47:22.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.963+0000 7fe0bd110640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fe0b815ef70 con 0x7fe0b815e7d0 2024-03-17T23:47:22.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.963+0000 7fe0b6d9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe0b815df40 0x7fe0b80b97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:22.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.963+0000 7fe0b659c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fe0b815e320 0x7fe0b8162160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34826/0 (socket says 172.21.15.150:34826) 2024-03-17T23:47:22.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.963+0000 7fe0b6d9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe0b815df40 0x7fe0b80b97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40562/0 (socket says 172.21.15.150:40562) 2024-03-17T23:47:22.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.963+0000 7fe0b659c640 1 -- 172.21.15.150:0/2865329274 learned_addr learned my addr 172.21.15.150:0/2865329274 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:22.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.963+0000 7fe0b759e640 1 --2- 172.21.15.150:0/2865329274 >> v2:172.21.15.157:3300/0 conn(0x7fe0b815e7d0 0x7fe0b80d2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:22.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.964+0000 7fe0b6d9d640 1 -- 172.21.15.150:0/2865329274 >> v2:172.21.15.157:3300/0 conn(0x7fe0b815e7d0 msgr2=0x7fe0b80d2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:22.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.964+0000 7fe0b6d9d640 1 --2- 172.21.15.150:0/2865329274 >> v2:172.21.15.157:3300/0 conn(0x7fe0b815e7d0 0x7fe0b80d2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.964+0000 7fe0b6d9d640 1 -- 172.21.15.150:0/2865329274 >> v1:172.21.15.150:6789/0 conn(0x7fe0b815e320 legacy=0x7fe0b8162160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:22.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.964+0000 7fe0b6d9d640 1 -- 172.21.15.150:0/2865329274 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0b815f250 con 0x7fe0b815df40 2024-03-17T23:47:22.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.964+0000 7fe0b5d9b640 1 -- 172.21.15.150:0/2865329274 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3625907303 0 0) 0x7fe0b815ec90 con 0x7fe0b815e320 2024-03-17T23:47:22.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.964+0000 7fe0b6d9d640 1 --2- 172.21.15.150:0/2865329274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0b815df40 0x7fe0b80b97f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe0b8073880 tx=0x7fe0a802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46b81172a223f1bc server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:22.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.964+0000 7fe0b5d9b640 1 -- 172.21.15.150:0/2865329274 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8039070 con 0x7fe0b815df40 2024-03-17T23:47:22.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.964+0000 7fe0b5d9b640 1 -- 172.21.15.150:0/2865329274 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0a802cd00 con 0x7fe0b815df40 2024-03-17T23:47:22.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.964+0000 7fe0b5d9b640 1 -- 172.21.15.150:0/2865329274 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8034660 con 0x7fe0b815df40 2024-03-17T23:47:22.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.965+0000 7fe0bd110640 1 -- 172.21.15.150:0/2865329274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0b815df40 msgr2=0x7fe0b80b97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:22.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.965+0000 7fe0bd110640 1 --2- 172.21.15.150:0/2865329274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0b815df40 0x7fe0b80b97f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe0b8073880 tx=0x7fe0a802bf50 comp rx=0 tx=0).stop 2024-03-17T23:47:22.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.965+0000 7fe0bd110640 1 -- 172.21.15.150:0/2865329274 shutdown_connections 2024-03-17T23:47:22.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.965+0000 7fe0bd110640 1 --2- 172.21.15.150:0/2865329274 >> v2:172.21.15.157:3300/0 conn(0x7fe0b815e7d0 0x7fe0b80d2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.965+0000 7fe0bd110640 1 --2- 172.21.15.150:0/2865329274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0b815df40 0x7fe0b80b97f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.965+0000 7fe0bd110640 1 -- 172.21.15.150:0/2865329274 >> 172.21.15.150:0/2865329274 conn(0x7fe0b8072ef0 msgr2=0x7fe0b80d3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:22.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.965+0000 7fe0bd110640 1 -- 172.21.15.150:0/2865329274 shutdown_connections 2024-03-17T23:47:22.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.965+0000 7fe0bd110640 1 -- 172.21.15.150:0/2865329274 wait complete. 2024-03-17T23:47:22.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.966+0000 7fe0bd110640 1 Processor -- start 2024-03-17T23:47:22.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.967+0000 7fe0bd110640 1 -- start start 2024-03-17T23:47:22.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.967+0000 7fe0bd110640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe0b815df40 0x7fe0b814b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:22.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.967+0000 7fe0bd110640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0b815e320 0x7fe0b80b5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:22.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.967+0000 7fe0bd110640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe0b815e7d0 0x7fe0b81f44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:22.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.967+0000 7fe0bd110640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b8162870 con 0x7fe0b815e320 2024-03-17T23:47:22.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.967+0000 7fe0bd110640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b80d2640 con 0x7fe0b815e7d0 2024-03-17T23:47:22.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.967+0000 7fe0bd110640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe0b80b3720 con 0x7fe0b815df40 2024-03-17T23:47:22.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.967+0000 7fe0b659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0b815e320 0x7fe0b80b5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:22.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.967+0000 7fe0b659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0b815e320 0x7fe0b80b5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40574/0 (socket says 172.21.15.150:40574) 2024-03-17T23:47:22.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.967+0000 7fe0b659c640 1 -- 172.21.15.150:0/495613610 learned_addr learned my addr 172.21.15.150:0/495613610 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:22.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.967+0000 7fe0b6d9d640 1 --2- 172.21.15.150:0/495613610 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe0b815df40 0x7fe0b814b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:22.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.967+0000 7fe0b759e640 1 --2- 172.21.15.150:0/495613610 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe0b815e7d0 0x7fe0b81f44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:22.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.968+0000 7fe0b659c640 1 -- 172.21.15.150:0/495613610 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe0b815df40 msgr2=0x7fe0b814b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:22.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.968+0000 7fe0b659c640 1 --2- 172.21.15.150:0/495613610 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe0b815df40 0x7fe0b814b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.968+0000 7fe0b659c640 1 -- 172.21.15.150:0/495613610 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe0b815e7d0 msgr2=0x7fe0b81f44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:22.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.968+0000 7fe0b659c640 1 --2- 172.21.15.150:0/495613610 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe0b815e7d0 0x7fe0b81f44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.968+0000 7fe0b659c640 1 -- 172.21.15.150:0/495613610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0b80b39c0 con 0x7fe0b815e320 2024-03-17T23:47:22.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.968+0000 7fe0b659c640 1 --2- 172.21.15.150:0/495613610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0b815e320 0x7fe0b80b5fc0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac00c4a0 tx=0x7fe0ac00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:22.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.968+0000 7fe09f7fe640 1 -- 172.21.15.150:0/495613610 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac018070 con 0x7fe0b815e320 2024-03-17T23:47:22.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.968+0000 7fe09f7fe640 1 -- 172.21.15.150:0/495613610 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0ac0074e0 con 0x7fe0b815e320 2024-03-17T23:47:22.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.968+0000 7fe0bd110640 1 -- 172.21.15.150:0/495613610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0b80b3ca0 con 0x7fe0b815e320 2024-03-17T23:47:22.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.968+0000 7fe09f7fe640 1 -- 172.21.15.150:0/495613610 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac004a20 con 0x7fe0b815e320 2024-03-17T23:47:22.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.968+0000 7fe0bd110640 1 -- 172.21.15.150:0/495613610 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0b80b4260 con 0x7fe0b815e320 2024-03-17T23:47:22.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.970+0000 7fe09f7fe640 1 -- 172.21.15.150:0/495613610 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe0ac01b070 con 0x7fe0b815e320 2024-03-17T23:47:22.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.970+0000 7fe09f7fe640 1 --2- 172.21.15.150:0/495613610 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe0a406bbd0 0x7fe0a406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:22.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.970+0000 7fe09f7fe640 1 -- 172.21.15.150:0/495613610 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fe0ac089f90 con 0x7fe0b815e320 2024-03-17T23:47:22.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.970+0000 7fe0b6d9d640 1 --2- 172.21.15.150:0/495613610 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe0a406bbd0 0x7fe0a406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:22.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.971+0000 7fe0b6d9d640 1 --2- 172.21.15.150:0/495613610 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe0a406bbd0 0x7fe0a406e090 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8036000 tx=0x7fe0a8030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:22.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.971+0000 7fe0bfddc740 1 Processor -- start 2024-03-17T23:47:22.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.971+0000 7fe0bfddc740 1 -- start start 2024-03-17T23:47:22.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.972+0000 7fe0bfddc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556965d1dc80 0x556965cf0c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:22.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.972+0000 7fe0bfddc740 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x556965d1e060 0x556965d20a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:22.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.972+0000 7fe0bfddc740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965cf1160 0x556965d20f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:22.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.972+0000 7fe0bfddc740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x556965c59c80 con 0x556965cf1160 2024-03-17T23:47:22.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.972+0000 7fe0bfddc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x556965c02aa0 con 0x556965d1dc80 2024-03-17T23:47:22.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.972+0000 7fe0bfddc740 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x556965c55d20 con 0x556965d1e060 2024-03-17T23:47:22.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.972+0000 7fe0b759e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965cf1160 0x556965d20f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:22.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.972+0000 7fe0b6d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556965d1dc80 0x556965cf0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:22.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.972+0000 7fe0b659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x556965d1e060 0x556965d20a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:22.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.972+0000 7fe0b659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x556965d1e060 0x556965d20a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:55648/0 (socket says 172.21.15.150:55648) 2024-03-17T23:47:22.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.972+0000 7fe0b759e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965cf1160 0x556965d20f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40588/0 (socket says 172.21.15.150:40588) 2024-03-17T23:47:22.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.972+0000 7fe0b659c640 1 -- 172.21.15.150:0/2844401479 learned_addr learned my addr 172.21.15.150:0/2844401479 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:22.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.972+0000 7fe0b6d9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556965d1dc80 0x556965cf0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:42494/0 (socket says 172.21.15.150:42494) 2024-03-17T23:47:22.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.972+0000 7fe0b6d9d640 1 -- 172.21.15.150:0/2844401479 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x556965d1e060 msgr2=0x556965d20a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:22.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.972+0000 7fe0b6d9d640 1 --2- 172.21.15.150:0/2844401479 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x556965d1e060 0x556965d20a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.972+0000 7fe0b6d9d640 1 -- 172.21.15.150:0/2844401479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965cf1160 msgr2=0x556965d20f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:22.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.972+0000 7fe0b6d9d640 1 --2- 172.21.15.150:0/2844401479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965cf1160 0x556965d20f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.972+0000 7fe0b6d9d640 1 -- 172.21.15.150:0/2844401479 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556965c24ff0 con 0x556965d1dc80 2024-03-17T23:47:22.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.972+0000 7fe0b659c640 1 --2- 172.21.15.150:0/2844401479 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x556965d1e060 0x556965d20a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:22.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.972+0000 7fe0b6d9d640 1 --2- 172.21.15.150:0/2844401479 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556965d1dc80 0x556965cf0c20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe0b80b68d0 tx=0x7fe0a802e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72007c42b3bbafe7 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:22.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.973+0000 7fe09cff9640 1 -- 172.21.15.150:0/2844401479 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8051070 con 0x556965d1dc80 2024-03-17T23:47:22.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.973+0000 7fe09cff9640 1 -- 172.21.15.150:0/2844401479 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0a803a040 con 0x556965d1dc80 2024-03-17T23:47:22.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.973+0000 7fe09cff9640 1 -- 172.21.15.150:0/2844401479 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8039040 con 0x556965d1dc80 2024-03-17T23:47:22.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.973+0000 7fe0bfddc740 1 -- 172.21.15.150:0/2844401479 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556965d1dc80 msgr2=0x556965cf0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:22.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.973+0000 7fe0bfddc740 1 --2- 172.21.15.150:0/2844401479 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556965d1dc80 0x556965cf0c20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe0b80b68d0 tx=0x7fe0a802e000 comp rx=0 tx=0).stop 2024-03-17T23:47:22.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.973+0000 7fe0bfddc740 1 -- 172.21.15.150:0/2844401479 shutdown_connections 2024-03-17T23:47:22.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.973+0000 7fe0bfddc740 1 --2- 172.21.15.150:0/2844401479 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965cf1160 0x556965d20f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.973+0000 7fe0bfddc740 1 --2- 172.21.15.150:0/2844401479 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x556965d1e060 0x556965d20a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.973+0000 7fe0bfddc740 1 --2- 172.21.15.150:0/2844401479 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556965d1dc80 0x556965cf0c20 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.973+0000 7fe0bfddc740 1 -- 172.21.15.150:0/2844401479 >> 172.21.15.150:0/2844401479 conn(0x556965cf2e90 msgr2=0x556965d3ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:22.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.973+0000 7fe0bfddc740 1 -- 172.21.15.150:0/2844401479 shutdown_connections 2024-03-17T23:47:22.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.973+0000 7fe0bfddc740 1 -- 172.21.15.150:0/2844401479 wait complete. 2024-03-17T23:47:22.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.974+0000 7fe0bfddc740 1 -- start start 2024-03-17T23:47:22.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.974+0000 7fe0bfddc740 1 Processor -- start 2024-03-17T23:47:22.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.975+0000 7fe0bfddc740 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x556965d1dc80 0x556965d1b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:22.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.975+0000 7fe0bfddc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556965d1e060 0x556965d25540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:22.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.975+0000 7fe0bfddc740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965d25a80 0x556965d27090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:22.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.975+0000 7fe0bfddc740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x556965cf5530 con 0x556965d25a80 2024-03-17T23:47:22.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.975+0000 7fe0bfddc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x556965c15ef0 con 0x556965d1e060 2024-03-17T23:47:22.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.975+0000 7fe0bfddc740 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x556965cf7d70 con 0x556965d1dc80 2024-03-17T23:47:22.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.975+0000 7fe0bfddc740 10 client.0 _ll_register_callbacks cb 0x556965c10dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:47:22.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.975+0000 7fe0b759e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965d25a80 0x556965d27090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:22.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.975+0000 7fe0b659c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556965d1e060 0x556965d25540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:22.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.975+0000 7fe0b6d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x556965d1dc80 0x556965d1b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:22.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.975+0000 7fe0b759e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965d25a80 0x556965d27090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40604/0 (socket says 172.21.15.150:40604) 2024-03-17T23:47:22.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.975+0000 7fe0b759e640 1 -- 172.21.15.150:0/887661352 learned_addr learned my addr 172.21.15.150:0/887661352 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:22.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.975+0000 7fe0b759e640 1 -- 172.21.15.150:0/887661352 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x556965d1dc80 msgr2=0x556965d1b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:22.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.975+0000 7fe0b759e640 1 --2- 172.21.15.150:0/887661352 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x556965d1dc80 0x556965d1b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.975+0000 7fe0b759e640 1 -- 172.21.15.150:0/887661352 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556965d1e060 msgr2=0x556965d25540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:22.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.975+0000 7fe0b759e640 1 --2- 172.21.15.150:0/887661352 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556965d1e060 0x556965d25540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.975+0000 7fe0b759e640 1 -- 172.21.15.150:0/887661352 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556965c22a30 con 0x556965d25a80 2024-03-17T23:47:22.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.975+0000 7fe0b659c640 1 --2- 172.21.15.150:0/887661352 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556965d1e060 0x556965d25540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:22.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.975+0000 7fe0b6d9d640 1 --2- 172.21.15.150:0/887661352 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x556965d1dc80 0x556965d1b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:22.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.975+0000 7fe0b759e640 1 --2- 172.21.15.150:0/887661352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965d25a80 0x556965d27090 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0014800 tx=0x7fe0b0014c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7997c669d0a393d server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:22.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.975+0000 7fe095ffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-17T23:47:22.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.975+0000 7fe095ffb640 1 -- 172.21.15.150:0/887661352 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b00129a0 con 0x556965d25a80 2024-03-17T23:47:22.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.975+0000 7fe095ffb640 1 -- 172.21.15.150:0/887661352 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fe0b0012b40 con 0x556965d25a80 2024-03-17T23:47:22.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.975+0000 7fe0bfddc740 1 -- 172.21.15.150:0/887661352 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0a803a040 con 0x556965d25a80 2024-03-17T23:47:22.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.976+0000 7fe095ffb640 1 -- 172.21.15.150:0/887661352 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7fe0b0027050 con 0x556965d25a80 2024-03-17T23:47:22.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.976+0000 7fe0bfddc740 10 client.4824 fetch_fsmap learned FSMap version 8 2024-03-17T23:47:22.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.976+0000 7fe0bfddc740 1 -- 172.21.15.150:0/887661352 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x55696596e2d0 con 0x556965d25a80 2024-03-17T23:47:22.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.976+0000 7fe095ffb640 1 -- 172.21.15.150:0/887661352 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0a803a040 con 0x556965d25a80 2024-03-17T23:47:22.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.976+0000 7fe095ffb640 1 -- 172.21.15.150:0/887661352 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7fe0b002a020 con 0x556965d25a80 2024-03-17T23:47:22.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.976+0000 7fe0bfddc740 10 client.4824 fetch_fsmap finished waiting for FSMap version 8 2024-03-17T23:47:22.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.976+0000 7fe0bfddc740 10 client.4824 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-17T23:47:22.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.976+0000 7fe0bfddc740 20 client.4824 populate_metadata read hostname 'smithi150' 2024-03-17T23:47:22.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.976+0000 7fe0bfddc740 1 --2- 172.21.15.150:0/887661352 >> [v2:172.21.15.157:6832/1625058792,v1:172.21.15.157:6834/1625058792] conn(0x556965d22a60 0x556965d4f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:22.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.976+0000 7fe0bfddc740 4 client.4824 mds_command: new command op to 4297 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-17T23:47:22.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.976+0000 7fe0bfddc740 1 -- 172.21.15.150:0/887661352 --> [v2:172.21.15.157:6832/1625058792,v1:172.21.15.157:6834/1625058792] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556965c671c0 con 0x556965d22a60 2024-03-17T23:47:22.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.976+0000 7fe0b6d9d640 1 --2- 172.21.15.150:0/887661352 >> [v2:172.21.15.157:6832/1625058792,v1:172.21.15.157:6834/1625058792] conn(0x556965d22a60 0x556965d4f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:22.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.976+0000 7fe0b6d9d640 1 --2- 172.21.15.150:0/887661352 >> [v2:172.21.15.157:6832/1625058792,v1:172.21.15.157:6834/1625058792] conn(0x556965d22a60 0x556965d4f100 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48c80df568f9fab8 server_cookie=d96ec6d389c72328 in_seq=0 out_seq=0 2024-03-17T23:47:22.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.976+0000 7fe095ffb640 10 client.4824 ms_handle_connect on v2:172.21.15.157:6832/1625058792 2024-03-17T23:47:22.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.977+0000 7fe095ffb640 1 -- 172.21.15.150:0/887661352 <== mds.0 v2:172.21.15.157:6832/1625058792 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556965c671c0 con 0x556965d22a60 2024-03-17T23:47:22.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.977+0000 7fe095ffb640 10 client.4824 handle_command_reply: tid=0 2024-03-17T23:47:22.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.977+0000 7fe0bfddc740 1 client.4824 shutdown 2024-03-17T23:47:22.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.977+0000 7fe0bfddc740 1 -- 172.21.15.150:0/887661352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965d25a80 msgr2=0x556965d27090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:22.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.977+0000 7fe0bfddc740 1 --2- 172.21.15.150:0/887661352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965d25a80 0x556965d27090 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0014800 tx=0x7fe0b0014c00 comp rx=0 tx=0).stop 2024-03-17T23:47:22.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.977+0000 7fe0bfddc740 1 -- 172.21.15.150:0/887661352 shutdown_connections 2024-03-17T23:47:22.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.977+0000 7fe0bfddc740 1 --2- 172.21.15.150:0/887661352 >> [v2:172.21.15.157:6832/1625058792,v1:172.21.15.157:6834/1625058792] conn(0x556965d22a60 0x556965d4f100 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.977+0000 7fe0bfddc740 1 --2- 172.21.15.150:0/887661352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965d25a80 0x556965d27090 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.977+0000 7fe0bfddc740 1 --2- 172.21.15.150:0/887661352 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556965d1e060 0x556965d25540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.977+0000 7fe095ffb640 0 client.4824 ms_handle_reset on v2:172.21.15.157:6832/1625058792 2024-03-17T23:47:22.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.977+0000 7fe0bfddc740 1 --2- 172.21.15.150:0/887661352 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x556965d1dc80 0x556965d1b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:22.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.977+0000 7fe0bfddc740 1 -- 172.21.15.150:0/887661352 >> 172.21.15.150:0/887661352 conn(0x556965cf1160 msgr2=0x556965cf4c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:22.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.977+0000 7fe0bfddc740 1 -- 172.21.15.150:0/887661352 shutdown_connections 2024-03-17T23:47:22.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.977+0000 7fe0bfddc740 1 -- 172.21.15.150:0/887661352 wait complete. 2024-03-17T23:47:22.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.977+0000 7fe0bfddc740 20 client.4824 trim_cache size 0 max 16384 2024-03-17T23:47:22.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.995+0000 7fe0bfddc740 1 Processor -- start 2024-03-17T23:47:22.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.995+0000 7fe0bfddc740 1 -- start start 2024-03-17T23:47:23.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.996+0000 7fe0bfddc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556965d1dc80 0x556965d27040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:23.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.996+0000 7fe0bfddc740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965d1e060 0x556965cf6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:23.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.996+0000 7fe0bfddc740 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x556965d25a80 0x556965d27aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:23.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.996+0000 7fe0bfddc740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b80c9dc0 con 0x556965d1e060 2024-03-17T23:47:23.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.996+0000 7fe0bfddc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b81f87a0 con 0x556965d1dc80 2024-03-17T23:47:23.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.996+0000 7fe0bfddc740 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe0b815c270 con 0x556965d25a80 2024-03-17T23:47:23.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.996+0000 7fe0b659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965d1e060 0x556965cf6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:23.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.996+0000 7fe0b659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965d1e060 0x556965cf6bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40612/0 (socket says 172.21.15.150:40612) 2024-03-17T23:47:23.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.996+0000 7fe0b659c640 1 -- 172.21.15.150:0/2570659559 learned_addr learned my addr 172.21.15.150:0/2570659559 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:23.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.996+0000 7fe0b759e640 1 --2- 172.21.15.150:0/2570659559 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x556965d25a80 0x556965d27aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:23.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.996+0000 7fe0b6d9d640 1 --2- 172.21.15.150:0/2570659559 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556965d1dc80 0x556965d27040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:23.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.996+0000 7fe0b659c640 1 -- 172.21.15.150:0/2570659559 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x556965d25a80 msgr2=0x556965d27aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:23.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.996+0000 7fe0b659c640 1 --2- 172.21.15.150:0/2570659559 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x556965d25a80 0x556965d27aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:23.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.996+0000 7fe0b659c640 1 -- 172.21.15.150:0/2570659559 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556965d1dc80 msgr2=0x556965d27040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:23.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.996+0000 7fe0b659c640 1 --2- 172.21.15.150:0/2570659559 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556965d1dc80 0x556965d27040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:23.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.996+0000 7fe0b659c640 1 -- 172.21.15.150:0/2570659559 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556965a386b0 con 0x556965d1e060 2024-03-17T23:47:23.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.997+0000 7fe0b659c640 1 --2- 172.21.15.150:0/2570659559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965d1e060 0x556965cf6bb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac0602b0 tx=0x7fe0ac009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72ae779d6da5e9e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:23.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.997+0000 7fe097fff640 1 -- 172.21.15.150:0/2570659559 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac004bc0 con 0x556965d1e060 2024-03-17T23:47:23.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.997+0000 7fe097fff640 1 -- 172.21.15.150:0/2570659559 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0ac004d60 con 0x556965d1e060 2024-03-17T23:47:23.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.997+0000 7fe097fff640 1 -- 172.21.15.150:0/2570659559 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac081d70 con 0x556965d1e060 2024-03-17T23:47:23.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.997+0000 7fe0bfddc740 1 -- 172.21.15.150:0/2570659559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965d1e060 msgr2=0x556965cf6bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:23.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.997+0000 7fe0bfddc740 1 --2- 172.21.15.150:0/2570659559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965d1e060 0x556965cf6bb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac0602b0 tx=0x7fe0ac009040 comp rx=0 tx=0).stop 2024-03-17T23:47:23.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.997+0000 7fe0bfddc740 1 -- 172.21.15.150:0/2570659559 shutdown_connections 2024-03-17T23:47:23.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.997+0000 7fe0bfddc740 1 --2- 172.21.15.150:0/2570659559 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x556965d25a80 0x556965d27aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:23.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.997+0000 7fe0bfddc740 1 --2- 172.21.15.150:0/2570659559 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965d1e060 0x556965cf6bb0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:23.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.997+0000 7fe0bfddc740 1 --2- 172.21.15.150:0/2570659559 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556965d1dc80 0x556965d27040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:23.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.997+0000 7fe0bfddc740 1 -- 172.21.15.150:0/2570659559 >> 172.21.15.150:0/2570659559 conn(0x556965cf1160 msgr2=0x556965cec210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:23.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.997+0000 7fe0bfddc740 1 -- 172.21.15.150:0/2570659559 shutdown_connections 2024-03-17T23:47:23.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.997+0000 7fe0bfddc740 1 -- 172.21.15.150:0/2570659559 wait complete. 2024-03-17T23:47:23.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.998+0000 7fe0bfddc740 1 -- start start 2024-03-17T23:47:23.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.998+0000 7fe0bfddc740 1 Processor -- start 2024-03-17T23:47:23.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.999+0000 7fe0bfddc740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965d1dc80 0x556965d3d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:23.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.999+0000 7fe0bfddc740 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x556965d1e060 0x556965ceff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:23.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.999+0000 7fe0bfddc740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556965d25a80 0x556965d20450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:23.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.999+0000 7fe0bfddc740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x556965d228e0 con 0x556965d1dc80 2024-03-17T23:47:23.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.999+0000 7fe0bfddc740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x556965d223c0 con 0x556965d25a80 2024-03-17T23:47:23.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.999+0000 7fe0bfddc740 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x556965d22f30 con 0x556965d1e060 2024-03-17T23:47:23.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.999+0000 7fe0bfddc740 10 client.0 _ll_register_callbacks cb 0x7fe0b00097e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:47:23.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.999+0000 7fe0b6d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965d1dc80 0x556965d3d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:23.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.999+0000 7fe0b6d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965d1dc80 0x556965d3d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40616/0 (socket says 172.21.15.150:40616) 2024-03-17T23:47:23.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.999+0000 7fe0b6d9d640 1 -- 172.21.15.150:0/3964780025 learned_addr learned my addr 172.21.15.150:0/3964780025 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:23.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.999+0000 7fe0b759e640 1 --2- 172.21.15.150:0/3964780025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556965d25a80 0x556965d20450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:23.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.999+0000 7fe0b659c640 1 --2- 172.21.15.150:0/3964780025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x556965d1e060 0x556965ceff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:23.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.999+0000 7fe0b6d9d640 1 -- 172.21.15.150:0/3964780025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x556965d1e060 msgr2=0x556965ceff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:23.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.999+0000 7fe0b6d9d640 1 --2- 172.21.15.150:0/3964780025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x556965d1e060 0x556965ceff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:23.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.999+0000 7fe0b6d9d640 1 -- 172.21.15.150:0/3964780025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556965d25a80 msgr2=0x556965d20450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:23.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.999+0000 7fe0b6d9d640 1 --2- 172.21.15.150:0/3964780025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556965d25a80 0x556965d20450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:23.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.999+0000 7fe0b6d9d640 1 -- 172.21.15.150:0/3964780025 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556965d3dcd0 con 0x556965d1dc80 2024-03-17T23:47:23.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.999+0000 7fe0b759e640 1 --2- 172.21.15.150:0/3964780025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556965d25a80 0x556965d20450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:23.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.999+0000 7fe0b6d9d640 1 --2- 172.21.15.150:0/3964780025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965d1dc80 0x556965d3d790 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8000f80 tx=0x7fe0a8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f4b485011c5811e server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:23.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.999+0000 7fe095ffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-17T23:47:23.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.999+0000 7fe095ffb640 1 -- 172.21.15.150:0/3964780025 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8060070 con 0x556965d1dc80 2024-03-17T23:47:23.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.999+0000 7fe095ffb640 1 -- 172.21.15.150:0/3964780025 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fe0a805a2f0 con 0x556965d1dc80 2024-03-17T23:47:23.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:22.999+0000 7fe0bfddc740 1 -- 172.21.15.150:0/3964780025 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0ac004d60 con 0x556965d1dc80 2024-03-17T23:47:23.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.000+0000 7fe095ffb640 1 -- 172.21.15.150:0/3964780025 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7fe0a8068050 con 0x556965d1dc80 2024-03-17T23:47:23.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.000+0000 7fe0bfddc740 10 client.4830 fetch_fsmap learned FSMap version 8 2024-03-17T23:47:23.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.000+0000 7fe0bfddc740 1 -- 172.21.15.150:0/3964780025 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x556965d1d480 con 0x556965d1dc80 2024-03-17T23:47:23.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.000+0000 7fe095ffb640 1 -- 172.21.15.150:0/3964780025 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0ac004d60 con 0x556965d1dc80 2024-03-17T23:47:23.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.000+0000 7fe095ffb640 1 -- 172.21.15.150:0/3964780025 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7fe0a806b020 con 0x556965d1dc80 2024-03-17T23:47:23.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.000+0000 7fe0bfddc740 10 client.4830 fetch_fsmap finished waiting for FSMap version 8 2024-03-17T23:47:23.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.000+0000 7fe0bfddc740 10 client.4830 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-17T23:47:23.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.000+0000 7fe0bfddc740 20 client.4830 populate_metadata read hostname 'smithi150' 2024-03-17T23:47:23.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.000+0000 7fe0bfddc740 1 --2- 172.21.15.150:0/3964780025 >> [v2:172.21.15.157:6832/1625058792,v1:172.21.15.157:6834/1625058792] conn(0x556965d3b2d0 0x556965d3b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:23.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.000+0000 7fe0bfddc740 4 client.4830 mds_command: new command op to 4297 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-17T23:47:23.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.000+0000 7fe0bfddc740 1 -- 172.21.15.150:0/3964780025 --> [v2:172.21.15.157:6832/1625058792,v1:172.21.15.157:6834/1625058792] -- command(tid 0: {"prefix": "status"}) -- 0x556965c61510 con 0x556965d3b2d0 2024-03-17T23:47:23.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.000+0000 7fe0b659c640 1 --2- 172.21.15.150:0/3964780025 >> [v2:172.21.15.157:6832/1625058792,v1:172.21.15.157:6834/1625058792] conn(0x556965d3b2d0 0x556965d3b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:23.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.000+0000 7fe095ffb640 10 client.4830 ms_handle_connect on v2:172.21.15.157:6832/1625058792 2024-03-17T23:47:23.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.000+0000 7fe0b659c640 1 --2- 172.21.15.150:0/3964780025 >> [v2:172.21.15.157:6832/1625058792,v1:172.21.15.157:6834/1625058792] conn(0x556965d3b2d0 0x556965d3b6b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5108035a8609a160 server_cookie=f58e9a9e745b9b55 in_seq=0 out_seq=0 2024-03-17T23:47:23.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe095ffb640 1 -- 172.21.15.150:0/3964780025 <== mds.0 v2:172.21.15.157:6832/1625058792 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x556965c61510 con 0x556965d3b2d0 2024-03-17T23:47:23.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe095ffb640 10 client.4830 handle_command_reply: tid=0 2024-03-17T23:47:23.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe0bfddc740 1 client.4830 shutdown 2024-03-17T23:47:23.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe0bfddc740 1 -- 172.21.15.150:0/3964780025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965d1dc80 msgr2=0x556965d3d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:23.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe0bfddc740 1 --2- 172.21.15.150:0/3964780025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965d1dc80 0x556965d3d790 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8000f80 tx=0x7fe0a8004ab0 comp rx=0 tx=0).stop 2024-03-17T23:47:23.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe0bfddc740 1 -- 172.21.15.150:0/3964780025 shutdown_connections 2024-03-17T23:47:23.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe0bfddc740 1 --2- 172.21.15.150:0/3964780025 >> [v2:172.21.15.157:6832/1625058792,v1:172.21.15.157:6834/1625058792] conn(0x556965d3b2d0 0x556965d3b6b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:23.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe0bfddc740 1 --2- 172.21.15.150:0/3964780025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556965d25a80 0x556965d20450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:23.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe095ffb640 0 client.4830 ms_handle_reset on v2:172.21.15.157:6832/1625058792 2024-03-17T23:47:23.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe0bfddc740 1 --2- 172.21.15.150:0/3964780025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x556965d1e060 0x556965ceff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:23.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe0bfddc740 1 --2- 172.21.15.150:0/3964780025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x556965d1dc80 0x556965d3d790 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:23.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe0bfddc740 1 -- 172.21.15.150:0/3964780025 >> 172.21.15.150:0/3964780025 conn(0x556965cf1160 msgr2=0x556965d5f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:23.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe0bfddc740 1 -- 172.21.15.150:0/3964780025 shutdown_connections 2024-03-17T23:47:23.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe0bfddc740 1 -- 172.21.15.150:0/3964780025 wait complete. 2024-03-17T23:47:23.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe0bfddc740 20 client.4830 trim_cache size 0 max 16384 2024-03-17T23:47:23.013 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-17T23:47:23.014 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_fsid": "a1c5acd3-e97f-470b-8226-335736a7795b", 2024-03-17T23:47:23.014 INFO:teuthology.orchestra.run.smithi150.stdout: "whoami": 0, 2024-03-17T23:47:23.014 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 4297, 2024-03-17T23:47:23.014 INFO:teuthology.orchestra.run.smithi150.stdout: "want_state": "up:active", 2024-03-17T23:47:23.014 INFO:teuthology.orchestra.run.smithi150.stdout: "state": "up:active", 2024-03-17T23:47:23.014 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "cephfs", 2024-03-17T23:47:23.014 INFO:teuthology.orchestra.run.smithi150.stdout: "rank_uptime": 27.078615767999999, 2024-03-17T23:47:23.014 INFO:teuthology.orchestra.run.smithi150.stdout: "mdsmap_epoch": 8, 2024-03-17T23:47:23.014 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch": 26, 2024-03-17T23:47:23.014 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch_barrier": 26, 2024-03-17T23:47:23.014 INFO:teuthology.orchestra.run.smithi150.stdout: "uptime": 33.808541193000003 2024-03-17T23:47:23.015 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-17T23:47:23.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe0bd110640 1 -- 172.21.15.150:0/495613610 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe0a406bbd0 msgr2=0x7fe0a406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:23.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe0bd110640 1 --2- 172.21.15.150:0/495613610 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe0a406bbd0 0x7fe0a406e090 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8036000 tx=0x7fe0a8030000 comp rx=0 tx=0).stop 2024-03-17T23:47:23.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe0bd110640 1 -- 172.21.15.150:0/495613610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0b815e320 msgr2=0x7fe0b80b5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:23.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe0bd110640 1 --2- 172.21.15.150:0/495613610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0b815e320 0x7fe0b80b5fc0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac00c4a0 tx=0x7fe0ac00c960 comp rx=0 tx=0).stop 2024-03-17T23:47:23.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe0bd110640 1 -- 172.21.15.150:0/495613610 shutdown_connections 2024-03-17T23:47:23.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe0bd110640 1 --2- 172.21.15.150:0/495613610 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe0a406bbd0 0x7fe0a406e090 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:23.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe0bd110640 1 --2- 172.21.15.150:0/495613610 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe0b815e7d0 0x7fe0b81f44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:23.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe0bd110640 1 --2- 172.21.15.150:0/495613610 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0b815e320 0x7fe0b80b5fc0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:23.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe0bd110640 1 --2- 172.21.15.150:0/495613610 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe0b815df40 0x7fe0b814b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:23.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe0bd110640 1 -- 172.21.15.150:0/495613610 >> 172.21.15.150:0/495613610 conn(0x7fe0b8072ef0 msgr2=0x7fe0b80b6ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:23.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe0bd110640 1 -- 172.21.15.150:0/495613610 shutdown_connections 2024-03-17T23:47:23.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.001+0000 7fe0bd110640 1 -- 172.21.15.150:0/495613610 wait complete. 2024-03-17T23:47:23.018 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-17T23:47:23.032 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-17T23:47:23.032 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-17T23:47:23.032 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-17T23:47:23.032 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-17T23:47:23.032 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-17T23:47:23.032 INFO:tasks.ceph_fuse:client.2 config is {} 2024-03-17T23:47:23.033 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-17T23:47:23.033 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-17T23:47:23.033 INFO:tasks.ceph_fuse:client.1 config is {} 2024-03-17T23:47:23.033 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-17T23:47:23.033 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-17T23:47:23.033 INFO:tasks.ceph_fuse:client.3 config is {} 2024-03-17T23:47:23.033 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-17T23:47:23.033 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-17T23:47:23.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:23.033 DEBUG:teuthology.orchestra.run.smithi157:> ip netns list 2024-03-17T23:47:23.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:23.049 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip link delete ceph-brx 2024-03-17T23:47:23.118 INFO:teuthology.orchestra.run.smithi157.stderr:Cannot find device "ceph-brx" 2024-03-17T23:47:23.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-17T23:47:23.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:23.120 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-03-17T23:47:23.135 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-17T23:47:23.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:23.136 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-17T23:47:23.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:23.204 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link delete ceph-brx 2024-03-17T23:47:23.296 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-17T23:47:23.297 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-17T23:47:23.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.398+0000 7f6e8a587640 1 Processor -- start 2024-03-17T23:47:23.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.398+0000 7f6e8a587640 1 -- start start 2024-03-17T23:47:23.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.399+0000 7f6e8a587640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6e84144bb0 0x7f6e8415ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:23.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.399+0000 7f6e8a587640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6e840ce6a0 0x7f6e84165a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:23.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.399+0000 7f6e8a587640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e84145170 con 0x7f6e8415d390 2024-03-17T23:47:23.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.399+0000 7f6e8a587640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f6e84145450 con 0x7f6e84144bb0 2024-03-17T23:47:23.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.399+0000 7f6e8a587640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6e84145730 con 0x7f6e840ce6a0 2024-03-17T23:47:23.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.399+0000 7f6e837fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f6e8415d390 0x7f6e840cdf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34842/0 (socket says 172.21.15.150:34842) 2024-03-17T23:47:23.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.399+0000 7f6e837fe640 1 -- 172.21.15.150:0/2248653729 learned_addr learned my addr 172.21.15.150:0/2248653729 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:23.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.399+0000 7f6e83fff640 1 --2- 172.21.15.150:0/2248653729 >> v2:172.21.15.157:3300/0 conn(0x7f6e84144bb0 0x7f6e8415ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:23.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.399+0000 7f6e88b24640 1 --2- 172.21.15.150:0/2248653729 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6e840ce6a0 0x7f6e84165a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:23.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.399+0000 7f6e82ffd640 1 -- 172.21.15.150:0/2248653729 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 283060778 0 0) 0x7f6e84145170 con 0x7f6e8415d390 2024-03-17T23:47:23.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.399+0000 7f6e82ffd640 1 -- 172.21.15.150:0/2248653729 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e6c003520 con 0x7f6e8415d390 2024-03-17T23:47:23.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.399+0000 7f6e83fff640 1 -- 172.21.15.150:0/2248653729 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6e840ce6a0 msgr2=0x7f6e84165a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:23.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.399+0000 7f6e83fff640 1 --2- 172.21.15.150:0/2248653729 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6e840ce6a0 0x7f6e84165a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:23.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.399+0000 7f6e83fff640 1 -- 172.21.15.150:0/2248653729 >> v1:172.21.15.150:6789/0 conn(0x7f6e8415d390 legacy=0x7f6e840cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:23.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.399+0000 7f6e83fff640 1 -- 172.21.15.150:0/2248653729 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e8406bd60 con 0x7f6e84144bb0 2024-03-17T23:47:23.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.399+0000 7f6e88b24640 1 --2- 172.21.15.150:0/2248653729 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6e840ce6a0 0x7f6e84165a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:23.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.400+0000 7f6e83fff640 1 --2- 172.21.15.150:0/2248653729 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6e84144bb0 0x7f6e8415ce50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6e740097a0 tx=0x7f6e7402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=674a7e30fa1d3f5c server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:23.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.400+0000 7f6e82ffd640 1 -- 172.21.15.150:0/2248653729 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e740315a0 con 0x7f6e84144bb0 2024-03-17T23:47:23.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.400+0000 7f6e82ffd640 1 -- 172.21.15.150:0/2248653729 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e74037070 con 0x7f6e84144bb0 2024-03-17T23:47:23.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.400+0000 7f6e82ffd640 1 -- 172.21.15.150:0/2248653729 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e74031940 con 0x7f6e84144bb0 2024-03-17T23:47:23.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.401+0000 7f6e8a587640 1 -- 172.21.15.150:0/2248653729 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6e84144bb0 msgr2=0x7f6e8415ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:23.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.401+0000 7f6e8a587640 1 --2- 172.21.15.150:0/2248653729 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6e84144bb0 0x7f6e8415ce50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6e740097a0 tx=0x7f6e7402b840 comp rx=0 tx=0).stop 2024-03-17T23:47:23.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.401+0000 7f6e8a587640 1 -- 172.21.15.150:0/2248653729 shutdown_connections 2024-03-17T23:47:23.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.401+0000 7f6e8a587640 1 --2- 172.21.15.150:0/2248653729 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6e840ce6a0 0x7f6e84165a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:23.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.401+0000 7f6e8a587640 1 --2- 172.21.15.150:0/2248653729 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6e84144bb0 0x7f6e8415ce50 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:23.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.401+0000 7f6e8a587640 1 -- 172.21.15.150:0/2248653729 >> 172.21.15.150:0/2248653729 conn(0x7f6e8406b190 msgr2=0x7f6e8406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:23.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.401+0000 7f6e8a587640 1 -- 172.21.15.150:0/2248653729 shutdown_connections 2024-03-17T23:47:23.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.401+0000 7f6e8a587640 1 -- 172.21.15.150:0/2248653729 wait complete. 2024-03-17T23:47:23.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.402+0000 7f6e8a587640 1 Processor -- start 2024-03-17T23:47:23.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.402+0000 7f6e8a587640 1 -- start start 2024-03-17T23:47:23.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.403+0000 7f6e8a587640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6e840ce6a0 0x7f6e8415aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:23.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.403+0000 7f6e8a587640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6e84144bb0 0x7f6e841591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:23.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.403+0000 7f6e8a587640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e8415d390 0x7f6e841596f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:23.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.403+0000 7f6e8a587640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6e84165f70 con 0x7f6e8415d390 2024-03-17T23:47:23.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.403+0000 7f6e8a587640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8406f610 con 0x7f6e84144bb0 2024-03-17T23:47:23.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.403+0000 7f6e8a587640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6e841660f0 con 0x7f6e840ce6a0 2024-03-17T23:47:23.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.403+0000 7f6e88b24640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e8415d390 0x7f6e841596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:23.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.403+0000 7f6e83fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6e840ce6a0 0x7f6e8415aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:23.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.403+0000 7f6e837fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6e84144bb0 0x7f6e841591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:23.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.403+0000 7f6e83fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6e840ce6a0 0x7f6e8415aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:55704/0 (socket says 172.21.15.150:55704) 2024-03-17T23:47:23.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.403+0000 7f6e83fff640 1 -- 172.21.15.150:0/250977871 learned_addr learned my addr 172.21.15.150:0/250977871 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:23.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.403+0000 7f6e837fe640 1 -- 172.21.15.150:0/250977871 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6e840ce6a0 msgr2=0x7f6e8415aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:23.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.403+0000 7f6e837fe640 1 --2- 172.21.15.150:0/250977871 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6e840ce6a0 0x7f6e8415aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:23.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.403+0000 7f6e837fe640 1 -- 172.21.15.150:0/250977871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e8415d390 msgr2=0x7f6e841596f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:23.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.403+0000 7f6e837fe640 1 --2- 172.21.15.150:0/250977871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e8415d390 0x7f6e841596f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:23.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.403+0000 7f6e837fe640 1 -- 172.21.15.150:0/250977871 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e840aeca0 con 0x7f6e84144bb0 2024-03-17T23:47:23.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.403+0000 7f6e83fff640 1 --2- 172.21.15.150:0/250977871 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6e840ce6a0 0x7f6e8415aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:23.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.403+0000 7f6e837fe640 1 --2- 172.21.15.150:0/250977871 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6e84144bb0 0x7f6e841591b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6e7800c430 tx=0x7f6e7800c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:23.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.403+0000 7f6e80ff9640 1 -- 172.21.15.150:0/250977871 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e78017070 con 0x7f6e84144bb0 2024-03-17T23:47:23.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.404+0000 7f6e80ff9640 1 -- 172.21.15.150:0/250977871 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e78005ce0 con 0x7f6e84144bb0 2024-03-17T23:47:23.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.404+0000 7f6e8a587640 1 -- 172.21.15.150:0/250977871 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e840aef80 con 0x7f6e84144bb0 2024-03-17T23:47:23.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.404+0000 7f6e80ff9640 1 -- 172.21.15.150:0/250977871 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e780072d0 con 0x7f6e84144bb0 2024-03-17T23:47:23.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.404+0000 7f6e8a587640 1 -- 172.21.15.150:0/250977871 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e840a9f20 con 0x7f6e84144bb0 2024-03-17T23:47:23.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.404+0000 7f6e8a587640 1 -- 172.21.15.150:0/250977871 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e840681a0 con 0x7f6e84144bb0 2024-03-17T23:47:23.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.405+0000 7f6e80ff9640 1 -- 172.21.15.150:0/250977871 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6e7801a070 con 0x7f6e84144bb0 2024-03-17T23:47:23.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.405+0000 7f6e80ff9640 1 --2- 172.21.15.150:0/250977871 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6e7006bbd0 0x7f6e7006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:23.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.405+0000 7f6e80ff9640 1 -- 172.21.15.150:0/250977871 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f6e7808a120 con 0x7f6e84144bb0 2024-03-17T23:47:23.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.405+0000 7f6e83fff640 1 --2- 172.21.15.150:0/250977871 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6e7006bbd0 0x7f6e7006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:23.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.406+0000 7f6e83fff640 1 --2- 172.21.15.150:0/250977871 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6e7006bbd0 0x7f6e7006e090 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6e74000c00 tx=0x7f6e74008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:23.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.414+0000 7f6e80ff9640 1 -- 172.21.15.150:0/250977871 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6e78058160 con 0x7f6e84144bb0 2024-03-17T23:47:23.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.623+0000 7f6e8a587640 1 -- 172.21.15.150:0/250977871 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f6e8413db50 con 0x7f6e84144bb0 2024-03-17T23:47:23.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.623+0000 7f6e80ff9640 1 -- 172.21.15.150:0/250977871 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7f6e7805c200 con 0x7f6e84144bb0 2024-03-17T23:47:23.624 INFO:teuthology.orchestra.run.smithi150.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-17T23:47:23.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.625+0000 7f6e8a587640 1 -- 172.21.15.150:0/250977871 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6e7006bbd0 msgr2=0x7f6e7006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:23.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.625+0000 7f6e8a587640 1 --2- 172.21.15.150:0/250977871 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6e7006bbd0 0x7f6e7006e090 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6e74000c00 tx=0x7f6e74008030 comp rx=0 tx=0).stop 2024-03-17T23:47:23.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.625+0000 7f6e8a587640 1 -- 172.21.15.150:0/250977871 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6e84144bb0 msgr2=0x7f6e841591b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:23.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.625+0000 7f6e8a587640 1 --2- 172.21.15.150:0/250977871 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6e84144bb0 0x7f6e841591b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6e7800c430 tx=0x7f6e7800c830 comp rx=0 tx=0).stop 2024-03-17T23:47:23.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.625+0000 7f6e8a587640 1 -- 172.21.15.150:0/250977871 shutdown_connections 2024-03-17T23:47:23.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.625+0000 7f6e8a587640 1 --2- 172.21.15.150:0/250977871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e8415d390 0x7f6e841596f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:23.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.625+0000 7f6e8a587640 1 --2- 172.21.15.150:0/250977871 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6e7006bbd0 0x7f6e7006e090 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:23.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.625+0000 7f6e8a587640 1 --2- 172.21.15.150:0/250977871 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6e84144bb0 0x7f6e841591b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:23.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.625+0000 7f6e8a587640 1 --2- 172.21.15.150:0/250977871 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6e840ce6a0 0x7f6e8415aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:23.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.625+0000 7f6e8a587640 1 -- 172.21.15.150:0/250977871 >> 172.21.15.150:0/250977871 conn(0x7f6e8406b190 msgr2=0x7f6e84143dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:23.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.625+0000 7f6e8a587640 1 -- 172.21.15.150:0/250977871 shutdown_connections 2024-03-17T23:47:23.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23.625+0000 7f6e8a587640 1 -- 172.21.15.150:0/250977871 wait complete. 2024-03-17T23:47:23.636 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-17T23:47:23.636 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-17T23:47:23.636 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi150.front.sepia.ceph.com 2024-03-17T23:47:23.637 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-17T23:47:23.637 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-17T23:47:23.637 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-17T23:47:23.637 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-17T23:47:23.637 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-17T23:47:23.637 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-17T23:47:23.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:23.637 DEBUG:teuthology.orchestra.run.smithi150:> ip addr 2024-03-17T23:47:23.653 INFO:teuthology.orchestra.run.smithi150.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-17T23:47:23.654 INFO:teuthology.orchestra.run.smithi150.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-17T23:47:23.654 INFO:teuthology.orchestra.run.smithi150.stdout: inet 127.0.0.1/8 scope host lo 2024-03-17T23:47:23.654 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:23.654 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 ::1/128 scope host 2024-03-17T23:47:23.655 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:23.655 INFO:teuthology.orchestra.run.smithi150.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:47:23.655 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bc brd ff:ff:ff:ff:ff:ff 2024-03-17T23:47:23.655 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f0 2024-03-17T23:47:23.655 INFO:teuthology.orchestra.run.smithi150.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:47:23.655 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bd brd ff:ff:ff:ff:ff:ff 2024-03-17T23:47:23.655 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f1 2024-03-17T23:47:23.655 INFO:teuthology.orchestra.run.smithi150.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:47:23.656 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:34 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:47:23.656 INFO:teuthology.orchestra.run.smithi150.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-17T23:47:23.656 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:35 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:47:23.656 INFO:teuthology.orchestra.run.smithi150.stdout: inet 172.21.15.150/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-17T23:47:23.656 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft 42411sec preferred_lft 42411sec 2024-03-17T23:47:23.656 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::ec4:7aff:fe88:6e35/64 scope link 2024-03-17T23:47:23.656 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:23.657 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-17T23:47:23.657 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:47:23.657 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-03-17T23:47:23.657 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link add name ceph-brx type bridge 2024-03-17T23:47:23.657 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip addr flush dev ceph-brx 2024-03-17T23:47:23.657 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set ceph-brx up 2024-03-17T23:47:23.657 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-17T23:47:23.657 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-03-17T23:47:23.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:47:23.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:23.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:23.802 DEBUG:teuthology.orchestra.run.smithi150:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-17T23:47:23.827 INFO:teuthology.orchestra.run.smithi150.stdout:1 2024-03-17T23:47:23.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:23.829 DEBUG:teuthology.orchestra.run.smithi150:> route 2024-03-17T23:47:23.887 INFO:teuthology.orchestra.run.smithi150.stdout:Kernel IP routing table 2024-03-17T23:47:23.887 INFO:teuthology.orchestra.run.smithi150.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-17T23:47:23.887 INFO:teuthology.orchestra.run.smithi150.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-17T23:47:23.887 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-17T23:47:23.887 INFO:teuthology.orchestra.run.smithi150.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-17T23:47:23.887 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:47:23.888 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-03-17T23:47:23.888 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-17T23:47:23.888 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-17T23:47:23.888 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-17T23:47:23.888 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-03-17T23:47:23.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:47:24.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:24.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:24.024 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-03-17T23:47:24.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:24.039 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list-id 2024-03-17T23:47:24.094 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:47:24.094 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-03-17T23:47:24.094 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:47:24.095 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-17T23:47:24.095 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-03-17T23:47:24.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:24 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:47:24.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:24.215 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-17T23:47:24.216 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:47:24.216 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-03-17T23:47:24.216 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-17T23:47:24.216 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-17T23:47:24.216 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-17T23:47:24.216 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-17T23:47:24.216 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-17T23:47:24.216 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-03-17T23:47:24.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:24 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:47:24.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:24.340 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:47:24.340 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-03-17T23:47:24.340 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set brx.0 up 2024-03-17T23:47:24.340 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set dev brx.0 master ceph-brx 2024-03-17T23:47:24.340 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-03-17T23:47:24.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:24 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:47:24.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:24.423 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-17T23:47:24.423 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-17T23:47:24.423 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-17T23:47:24.438 INFO:teuthology.orchestra.run.smithi150.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-17T23:47:24.438 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-17T23:47:24.438 DEBUG:teuthology.orchestra.run.smithi150:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-17T23:47:24.493 DEBUG:teuthology.orchestra.run.smithi150:> sudo modprobe fuse 2024-03-17T23:47:24.561 DEBUG:teuthology.orchestra.run.smithi150:> cat /proc/self/mounts | awk '{print $2}' 2024-03-17T23:47:24.617 INFO:teuthology.orchestra.run.smithi150.stdout:/proc 2024-03-17T23:47:24.617 INFO:teuthology.orchestra.run.smithi150.stdout:/sys 2024-03-17T23:47:24.617 INFO:teuthology.orchestra.run.smithi150.stdout:/dev 2024-03-17T23:47:24.617 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/security 2024-03-17T23:47:24.617 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/shm 2024-03-17T23:47:24.617 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/pts 2024-03-17T23:47:24.617 INFO:teuthology.orchestra.run.smithi150.stdout:/run 2024-03-17T23:47:24.618 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/cgroup 2024-03-17T23:47:24.618 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/pstore 2024-03-17T23:47:24.618 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/bpf 2024-03-17T23:47:24.618 INFO:teuthology.orchestra.run.smithi150.stdout:/ 2024-03-17T23:47:24.618 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/selinux 2024-03-17T23:47:24.618 INFO:teuthology.orchestra.run.smithi150.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:47:24.618 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/hugepages 2024-03-17T23:47:24.618 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/mqueue 2024-03-17T23:47:24.618 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/debug 2024-03-17T23:47:24.618 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/tracing 2024-03-17T23:47:24.618 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-17T23:47:24.618 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/fuse/connections 2024-03-17T23:47:24.618 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/config 2024-03-17T23:47:24.619 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-sysctl.service 2024-03-17T23:47:24.619 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-17T23:47:24.619 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/nfs/rpc_pipefs 2024-03-17T23:47:24.619 INFO:teuthology.orchestra.run.smithi150.stdout:/run/user/1000 2024-03-17T23:47:24.619 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph 2024-03-17T23:47:24.619 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-0 2024-03-17T23:47:24.619 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-1 2024-03-17T23:47:24.619 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-2 2024-03-17T23:47:24.619 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-3 2024-03-17T23:47:24.619 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns 2024-03-17T23:47:24.619 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:47:24.619 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:47:24.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:24.620 DEBUG:teuthology.orchestra.run.smithi150:> ls /sys/fs/fuse/connections 2024-03-17T23:47:24.674 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-17T23:47:24.675 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-17T23:47:24.717 DEBUG:teuthology.orchestra.run.smithi150:> sudo modprobe fuse 2024-03-17T23:47:24.743 DEBUG:teuthology.orchestra.run.smithi150:> cat /proc/self/mounts | awk '{print $2}' 2024-03-17T23:47:24.800 INFO:teuthology.orchestra.run.smithi150.stdout:/proc 2024-03-17T23:47:24.801 INFO:teuthology.orchestra.run.smithi150.stdout:/sys 2024-03-17T23:47:24.801 INFO:teuthology.orchestra.run.smithi150.stdout:/dev 2024-03-17T23:47:24.801 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/security 2024-03-17T23:47:24.801 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/shm 2024-03-17T23:47:24.801 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/pts 2024-03-17T23:47:24.801 INFO:teuthology.orchestra.run.smithi150.stdout:/run 2024-03-17T23:47:24.802 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/cgroup 2024-03-17T23:47:24.802 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/pstore 2024-03-17T23:47:24.802 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/bpf 2024-03-17T23:47:24.802 INFO:teuthology.orchestra.run.smithi150.stdout:/ 2024-03-17T23:47:24.802 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/selinux 2024-03-17T23:47:24.802 INFO:teuthology.orchestra.run.smithi150.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:47:24.803 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/hugepages 2024-03-17T23:47:24.803 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/mqueue 2024-03-17T23:47:24.803 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/debug 2024-03-17T23:47:24.803 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/tracing 2024-03-17T23:47:24.803 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-17T23:47:24.803 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/fuse/connections 2024-03-17T23:47:24.804 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/config 2024-03-17T23:47:24.804 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-sysctl.service 2024-03-17T23:47:24.804 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-17T23:47:24.804 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/nfs/rpc_pipefs 2024-03-17T23:47:24.804 INFO:teuthology.orchestra.run.smithi150.stdout:/run/user/1000 2024-03-17T23:47:24.804 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph 2024-03-17T23:47:24.804 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-0 2024-03-17T23:47:24.805 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-1 2024-03-17T23:47:24.805 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-2 2024-03-17T23:47:24.805 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-3 2024-03-17T23:47:24.805 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns 2024-03-17T23:47:24.805 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:47:24.805 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:47:24.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:24.806 DEBUG:teuthology.orchestra.run.smithi150:> ls /sys/fs/fuse/connections 2024-03-17T23:47:24.808 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:2024-03-17T23:47:24.802+0000 7f7fac5f6480 -1 init, newargv = 0x560c6a6b82d0 newargc=15 2024-03-17T23:47:24.808 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:ceph-fuse[48272]: starting ceph client 2024-03-17T23:47:24.808 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:ceph-fuse[48272]: starting fuse 2024-03-17T23:47:24.822 INFO:teuthology.orchestra.run.smithi150.stdout:43 2024-03-17T23:47:24.822 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-17T23:47:24.823 DEBUG:teuthology.orchestra.run.smithi150:> sudo stdin-killer -- python3 -c ' 2024-03-17T23:47:24.823 DEBUG:teuthology.orchestra.run.smithi150:> import glob 2024-03-17T23:47:24.823 DEBUG:teuthology.orchestra.run.smithi150:> import re 2024-03-17T23:47:24.823 DEBUG:teuthology.orchestra.run.smithi150:> import os 2024-03-17T23:47:24.823 DEBUG:teuthology.orchestra.run.smithi150:> import subprocess 2024-03-17T23:47:24.823 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:47:24.823 DEBUG:teuthology.orchestra.run.smithi150:> def _find_admin_socket(client_name): 2024-03-17T23:47:24.824 DEBUG:teuthology.orchestra.run.smithi150:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-17T23:47:24.824 DEBUG:teuthology.orchestra.run.smithi150:> files = glob.glob(asok_path) 2024-03-17T23:47:24.824 DEBUG:teuthology.orchestra.run.smithi150:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-17T23:47:24.824 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:47:24.824 DEBUG:teuthology.orchestra.run.smithi150:> # Given a non-glob path, it better be there 2024-03-17T23:47:24.824 DEBUG:teuthology.orchestra.run.smithi150:> if "*" not in asok_path: 2024-03-17T23:47:24.824 DEBUG:teuthology.orchestra.run.smithi150:> assert(len(files) == 1) 2024-03-17T23:47:24.824 DEBUG:teuthology.orchestra.run.smithi150:> return files[0] 2024-03-17T23:47:24.824 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:47:24.825 DEBUG:teuthology.orchestra.run.smithi150:> for f in files: 2024-03-17T23:47:24.825 DEBUG:teuthology.orchestra.run.smithi150:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-17T23:47:24.825 DEBUG:teuthology.orchestra.run.smithi150:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-17T23:47:24.825 DEBUG:teuthology.orchestra.run.smithi150:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-17T23:47:24.825 DEBUG:teuthology.orchestra.run.smithi150:> contents = proc_f.read() 2024-03-17T23:47:24.825 DEBUG:teuthology.orchestra.run.smithi150:> if mountpoint in contents: 2024-03-17T23:47:24.825 DEBUG:teuthology.orchestra.run.smithi150:> return f 2024-03-17T23:47:24.826 DEBUG:teuthology.orchestra.run.smithi150:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-17T23:47:24.826 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:47:24.826 DEBUG:teuthology.orchestra.run.smithi150:> print(_find_admin_socket("client.0")) 2024-03-17T23:47:24.826 DEBUG:teuthology.orchestra.run.smithi150:> ' 2024-03-17T23:47:24.943 INFO:teuthology.orchestra.run.smithi150.stdout:/var/run/ceph/ceph-client.0.48272.asok 2024-03-17T23:47:24.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:24.953 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.48272.asok 2024-03-17T23:47:24.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:24.953 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.48272.asok status 2024-03-17T23:47:25.089 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-17T23:47:25.090 INFO:teuthology.orchestra.run.smithi150.stdout: "metadata": { 2024-03-17T23:47:25.090 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-17T23:47:25.090 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-17T23:47:25.090 INFO:teuthology.orchestra.run.smithi150.stdout: "entity_id": "0", 2024-03-17T23:47:25.090 INFO:teuthology.orchestra.run.smithi150.stdout: "hostname": "smithi150", 2024-03-17T23:47:25.090 INFO:teuthology.orchestra.run.smithi150.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-17T23:47:25.090 INFO:teuthology.orchestra.run.smithi150.stdout: "pid": "48272", 2024-03-17T23:47:25.090 INFO:teuthology.orchestra.run.smithi150.stdout: "root": "/" 2024-03-17T23:47:25.091 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:47:25.091 INFO:teuthology.orchestra.run.smithi150.stdout: "dentry_count": 0, 2024-03-17T23:47:25.091 INFO:teuthology.orchestra.run.smithi150.stdout: "dentry_pinned_count": 0, 2024-03-17T23:47:25.091 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 4842, 2024-03-17T23:47:25.091 INFO:teuthology.orchestra.run.smithi150.stdout: "inst": { 2024-03-17T23:47:25.091 INFO:teuthology.orchestra.run.smithi150.stdout: "name": { 2024-03-17T23:47:25.091 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "client", 2024-03-17T23:47:25.091 INFO:teuthology.orchestra.run.smithi150.stdout: "num": 4842 2024-03-17T23:47:25.091 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:47:25.092 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": { 2024-03-17T23:47:25.092 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v1", 2024-03-17T23:47:25.092 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "192.168.0.1:0", 2024-03-17T23:47:25.092 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3771416655 2024-03-17T23:47:25.092 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:47:25.092 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:47:25.092 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": { 2024-03-17T23:47:25.092 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v1", 2024-03-17T23:47:25.093 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "192.168.0.1:0", 2024-03-17T23:47:25.093 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3771416655 2024-03-17T23:47:25.093 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:47:25.093 INFO:teuthology.orchestra.run.smithi150.stdout: "inst_str": "client.4842 192.168.0.1:0/3771416655", 2024-03-17T23:47:25.093 INFO:teuthology.orchestra.run.smithi150.stdout: "addr_str": "192.168.0.1:0/3771416655", 2024-03-17T23:47:25.093 INFO:teuthology.orchestra.run.smithi150.stdout: "inode_count": 1, 2024-03-17T23:47:25.094 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_epoch": 8, 2024-03-17T23:47:25.094 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_epoch": 26, 2024-03-17T23:47:25.094 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_epoch_barrier": 0, 2024-03-17T23:47:25.094 INFO:teuthology.orchestra.run.smithi150.stdout: "blocklisted": false, 2024-03-17T23:47:25.094 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "cephfs" 2024-03-17T23:47:25.094 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-17T23:47:25.098 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-17T23:47:25.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.200+0000 7f0b4795b640 1 Processor -- start 2024-03-17T23:47:25.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.200+0000 7f0b4795b640 1 -- start start 2024-03-17T23:47:25.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.201+0000 7f0b4795b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0b40155320 0x7f0b4015bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:25.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.201+0000 7f0b4795b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b4006dd30 0x7f0b400ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:25.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.201+0000 7f0b4795b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b400cd250 0x7f0b4015e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:25.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.201+0000 7f0b4795b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f0b4015c550 con 0x7f0b400cd250 2024-03-17T23:47:25.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.201+0000 7f0b4795b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f0b400cf3b0 con 0x7f0b40155320 2024-03-17T23:47:25.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.201+0000 7f0b4795b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0b400cf690 con 0x7f0b4006dd30 2024-03-17T23:47:25.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.201+0000 7f0b45ef8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b400cd250 0x7f0b4015e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:25.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.201+0000 7f0b44ef6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b4006dd30 0x7f0b400ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:25.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.201+0000 7f0b456f7640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0b40155320 0x7f0b4015bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:25.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.201+0000 7f0b45ef8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b400cd250 0x7f0b4015e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40644/0 (socket says 172.21.15.150:40644) 2024-03-17T23:47:25.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.201+0000 7f0b44ef6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b4006dd30 0x7f0b400ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:55706/0 (socket says 172.21.15.150:55706) 2024-03-17T23:47:25.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.201+0000 7f0b45ef8640 1 -- 172.21.15.150:0/1684014001 learned_addr learned my addr 172.21.15.150:0/1684014001 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:25.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.201+0000 7f0b456f7640 1 -- 172.21.15.150:0/1684014001 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b4006dd30 msgr2=0x7f0b400ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:25.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.201+0000 7f0b456f7640 1 --2- 172.21.15.150:0/1684014001 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b4006dd30 0x7f0b400ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:25.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.201+0000 7f0b456f7640 1 -- 172.21.15.150:0/1684014001 >> v2:172.21.15.150:3300/0 conn(0x7f0b400cd250 msgr2=0x7f0b4015e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:25.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.201+0000 7f0b456f7640 1 --2- 172.21.15.150:0/1684014001 >> v2:172.21.15.150:3300/0 conn(0x7f0b400cd250 0x7f0b4015e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:25.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.201+0000 7f0b456f7640 1 -- 172.21.15.150:0/1684014001 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b4015c7a0 con 0x7f0b40155320 2024-03-17T23:47:25.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.201+0000 7f0b45ef8640 1 --2- 172.21.15.150:0/1684014001 >> v2:172.21.15.150:3300/0 conn(0x7f0b400cd250 0x7f0b4015e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:25.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.201+0000 7f0b44ef6640 1 --2- 172.21.15.150:0/1684014001 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b4006dd30 0x7f0b400ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:25.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.202+0000 7f0b456f7640 1 --2- 172.21.15.150:0/1684014001 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b40155320 0x7f0b4015bfd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0b4006d5a0 tx=0x7f0b3c02e010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba222fd8cc60abd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:25.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.202+0000 7f0b37fff640 1 -- 172.21.15.150:0/1684014001 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c03b070 con 0x7f0b40155320 2024-03-17T23:47:25.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.202+0000 7f0b37fff640 1 -- 172.21.15.150:0/1684014001 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b3c02edc0 con 0x7f0b40155320 2024-03-17T23:47:25.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.202+0000 7f0b37fff640 1 -- 172.21.15.150:0/1684014001 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c036660 con 0x7f0b40155320 2024-03-17T23:47:25.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.203+0000 7f0b4795b640 1 -- 172.21.15.150:0/1684014001 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b40155320 msgr2=0x7f0b4015bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:25.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.203+0000 7f0b4795b640 1 --2- 172.21.15.150:0/1684014001 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b40155320 0x7f0b4015bfd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0b4006d5a0 tx=0x7f0b3c02e010 comp rx=0 tx=0).stop 2024-03-17T23:47:25.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.203+0000 7f0b4795b640 1 -- 172.21.15.150:0/1684014001 shutdown_connections 2024-03-17T23:47:25.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.203+0000 7f0b4795b640 1 --2- 172.21.15.150:0/1684014001 >> v2:172.21.15.150:3300/0 conn(0x7f0b400cd250 0x7f0b4015e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:25.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.203+0000 7f0b4795b640 1 --2- 172.21.15.150:0/1684014001 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b4006dd30 0x7f0b400ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:25.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.203+0000 7f0b4795b640 1 --2- 172.21.15.150:0/1684014001 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b40155320 0x7f0b4015bfd0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:25.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.203+0000 7f0b4795b640 1 -- 172.21.15.150:0/1684014001 >> 172.21.15.150:0/1684014001 conn(0x7f0b4006b190 msgr2=0x7f0b4006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:25.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.203+0000 7f0b4795b640 1 -- 172.21.15.150:0/1684014001 shutdown_connections 2024-03-17T23:47:25.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.203+0000 7f0b4795b640 1 -- 172.21.15.150:0/1684014001 wait complete. 2024-03-17T23:47:25.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.204+0000 7f0b4795b640 1 Processor -- start 2024-03-17T23:47:25.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.204+0000 7f0b4795b640 1 -- start start 2024-03-17T23:47:25.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.205+0000 7f0b4795b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b4006dd30 0x7f0b40158510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:25.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.205+0000 7f0b4795b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b400cd250 0x7f0b4015d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:25.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.205+0000 7f0b4795b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b40155320 0x7f0b4015eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:25.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.205+0000 7f0b4795b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0b40155700 con 0x7f0b400cd250 2024-03-17T23:47:25.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.205+0000 7f0b4795b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0b4006e190 con 0x7f0b4006dd30 2024-03-17T23:47:25.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.205+0000 7f0b4795b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0b40155880 con 0x7f0b40155320 2024-03-17T23:47:25.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.205+0000 7f0b44ef6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b400cd250 0x7f0b4015d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:25.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.205+0000 7f0b44ef6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b400cd250 0x7f0b4015d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40652/0 (socket says 172.21.15.150:40652) 2024-03-17T23:47:25.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.205+0000 7f0b44ef6640 1 -- 172.21.15.150:0/463307508 learned_addr learned my addr 172.21.15.150:0/463307508 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:25.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.205+0000 7f0b456f7640 1 --2- 172.21.15.150:0/463307508 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b4006dd30 0x7f0b40158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:25.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.205+0000 7f0b45ef8640 1 --2- 172.21.15.150:0/463307508 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b40155320 0x7f0b4015eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:25.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.205+0000 7f0b45ef8640 1 -- 172.21.15.150:0/463307508 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b4006dd30 msgr2=0x7f0b40158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:25.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.205+0000 7f0b45ef8640 1 --2- 172.21.15.150:0/463307508 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b4006dd30 0x7f0b40158510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:25.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.205+0000 7f0b45ef8640 1 -- 172.21.15.150:0/463307508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b400cd250 msgr2=0x7f0b4015d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:25.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.205+0000 7f0b45ef8640 1 --2- 172.21.15.150:0/463307508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b400cd250 0x7f0b4015d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:25.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.205+0000 7f0b45ef8640 1 -- 172.21.15.150:0/463307508 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b4015d860 con 0x7f0b40155320 2024-03-17T23:47:25.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.205+0000 7f0b44ef6640 1 --2- 172.21.15.150:0/463307508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b400cd250 0x7f0b4015d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:25.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.205+0000 7f0b456f7640 1 --2- 172.21.15.150:0/463307508 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b4006dd30 0x7f0b40158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:25.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.205+0000 7f0b45ef8640 1 --2- 172.21.15.150:0/463307508 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b40155320 0x7f0b4015eb80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0b3800efc0 tx=0x7f0b3800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:25.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.206+0000 7f0b35ffb640 1 -- 172.21.15.150:0/463307508 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3800cb20 con 0x7f0b40155320 2024-03-17T23:47:25.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.206+0000 7f0b35ffb640 1 -- 172.21.15.150:0/463307508 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b3800cce0 con 0x7f0b40155320 2024-03-17T23:47:25.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.206+0000 7f0b4795b640 1 -- 172.21.15.150:0/463307508 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b4015f0c0 con 0x7f0b40155320 2024-03-17T23:47:25.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.206+0000 7f0b35ffb640 1 -- 172.21.15.150:0/463307508 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b38003bd0 con 0x7f0b40155320 2024-03-17T23:47:25.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.206+0000 7f0b4795b640 1 -- 172.21.15.150:0/463307508 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b4015ddc0 con 0x7f0b40155320 2024-03-17T23:47:25.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.207+0000 7f0b35ffb640 1 -- 172.21.15.150:0/463307508 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0b3801b030 con 0x7f0b40155320 2024-03-17T23:47:25.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.207+0000 7f0b4795b640 1 -- 172.21.15.150:0/463307508 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b400681a0 con 0x7f0b40155320 2024-03-17T23:47:25.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.207+0000 7f0b35ffb640 1 --2- 172.21.15.150:0/463307508 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b2406bbd0 0x7f0b2406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:25.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.207+0000 7f0b456f7640 1 --2- 172.21.15.150:0/463307508 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b2406bbd0 0x7f0b2406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:25.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.207+0000 7f0b35ffb640 1 -- 172.21.15.150:0/463307508 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f0b3808afd0 con 0x7f0b40155320 2024-03-17T23:47:25.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.208+0000 7f0b456f7640 1 --2- 172.21.15.150:0/463307508 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b2406bbd0 0x7f0b2406e090 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c038000 tx=0x7f0b3c032000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:25.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.212+0000 7f0b35ffb640 1 -- 172.21.15.150:0/463307508 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0b38022020 con 0x7f0b40155320 2024-03-17T23:47:25.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.424+0000 7f0b4795b640 1 -- 172.21.15.150:0/463307508 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f0b40152d80 con 0x7f0b40155320 2024-03-17T23:47:25.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.424+0000 7f0b35ffb640 1 -- 172.21.15.150:0/463307508 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7f0b4015f0c0 con 0x7f0b40155320 2024-03-17T23:47:25.425 INFO:teuthology.orchestra.run.smithi150.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-17T23:47:25.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.427+0000 7f0b4795b640 1 -- 172.21.15.150:0/463307508 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b2406bbd0 msgr2=0x7f0b2406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:25.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.427+0000 7f0b4795b640 1 --2- 172.21.15.150:0/463307508 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b2406bbd0 0x7f0b2406e090 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c038000 tx=0x7f0b3c032000 comp rx=0 tx=0).stop 2024-03-17T23:47:25.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.427+0000 7f0b4795b640 1 -- 172.21.15.150:0/463307508 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b40155320 msgr2=0x7f0b4015eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:25.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.427+0000 7f0b4795b640 1 --2- 172.21.15.150:0/463307508 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b40155320 0x7f0b4015eb80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0b3800efc0 tx=0x7f0b3800c490 comp rx=0 tx=0).stop 2024-03-17T23:47:25.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.427+0000 7f0b4795b640 1 -- 172.21.15.150:0/463307508 shutdown_connections 2024-03-17T23:47:25.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.427+0000 7f0b4795b640 1 --2- 172.21.15.150:0/463307508 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b40155320 0x7f0b4015eb80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:25.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.427+0000 7f0b4795b640 1 --2- 172.21.15.150:0/463307508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b400cd250 0x7f0b4015d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:25.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.427+0000 7f0b4795b640 1 --2- 172.21.15.150:0/463307508 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b2406bbd0 0x7f0b2406e090 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:25.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.427+0000 7f0b4795b640 1 --2- 172.21.15.150:0/463307508 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b4006dd30 0x7f0b40158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:25.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.427+0000 7f0b4795b640 1 -- 172.21.15.150:0/463307508 >> 172.21.15.150:0/463307508 conn(0x7f0b4006b190 msgr2=0x7f0b40151d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:25.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.427+0000 7f0b4795b640 1 -- 172.21.15.150:0/463307508 shutdown_connections 2024-03-17T23:47:25.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25.427+0000 7f0b4795b640 1 -- 172.21.15.150:0/463307508 wait complete. 2024-03-17T23:47:25.438 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-17T23:47:25.438 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-17T23:47:25.438 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi150.front.sepia.ceph.com 2024-03-17T23:47:25.438 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-17T23:47:25.438 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-17T23:47:25.438 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-17T23:47:25.438 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-17T23:47:25.438 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-17T23:47:25.439 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-17T23:47:25.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:25.439 DEBUG:teuthology.orchestra.run.smithi150:> ip addr 2024-03-17T23:47:25.454 INFO:teuthology.orchestra.run.smithi150.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-17T23:47:25.454 INFO:teuthology.orchestra.run.smithi150.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-17T23:47:25.454 INFO:teuthology.orchestra.run.smithi150.stdout: inet 127.0.0.1/8 scope host lo 2024-03-17T23:47:25.454 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:25.455 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 ::1/128 scope host 2024-03-17T23:47:25.455 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:25.455 INFO:teuthology.orchestra.run.smithi150.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:47:25.455 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bc brd ff:ff:ff:ff:ff:ff 2024-03-17T23:47:25.455 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f0 2024-03-17T23:47:25.455 INFO:teuthology.orchestra.run.smithi150.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:47:25.455 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bd brd ff:ff:ff:ff:ff:ff 2024-03-17T23:47:25.455 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f1 2024-03-17T23:47:25.455 INFO:teuthology.orchestra.run.smithi150.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:47:25.455 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:34 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:47:25.455 INFO:teuthology.orchestra.run.smithi150.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-17T23:47:25.455 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:35 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:47:25.455 INFO:teuthology.orchestra.run.smithi150.stdout: inet 172.21.15.150/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-17T23:47:25.456 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft 42409sec preferred_lft 42409sec 2024-03-17T23:47:25.456 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::ec4:7aff:fe88:6e35/64 scope link 2024-03-17T23:47:25.456 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:25.456 INFO:teuthology.orchestra.run.smithi150.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-17T23:47:25.456 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether f6:2d:aa:e3:d6:a3 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:47:25.456 INFO:teuthology.orchestra.run.smithi150.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-17T23:47:25.456 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:25.456 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::bc2f:65ff:fe6d:abc3/64 scope link 2024-03-17T23:47:25.456 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:25.456 INFO:teuthology.orchestra.run.smithi150.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-17T23:47:25.456 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether f6:2d:aa:e3:d6:a3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:47:25.457 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::f42d:aaff:fee3:d6a3/64 scope link tentative 2024-03-17T23:47:25.457 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:25.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:25.457 DEBUG:teuthology.orchestra.run.smithi150:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-17T23:47:25.520 INFO:teuthology.orchestra.run.smithi150.stdout:1 2024-03-17T23:47:25.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:25.521 DEBUG:teuthology.orchestra.run.smithi150:> route 2024-03-17T23:47:25.577 INFO:teuthology.orchestra.run.smithi150.stdout:Kernel IP routing table 2024-03-17T23:47:25.577 INFO:teuthology.orchestra.run.smithi150.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-17T23:47:25.577 INFO:teuthology.orchestra.run.smithi150.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-17T23:47:25.577 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-17T23:47:25.578 INFO:teuthology.orchestra.run.smithi150.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-17T23:47:25.578 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:47:25.578 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-03-17T23:47:25.578 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-17T23:47:25.578 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-17T23:47:25.578 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-17T23:47:25.578 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-03-17T23:47:25.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:47:25.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:25.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:25.712 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-03-17T23:47:25.726 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-17T23:47:25.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:25.727 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list-id 2024-03-17T23:47:25.782 INFO:teuthology.orchestra.run.smithi150.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-17T23:47:25.783 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:47:25.783 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-03-17T23:47:25.783 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:47:25.783 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-17T23:47:25.783 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-03-17T23:47:25.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:47:25.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:25.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:25.905 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-17T23:47:25.931 INFO:teuthology.orchestra.run.smithi150.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-17T23:47:25.932 INFO:teuthology.orchestra.run.smithi150.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-17T23:47:25.932 INFO:teuthology.orchestra.run.smithi150.stdout: inet 127.0.0.1/8 scope host lo 2024-03-17T23:47:25.932 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:25.932 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 ::1/128 scope host 2024-03-17T23:47:25.932 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:25.932 INFO:teuthology.orchestra.run.smithi150.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-17T23:47:25.932 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 3a:27:2e:ee:83:fd brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-17T23:47:25.932 INFO:teuthology.orchestra.run.smithi150.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-17T23:47:25.932 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:25.932 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::3827:2eff:feee:83fd/64 scope link 2024-03-17T23:47:25.932 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:25.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:25.934 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-17T23:47:26.000 INFO:teuthology.orchestra.run.smithi150.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-17T23:47:26.000 INFO:teuthology.orchestra.run.smithi150.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-17T23:47:26.001 INFO:teuthology.orchestra.run.smithi150.stdout: inet 127.0.0.1/8 scope host lo 2024-03-17T23:47:26.001 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:26.001 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 ::1/128 scope host 2024-03-17T23:47:26.001 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:26.001 INFO:teuthology.orchestra.run.smithi150.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-17T23:47:26.001 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 3a:27:2e:ee:83:fd brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-17T23:47:26.001 INFO:teuthology.orchestra.run.smithi150.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-17T23:47:26.001 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:26.001 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::3827:2eff:feee:83fd/64 scope link 2024-03-17T23:47:26.001 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:26.002 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-17T23:47:26.003 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:47:26.003 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-03-17T23:47:26.003 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-17T23:47:26.003 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-17T23:47:26.003 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-17T23:47:26.003 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-17T23:47:26.003 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-17T23:47:26.003 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-03-17T23:47:26.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:47:26.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:26.167 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:47:26.167 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-03-17T23:47:26.167 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set brx.1 up 2024-03-17T23:47:26.167 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set dev brx.1 master ceph-brx 2024-03-17T23:47:26.167 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-03-17T23:47:26.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:47:26.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:26.250 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-03-17T23:47:26.250 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-17T23:47:26.250 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-17T23:47:26.264 INFO:teuthology.orchestra.run.smithi150.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-17T23:47:26.265 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-17T23:47:26.265 DEBUG:teuthology.orchestra.run.smithi150:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-17T23:47:26.320 DEBUG:teuthology.orchestra.run.smithi150:> sudo modprobe fuse 2024-03-17T23:47:26.388 DEBUG:teuthology.orchestra.run.smithi150:> cat /proc/self/mounts | awk '{print $2}' 2024-03-17T23:47:26.444 INFO:teuthology.orchestra.run.smithi150.stdout:/proc 2024-03-17T23:47:26.444 INFO:teuthology.orchestra.run.smithi150.stdout:/sys 2024-03-17T23:47:26.444 INFO:teuthology.orchestra.run.smithi150.stdout:/dev 2024-03-17T23:47:26.445 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/security 2024-03-17T23:47:26.445 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/shm 2024-03-17T23:47:26.445 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/pts 2024-03-17T23:47:26.445 INFO:teuthology.orchestra.run.smithi150.stdout:/run 2024-03-17T23:47:26.445 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/cgroup 2024-03-17T23:47:26.445 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/pstore 2024-03-17T23:47:26.445 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/bpf 2024-03-17T23:47:26.445 INFO:teuthology.orchestra.run.smithi150.stdout:/ 2024-03-17T23:47:26.445 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/selinux 2024-03-17T23:47:26.445 INFO:teuthology.orchestra.run.smithi150.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:47:26.446 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/hugepages 2024-03-17T23:47:26.446 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/mqueue 2024-03-17T23:47:26.446 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/debug 2024-03-17T23:47:26.446 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/tracing 2024-03-17T23:47:26.446 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-17T23:47:26.446 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/fuse/connections 2024-03-17T23:47:26.446 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/config 2024-03-17T23:47:26.446 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-sysctl.service 2024-03-17T23:47:26.446 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-17T23:47:26.446 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/nfs/rpc_pipefs 2024-03-17T23:47:26.446 INFO:teuthology.orchestra.run.smithi150.stdout:/run/user/1000 2024-03-17T23:47:26.447 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph 2024-03-17T23:47:26.447 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-0 2024-03-17T23:47:26.447 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-1 2024-03-17T23:47:26.447 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-2 2024-03-17T23:47:26.447 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-3 2024-03-17T23:47:26.447 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns 2024-03-17T23:47:26.447 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:47:26.447 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:47:26.447 INFO:teuthology.orchestra.run.smithi150.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-17T23:47:26.447 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:47:26.447 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:47:26.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:26.448 DEBUG:teuthology.orchestra.run.smithi150:> ls /sys/fs/fuse/connections 2024-03-17T23:47:26.499 INFO:teuthology.orchestra.run.smithi150.stdout:43 2024-03-17T23:47:26.500 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-03-17T23:47:26.500 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-03-17T23:47:26.542 DEBUG:teuthology.orchestra.run.smithi150:> sudo modprobe fuse 2024-03-17T23:47:26.569 DEBUG:teuthology.orchestra.run.smithi150:> cat /proc/self/mounts | awk '{print $2}' 2024-03-17T23:47:26.624 INFO:teuthology.orchestra.run.smithi150.stdout:/proc 2024-03-17T23:47:26.624 INFO:teuthology.orchestra.run.smithi150.stdout:/sys 2024-03-17T23:47:26.624 INFO:teuthology.orchestra.run.smithi150.stdout:/dev 2024-03-17T23:47:26.624 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/security 2024-03-17T23:47:26.625 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/shm 2024-03-17T23:47:26.625 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/pts 2024-03-17T23:47:26.625 INFO:teuthology.orchestra.run.smithi150.stdout:/run 2024-03-17T23:47:26.625 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/cgroup 2024-03-17T23:47:26.625 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/pstore 2024-03-17T23:47:26.625 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/bpf 2024-03-17T23:47:26.625 INFO:teuthology.orchestra.run.smithi150.stdout:/ 2024-03-17T23:47:26.625 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/selinux 2024-03-17T23:47:26.625 INFO:teuthology.orchestra.run.smithi150.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:47:26.625 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/hugepages 2024-03-17T23:47:26.625 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/mqueue 2024-03-17T23:47:26.625 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/debug 2024-03-17T23:47:26.626 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/tracing 2024-03-17T23:47:26.626 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-17T23:47:26.626 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/fuse/connections 2024-03-17T23:47:26.626 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/config 2024-03-17T23:47:26.626 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-sysctl.service 2024-03-17T23:47:26.626 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-17T23:47:26.626 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/nfs/rpc_pipefs 2024-03-17T23:47:26.626 INFO:teuthology.orchestra.run.smithi150.stdout:/run/user/1000 2024-03-17T23:47:26.626 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph 2024-03-17T23:47:26.626 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-0 2024-03-17T23:47:26.626 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-1 2024-03-17T23:47:26.626 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-2 2024-03-17T23:47:26.627 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-3 2024-03-17T23:47:26.627 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns 2024-03-17T23:47:26.627 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:47:26.627 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:47:26.627 INFO:teuthology.orchestra.run.smithi150.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-17T23:47:26.627 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:47:26.627 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:47:26.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:26.628 DEBUG:teuthology.orchestra.run.smithi150:> ls /sys/fs/fuse/connections 2024-03-17T23:47:26.629 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi150.stderr:2024-03-17T23:47:26.627+0000 7f5ea303c480 -1 init, newargv = 0x55a8553b4f30 newargc=15 2024-03-17T23:47:26.629 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi150.stderr:ceph-fuse[48770]: starting ceph client 2024-03-17T23:47:26.632 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi150.stderr:ceph-fuse[48770]: starting fuse 2024-03-17T23:47:26.646 INFO:teuthology.orchestra.run.smithi150.stdout:43 2024-03-17T23:47:26.646 INFO:teuthology.orchestra.run.smithi150.stdout:44 2024-03-17T23:47:26.646 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-03-17T23:47:26.646 DEBUG:teuthology.orchestra.run.smithi150:> sudo stdin-killer -- python3 -c ' 2024-03-17T23:47:26.646 DEBUG:teuthology.orchestra.run.smithi150:> import glob 2024-03-17T23:47:26.646 DEBUG:teuthology.orchestra.run.smithi150:> import re 2024-03-17T23:47:26.647 DEBUG:teuthology.orchestra.run.smithi150:> import os 2024-03-17T23:47:26.647 DEBUG:teuthology.orchestra.run.smithi150:> import subprocess 2024-03-17T23:47:26.647 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:47:26.647 DEBUG:teuthology.orchestra.run.smithi150:> def _find_admin_socket(client_name): 2024-03-17T23:47:26.647 DEBUG:teuthology.orchestra.run.smithi150:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-03-17T23:47:26.647 DEBUG:teuthology.orchestra.run.smithi150:> files = glob.glob(asok_path) 2024-03-17T23:47:26.647 DEBUG:teuthology.orchestra.run.smithi150:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-03-17T23:47:26.647 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:47:26.647 DEBUG:teuthology.orchestra.run.smithi150:> # Given a non-glob path, it better be there 2024-03-17T23:47:26.647 DEBUG:teuthology.orchestra.run.smithi150:> if "*" not in asok_path: 2024-03-17T23:47:26.647 DEBUG:teuthology.orchestra.run.smithi150:> assert(len(files) == 1) 2024-03-17T23:47:26.647 DEBUG:teuthology.orchestra.run.smithi150:> return files[0] 2024-03-17T23:47:26.647 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:47:26.648 DEBUG:teuthology.orchestra.run.smithi150:> for f in files: 2024-03-17T23:47:26.648 DEBUG:teuthology.orchestra.run.smithi150:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-17T23:47:26.648 DEBUG:teuthology.orchestra.run.smithi150:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-17T23:47:26.648 DEBUG:teuthology.orchestra.run.smithi150:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-17T23:47:26.648 DEBUG:teuthology.orchestra.run.smithi150:> contents = proc_f.read() 2024-03-17T23:47:26.648 DEBUG:teuthology.orchestra.run.smithi150:> if mountpoint in contents: 2024-03-17T23:47:26.648 DEBUG:teuthology.orchestra.run.smithi150:> return f 2024-03-17T23:47:26.648 DEBUG:teuthology.orchestra.run.smithi150:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-17T23:47:26.648 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:47:26.648 DEBUG:teuthology.orchestra.run.smithi150:> print(_find_admin_socket("client.2")) 2024-03-17T23:47:26.648 DEBUG:teuthology.orchestra.run.smithi150:> ' 2024-03-17T23:47:26.765 INFO:teuthology.orchestra.run.smithi150.stdout:/var/run/ceph/ceph-client.2.48770.asok 2024-03-17T23:47:26.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:26.775 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.48770.asok 2024-03-17T23:47:26.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:26.775 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.48770.asok status 2024-03-17T23:47:26.911 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-17T23:47:26.911 INFO:teuthology.orchestra.run.smithi150.stdout: "metadata": { 2024-03-17T23:47:26.911 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-17T23:47:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-17T23:47:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: "entity_id": "2", 2024-03-17T23:47:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: "hostname": "smithi150", 2024-03-17T23:47:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-03-17T23:47:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: "pid": "48770", 2024-03-17T23:47:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: "root": "/" 2024-03-17T23:47:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:47:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: "dentry_count": 0, 2024-03-17T23:47:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: "dentry_pinned_count": 0, 2024-03-17T23:47:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 4854, 2024-03-17T23:47:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: "inst": { 2024-03-17T23:47:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: "name": { 2024-03-17T23:47:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "client", 2024-03-17T23:47:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: "num": 4854 2024-03-17T23:47:26.912 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:47:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": { 2024-03-17T23:47:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v1", 2024-03-17T23:47:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "192.168.0.2:0", 2024-03-17T23:47:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3583696601 2024-03-17T23:47:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:47:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:47:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": { 2024-03-17T23:47:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v1", 2024-03-17T23:47:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "192.168.0.2:0", 2024-03-17T23:47:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3583696601 2024-03-17T23:47:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:47:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: "inst_str": "client.4854 192.168.0.2:0/3583696601", 2024-03-17T23:47:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: "addr_str": "192.168.0.2:0/3583696601", 2024-03-17T23:47:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: "inode_count": 1, 2024-03-17T23:47:26.913 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_epoch": 8, 2024-03-17T23:47:26.914 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_epoch": 26, 2024-03-17T23:47:26.914 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_epoch_barrier": 0, 2024-03-17T23:47:26.914 INFO:teuthology.orchestra.run.smithi150.stdout: "blocklisted": false, 2024-03-17T23:47:26.914 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "cephfs" 2024-03-17T23:47:26.914 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-17T23:47:26.920 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-17T23:47:27.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.024+0000 7fb120d60640 1 Processor -- start 2024-03-17T23:47:27.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.024+0000 7fb120d60640 1 -- start start 2024-03-17T23:47:27.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.025+0000 7fb120d60640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb11c15be40 0x7fb11c09a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:27.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.025+0000 7fb120d60640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb11c09ad00 con 0x7fb11c0d0240 2024-03-17T23:47:27.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.025+0000 7fb120d60640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb11c09b260 con 0x7fb11c0c8210 2024-03-17T23:47:27.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.025+0000 7fb120d60640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fb11c0c5d70 con 0x7fb11c15be40 2024-03-17T23:47:27.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.025+0000 7fb11a59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb11c15be40 0x7fb11c09a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:27.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.025+0000 7fb11ad9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb11c0d0240 0x7fb11c1659e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33658/0 (socket says 172.21.15.150:33658) 2024-03-17T23:47:27.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.025+0000 7fb11ad9d640 1 -- 172.21.15.150:0/353810310 learned_addr learned my addr 172.21.15.150:0/353810310 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:27.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.025+0000 7fb11a59c640 1 -- 172.21.15.150:0/353810310 >> v1:172.21.15.157:6789/0 conn(0x7fb11c0c8210 legacy=0x7fb11c0cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:27.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.025+0000 7fb11959a640 1 -- 172.21.15.150:0/353810310 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3570776154 0 0) 0x7fb11c09b260 con 0x7fb11c0c8210 2024-03-17T23:47:27.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.025+0000 7fb11a59c640 1 -- 172.21.15.150:0/353810310 >> v1:172.21.15.150:6789/0 conn(0x7fb11c0d0240 legacy=0x7fb11c1659e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:27.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.025+0000 7fb11a59c640 1 -- 172.21.15.150:0/353810310 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb11c09af80 con 0x7fb11c15be40 2024-03-17T23:47:27.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.026+0000 7fb11a59c640 1 --2- 172.21.15.150:0/353810310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb11c15be40 0x7fb11c09a7c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb10c0097a0 tx=0x7fb10c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40ba3649ff9ee5a server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:27.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.026+0000 7fb11959a640 1 -- 172.21.15.150:0/353810310 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10c034070 con 0x7fb11c15be40 2024-03-17T23:47:27.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.026+0000 7fb11959a640 1 -- 172.21.15.150:0/353810310 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb10c030610 con 0x7fb11c15be40 2024-03-17T23:47:27.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.026+0000 7fb11959a640 1 -- 172.21.15.150:0/353810310 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10c0309e0 con 0x7fb11c15be40 2024-03-17T23:47:27.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.027+0000 7fb120d60640 1 -- 172.21.15.150:0/353810310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb11c15be40 msgr2=0x7fb11c09a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:27.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.027+0000 7fb120d60640 1 --2- 172.21.15.150:0/353810310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb11c15be40 0x7fb11c09a7c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb10c0097a0 tx=0x7fb10c02b960 comp rx=0 tx=0).stop 2024-03-17T23:47:27.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.027+0000 7fb120d60640 1 -- 172.21.15.150:0/353810310 shutdown_connections 2024-03-17T23:47:27.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.027+0000 7fb120d60640 1 --2- 172.21.15.150:0/353810310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb11c15be40 0x7fb11c09a7c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:27.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.027+0000 7fb120d60640 1 -- 172.21.15.150:0/353810310 >> 172.21.15.150:0/353810310 conn(0x7fb11c0b3620 msgr2=0x7fb11c0b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:27.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.027+0000 7fb120d60640 1 -- 172.21.15.150:0/353810310 shutdown_connections 2024-03-17T23:47:27.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.027+0000 7fb120d60640 1 -- 172.21.15.150:0/353810310 wait complete. 2024-03-17T23:47:27.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.028+0000 7fb120d60640 1 Processor -- start 2024-03-17T23:47:27.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.028+0000 7fb120d60640 1 -- start start 2024-03-17T23:47:27.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.029+0000 7fb120d60640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb11c0c8210 0x7fb11c147ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:27.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.029+0000 7fb120d60640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb11c0d0240 0x7fb11c15de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:27.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.029+0000 7fb120d60640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb11c15be40 0x7fb11c151c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:27.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.029+0000 7fb120d60640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb11c15c220 con 0x7fb11c0c8210 2024-03-17T23:47:27.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.029+0000 7fb120d60640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb11c0c5a00 con 0x7fb11c0d0240 2024-03-17T23:47:27.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.029+0000 7fb120d60640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb11c0d06d0 con 0x7fb11c15be40 2024-03-17T23:47:27.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.029+0000 7fb11a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb11c0c8210 0x7fb11c147ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:27.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.029+0000 7fb11a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb11c0c8210 0x7fb11c147ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46004/0 (socket says 172.21.15.150:46004) 2024-03-17T23:47:27.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.029+0000 7fb11a59c640 1 -- 172.21.15.150:0/1513976140 learned_addr learned my addr 172.21.15.150:0/1513976140 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:27.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.029+0000 7fb11ad9d640 1 --2- 172.21.15.150:0/1513976140 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb11c15be40 0x7fb11c151c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:27.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.029+0000 7fb119d9b640 1 --2- 172.21.15.150:0/1513976140 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb11c0d0240 0x7fb11c15de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:27.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.030+0000 7fb11ad9d640 1 -- 172.21.15.150:0/1513976140 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb11c0d0240 msgr2=0x7fb11c15de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:27.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.030+0000 7fb11ad9d640 1 --2- 172.21.15.150:0/1513976140 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb11c0d0240 0x7fb11c15de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:27.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.030+0000 7fb11ad9d640 1 -- 172.21.15.150:0/1513976140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb11c0c8210 msgr2=0x7fb11c147ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:27.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.030+0000 7fb11ad9d640 1 --2- 172.21.15.150:0/1513976140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb11c0c8210 0x7fb11c147ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:27.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.030+0000 7fb11ad9d640 1 -- 172.21.15.150:0/1513976140 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb11c148400 con 0x7fb11c15be40 2024-03-17T23:47:27.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.030+0000 7fb11a59c640 1 --2- 172.21.15.150:0/1513976140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb11c0c8210 0x7fb11c147ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:27.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.030+0000 7fb11ad9d640 1 --2- 172.21.15.150:0/1513976140 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb11c15be40 0x7fb11c151c10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb11400cad0 tx=0x7fb11400cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:27.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.030+0000 7fb10affd640 1 -- 172.21.15.150:0/1513976140 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb114017070 con 0x7fb11c15be40 2024-03-17T23:47:27.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.030+0000 7fb10affd640 1 -- 172.21.15.150:0/1513976140 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb11400ac30 con 0x7fb11c15be40 2024-03-17T23:47:27.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.030+0000 7fb10affd640 1 -- 172.21.15.150:0/1513976140 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb114012400 con 0x7fb11c15be40 2024-03-17T23:47:27.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.030+0000 7fb120d60640 1 -- 172.21.15.150:0/1513976140 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb11c15e910 con 0x7fb11c15be40 2024-03-17T23:47:27.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.030+0000 7fb120d60640 1 -- 172.21.15.150:0/1513976140 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb11c0b5650 con 0x7fb11c15be40 2024-03-17T23:47:27.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.031+0000 7fb120d60640 1 -- 172.21.15.150:0/1513976140 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb11c0688b0 con 0x7fb11c15be40 2024-03-17T23:47:27.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.031+0000 7fb10affd640 1 -- 172.21.15.150:0/1513976140 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb11401a070 con 0x7fb11c15be40 2024-03-17T23:47:27.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.032+0000 7fb10affd640 1 --2- 172.21.15.150:0/1513976140 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb0fc06bbd0 0x7fb0fc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:27.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.032+0000 7fb10affd640 1 -- 172.21.15.150:0/1513976140 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fb11408a9e0 con 0x7fb11c15be40 2024-03-17T23:47:27.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.032+0000 7fb11a59c640 1 --2- 172.21.15.150:0/1513976140 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb0fc06bbd0 0x7fb0fc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:27.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.032+0000 7fb11a59c640 1 --2- 172.21.15.150:0/1513976140 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb0fc06bbd0 0x7fb0fc06e090 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb10c009560 tx=0x7fb10c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:27.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.036+0000 7fb10affd640 1 -- 172.21.15.150:0/1513976140 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb11c09ad00 con 0x7fb11c15be40 2024-03-17T23:47:27.095 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-17T23:47:27.099 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-17T23:47:27.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.253+0000 7fb120d60640 1 -- 172.21.15.150:0/1513976140 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fb11c071ca0 con 0x7fb11c15be40 2024-03-17T23:47:27.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.254+0000 7fb10affd640 1 -- 172.21.15.150:0/1513976140 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7fb114058aa0 con 0x7fb11c15be40 2024-03-17T23:47:27.255 INFO:teuthology.orchestra.run.smithi150.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-17T23:47:27.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.256+0000 7fb120d60640 1 -- 172.21.15.150:0/1513976140 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb0fc06bbd0 msgr2=0x7fb0fc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:27.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.256+0000 7fb120d60640 1 --2- 172.21.15.150:0/1513976140 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb0fc06bbd0 0x7fb0fc06e090 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb10c009560 tx=0x7fb10c03b040 comp rx=0 tx=0).stop 2024-03-17T23:47:27.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.256+0000 7fb120d60640 1 -- 172.21.15.150:0/1513976140 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb11c15be40 msgr2=0x7fb11c151c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:27.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.256+0000 7fb120d60640 1 --2- 172.21.15.150:0/1513976140 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb11c15be40 0x7fb11c151c10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb11400cad0 tx=0x7fb11400cf90 comp rx=0 tx=0).stop 2024-03-17T23:47:27.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.256+0000 7fb120d60640 1 -- 172.21.15.150:0/1513976140 shutdown_connections 2024-03-17T23:47:27.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.256+0000 7fb120d60640 1 --2- 172.21.15.150:0/1513976140 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb11c15be40 0x7fb11c151c10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:27.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.256+0000 7fb120d60640 1 --2- 172.21.15.150:0/1513976140 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb0fc06bbd0 0x7fb0fc06e090 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:27.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.256+0000 7fb120d60640 1 --2- 172.21.15.150:0/1513976140 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb11c0d0240 0x7fb11c15de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:27.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.256+0000 7fb120d60640 1 --2- 172.21.15.150:0/1513976140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb11c0c8210 0x7fb11c147ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:27.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.256+0000 7fb120d60640 1 -- 172.21.15.150:0/1513976140 >> 172.21.15.150:0/1513976140 conn(0x7fb11c0b3620 msgr2=0x7fb11c0cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:27.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.256+0000 7fb120d60640 1 -- 172.21.15.150:0/1513976140 shutdown_connections 2024-03-17T23:47:27.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:27.256+0000 7fb120d60640 1 -- 172.21.15.150:0/1513976140 wait complete. 2024-03-17T23:47:27.268 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-17T23:47:27.268 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-17T23:47:27.268 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi157.front.sepia.ceph.com 2024-03-17T23:47:27.268 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-17T23:47:27.268 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-17T23:47:27.268 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-17T23:47:27.268 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-17T23:47:27.268 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-17T23:47:27.268 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-17T23:47:27.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:27.268 DEBUG:teuthology.orchestra.run.smithi157:> ip addr 2024-03-17T23:47:27.283 INFO:teuthology.orchestra.run.smithi157.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-17T23:47:27.283 INFO:teuthology.orchestra.run.smithi157.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-17T23:47:27.283 INFO:teuthology.orchestra.run.smithi157.stdout: inet 127.0.0.1/8 scope host lo 2024-03-17T23:47:27.284 INFO:teuthology.orchestra.run.smithi157.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:27.284 INFO:teuthology.orchestra.run.smithi157.stdout: inet6 ::1/128 scope host 2024-03-17T23:47:27.284 INFO:teuthology.orchestra.run.smithi157.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:27.284 INFO:teuthology.orchestra.run.smithi157.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:47:27.284 INFO:teuthology.orchestra.run.smithi157.stdout: link/ether 0c:c4:7a:d9:37:98 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:47:27.284 INFO:teuthology.orchestra.run.smithi157.stdout: altname enp7s0f0 2024-03-17T23:47:27.284 INFO:teuthology.orchestra.run.smithi157.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:47:27.284 INFO:teuthology.orchestra.run.smithi157.stdout: link/ether 0c:c4:7a:d9:37:99 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:47:27.284 INFO:teuthology.orchestra.run.smithi157.stdout: altname enp7s0f1 2024-03-17T23:47:27.284 INFO:teuthology.orchestra.run.smithi157.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:47:27.285 INFO:teuthology.orchestra.run.smithi157.stdout: link/ether 0c:c4:7a:88:7e:a0 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:47:27.285 INFO:teuthology.orchestra.run.smithi157.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-17T23:47:27.285 INFO:teuthology.orchestra.run.smithi157.stdout: link/ether 0c:c4:7a:88:7e:a1 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:47:27.285 INFO:teuthology.orchestra.run.smithi157.stdout: inet 172.21.15.157/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-17T23:47:27.285 INFO:teuthology.orchestra.run.smithi157.stdout: valid_lft 42417sec preferred_lft 42417sec 2024-03-17T23:47:27.285 INFO:teuthology.orchestra.run.smithi157.stdout: inet6 fe80::ec4:7aff:fe88:7ea1/64 scope link 2024-03-17T23:47:27.285 INFO:teuthology.orchestra.run.smithi157.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:27.286 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-17T23:47:27.286 DEBUG:teuthology.orchestra.run.smithi157:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:47:27.286 DEBUG:teuthology.orchestra.run.smithi157:> set -e 2024-03-17T23:47:27.286 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip link add name ceph-brx type bridge 2024-03-17T23:47:27.286 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip addr flush dev ceph-brx 2024-03-17T23:47:27.286 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip link set ceph-brx up 2024-03-17T23:47:27.286 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-17T23:47:27.287 DEBUG:teuthology.orchestra.run.smithi157:> ') 2024-03-17T23:47:27.376 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:47:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:47:27.478 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:47:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:27.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:27.485 DEBUG:teuthology.orchestra.run.smithi157:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-17T23:47:27.513 INFO:teuthology.orchestra.run.smithi157.stdout:1 2024-03-17T23:47:27.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:27.515 DEBUG:teuthology.orchestra.run.smithi157:> route 2024-03-17T23:47:27.599 INFO:teuthology.orchestra.run.smithi157.stdout:Kernel IP routing table 2024-03-17T23:47:27.600 INFO:teuthology.orchestra.run.smithi157.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-17T23:47:27.600 INFO:teuthology.orchestra.run.smithi157.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-17T23:47:27.600 INFO:teuthology.orchestra.run.smithi157.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-17T23:47:27.600 INFO:teuthology.orchestra.run.smithi157.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-17T23:47:27.600 DEBUG:teuthology.orchestra.run.smithi157:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:47:27.601 DEBUG:teuthology.orchestra.run.smithi157:> set -e 2024-03-17T23:47:27.601 DEBUG:teuthology.orchestra.run.smithi157:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-17T23:47:27.601 DEBUG:teuthology.orchestra.run.smithi157:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-17T23:47:27.601 DEBUG:teuthology.orchestra.run.smithi157:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-17T23:47:27.601 DEBUG:teuthology.orchestra.run.smithi157:> ') 2024-03-17T23:47:27.687 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:47:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:47:27.926 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:47:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:27.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:27.934 DEBUG:teuthology.orchestra.run.smithi157:> ip netns list 2024-03-17T23:47:27.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:27.954 DEBUG:teuthology.orchestra.run.smithi157:> ip netns list-id 2024-03-17T23:47:28.009 DEBUG:teuthology.orchestra.run.smithi157:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:47:28.009 DEBUG:teuthology.orchestra.run.smithi157:> set -e 2024-03-17T23:47:28.009 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-17T23:47:28.009 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-17T23:47:28.009 DEBUG:teuthology.orchestra.run.smithi157:> ') 2024-03-17T23:47:28.099 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:47:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:47:28.128 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:47:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:28.134 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-17T23:47:28.134 DEBUG:teuthology.orchestra.run.smithi157:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:47:28.134 DEBUG:teuthology.orchestra.run.smithi157:> set -e 2024-03-17T23:47:28.135 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-17T23:47:28.135 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-17T23:47:28.135 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-17T23:47:28.135 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-17T23:47:28.135 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-17T23:47:28.135 DEBUG:teuthology.orchestra.run.smithi157:> ') 2024-03-17T23:47:28.183 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:47:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:47:28.287 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:47:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:28.294 DEBUG:teuthology.orchestra.run.smithi157:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:47:28.294 DEBUG:teuthology.orchestra.run.smithi157:> set -e 2024-03-17T23:47:28.294 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip link set brx.0 up 2024-03-17T23:47:28.294 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip link set dev brx.0 master ceph-brx 2024-03-17T23:47:28.294 DEBUG:teuthology.orchestra.run.smithi157:> ') 2024-03-17T23:47:28.344 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:47:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:47:28.375 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:47:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:28.383 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-03-17T23:47:28.384 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-17T23:47:28.384 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-17T23:47:28.409 INFO:teuthology.orchestra.run.smithi157.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-17T23:47:28.409 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-17T23:47:28.409 DEBUG:teuthology.orchestra.run.smithi157:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-17T23:47:28.464 DEBUG:teuthology.orchestra.run.smithi157:> sudo modprobe fuse 2024-03-17T23:47:28.533 DEBUG:teuthology.orchestra.run.smithi157:> cat /proc/self/mounts | awk '{print $2}' 2024-03-17T23:47:28.589 INFO:teuthology.orchestra.run.smithi157.stdout:/proc 2024-03-17T23:47:28.589 INFO:teuthology.orchestra.run.smithi157.stdout:/sys 2024-03-17T23:47:28.589 INFO:teuthology.orchestra.run.smithi157.stdout:/dev 2024-03-17T23:47:28.589 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/kernel/security 2024-03-17T23:47:28.589 INFO:teuthology.orchestra.run.smithi157.stdout:/dev/shm 2024-03-17T23:47:28.589 INFO:teuthology.orchestra.run.smithi157.stdout:/dev/pts 2024-03-17T23:47:28.589 INFO:teuthology.orchestra.run.smithi157.stdout:/run 2024-03-17T23:47:28.589 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/fs/cgroup 2024-03-17T23:47:28.589 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/fs/pstore 2024-03-17T23:47:28.589 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/fs/bpf 2024-03-17T23:47:28.589 INFO:teuthology.orchestra.run.smithi157.stdout:/ 2024-03-17T23:47:28.589 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/fs/selinux 2024-03-17T23:47:28.590 INFO:teuthology.orchestra.run.smithi157.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:47:28.590 INFO:teuthology.orchestra.run.smithi157.stdout:/dev/mqueue 2024-03-17T23:47:28.590 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/kernel/debug 2024-03-17T23:47:28.590 INFO:teuthology.orchestra.run.smithi157.stdout:/dev/hugepages 2024-03-17T23:47:28.590 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/kernel/tracing 2024-03-17T23:47:28.590 INFO:teuthology.orchestra.run.smithi157.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-17T23:47:28.590 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/fs/fuse/connections 2024-03-17T23:47:28.590 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/kernel/config 2024-03-17T23:47:28.590 INFO:teuthology.orchestra.run.smithi157.stdout:/run/credentials/systemd-sysctl.service 2024-03-17T23:47:28.590 INFO:teuthology.orchestra.run.smithi157.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-17T23:47:28.590 INFO:teuthology.orchestra.run.smithi157.stdout:/var/lib/nfs/rpc_pipefs 2024-03-17T23:47:28.590 INFO:teuthology.orchestra.run.smithi157.stdout:/run/user/1000 2024-03-17T23:47:28.590 INFO:teuthology.orchestra.run.smithi157.stdout:/var/lib/ceph 2024-03-17T23:47:28.591 INFO:teuthology.orchestra.run.smithi157.stdout:/var/lib/ceph/osd/ceph-4 2024-03-17T23:47:28.591 INFO:teuthology.orchestra.run.smithi157.stdout:/var/lib/ceph/osd/ceph-5 2024-03-17T23:47:28.591 INFO:teuthology.orchestra.run.smithi157.stdout:/var/lib/ceph/osd/ceph-6 2024-03-17T23:47:28.591 INFO:teuthology.orchestra.run.smithi157.stdout:/var/lib/ceph/osd/ceph-7 2024-03-17T23:47:28.591 INFO:teuthology.orchestra.run.smithi157.stdout:/run/netns 2024-03-17T23:47:28.591 INFO:teuthology.orchestra.run.smithi157.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-17T23:47:28.591 INFO:teuthology.orchestra.run.smithi157.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-17T23:47:28.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:28.591 DEBUG:teuthology.orchestra.run.smithi157:> ls /sys/fs/fuse/connections 2024-03-17T23:47:28.645 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-17T23:47:28.645 DEBUG:teuthology.orchestra.run.smithi157:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-17T23:47:28.688 DEBUG:teuthology.orchestra.run.smithi157:> sudo modprobe fuse 2024-03-17T23:47:28.714 DEBUG:teuthology.orchestra.run.smithi157:> cat /proc/self/mounts | awk '{print $2}' 2024-03-17T23:47:28.773 INFO:teuthology.orchestra.run.smithi157.stdout:/proc 2024-03-17T23:47:28.773 INFO:teuthology.orchestra.run.smithi157.stdout:/sys 2024-03-17T23:47:28.773 INFO:teuthology.orchestra.run.smithi157.stdout:/dev 2024-03-17T23:47:28.773 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/kernel/security 2024-03-17T23:47:28.774 INFO:teuthology.orchestra.run.smithi157.stdout:/dev/shm 2024-03-17T23:47:28.774 INFO:teuthology.orchestra.run.smithi157.stdout:/dev/pts 2024-03-17T23:47:28.774 INFO:teuthology.orchestra.run.smithi157.stdout:/run 2024-03-17T23:47:28.774 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/fs/cgroup 2024-03-17T23:47:28.774 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/fs/pstore 2024-03-17T23:47:28.774 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/fs/bpf 2024-03-17T23:47:28.774 INFO:teuthology.orchestra.run.smithi157.stdout:/ 2024-03-17T23:47:28.774 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/fs/selinux 2024-03-17T23:47:28.774 INFO:teuthology.orchestra.run.smithi157.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:47:28.774 INFO:teuthology.orchestra.run.smithi157.stdout:/dev/mqueue 2024-03-17T23:47:28.774 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/kernel/debug 2024-03-17T23:47:28.774 INFO:teuthology.orchestra.run.smithi157.stdout:/dev/hugepages 2024-03-17T23:47:28.774 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/kernel/tracing 2024-03-17T23:47:28.775 INFO:teuthology.orchestra.run.smithi157.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-17T23:47:28.775 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/fs/fuse/connections 2024-03-17T23:47:28.775 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/kernel/config 2024-03-17T23:47:28.775 INFO:teuthology.orchestra.run.smithi157.stdout:/run/credentials/systemd-sysctl.service 2024-03-17T23:47:28.775 INFO:teuthology.orchestra.run.smithi157.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-17T23:47:28.775 INFO:teuthology.orchestra.run.smithi157.stdout:/var/lib/nfs/rpc_pipefs 2024-03-17T23:47:28.775 INFO:teuthology.orchestra.run.smithi157.stdout:/run/user/1000 2024-03-17T23:47:28.775 INFO:teuthology.orchestra.run.smithi157.stdout:/var/lib/ceph 2024-03-17T23:47:28.775 INFO:teuthology.orchestra.run.smithi157.stdout:/var/lib/ceph/osd/ceph-4 2024-03-17T23:47:28.775 INFO:teuthology.orchestra.run.smithi157.stdout:/var/lib/ceph/osd/ceph-5 2024-03-17T23:47:28.775 INFO:teuthology.orchestra.run.smithi157.stdout:/var/lib/ceph/osd/ceph-6 2024-03-17T23:47:28.775 INFO:teuthology.orchestra.run.smithi157.stdout:/var/lib/ceph/osd/ceph-7 2024-03-17T23:47:28.775 INFO:teuthology.orchestra.run.smithi157.stdout:/run/netns 2024-03-17T23:47:28.775 INFO:teuthology.orchestra.run.smithi157.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-17T23:47:28.776 INFO:teuthology.orchestra.run.smithi157.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-17T23:47:28.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:28.776 DEBUG:teuthology.orchestra.run.smithi157:> ls /sys/fs/fuse/connections 2024-03-17T23:47:28.777 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi157.stderr:2024-03-17T23:47:28.776+0000 7ff15c635480 -1 init, newargv = 0x55bba8020f20 newargc=15 2024-03-17T23:47:28.777 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi157.stderr:ceph-fuse[43322]: starting ceph client 2024-03-17T23:47:28.783 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi157.stderr:ceph-fuse[43322]: starting fuse 2024-03-17T23:47:28.792 INFO:teuthology.orchestra.run.smithi157.stdout:42 2024-03-17T23:47:28.793 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-03-17T23:47:28.793 DEBUG:teuthology.orchestra.run.smithi157:> sudo stdin-killer -- python3 -c ' 2024-03-17T23:47:28.793 DEBUG:teuthology.orchestra.run.smithi157:> import glob 2024-03-17T23:47:28.794 DEBUG:teuthology.orchestra.run.smithi157:> import re 2024-03-17T23:47:28.794 DEBUG:teuthology.orchestra.run.smithi157:> import os 2024-03-17T23:47:28.794 DEBUG:teuthology.orchestra.run.smithi157:> import subprocess 2024-03-17T23:47:28.794 DEBUG:teuthology.orchestra.run.smithi157:> 2024-03-17T23:47:28.794 DEBUG:teuthology.orchestra.run.smithi157:> def _find_admin_socket(client_name): 2024-03-17T23:47:28.794 DEBUG:teuthology.orchestra.run.smithi157:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-17T23:47:28.794 DEBUG:teuthology.orchestra.run.smithi157:> files = glob.glob(asok_path) 2024-03-17T23:47:28.794 DEBUG:teuthology.orchestra.run.smithi157:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-17T23:47:28.794 DEBUG:teuthology.orchestra.run.smithi157:> 2024-03-17T23:47:28.795 DEBUG:teuthology.orchestra.run.smithi157:> # Given a non-glob path, it better be there 2024-03-17T23:47:28.795 DEBUG:teuthology.orchestra.run.smithi157:> if "*" not in asok_path: 2024-03-17T23:47:28.795 DEBUG:teuthology.orchestra.run.smithi157:> assert(len(files) == 1) 2024-03-17T23:47:28.795 DEBUG:teuthology.orchestra.run.smithi157:> return files[0] 2024-03-17T23:47:28.795 DEBUG:teuthology.orchestra.run.smithi157:> 2024-03-17T23:47:28.795 DEBUG:teuthology.orchestra.run.smithi157:> for f in files: 2024-03-17T23:47:28.795 DEBUG:teuthology.orchestra.run.smithi157:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-17T23:47:28.795 DEBUG:teuthology.orchestra.run.smithi157:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-17T23:47:28.795 DEBUG:teuthology.orchestra.run.smithi157:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-17T23:47:28.795 DEBUG:teuthology.orchestra.run.smithi157:> contents = proc_f.read() 2024-03-17T23:47:28.795 DEBUG:teuthology.orchestra.run.smithi157:> if mountpoint in contents: 2024-03-17T23:47:28.795 DEBUG:teuthology.orchestra.run.smithi157:> return f 2024-03-17T23:47:28.795 DEBUG:teuthology.orchestra.run.smithi157:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-17T23:47:28.795 DEBUG:teuthology.orchestra.run.smithi157:> 2024-03-17T23:47:28.795 DEBUG:teuthology.orchestra.run.smithi157:> print(_find_admin_socket("client.1")) 2024-03-17T23:47:28.796 DEBUG:teuthology.orchestra.run.smithi157:> ' 2024-03-17T23:47:28.913 INFO:teuthology.orchestra.run.smithi157.stdout:/var/run/ceph/ceph-client.1.43322.asok 2024-03-17T23:47:28.916 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:47:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:28.923 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.43322.asok 2024-03-17T23:47:28.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:28.923 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.43322.asok status 2024-03-17T23:47:29.062 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-03-17T23:47:29.095 INFO:teuthology.orchestra.run.smithi157.stdout: "metadata": { 2024-03-17T23:47:29.095 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-17T23:47:29.095 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-17T23:47:29.095 INFO:teuthology.orchestra.run.smithi157.stdout: "entity_id": "1", 2024-03-17T23:47:29.095 INFO:teuthology.orchestra.run.smithi157.stdout: "hostname": "smithi157", 2024-03-17T23:47:29.096 INFO:teuthology.orchestra.run.smithi157.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-17T23:47:29.096 INFO:teuthology.orchestra.run.smithi157.stdout: "pid": "43322", 2024-03-17T23:47:29.096 INFO:teuthology.orchestra.run.smithi157.stdout: "root": "/" 2024-03-17T23:47:29.096 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-03-17T23:47:29.096 INFO:teuthology.orchestra.run.smithi157.stdout: "dentry_count": 0, 2024-03-17T23:47:29.096 INFO:teuthology.orchestra.run.smithi157.stdout: "dentry_pinned_count": 0, 2024-03-17T23:47:29.096 INFO:teuthology.orchestra.run.smithi157.stdout: "id": 4697, 2024-03-17T23:47:29.096 INFO:teuthology.orchestra.run.smithi157.stdout: "inst": { 2024-03-17T23:47:29.096 INFO:teuthology.orchestra.run.smithi157.stdout: "name": { 2024-03-17T23:47:29.097 INFO:teuthology.orchestra.run.smithi157.stdout: "type": "client", 2024-03-17T23:47:29.097 INFO:teuthology.orchestra.run.smithi157.stdout: "num": 4697 2024-03-17T23:47:29.097 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-03-17T23:47:29.097 INFO:teuthology.orchestra.run.smithi157.stdout: "addr": { 2024-03-17T23:47:29.097 INFO:teuthology.orchestra.run.smithi157.stdout: "type": "v1", 2024-03-17T23:47:29.097 INFO:teuthology.orchestra.run.smithi157.stdout: "addr": "192.168.0.1:0", 2024-03-17T23:47:29.097 INFO:teuthology.orchestra.run.smithi157.stdout: "nonce": 1950132301 2024-03-17T23:47:29.097 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-03-17T23:47:29.097 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-03-17T23:47:29.097 INFO:teuthology.orchestra.run.smithi157.stdout: "addr": { 2024-03-17T23:47:29.097 INFO:teuthology.orchestra.run.smithi157.stdout: "type": "v1", 2024-03-17T23:47:29.097 INFO:teuthology.orchestra.run.smithi157.stdout: "addr": "192.168.0.1:0", 2024-03-17T23:47:29.097 INFO:teuthology.orchestra.run.smithi157.stdout: "nonce": 1950132301 2024-03-17T23:47:29.097 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-03-17T23:47:29.098 INFO:teuthology.orchestra.run.smithi157.stdout: "inst_str": "client.4697 192.168.0.1:0/1950132301", 2024-03-17T23:47:29.098 INFO:teuthology.orchestra.run.smithi157.stdout: "addr_str": "192.168.0.1:0/1950132301", 2024-03-17T23:47:29.098 INFO:teuthology.orchestra.run.smithi157.stdout: "inode_count": 1, 2024-03-17T23:47:29.098 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_epoch": 8, 2024-03-17T23:47:29.098 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_epoch": 26, 2024-03-17T23:47:29.098 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_epoch_barrier": 0, 2024-03-17T23:47:29.098 INFO:teuthology.orchestra.run.smithi157.stdout: "blocklisted": false, 2024-03-17T23:47:29.098 INFO:teuthology.orchestra.run.smithi157.stdout: "fs_name": "cephfs" 2024-03-17T23:47:29.098 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-03-17T23:47:29.100 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-17T23:47:29.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.214+0000 7fe8889a9640 1 Processor -- start 2024-03-17T23:47:29.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.214+0000 7fe8889a9640 1 -- start start 2024-03-17T23:47:29.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.215+0000 7fe8889a9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe88415de30 0x7fe8841619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:29.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.215+0000 7fe8889a9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe8840cf120 0x7fe8840d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:29.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.215+0000 7fe8889a9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8840b9cf0 con 0x7fe8840b9860 2024-03-17T23:47:29.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.215+0000 7fe8889a9640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fe884162940 con 0x7fe88415de30 2024-03-17T23:47:29.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.215+0000 7fe8889a9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe8841620a0 con 0x7fe8840cf120 2024-03-17T23:47:29.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.215+0000 7fe881d9b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fe8840b9860 0x7fe8840cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33666/0 (socket says 172.21.15.150:33666) 2024-03-17T23:47:29.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.215+0000 7fe881d9b640 1 -- 172.21.15.150:0/4149254169 learned_addr learned my addr 172.21.15.150:0/4149254169 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:29.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.215+0000 7fe882d9d640 1 --2- 172.21.15.150:0/4149254169 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe8840cf120 0x7fe8840d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:29.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.215+0000 7fe88259c640 1 --2- 172.21.15.150:0/4149254169 >> v2:172.21.15.157:3300/0 conn(0x7fe88415de30 0x7fe8841619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:29.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.215+0000 7fe88159a640 1 -- 172.21.15.150:0/4149254169 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1662101484 0 0) 0x7fe8840b9cf0 con 0x7fe8840b9860 2024-03-17T23:47:29.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.215+0000 7fe88159a640 1 -- 172.21.15.150:0/4149254169 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe86c003520 con 0x7fe8840b9860 2024-03-17T23:47:29.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.215+0000 7fe882d9d640 1 -- 172.21.15.150:0/4149254169 >> v2:172.21.15.157:3300/0 conn(0x7fe88415de30 msgr2=0x7fe8841619a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:29.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.215+0000 7fe882d9d640 1 --2- 172.21.15.150:0/4149254169 >> v2:172.21.15.157:3300/0 conn(0x7fe88415de30 0x7fe8841619a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:29.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.216+0000 7fe882d9d640 1 -- 172.21.15.150:0/4149254169 >> v1:172.21.15.150:6789/0 conn(0x7fe8840b9860 legacy=0x7fe8840cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:29.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.216+0000 7fe882d9d640 1 -- 172.21.15.150:0/4149254169 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe884162380 con 0x7fe8840cf120 2024-03-17T23:47:29.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.216+0000 7fe882d9d640 1 --2- 172.21.15.150:0/4149254169 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe8840cf120 0x7fe8840d2690 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009d90 tx=0x7fe87c02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9e16426516a169c server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:29.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.216+0000 7fe88159a640 1 -- 172.21.15.150:0/4149254169 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87c039070 con 0x7fe8840cf120 2024-03-17T23:47:29.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.217+0000 7fe88159a640 1 -- 172.21.15.150:0/4149254169 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe87c02cbc0 con 0x7fe8840cf120 2024-03-17T23:47:29.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.217+0000 7fe88159a640 1 -- 172.21.15.150:0/4149254169 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87c034440 con 0x7fe8840cf120 2024-03-17T23:47:29.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.217+0000 7fe8889a9640 1 -- 172.21.15.150:0/4149254169 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe8840cf120 msgr2=0x7fe8840d2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:29.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.217+0000 7fe8889a9640 1 --2- 172.21.15.150:0/4149254169 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe8840cf120 0x7fe8840d2690 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009d90 tx=0x7fe87c02be10 comp rx=0 tx=0).stop 2024-03-17T23:47:29.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.217+0000 7fe8889a9640 1 -- 172.21.15.150:0/4149254169 shutdown_connections 2024-03-17T23:47:29.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.217+0000 7fe8889a9640 1 --2- 172.21.15.150:0/4149254169 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe8840cf120 0x7fe8840d2690 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:29.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.217+0000 7fe8889a9640 1 --2- 172.21.15.150:0/4149254169 >> v2:172.21.15.157:3300/0 conn(0x7fe88415de30 0x7fe8841619a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:29.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.217+0000 7fe8889a9640 1 -- 172.21.15.150:0/4149254169 >> 172.21.15.150:0/4149254169 conn(0x7fe8840730b0 msgr2=0x7fe8840d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:29.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.217+0000 7fe8889a9640 1 -- 172.21.15.150:0/4149254169 shutdown_connections 2024-03-17T23:47:29.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.217+0000 7fe8889a9640 1 -- 172.21.15.150:0/4149254169 wait complete. 2024-03-17T23:47:29.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.218+0000 7fe8889a9640 1 Processor -- start 2024-03-17T23:47:29.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.218+0000 7fe8889a9640 1 -- start start 2024-03-17T23:47:29.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.219+0000 7fe8889a9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe8840b9860 0x7fe884155160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:29.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.219+0000 7fe8889a9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8840cf120 0x7fe8841588c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:29.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.219+0000 7fe8889a9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe88415de30 0x7fe88415fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:29.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.219+0000 7fe8889a9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe88415e210 con 0x7fe8840cf120 2024-03-17T23:47:29.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.219+0000 7fe8889a9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe8840d2170 con 0x7fe88415de30 2024-03-17T23:47:29.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.219+0000 7fe8889a9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe8840d2bd0 con 0x7fe8840b9860 2024-03-17T23:47:29.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.219+0000 7fe881d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8840cf120 0x7fe8841588c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:29.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.219+0000 7fe881d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8840cf120 0x7fe8841588c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46020/0 (socket says 172.21.15.150:46020) 2024-03-17T23:47:29.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.219+0000 7fe881d9b640 1 -- 172.21.15.150:0/4229498871 learned_addr learned my addr 172.21.15.150:0/4229498871 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:29.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.219+0000 7fe88259c640 1 --2- 172.21.15.150:0/4229498871 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe8840b9860 0x7fe884155160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:29.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.219+0000 7fe882d9d640 1 --2- 172.21.15.150:0/4229498871 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe88415de30 0x7fe88415fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:29.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.220+0000 7fe881d9b640 1 -- 172.21.15.150:0/4229498871 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe8840b9860 msgr2=0x7fe884155160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:29.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.220+0000 7fe881d9b640 1 --2- 172.21.15.150:0/4229498871 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe8840b9860 0x7fe884155160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:29.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.220+0000 7fe881d9b640 1 -- 172.21.15.150:0/4229498871 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe88415de30 msgr2=0x7fe88415fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:29.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.220+0000 7fe881d9b640 1 --2- 172.21.15.150:0/4229498871 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe88415de30 0x7fe88415fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:29.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.220+0000 7fe881d9b640 1 -- 172.21.15.150:0/4229498871 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe88415e9a0 con 0x7fe8840cf120 2024-03-17T23:47:29.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.220+0000 7fe882d9d640 1 --2- 172.21.15.150:0/4229498871 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe88415de30 0x7fe88415fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:29.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.220+0000 7fe881d9b640 1 --2- 172.21.15.150:0/4229498871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8840cf120 0x7fe8841588c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe87800c430 tx=0x7fe87800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:29.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.220+0000 7fe872ffd640 1 -- 172.21.15.150:0/4229498871 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe878017070 con 0x7fe8840cf120 2024-03-17T23:47:29.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.220+0000 7fe872ffd640 1 -- 172.21.15.150:0/4229498871 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe878005ce0 con 0x7fe8840cf120 2024-03-17T23:47:29.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.220+0000 7fe872ffd640 1 -- 172.21.15.150:0/4229498871 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8780072f0 con 0x7fe8840cf120 2024-03-17T23:47:29.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.220+0000 7fe8889a9640 1 -- 172.21.15.150:0/4229498871 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe88415ece0 con 0x7fe8840cf120 2024-03-17T23:47:29.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.220+0000 7fe8889a9640 1 -- 172.21.15.150:0/4229498871 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe88415f520 con 0x7fe8840cf120 2024-03-17T23:47:29.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.221+0000 7fe8889a9640 1 -- 172.21.15.150:0/4229498871 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8840681a0 con 0x7fe8840cf120 2024-03-17T23:47:29.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.222+0000 7fe872ffd640 1 -- 172.21.15.150:0/4229498871 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe87801a070 con 0x7fe8840cf120 2024-03-17T23:47:29.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.222+0000 7fe872ffd640 1 --2- 172.21.15.150:0/4229498871 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe86406bbd0 0x7fe86406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:29.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.222+0000 7fe872ffd640 1 -- 172.21.15.150:0/4229498871 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fe87808a240 con 0x7fe8840cf120 2024-03-17T23:47:29.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.222+0000 7fe88259c640 1 --2- 172.21.15.150:0/4229498871 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe86406bbd0 0x7fe86406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:29.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.222+0000 7fe88259c640 1 --2- 172.21.15.150:0/4229498871 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe86406bbd0 0x7fe86406e090 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe874009aa0 tx=0x7fe874009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:29.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.226+0000 7fe872ffd640 1 -- 172.21.15.150:0/4229498871 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe88415ece0 con 0x7fe8840cf120 2024-03-17T23:47:29.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.437+0000 7fe8889a9640 1 -- 172.21.15.150:0/4229498871 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe884161bb0 con 0x7fe8840cf120 2024-03-17T23:47:29.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.438+0000 7fe872ffd640 1 -- 172.21.15.150:0/4229498871 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7fe878058320 con 0x7fe8840cf120 2024-03-17T23:47:29.438 INFO:teuthology.orchestra.run.smithi150.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-17T23:47:29.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.440+0000 7fe8889a9640 1 -- 172.21.15.150:0/4229498871 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe86406bbd0 msgr2=0x7fe86406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:29.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.440+0000 7fe8889a9640 1 --2- 172.21.15.150:0/4229498871 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe86406bbd0 0x7fe86406e090 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe874009aa0 tx=0x7fe874009190 comp rx=0 tx=0).stop 2024-03-17T23:47:29.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.440+0000 7fe8889a9640 1 -- 172.21.15.150:0/4229498871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8840cf120 msgr2=0x7fe8841588c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:29.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.440+0000 7fe8889a9640 1 --2- 172.21.15.150:0/4229498871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8840cf120 0x7fe8841588c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe87800c430 tx=0x7fe87800c8f0 comp rx=0 tx=0).stop 2024-03-17T23:47:29.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.440+0000 7fe8889a9640 1 -- 172.21.15.150:0/4229498871 shutdown_connections 2024-03-17T23:47:29.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.440+0000 7fe8889a9640 1 --2- 172.21.15.150:0/4229498871 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe86406bbd0 0x7fe86406e090 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:29.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.440+0000 7fe8889a9640 1 --2- 172.21.15.150:0/4229498871 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe88415de30 0x7fe88415fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:29.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.440+0000 7fe8889a9640 1 --2- 172.21.15.150:0/4229498871 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8840cf120 0x7fe8841588c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:29.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.440+0000 7fe8889a9640 1 --2- 172.21.15.150:0/4229498871 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe8840b9860 0x7fe884155160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:29.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.440+0000 7fe8889a9640 1 -- 172.21.15.150:0/4229498871 >> 172.21.15.150:0/4229498871 conn(0x7fe8840730b0 msgr2=0x7fe8840d3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:29.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.440+0000 7fe8889a9640 1 -- 172.21.15.150:0/4229498871 shutdown_connections 2024-03-17T23:47:29.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:29.440+0000 7fe8889a9640 1 -- 172.21.15.150:0/4229498871 wait complete. 2024-03-17T23:47:29.451 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-17T23:47:29.451 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-17T23:47:29.451 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi157.front.sepia.ceph.com 2024-03-17T23:47:29.452 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-17T23:47:29.452 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-17T23:47:29.452 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-17T23:47:29.452 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-17T23:47:29.452 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-17T23:47:29.452 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-17T23:47:29.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:29.452 DEBUG:teuthology.orchestra.run.smithi157:> ip addr 2024-03-17T23:47:29.468 INFO:teuthology.orchestra.run.smithi157.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-17T23:47:29.468 INFO:teuthology.orchestra.run.smithi157.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-17T23:47:29.468 INFO:teuthology.orchestra.run.smithi157.stdout: inet 127.0.0.1/8 scope host lo 2024-03-17T23:47:29.468 INFO:teuthology.orchestra.run.smithi157.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:29.468 INFO:teuthology.orchestra.run.smithi157.stdout: inet6 ::1/128 scope host 2024-03-17T23:47:29.468 INFO:teuthology.orchestra.run.smithi157.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:29.468 INFO:teuthology.orchestra.run.smithi157.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:47:29.468 INFO:teuthology.orchestra.run.smithi157.stdout: link/ether 0c:c4:7a:d9:37:98 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:47:29.468 INFO:teuthology.orchestra.run.smithi157.stdout: altname enp7s0f0 2024-03-17T23:47:29.468 INFO:teuthology.orchestra.run.smithi157.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:47:29.468 INFO:teuthology.orchestra.run.smithi157.stdout: link/ether 0c:c4:7a:d9:37:99 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:47:29.468 INFO:teuthology.orchestra.run.smithi157.stdout: altname enp7s0f1 2024-03-17T23:47:29.468 INFO:teuthology.orchestra.run.smithi157.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:47:29.469 INFO:teuthology.orchestra.run.smithi157.stdout: link/ether 0c:c4:7a:88:7e:a0 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:47:29.469 INFO:teuthology.orchestra.run.smithi157.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-17T23:47:29.469 INFO:teuthology.orchestra.run.smithi157.stdout: link/ether 0c:c4:7a:88:7e:a1 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:47:29.469 INFO:teuthology.orchestra.run.smithi157.stdout: inet 172.21.15.157/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-17T23:47:29.469 INFO:teuthology.orchestra.run.smithi157.stdout: valid_lft 42415sec preferred_lft 42415sec 2024-03-17T23:47:29.469 INFO:teuthology.orchestra.run.smithi157.stdout: inet6 fe80::ec4:7aff:fe88:7ea1/64 scope link 2024-03-17T23:47:29.469 INFO:teuthology.orchestra.run.smithi157.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:29.469 INFO:teuthology.orchestra.run.smithi157.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-17T23:47:29.469 INFO:teuthology.orchestra.run.smithi157.stdout: link/ether 1a:4e:4e:b1:dd:6c brd ff:ff:ff:ff:ff:ff 2024-03-17T23:47:29.469 INFO:teuthology.orchestra.run.smithi157.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-17T23:47:29.469 INFO:teuthology.orchestra.run.smithi157.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:29.469 INFO:teuthology.orchestra.run.smithi157.stdout: inet6 fe80::485d:b6ff:feea:14c3/64 scope link 2024-03-17T23:47:29.469 INFO:teuthology.orchestra.run.smithi157.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:29.469 INFO:teuthology.orchestra.run.smithi157.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-17T23:47:29.469 INFO:teuthology.orchestra.run.smithi157.stdout: link/ether 1a:4e:4e:b1:dd:6c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-17T23:47:29.470 INFO:teuthology.orchestra.run.smithi157.stdout: inet6 fe80::184e:4eff:feb1:dd6c/64 scope link tentative 2024-03-17T23:47:29.470 INFO:teuthology.orchestra.run.smithi157.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:29.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:29.470 DEBUG:teuthology.orchestra.run.smithi157:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-17T23:47:29.534 INFO:teuthology.orchestra.run.smithi157.stdout:1 2024-03-17T23:47:29.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:29.535 DEBUG:teuthology.orchestra.run.smithi157:> route 2024-03-17T23:47:29.592 INFO:teuthology.orchestra.run.smithi157.stdout:Kernel IP routing table 2024-03-17T23:47:29.592 INFO:teuthology.orchestra.run.smithi157.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-17T23:47:29.592 INFO:teuthology.orchestra.run.smithi157.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-17T23:47:29.592 INFO:teuthology.orchestra.run.smithi157.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-17T23:47:29.593 INFO:teuthology.orchestra.run.smithi157.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-17T23:47:29.593 DEBUG:teuthology.orchestra.run.smithi157:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:47:29.593 DEBUG:teuthology.orchestra.run.smithi157:> set -e 2024-03-17T23:47:29.593 DEBUG:teuthology.orchestra.run.smithi157:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-17T23:47:29.593 DEBUG:teuthology.orchestra.run.smithi157:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-17T23:47:29.593 DEBUG:teuthology.orchestra.run.smithi157:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-17T23:47:29.593 DEBUG:teuthology.orchestra.run.smithi157:> ') 2024-03-17T23:47:29.681 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:47:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:47:29.722 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:47:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:29.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:29.730 DEBUG:teuthology.orchestra.run.smithi157:> ip netns list 2024-03-17T23:47:29.746 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-17T23:47:29.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:29.747 DEBUG:teuthology.orchestra.run.smithi157:> ip netns list-id 2024-03-17T23:47:29.801 INFO:teuthology.orchestra.run.smithi157.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-17T23:47:29.801 DEBUG:teuthology.orchestra.run.smithi157:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:47:29.801 DEBUG:teuthology.orchestra.run.smithi157:> set -e 2024-03-17T23:47:29.801 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-17T23:47:29.802 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-17T23:47:29.802 DEBUG:teuthology.orchestra.run.smithi157:> ') 2024-03-17T23:47:29.889 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:47:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:47:29.917 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:47:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:29.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:29.924 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-17T23:47:29.951 INFO:teuthology.orchestra.run.smithi157.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-17T23:47:29.951 INFO:teuthology.orchestra.run.smithi157.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-17T23:47:29.951 INFO:teuthology.orchestra.run.smithi157.stdout: inet 127.0.0.1/8 scope host lo 2024-03-17T23:47:29.951 INFO:teuthology.orchestra.run.smithi157.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:29.951 INFO:teuthology.orchestra.run.smithi157.stdout: inet6 ::1/128 scope host 2024-03-17T23:47:29.951 INFO:teuthology.orchestra.run.smithi157.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:29.951 INFO:teuthology.orchestra.run.smithi157.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-17T23:47:29.951 INFO:teuthology.orchestra.run.smithi157.stdout: link/ether 12:e3:10:6e:d2:51 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-17T23:47:29.951 INFO:teuthology.orchestra.run.smithi157.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-17T23:47:29.951 INFO:teuthology.orchestra.run.smithi157.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:29.952 INFO:teuthology.orchestra.run.smithi157.stdout: inet6 fe80::10e3:10ff:fe6e:d251/64 scope link 2024-03-17T23:47:29.952 INFO:teuthology.orchestra.run.smithi157.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:29.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:29.953 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-17T23:47:30.021 INFO:teuthology.orchestra.run.smithi157.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-17T23:47:30.021 INFO:teuthology.orchestra.run.smithi157.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-17T23:47:30.021 INFO:teuthology.orchestra.run.smithi157.stdout: inet 127.0.0.1/8 scope host lo 2024-03-17T23:47:30.021 INFO:teuthology.orchestra.run.smithi157.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:30.021 INFO:teuthology.orchestra.run.smithi157.stdout: inet6 ::1/128 scope host 2024-03-17T23:47:30.021 INFO:teuthology.orchestra.run.smithi157.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:30.021 INFO:teuthology.orchestra.run.smithi157.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-17T23:47:30.021 INFO:teuthology.orchestra.run.smithi157.stdout: link/ether 12:e3:10:6e:d2:51 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-17T23:47:30.021 INFO:teuthology.orchestra.run.smithi157.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-17T23:47:30.022 INFO:teuthology.orchestra.run.smithi157.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:30.022 INFO:teuthology.orchestra.run.smithi157.stdout: inet6 fe80::10e3:10ff:fe6e:d251/64 scope link 2024-03-17T23:47:30.022 INFO:teuthology.orchestra.run.smithi157.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:47:30.023 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-17T23:47:30.023 DEBUG:teuthology.orchestra.run.smithi157:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:47:30.023 DEBUG:teuthology.orchestra.run.smithi157:> set -e 2024-03-17T23:47:30.023 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-17T23:47:30.023 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-17T23:47:30.023 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-17T23:47:30.023 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-17T23:47:30.023 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-17T23:47:30.023 DEBUG:teuthology.orchestra.run.smithi157:> ') 2024-03-17T23:47:30.112 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:47:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:47:30.186 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:47:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:30.193 DEBUG:teuthology.orchestra.run.smithi157:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:47:30.193 DEBUG:teuthology.orchestra.run.smithi157:> set -e 2024-03-17T23:47:30.193 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip link set brx.1 up 2024-03-17T23:47:30.193 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip link set dev brx.1 master ceph-brx 2024-03-17T23:47:30.193 DEBUG:teuthology.orchestra.run.smithi157:> ') 2024-03-17T23:47:30.242 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:47:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:47:30.269 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:47:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:30.275 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-03-17T23:47:30.276 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-17T23:47:30.276 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-17T23:47:30.290 INFO:teuthology.orchestra.run.smithi157.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-17T23:47:30.290 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-17T23:47:30.291 DEBUG:teuthology.orchestra.run.smithi157:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-17T23:47:30.344 DEBUG:teuthology.orchestra.run.smithi157:> sudo modprobe fuse 2024-03-17T23:47:30.412 DEBUG:teuthology.orchestra.run.smithi157:> cat /proc/self/mounts | awk '{print $2}' 2024-03-17T23:47:30.468 INFO:teuthology.orchestra.run.smithi157.stdout:/proc 2024-03-17T23:47:30.468 INFO:teuthology.orchestra.run.smithi157.stdout:/sys 2024-03-17T23:47:30.468 INFO:teuthology.orchestra.run.smithi157.stdout:/dev 2024-03-17T23:47:30.468 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/kernel/security 2024-03-17T23:47:30.468 INFO:teuthology.orchestra.run.smithi157.stdout:/dev/shm 2024-03-17T23:47:30.469 INFO:teuthology.orchestra.run.smithi157.stdout:/dev/pts 2024-03-17T23:47:30.469 INFO:teuthology.orchestra.run.smithi157.stdout:/run 2024-03-17T23:47:30.469 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/fs/cgroup 2024-03-17T23:47:30.469 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/fs/pstore 2024-03-17T23:47:30.469 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/fs/bpf 2024-03-17T23:47:30.469 INFO:teuthology.orchestra.run.smithi157.stdout:/ 2024-03-17T23:47:30.469 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/fs/selinux 2024-03-17T23:47:30.469 INFO:teuthology.orchestra.run.smithi157.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:47:30.469 INFO:teuthology.orchestra.run.smithi157.stdout:/dev/mqueue 2024-03-17T23:47:30.469 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/kernel/debug 2024-03-17T23:47:30.469 INFO:teuthology.orchestra.run.smithi157.stdout:/dev/hugepages 2024-03-17T23:47:30.469 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/kernel/tracing 2024-03-17T23:47:30.469 INFO:teuthology.orchestra.run.smithi157.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-17T23:47:30.469 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/fs/fuse/connections 2024-03-17T23:47:30.470 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/kernel/config 2024-03-17T23:47:30.470 INFO:teuthology.orchestra.run.smithi157.stdout:/run/credentials/systemd-sysctl.service 2024-03-17T23:47:30.470 INFO:teuthology.orchestra.run.smithi157.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-17T23:47:30.470 INFO:teuthology.orchestra.run.smithi157.stdout:/var/lib/nfs/rpc_pipefs 2024-03-17T23:47:30.470 INFO:teuthology.orchestra.run.smithi157.stdout:/run/user/1000 2024-03-17T23:47:30.470 INFO:teuthology.orchestra.run.smithi157.stdout:/var/lib/ceph 2024-03-17T23:47:30.470 INFO:teuthology.orchestra.run.smithi157.stdout:/var/lib/ceph/osd/ceph-4 2024-03-17T23:47:30.470 INFO:teuthology.orchestra.run.smithi157.stdout:/var/lib/ceph/osd/ceph-5 2024-03-17T23:47:30.471 INFO:teuthology.orchestra.run.smithi157.stdout:/var/lib/ceph/osd/ceph-6 2024-03-17T23:47:30.471 INFO:teuthology.orchestra.run.smithi157.stdout:/var/lib/ceph/osd/ceph-7 2024-03-17T23:47:30.471 INFO:teuthology.orchestra.run.smithi157.stdout:/run/netns 2024-03-17T23:47:30.471 INFO:teuthology.orchestra.run.smithi157.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-17T23:47:30.471 INFO:teuthology.orchestra.run.smithi157.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-17T23:47:30.471 INFO:teuthology.orchestra.run.smithi157.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-17T23:47:30.471 INFO:teuthology.orchestra.run.smithi157.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-17T23:47:30.471 INFO:teuthology.orchestra.run.smithi157.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-17T23:47:30.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:30.472 DEBUG:teuthology.orchestra.run.smithi157:> ls /sys/fs/fuse/connections 2024-03-17T23:47:30.523 INFO:teuthology.orchestra.run.smithi157.stdout:42 2024-03-17T23:47:30.524 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2024-03-17T23:47:30.524 DEBUG:teuthology.orchestra.run.smithi157:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-03-17T23:47:30.566 DEBUG:teuthology.orchestra.run.smithi157:> sudo modprobe fuse 2024-03-17T23:47:30.593 DEBUG:teuthology.orchestra.run.smithi157:> cat /proc/self/mounts | awk '{print $2}' 2024-03-17T23:47:30.651 INFO:teuthology.orchestra.run.smithi157.stdout:/proc 2024-03-17T23:47:30.651 INFO:teuthology.orchestra.run.smithi157.stdout:/sys 2024-03-17T23:47:30.651 INFO:teuthology.orchestra.run.smithi157.stdout:/dev 2024-03-17T23:47:30.651 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/kernel/security 2024-03-17T23:47:30.652 INFO:teuthology.orchestra.run.smithi157.stdout:/dev/shm 2024-03-17T23:47:30.652 INFO:teuthology.orchestra.run.smithi157.stdout:/dev/pts 2024-03-17T23:47:30.652 INFO:teuthology.orchestra.run.smithi157.stdout:/run 2024-03-17T23:47:30.652 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/fs/cgroup 2024-03-17T23:47:30.652 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/fs/pstore 2024-03-17T23:47:30.652 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/fs/bpf 2024-03-17T23:47:30.652 INFO:teuthology.orchestra.run.smithi157.stdout:/ 2024-03-17T23:47:30.652 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/fs/selinux 2024-03-17T23:47:30.653 INFO:teuthology.orchestra.run.smithi157.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:47:30.653 INFO:teuthology.orchestra.run.smithi157.stdout:/dev/mqueue 2024-03-17T23:47:30.653 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/kernel/debug 2024-03-17T23:47:30.653 INFO:teuthology.orchestra.run.smithi157.stdout:/dev/hugepages 2024-03-17T23:47:30.653 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/kernel/tracing 2024-03-17T23:47:30.653 INFO:teuthology.orchestra.run.smithi157.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-17T23:47:30.654 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/fs/fuse/connections 2024-03-17T23:47:30.654 INFO:teuthology.orchestra.run.smithi157.stdout:/sys/kernel/config 2024-03-17T23:47:30.654 INFO:teuthology.orchestra.run.smithi157.stdout:/run/credentials/systemd-sysctl.service 2024-03-17T23:47:30.654 INFO:teuthology.orchestra.run.smithi157.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-17T23:47:30.654 INFO:teuthology.orchestra.run.smithi157.stdout:/var/lib/nfs/rpc_pipefs 2024-03-17T23:47:30.654 INFO:teuthology.orchestra.run.smithi157.stdout:/run/user/1000 2024-03-17T23:47:30.654 INFO:teuthology.orchestra.run.smithi157.stdout:/var/lib/ceph 2024-03-17T23:47:30.654 INFO:teuthology.orchestra.run.smithi157.stdout:/var/lib/ceph/osd/ceph-4 2024-03-17T23:47:30.654 INFO:teuthology.orchestra.run.smithi157.stdout:/var/lib/ceph/osd/ceph-5 2024-03-17T23:47:30.654 INFO:teuthology.orchestra.run.smithi157.stdout:/var/lib/ceph/osd/ceph-6 2024-03-17T23:47:30.654 INFO:teuthology.orchestra.run.smithi157.stdout:/var/lib/ceph/osd/ceph-7 2024-03-17T23:47:30.655 INFO:teuthology.orchestra.run.smithi157.stdout:/run/netns 2024-03-17T23:47:30.655 INFO:teuthology.orchestra.run.smithi157.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-17T23:47:30.655 INFO:teuthology.orchestra.run.smithi157.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-17T23:47:30.655 INFO:teuthology.orchestra.run.smithi157.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-17T23:47:30.655 INFO:teuthology.orchestra.run.smithi157.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-17T23:47:30.655 INFO:teuthology.orchestra.run.smithi157.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-17T23:47:30.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:30.655 DEBUG:teuthology.orchestra.run.smithi157:> ls /sys/fs/fuse/connections 2024-03-17T23:47:30.656 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi157.stderr:2024-03-17T23:47:30.650+0000 7f74bee42480 -1 init, newargv = 0x565493ab96a0 newargc=15 2024-03-17T23:47:30.656 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi157.stderr:ceph-fuse[43779]: starting ceph client 2024-03-17T23:47:30.656 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi157.stderr:ceph-fuse[43779]: starting fuse 2024-03-17T23:47:30.670 INFO:teuthology.orchestra.run.smithi157.stdout:42 2024-03-17T23:47:30.670 INFO:teuthology.orchestra.run.smithi157.stdout:44 2024-03-17T23:47:30.670 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 44] 2024-03-17T23:47:30.670 DEBUG:teuthology.orchestra.run.smithi157:> sudo stdin-killer -- python3 -c ' 2024-03-17T23:47:30.671 DEBUG:teuthology.orchestra.run.smithi157:> import glob 2024-03-17T23:47:30.671 DEBUG:teuthology.orchestra.run.smithi157:> import re 2024-03-17T23:47:30.671 DEBUG:teuthology.orchestra.run.smithi157:> import os 2024-03-17T23:47:30.671 DEBUG:teuthology.orchestra.run.smithi157:> import subprocess 2024-03-17T23:47:30.671 DEBUG:teuthology.orchestra.run.smithi157:> 2024-03-17T23:47:30.671 DEBUG:teuthology.orchestra.run.smithi157:> def _find_admin_socket(client_name): 2024-03-17T23:47:30.671 DEBUG:teuthology.orchestra.run.smithi157:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-03-17T23:47:30.671 DEBUG:teuthology.orchestra.run.smithi157:> files = glob.glob(asok_path) 2024-03-17T23:47:30.671 DEBUG:teuthology.orchestra.run.smithi157:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-03-17T23:47:30.671 DEBUG:teuthology.orchestra.run.smithi157:> 2024-03-17T23:47:30.671 DEBUG:teuthology.orchestra.run.smithi157:> # Given a non-glob path, it better be there 2024-03-17T23:47:30.671 DEBUG:teuthology.orchestra.run.smithi157:> if "*" not in asok_path: 2024-03-17T23:47:30.671 DEBUG:teuthology.orchestra.run.smithi157:> assert(len(files) == 1) 2024-03-17T23:47:30.671 DEBUG:teuthology.orchestra.run.smithi157:> return files[0] 2024-03-17T23:47:30.671 DEBUG:teuthology.orchestra.run.smithi157:> 2024-03-17T23:47:30.672 DEBUG:teuthology.orchestra.run.smithi157:> for f in files: 2024-03-17T23:47:30.672 DEBUG:teuthology.orchestra.run.smithi157:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-17T23:47:30.672 DEBUG:teuthology.orchestra.run.smithi157:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-17T23:47:30.672 DEBUG:teuthology.orchestra.run.smithi157:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-17T23:47:30.672 DEBUG:teuthology.orchestra.run.smithi157:> contents = proc_f.read() 2024-03-17T23:47:30.672 DEBUG:teuthology.orchestra.run.smithi157:> if mountpoint in contents: 2024-03-17T23:47:30.672 DEBUG:teuthology.orchestra.run.smithi157:> return f 2024-03-17T23:47:30.672 DEBUG:teuthology.orchestra.run.smithi157:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-17T23:47:30.672 DEBUG:teuthology.orchestra.run.smithi157:> 2024-03-17T23:47:30.672 DEBUG:teuthology.orchestra.run.smithi157:> print(_find_admin_socket("client.3")) 2024-03-17T23:47:30.672 DEBUG:teuthology.orchestra.run.smithi157:> ' 2024-03-17T23:47:30.793 INFO:teuthology.orchestra.run.smithi157.stdout:/var/run/ceph/ceph-client.3.43779.asok 2024-03-17T23:47:30.796 INFO:teuthology.orchestra.run.smithi157.stderr:2024-03-17T23:47:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:47:30.805 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.43779.asok 2024-03-17T23:47:30.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:30.806 DEBUG:teuthology.orchestra.run.smithi157:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.43779.asok status 2024-03-17T23:47:30.946 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2024-03-17T23:47:30.946 INFO:teuthology.orchestra.run.smithi157.stdout: "metadata": { 2024-03-17T23:47:30.946 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-17T23:47:30.947 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-17T23:47:30.947 INFO:teuthology.orchestra.run.smithi157.stdout: "entity_id": "3", 2024-03-17T23:47:30.947 INFO:teuthology.orchestra.run.smithi157.stdout: "hostname": "smithi157", 2024-03-17T23:47:30.947 INFO:teuthology.orchestra.run.smithi157.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-03-17T23:47:30.947 INFO:teuthology.orchestra.run.smithi157.stdout: "pid": "43779", 2024-03-17T23:47:30.947 INFO:teuthology.orchestra.run.smithi157.stdout: "root": "/" 2024-03-17T23:47:30.947 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-03-17T23:47:30.947 INFO:teuthology.orchestra.run.smithi157.stdout: "dentry_count": 0, 2024-03-17T23:47:30.947 INFO:teuthology.orchestra.run.smithi157.stdout: "dentry_pinned_count": 0, 2024-03-17T23:47:30.947 INFO:teuthology.orchestra.run.smithi157.stdout: "id": 4709, 2024-03-17T23:47:30.947 INFO:teuthology.orchestra.run.smithi157.stdout: "inst": { 2024-03-17T23:47:30.947 INFO:teuthology.orchestra.run.smithi157.stdout: "name": { 2024-03-17T23:47:30.947 INFO:teuthology.orchestra.run.smithi157.stdout: "type": "client", 2024-03-17T23:47:30.947 INFO:teuthology.orchestra.run.smithi157.stdout: "num": 4709 2024-03-17T23:47:30.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-03-17T23:47:30.948 INFO:teuthology.orchestra.run.smithi157.stdout: "addr": { 2024-03-17T23:47:30.948 INFO:teuthology.orchestra.run.smithi157.stdout: "type": "v1", 2024-03-17T23:47:30.948 INFO:teuthology.orchestra.run.smithi157.stdout: "addr": "192.168.0.2:0", 2024-03-17T23:47:30.948 INFO:teuthology.orchestra.run.smithi157.stdout: "nonce": 1822542679 2024-03-17T23:47:30.948 INFO:teuthology.orchestra.run.smithi157.stdout: } 2024-03-17T23:47:30.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-03-17T23:47:30.948 INFO:teuthology.orchestra.run.smithi157.stdout: "addr": { 2024-03-17T23:47:30.948 INFO:teuthology.orchestra.run.smithi157.stdout: "type": "v1", 2024-03-17T23:47:30.948 INFO:teuthology.orchestra.run.smithi157.stdout: "addr": "192.168.0.2:0", 2024-03-17T23:47:30.948 INFO:teuthology.orchestra.run.smithi157.stdout: "nonce": 1822542679 2024-03-17T23:47:30.948 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2024-03-17T23:47:30.948 INFO:teuthology.orchestra.run.smithi157.stdout: "inst_str": "client.4709 192.168.0.2:0/1822542679", 2024-03-17T23:47:30.948 INFO:teuthology.orchestra.run.smithi157.stdout: "addr_str": "192.168.0.2:0/1822542679", 2024-03-17T23:47:30.948 INFO:teuthology.orchestra.run.smithi157.stdout: "inode_count": 1, 2024-03-17T23:47:30.949 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_epoch": 8, 2024-03-17T23:47:30.949 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_epoch": 26, 2024-03-17T23:47:30.949 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_epoch_barrier": 0, 2024-03-17T23:47:30.949 INFO:teuthology.orchestra.run.smithi157.stdout: "blocklisted": false, 2024-03-17T23:47:30.949 INFO:teuthology.orchestra.run.smithi157.stdout: "fs_name": "cephfs" 2024-03-17T23:47:30.949 INFO:teuthology.orchestra.run.smithi157.stdout:} 2024-03-17T23:47:30.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:30.955 DEBUG:teuthology.orchestra.run.smithi150:> stat --file-system '--printf=%T 2024-03-17T23:47:30.955 DEBUG:teuthology.orchestra.run.smithi150:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-17T23:47:30.974 INFO:teuthology.orchestra.run.smithi150.stdout:fuseblk 2024-03-17T23:47:30.974 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-17T23:47:30.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:30.974 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-17T23:47:31.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:31.047 DEBUG:teuthology.orchestra.run.smithi150:> stat --file-system '--printf=%T 2024-03-17T23:47:31.047 DEBUG:teuthology.orchestra.run.smithi150:> ' -- /home/ubuntu/cephtest/mnt.2 2024-03-17T23:47:31.105 INFO:teuthology.orchestra.run.smithi150.stdout:fuseblk 2024-03-17T23:47:31.105 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-03-17T23:47:31.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:31.106 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-17T23:47:31.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:31.176 DEBUG:teuthology.orchestra.run.smithi157:> stat --file-system '--printf=%T 2024-03-17T23:47:31.176 DEBUG:teuthology.orchestra.run.smithi157:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-17T23:47:31.191 INFO:teuthology.orchestra.run.smithi157.stdout:fuseblk 2024-03-17T23:47:31.191 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-17T23:47:31.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:31.192 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-17T23:47:31.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:31.258 DEBUG:teuthology.orchestra.run.smithi157:> stat --file-system '--printf=%T 2024-03-17T23:47:31.258 DEBUG:teuthology.orchestra.run.smithi157:> ' -- /home/ubuntu/cephtest/mnt.3 2024-03-17T23:47:31.314 INFO:teuthology.orchestra.run.smithi157.stdout:fuseblk 2024-03-17T23:47:31.314 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-03-17T23:47:31.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:31.315 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-17T23:47:31.381 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-17T23:47:31.397 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:31.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.505+0000 7f53edaac640 1 Processor -- start 2024-03-17T23:47:31.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.505+0000 7f53edaac640 1 -- start start 2024-03-17T23:47:31.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.506+0000 7f53edaac640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f53e8155320 0x7f53e815bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:31.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.506+0000 7f53edaac640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f53e806dd30 0x7f53e80ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:31.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.506+0000 7f53edaac640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53e80cf3b0 con 0x7f53e80cd250 2024-03-17T23:47:31.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.506+0000 7f53edaac640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f53e815c550 con 0x7f53e8155320 2024-03-17T23:47:31.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.506+0000 7f53edaac640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f53e80cf690 con 0x7f53e806dd30 2024-03-17T23:47:31.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.506+0000 7f53e77fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f53e8155320 0x7f53e815bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:31.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.506+0000 7f53e77fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f53e8155320 0x7f53e815bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46030/0 (socket says 172.21.15.150:46030) 2024-03-17T23:47:31.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.506+0000 7f53e77fe640 1 -- 172.21.15.150:0/2820790859 learned_addr learned my addr 172.21.15.150:0/2820790859 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:31.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.506+0000 7f53e6ffd640 1 --2- 172.21.15.150:0/2820790859 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f53e806dd30 0x7f53e80ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:31.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.506+0000 7f53e77fe640 1 -- 172.21.15.150:0/2820790859 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f53e806dd30 msgr2=0x7f53e80ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:31.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.506+0000 7f53e77fe640 1 --2- 172.21.15.150:0/2820790859 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f53e806dd30 0x7f53e80ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:31.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.506+0000 7f53e77fe640 1 -- 172.21.15.150:0/2820790859 >> v1:172.21.15.157:6789/0 conn(0x7f53e80cd250 legacy=0x7f53e815e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:31.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.506+0000 7f53e77fe640 1 -- 172.21.15.150:0/2820790859 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53e815c7a0 con 0x7f53e8155320 2024-03-17T23:47:31.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.506+0000 7f53e67fc640 1 -- 172.21.15.150:0/2820790859 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 185721539 0 0) 0x7f53e80cf3b0 con 0x7f53e80cd250 2024-03-17T23:47:31.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.507+0000 7f53e77fe640 1 --2- 172.21.15.150:0/2820790859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53e8155320 0x7f53e815bfd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f53d8009870 tx=0x7f53d802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ae8b6e47cdc7add server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:31.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.507+0000 7f53e67fc640 1 -- 172.21.15.150:0/2820790859 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d8034070 con 0x7f53e8155320 2024-03-17T23:47:31.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.507+0000 7f53e67fc640 1 -- 172.21.15.150:0/2820790859 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53d8030840 con 0x7f53e8155320 2024-03-17T23:47:31.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.507+0000 7f53e67fc640 1 -- 172.21.15.150:0/2820790859 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d8030c10 con 0x7f53e8155320 2024-03-17T23:47:31.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.508+0000 7f53edaac640 1 -- 172.21.15.150:0/2820790859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53e8155320 msgr2=0x7f53e815bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:31.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.508+0000 7f53edaac640 1 --2- 172.21.15.150:0/2820790859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53e8155320 0x7f53e815bfd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f53d8009870 tx=0x7f53d802b980 comp rx=0 tx=0).stop 2024-03-17T23:47:31.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.508+0000 7f53edaac640 1 -- 172.21.15.150:0/2820790859 shutdown_connections 2024-03-17T23:47:31.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.508+0000 7f53edaac640 1 --2- 172.21.15.150:0/2820790859 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f53e806dd30 0x7f53e80ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:31.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.508+0000 7f53edaac640 1 --2- 172.21.15.150:0/2820790859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53e8155320 0x7f53e815bfd0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:31.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.508+0000 7f53edaac640 1 -- 172.21.15.150:0/2820790859 >> 172.21.15.150:0/2820790859 conn(0x7f53e806b190 msgr2=0x7f53e806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:31.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.508+0000 7f53edaac640 1 -- 172.21.15.150:0/2820790859 shutdown_connections 2024-03-17T23:47:31.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.508+0000 7f53edaac640 1 -- 172.21.15.150:0/2820790859 wait complete. 2024-03-17T23:47:31.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.509+0000 7f53edaac640 1 Processor -- start 2024-03-17T23:47:31.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.509+0000 7f53edaac640 1 -- start start 2024-03-17T23:47:31.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.510+0000 7f53edaac640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f53e806dd30 0x7f53e814ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:31.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.510+0000 7f53edaac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53e80cd250 0x7f53e8143510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:31.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.510+0000 7f53edaac640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f53e8155320 0x7f53e8156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:31.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.510+0000 7f53edaac640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f53e8155700 con 0x7f53e80cd250 2024-03-17T23:47:31.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.510+0000 7f53edaac640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f53e806e190 con 0x7f53e8155320 2024-03-17T23:47:31.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.510+0000 7f53edaac640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f53e8155880 con 0x7f53e806dd30 2024-03-17T23:47:31.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.510+0000 7f53e6ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53e80cd250 0x7f53e8143510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:31.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.510+0000 7f53e6ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53e80cd250 0x7f53e8143510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46040/0 (socket says 172.21.15.150:46040) 2024-03-17T23:47:31.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.510+0000 7f53e6ffd640 1 -- 172.21.15.150:0/2750980310 learned_addr learned my addr 172.21.15.150:0/2750980310 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:31.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.510+0000 7f53e77fe640 1 --2- 172.21.15.150:0/2750980310 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f53e806dd30 0x7f53e814ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:31.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.510+0000 7f53e7fff640 1 --2- 172.21.15.150:0/2750980310 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f53e8155320 0x7f53e8156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:31.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.511+0000 7f53e6ffd640 1 -- 172.21.15.150:0/2750980310 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f53e806dd30 msgr2=0x7f53e814ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:31.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.511+0000 7f53e6ffd640 1 --2- 172.21.15.150:0/2750980310 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f53e806dd30 0x7f53e814ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:31.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.511+0000 7f53e6ffd640 1 -- 172.21.15.150:0/2750980310 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f53e8155320 msgr2=0x7f53e8156380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:31.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.511+0000 7f53e6ffd640 1 --2- 172.21.15.150:0/2750980310 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f53e8155320 0x7f53e8156380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:31.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.511+0000 7f53e6ffd640 1 -- 172.21.15.150:0/2750980310 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53e80b3d10 con 0x7f53e80cd250 2024-03-17T23:47:31.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.511+0000 7f53e6ffd640 1 --2- 172.21.15.150:0/2750980310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53e80cd250 0x7f53e8143510 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f53d000e9e0 tx=0x7f53d000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:31.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.511+0000 7f53dffff640 1 -- 172.21.15.150:0/2750980310 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d0017070 con 0x7f53e80cd250 2024-03-17T23:47:31.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.511+0000 7f53dffff640 1 -- 172.21.15.150:0/2750980310 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53d000c9d0 con 0x7f53e80cd250 2024-03-17T23:47:31.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.511+0000 7f53dffff640 1 -- 172.21.15.150:0/2750980310 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d000cdc0 con 0x7f53e80cd250 2024-03-17T23:47:31.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.511+0000 7f53edaac640 1 -- 172.21.15.150:0/2750980310 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53e80b3ff0 con 0x7f53e80cd250 2024-03-17T23:47:31.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.511+0000 7f53edaac640 1 -- 172.21.15.150:0/2750980310 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53e8155e60 con 0x7f53e80cd250 2024-03-17T23:47:31.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.512+0000 7f53edaac640 1 -- 172.21.15.150:0/2750980310 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53e8067a40 con 0x7f53e80cd250 2024-03-17T23:47:31.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.512+0000 7f53dffff640 1 -- 172.21.15.150:0/2750980310 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f53d001a070 con 0x7f53e80cd250 2024-03-17T23:47:31.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.513+0000 7f53dffff640 1 --2- 172.21.15.150:0/2750980310 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f53c806bbd0 0x7f53c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:31.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.513+0000 7f53dffff640 1 -- 172.21.15.150:0/2750980310 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f53d008b0b0 con 0x7f53e80cd250 2024-03-17T23:47:31.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.513+0000 7f53e77fe640 1 --2- 172.21.15.150:0/2750980310 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f53c806bbd0 0x7f53c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:31.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.513+0000 7f53e77fe640 1 --2- 172.21.15.150:0/2750980310 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f53c806bbd0 0x7f53c806e090 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f53d8036000 tx=0x7f53d803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:31.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.517+0000 7f53dffff640 1 -- 172.21.15.150:0/2750980310 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f53e8067a40 con 0x7f53e80cd250 2024-03-17T23:47:31.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.691+0000 7f53edaac640 1 -- 172.21.15.150:0/2750980310 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f53e80b4fc0 con 0x7f53e80cd250 2024-03-17T23:47:31.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.692+0000 7f53dffff640 1 -- 172.21.15.150:0/2750980310 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18550 (secure 0 0 0) 0x7f53e80b4fc0 con 0x7f53e80cd250 2024-03-17T23:47:31.693 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:31.693 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:01.940019+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:31.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.694+0000 7f53edaac640 1 -- 172.21.15.150:0/2750980310 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f53c806bbd0 msgr2=0x7f53c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:31.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.694+0000 7f53edaac640 1 --2- 172.21.15.150:0/2750980310 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f53c806bbd0 0x7f53c806e090 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f53d8036000 tx=0x7f53d803b040 comp rx=0 tx=0).stop 2024-03-17T23:47:31.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.694+0000 7f53edaac640 1 -- 172.21.15.150:0/2750980310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53e80cd250 msgr2=0x7f53e8143510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:31.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.694+0000 7f53edaac640 1 --2- 172.21.15.150:0/2750980310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53e80cd250 0x7f53e8143510 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f53d000e9e0 tx=0x7f53d000eea0 comp rx=0 tx=0).stop 2024-03-17T23:47:31.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.695+0000 7f53edaac640 1 -- 172.21.15.150:0/2750980310 shutdown_connections 2024-03-17T23:47:31.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.695+0000 7f53edaac640 1 --2- 172.21.15.150:0/2750980310 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f53c806bbd0 0x7f53c806e090 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:31.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.695+0000 7f53edaac640 1 --2- 172.21.15.150:0/2750980310 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f53e8155320 0x7f53e8156380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:31.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.695+0000 7f53edaac640 1 --2- 172.21.15.150:0/2750980310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53e80cd250 0x7f53e8143510 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:31.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.695+0000 7f53edaac640 1 --2- 172.21.15.150:0/2750980310 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f53e806dd30 0x7f53e814ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:31.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.695+0000 7f53edaac640 1 -- 172.21.15.150:0/2750980310 >> 172.21.15.150:0/2750980310 conn(0x7f53e806b190 msgr2=0x7f53e80b09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:31.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.695+0000 7f53edaac640 1 -- 172.21.15.150:0/2750980310 shutdown_connections 2024-03-17T23:47:31.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.695+0000 7f53edaac640 1 -- 172.21.15.150:0/2750980310 wait complete. 2024-03-17T23:47:31.707 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:46:53.610678+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:46:54.227335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-17T23:46:57.949235+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-17T23:47:31.707 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:47:31.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.812+0000 7fd29d861640 1 Processor -- start 2024-03-17T23:47:31.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.812+0000 7fd29d861640 1 -- start start 2024-03-17T23:47:31.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.812+0000 7fd29d861640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd29815e000 0x7fd2980afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:31.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.812+0000 7fd29d861640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd29815e3e0 0x7fd2980b0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:31.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.812+0000 7fd29d861640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd29806bdf0 con 0x7fd2980b5170 2024-03-17T23:47:31.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.812+0000 7fd29d861640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd29806bba0 con 0x7fd29815e000 2024-03-17T23:47:31.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.812+0000 7fd29d861640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd29806c0d0 con 0x7fd29815e3e0 2024-03-17T23:47:31.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.813+0000 7fd2977fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd2980b5170 0x7fd2980d0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33680/0 (socket says 172.21.15.150:33680) 2024-03-17T23:47:31.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.813+0000 7fd2977fe640 1 -- 172.21.15.150:0/2450732484 learned_addr learned my addr 172.21.15.150:0/2450732484 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:31.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.813+0000 7fd296ffd640 1 --2- 172.21.15.150:0/2450732484 >> v2:172.21.15.157:3300/0 conn(0x7fd29815e000 0x7fd2980afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:31.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.813+0000 7fd28e7fc640 1 --2- 172.21.15.150:0/2450732484 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd29815e3e0 0x7fd2980b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:31.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.813+0000 7fd2967fc640 1 -- 172.21.15.150:0/2450732484 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 932780388 0 0) 0x7fd29806bdf0 con 0x7fd2980b5170 2024-03-17T23:47:31.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.813+0000 7fd2967fc640 1 -- 172.21.15.150:0/2450732484 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd284003520 con 0x7fd2980b5170 2024-03-17T23:47:31.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.813+0000 7fd296ffd640 1 -- 172.21.15.150:0/2450732484 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd29815e3e0 msgr2=0x7fd2980b0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:31.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.813+0000 7fd296ffd640 1 --2- 172.21.15.150:0/2450732484 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd29815e3e0 0x7fd2980b0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:31.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.813+0000 7fd296ffd640 1 -- 172.21.15.150:0/2450732484 >> v1:172.21.15.150:6789/0 conn(0x7fd2980b5170 legacy=0x7fd2980d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:31.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.813+0000 7fd296ffd640 1 -- 172.21.15.150:0/2450732484 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd29806c3b0 con 0x7fd29815e000 2024-03-17T23:47:31.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.813+0000 7fd28e7fc640 1 --2- 172.21.15.150:0/2450732484 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd29815e3e0 0x7fd2980b0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:31.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.814+0000 7fd296ffd640 1 --2- 172.21.15.150:0/2450732484 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd29815e000 0x7fd2980afc00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd2880097a0 tx=0x7fd28802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1fb232c959b47d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:31.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.814+0000 7fd2967fc640 1 -- 172.21.15.150:0/2450732484 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd288034070 con 0x7fd29815e000 2024-03-17T23:47:31.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.814+0000 7fd2967fc640 1 -- 172.21.15.150:0/2450732484 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd288030610 con 0x7fd29815e000 2024-03-17T23:47:31.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.814+0000 7fd2967fc640 1 -- 172.21.15.150:0/2450732484 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2880309e0 con 0x7fd29815e000 2024-03-17T23:47:31.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.814+0000 7fd29d861640 1 -- 172.21.15.150:0/2450732484 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd29815e000 msgr2=0x7fd2980afc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:31.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.814+0000 7fd29d861640 1 --2- 172.21.15.150:0/2450732484 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd29815e000 0x7fd2980afc00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd2880097a0 tx=0x7fd28802b960 comp rx=0 tx=0).stop 2024-03-17T23:47:31.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.814+0000 7fd29d861640 1 -- 172.21.15.150:0/2450732484 shutdown_connections 2024-03-17T23:47:31.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.814+0000 7fd29d861640 1 --2- 172.21.15.150:0/2450732484 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd29815e3e0 0x7fd2980b0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:31.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.814+0000 7fd29d861640 1 --2- 172.21.15.150:0/2450732484 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd29815e000 0x7fd2980afc00 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:31.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.814+0000 7fd29d861640 1 -- 172.21.15.150:0/2450732484 >> 172.21.15.150:0/2450732484 conn(0x7fd298099a70 msgr2=0x7fd2980b55b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:31.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.815+0000 7fd29d861640 1 -- 172.21.15.150:0/2450732484 shutdown_connections 2024-03-17T23:47:31.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.815+0000 7fd29d861640 1 -- 172.21.15.150:0/2450732484 wait complete. 2024-03-17T23:47:31.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.816+0000 7fd29d861640 1 Processor -- start 2024-03-17T23:47:31.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.816+0000 7fd29d861640 1 -- start start 2024-03-17T23:47:31.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.816+0000 7fd29d861640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2980b5170 0x7fd298143780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:31.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.816+0000 7fd29d861640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd29815e000 0x7fd2980ad1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:31.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.816+0000 7fd29d861640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd29815e3e0 0x7fd2980ad6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:31.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.816+0000 7fd29d861640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd2980b5e40 con 0x7fd29815e000 2024-03-17T23:47:31.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.816+0000 7fd29d861640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd2980b5cc0 con 0x7fd2980b5170 2024-03-17T23:47:31.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.816+0000 7fd29d861640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd2980ae320 con 0x7fd29815e3e0 2024-03-17T23:47:31.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.817+0000 7fd28e7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd29815e000 0x7fd2980ad1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:31.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.817+0000 7fd28e7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd29815e000 0x7fd2980ad1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46054/0 (socket says 172.21.15.150:46054) 2024-03-17T23:47:31.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.817+0000 7fd28e7fc640 1 -- 172.21.15.150:0/3453539402 learned_addr learned my addr 172.21.15.150:0/3453539402 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:31.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.817+0000 7fd296ffd640 1 --2- 172.21.15.150:0/3453539402 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2980b5170 0x7fd298143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:31.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.817+0000 7fd2977fe640 1 --2- 172.21.15.150:0/3453539402 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd29815e3e0 0x7fd2980ad6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:31.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.817+0000 7fd28e7fc640 1 -- 172.21.15.150:0/3453539402 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd29815e3e0 msgr2=0x7fd2980ad6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:31.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.817+0000 7fd28e7fc640 1 --2- 172.21.15.150:0/3453539402 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd29815e3e0 0x7fd2980ad6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:31.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.817+0000 7fd28e7fc640 1 -- 172.21.15.150:0/3453539402 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2980b5170 msgr2=0x7fd298143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:31.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.817+0000 7fd28e7fc640 1 --2- 172.21.15.150:0/3453539402 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2980b5170 0x7fd298143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:31.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.817+0000 7fd28e7fc640 1 -- 172.21.15.150:0/3453539402 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd298144ba0 con 0x7fd29815e000 2024-03-17T23:47:31.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.817+0000 7fd28e7fc640 1 --2- 172.21.15.150:0/3453539402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd29815e000 0x7fd2980ad1a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd28000e910 tx=0x7fd28000edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:31.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.817+0000 7fd294ff9640 1 -- 172.21.15.150:0/3453539402 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd280017070 con 0x7fd29815e000 2024-03-17T23:47:31.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.817+0000 7fd294ff9640 1 -- 172.21.15.150:0/3453539402 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd28000ca00 con 0x7fd29815e000 2024-03-17T23:47:31.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.817+0000 7fd29d861640 1 -- 172.21.15.150:0/3453539402 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd298144e80 con 0x7fd29815e000 2024-03-17T23:47:31.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.817+0000 7fd294ff9640 1 -- 172.21.15.150:0/3453539402 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28000cdb0 con 0x7fd29815e000 2024-03-17T23:47:31.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.817+0000 7fd29d861640 1 -- 172.21.15.150:0/3453539402 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd298145440 con 0x7fd29815e000 2024-03-17T23:47:31.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.818+0000 7fd29d861640 1 -- 172.21.15.150:0/3453539402 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd298153340 con 0x7fd29815e000 2024-03-17T23:47:31.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.819+0000 7fd294ff9640 1 -- 172.21.15.150:0/3453539402 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd28001a070 con 0x7fd29815e000 2024-03-17T23:47:31.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.819+0000 7fd294ff9640 1 --2- 172.21.15.150:0/3453539402 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd27806bbd0 0x7fd27806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:31.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.819+0000 7fd296ffd640 1 --2- 172.21.15.150:0/3453539402 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd27806bbd0 0x7fd27806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:31.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.819+0000 7fd294ff9640 1 -- 172.21.15.150:0/3453539402 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fd28008af00 con 0x7fd29815e000 2024-03-17T23:47:31.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.820+0000 7fd296ffd640 1 --2- 172.21.15.150:0/3453539402 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd27806bbd0 0x7fd27806e090 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd288036000 tx=0x7fd28803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:31.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.824+0000 7fd294ff9640 1 -- 172.21.15.150:0/3453539402 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd298153340 con 0x7fd29815e000 2024-03-17T23:47:31.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.998+0000 7fd29d861640 1 -- 172.21.15.150:0/3453539402 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd2980681a0 con 0x7fd29815e000 2024-03-17T23:47:31.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:31.999+0000 7fd294ff9640 1 -- 172.21.15.150:0/3453539402 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fd280059060 con 0x7fd29815e000 2024-03-17T23:47:31.999 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:47:32.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.001+0000 7fd29d861640 1 -- 172.21.15.150:0/3453539402 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd27806bbd0 msgr2=0x7fd27806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:32.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.001+0000 7fd29d861640 1 --2- 172.21.15.150:0/3453539402 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd27806bbd0 0x7fd27806e090 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd288036000 tx=0x7fd28803b040 comp rx=0 tx=0).stop 2024-03-17T23:47:32.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.001+0000 7fd29d861640 1 -- 172.21.15.150:0/3453539402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd29815e000 msgr2=0x7fd2980ad1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:32.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.001+0000 7fd29d861640 1 --2- 172.21.15.150:0/3453539402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd29815e000 0x7fd2980ad1a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd28000e910 tx=0x7fd28000edd0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.001+0000 7fd29d861640 1 -- 172.21.15.150:0/3453539402 shutdown_connections 2024-03-17T23:47:32.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.001+0000 7fd29d861640 1 --2- 172.21.15.150:0/3453539402 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd29815e3e0 0x7fd2980ad6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.001+0000 7fd29d861640 1 --2- 172.21.15.150:0/3453539402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd29815e000 0x7fd2980ad1a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.001+0000 7fd29d861640 1 --2- 172.21.15.150:0/3453539402 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd27806bbd0 0x7fd27806e090 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.001+0000 7fd29d861640 1 --2- 172.21.15.150:0/3453539402 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2980b5170 0x7fd298143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.001+0000 7fd29d861640 1 -- 172.21.15.150:0/3453539402 >> 172.21.15.150:0/3453539402 conn(0x7fd298099a70 msgr2=0x7fd2980b0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:32.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.001+0000 7fd29d861640 1 -- 172.21.15.150:0/3453539402 shutdown_connections 2024-03-17T23:47:32.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.001+0000 7fd29d861640 1 -- 172.21.15.150:0/3453539402 wait complete. 2024-03-17T23:47:32.013 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:47:32.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.116+0000 7fb31f3cd640 1 Processor -- start 2024-03-17T23:47:32.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.116+0000 7fb31f3cd640 1 -- start start 2024-03-17T23:47:32.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.117+0000 7fb31f3cd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb318155320 0x7fb31815bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:32.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.117+0000 7fb31f3cd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb31806dd30 0x7fb3180ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:32.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.117+0000 7fb31f3cd640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3180cf3b0 con 0x7fb3180cd250 2024-03-17T23:47:32.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.117+0000 7fb31f3cd640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fb31815c550 con 0x7fb31806dd30 2024-03-17T23:47:32.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.117+0000 7fb31f3cd640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb3180cf690 con 0x7fb318155320 2024-03-17T23:47:32.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.117+0000 7fb31d96a640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb3180cd250 0x7fb31815e980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33696/0 (socket says 172.21.15.150:33696) 2024-03-17T23:47:32.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.117+0000 7fb31d96a640 1 -- 172.21.15.150:0/3366167033 learned_addr learned my addr 172.21.15.150:0/3366167033 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:32.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.117+0000 7fb31d169640 1 --2- 172.21.15.150:0/3366167033 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb318155320 0x7fb31815bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:32.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.117+0000 7fb31c968640 1 --2- 172.21.15.150:0/3366167033 >> v2:172.21.15.157:3300/0 conn(0x7fb31806dd30 0x7fb3180ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:32.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.117+0000 7fb30ffff640 1 -- 172.21.15.150:0/3366167033 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3409560484 0 0) 0x7fb3180cf3b0 con 0x7fb3180cd250 2024-03-17T23:47:32.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.117+0000 7fb30ffff640 1 -- 172.21.15.150:0/3366167033 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb304003520 con 0x7fb3180cd250 2024-03-17T23:47:32.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.117+0000 7fb31d169640 1 -- 172.21.15.150:0/3366167033 >> v2:172.21.15.157:3300/0 conn(0x7fb31806dd30 msgr2=0x7fb3180ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:32.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.117+0000 7fb31d169640 1 --2- 172.21.15.150:0/3366167033 >> v2:172.21.15.157:3300/0 conn(0x7fb31806dd30 0x7fb3180ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.117+0000 7fb31d169640 1 -- 172.21.15.150:0/3366167033 >> v1:172.21.15.150:6789/0 conn(0x7fb3180cd250 legacy=0x7fb31815e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:32.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.117+0000 7fb31d169640 1 -- 172.21.15.150:0/3366167033 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb31815c7a0 con 0x7fb318155320 2024-03-17T23:47:32.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.118+0000 7fb31d169640 1 --2- 172.21.15.150:0/3366167033 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb318155320 0x7fb31815bfd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb3140097a0 tx=0x7fb31402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65263fc64fb2713b server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:32.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.118+0000 7fb30ffff640 1 -- 172.21.15.150:0/3366167033 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb314035020 con 0x7fb318155320 2024-03-17T23:47:32.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.118+0000 7fb30ffff640 1 -- 172.21.15.150:0/3366167033 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb314031830 con 0x7fb318155320 2024-03-17T23:47:32.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.118+0000 7fb30ffff640 1 -- 172.21.15.150:0/3366167033 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb314031be0 con 0x7fb318155320 2024-03-17T23:47:32.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.119+0000 7fb31f3cd640 1 -- 172.21.15.150:0/3366167033 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb318155320 msgr2=0x7fb31815bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:32.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.119+0000 7fb31f3cd640 1 --2- 172.21.15.150:0/3366167033 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb318155320 0x7fb31815bfd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb3140097a0 tx=0x7fb31402b8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.119+0000 7fb31f3cd640 1 -- 172.21.15.150:0/3366167033 shutdown_connections 2024-03-17T23:47:32.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.119+0000 7fb31f3cd640 1 --2- 172.21.15.150:0/3366167033 >> v2:172.21.15.157:3300/0 conn(0x7fb31806dd30 0x7fb3180ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.119+0000 7fb31f3cd640 1 --2- 172.21.15.150:0/3366167033 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb318155320 0x7fb31815bfd0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.119+0000 7fb31f3cd640 1 -- 172.21.15.150:0/3366167033 >> 172.21.15.150:0/3366167033 conn(0x7fb31806b190 msgr2=0x7fb31806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:32.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.119+0000 7fb31f3cd640 1 -- 172.21.15.150:0/3366167033 shutdown_connections 2024-03-17T23:47:32.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.119+0000 7fb31f3cd640 1 -- 172.21.15.150:0/3366167033 wait complete. 2024-03-17T23:47:32.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.120+0000 7fb31f3cd640 1 Processor -- start 2024-03-17T23:47:32.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.120+0000 7fb31f3cd640 1 -- start start 2024-03-17T23:47:32.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.121+0000 7fb31f3cd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb31806dd30 0x7fb31806e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:32.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.121+0000 7fb31f3cd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3180cd250 0x7fb31806eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:32.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.121+0000 7fb31f3cd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb318155320 0x7fb3181470c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:32.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.121+0000 7fb31f3cd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb318155700 con 0x7fb318155320 2024-03-17T23:47:32.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.121+0000 7fb31f3cd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb31806e190 con 0x7fb31806dd30 2024-03-17T23:47:32.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.121+0000 7fb31f3cd640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb318155880 con 0x7fb3180cd250 2024-03-17T23:47:32.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.121+0000 7fb31d96a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb318155320 0x7fb3181470c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:32.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.121+0000 7fb31d96a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb318155320 0x7fb3181470c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46068/0 (socket says 172.21.15.150:46068) 2024-03-17T23:47:32.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.121+0000 7fb31d96a640 1 -- 172.21.15.150:0/473746712 learned_addr learned my addr 172.21.15.150:0/473746712 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:32.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.121+0000 7fb31d169640 1 --2- 172.21.15.150:0/473746712 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb31806dd30 0x7fb31806e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:32.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.121+0000 7fb31c968640 1 --2- 172.21.15.150:0/473746712 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3180cd250 0x7fb31806eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:32.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.121+0000 7fb31d96a640 1 -- 172.21.15.150:0/473746712 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3180cd250 msgr2=0x7fb31806eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:32.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.121+0000 7fb31d96a640 1 --2- 172.21.15.150:0/473746712 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3180cd250 0x7fb31806eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.121+0000 7fb31d96a640 1 -- 172.21.15.150:0/473746712 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb31806dd30 msgr2=0x7fb31806e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:32.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.121+0000 7fb31d96a640 1 --2- 172.21.15.150:0/473746712 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb31806dd30 0x7fb31806e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.121+0000 7fb31d96a640 1 -- 172.21.15.150:0/473746712 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb318147e10 con 0x7fb318155320 2024-03-17T23:47:32.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.121+0000 7fb31d96a640 1 --2- 172.21.15.150:0/473746712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb318155320 0x7fb3181470c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb31000cb40 tx=0x7fb31000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:32.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.121+0000 7fb30dffb640 1 -- 172.21.15.150:0/473746712 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb310017070 con 0x7fb318155320 2024-03-17T23:47:32.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.121+0000 7fb30dffb640 1 -- 172.21.15.150:0/473746712 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb31000aca0 con 0x7fb318155320 2024-03-17T23:47:32.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.121+0000 7fb31f3cd640 1 -- 172.21.15.150:0/473746712 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb318144f10 con 0x7fb318155320 2024-03-17T23:47:32.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.121+0000 7fb30dffb640 1 -- 172.21.15.150:0/473746712 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb310012620 con 0x7fb318155320 2024-03-17T23:47:32.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.122+0000 7fb31f3cd640 1 -- 172.21.15.150:0/473746712 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3181454d0 con 0x7fb318155320 2024-03-17T23:47:32.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.122+0000 7fb31f3cd640 1 -- 172.21.15.150:0/473746712 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3180681a0 con 0x7fb318155320 2024-03-17T23:47:32.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.123+0000 7fb30dffb640 1 -- 172.21.15.150:0/473746712 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb31001a070 con 0x7fb318155320 2024-03-17T23:47:32.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.123+0000 7fb30dffb640 1 --2- 172.21.15.150:0/473746712 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb2fc06bbd0 0x7fb2fc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:32.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.123+0000 7fb30dffb640 1 -- 172.21.15.150:0/473746712 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fb31008abb0 con 0x7fb318155320 2024-03-17T23:47:32.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.123+0000 7fb31d169640 1 --2- 172.21.15.150:0/473746712 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb2fc06bbd0 0x7fb2fc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:32.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.123+0000 7fb31d169640 1 --2- 172.21.15.150:0/473746712 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb2fc06bbd0 0x7fb2fc06e090 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb314036000 tx=0x7fb31403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:32.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.127+0000 7fb30dffb640 1 -- 172.21.15.150:0/473746712 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb310058d10 con 0x7fb318155320 2024-03-17T23:47:32.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.301+0000 7fb31f3cd640 1 -- 172.21.15.150:0/473746712 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb3180b4f90 con 0x7fb318155320 2024-03-17T23:47:32.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.301+0000 7fb30dffb640 1 -- 172.21.15.150:0/473746712 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7fb31005cdb0 con 0x7fb318155320 2024-03-17T23:47:32.302 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:32.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.303+0000 7fb31f3cd640 1 -- 172.21.15.150:0/473746712 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb2fc06bbd0 msgr2=0x7fb2fc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:32.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.303+0000 7fb31f3cd640 1 --2- 172.21.15.150:0/473746712 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb2fc06bbd0 0x7fb2fc06e090 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb314036000 tx=0x7fb31403c040 comp rx=0 tx=0).stop 2024-03-17T23:47:32.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.303+0000 7fb31f3cd640 1 -- 172.21.15.150:0/473746712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb318155320 msgr2=0x7fb3181470c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:32.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.303+0000 7fb31f3cd640 1 --2- 172.21.15.150:0/473746712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb318155320 0x7fb3181470c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb31000cb40 tx=0x7fb31000a060 comp rx=0 tx=0).stop 2024-03-17T23:47:32.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.303+0000 7fb31f3cd640 1 -- 172.21.15.150:0/473746712 shutdown_connections 2024-03-17T23:47:32.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.303+0000 7fb31f3cd640 1 --2- 172.21.15.150:0/473746712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb318155320 0x7fb3181470c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.303+0000 7fb31f3cd640 1 --2- 172.21.15.150:0/473746712 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3180cd250 0x7fb31806eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.303+0000 7fb31f3cd640 1 --2- 172.21.15.150:0/473746712 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb2fc06bbd0 0x7fb2fc06e090 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.303+0000 7fb31f3cd640 1 --2- 172.21.15.150:0/473746712 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb31806dd30 0x7fb31806e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.303+0000 7fb31f3cd640 1 -- 172.21.15.150:0/473746712 >> 172.21.15.150:0/473746712 conn(0x7fb31806b190 msgr2=0x7fb318152d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:32.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.303+0000 7fb31f3cd640 1 -- 172.21.15.150:0/473746712 shutdown_connections 2024-03-17T23:47:32.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.304+0000 7fb31f3cd640 1 -- 172.21.15.150:0/473746712 wait complete. 2024-03-17T23:47:32.315 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:47:32.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.420+0000 7f6fc53cd640 1 Processor -- start 2024-03-17T23:47:32.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.420+0000 7f6fc53cd640 1 -- start start 2024-03-17T23:47:32.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.421+0000 7f6fc53cd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fc015de00 0x7f6fc00b9970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:32.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.421+0000 7f6fc53cd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6fc006c690 0x7f6fc00b5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:32.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.421+0000 7f6fc53cd640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fc006cec0 con 0x7f6fc00b5cd0 2024-03-17T23:47:32.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.421+0000 7f6fc53cd640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f6fc00b6cd0 con 0x7f6fc006c690 2024-03-17T23:47:32.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.421+0000 7f6fc53cd640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6fc00b6120 con 0x7f6fc015de00 2024-03-17T23:47:32.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.421+0000 7f6fbeffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fc015de00 0x7f6fc00b9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:32.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.421+0000 7f6fbeffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fc015de00 0x7f6fc00b9970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:36460/0 (socket says 172.21.15.150:36460) 2024-03-17T23:47:32.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.421+0000 7f6fb7fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6fc006c690 0x7f6fc00b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:32.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.421+0000 7f6fbeffd640 1 -- 172.21.15.150:0/3254348899 learned_addr learned my addr 172.21.15.150:0/3254348899 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:32.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.421+0000 7f6fbe7fc640 1 -- 172.21.15.150:0/3254348899 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2135106163 0 0) 0x7f6fc006cec0 con 0x7f6fc00b5cd0 2024-03-17T23:47:32.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.421+0000 7f6fbeffd640 1 -- 172.21.15.150:0/3254348899 >> v2:172.21.15.157:3300/0 conn(0x7f6fc006c690 msgr2=0x7f6fc00b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:32.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.421+0000 7f6fbeffd640 1 --2- 172.21.15.150:0/3254348899 >> v2:172.21.15.157:3300/0 conn(0x7f6fc006c690 0x7f6fc00b5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.421+0000 7f6fbeffd640 1 -- 172.21.15.150:0/3254348899 >> v1:172.21.15.157:6789/0 conn(0x7f6fc00b5cd0 legacy=0x7f6fc01619f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:32.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.421+0000 7f6fbeffd640 1 -- 172.21.15.150:0/3254348899 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fc006d1a0 con 0x7f6fc015de00 2024-03-17T23:47:32.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.421+0000 7f6fb7fff640 1 --2- 172.21.15.150:0/3254348899 >> v2:172.21.15.157:3300/0 conn(0x7f6fc006c690 0x7f6fc00b5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:32.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.422+0000 7f6fbeffd640 1 --2- 172.21.15.150:0/3254348899 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fc015de00 0x7f6fc00b9970 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0009870 tx=0x7f6fb002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5233027fdd3be44 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:32.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.422+0000 7f6fbe7fc640 1 -- 172.21.15.150:0/3254348899 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0034070 con 0x7f6fc015de00 2024-03-17T23:47:32.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.422+0000 7f6fbe7fc640 1 -- 172.21.15.150:0/3254348899 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fb0030840 con 0x7f6fc015de00 2024-03-17T23:47:32.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.422+0000 7f6fbe7fc640 1 -- 172.21.15.150:0/3254348899 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0030c10 con 0x7f6fc015de00 2024-03-17T23:47:32.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.423+0000 7f6fc53cd640 1 -- 172.21.15.150:0/3254348899 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fc015de00 msgr2=0x7f6fc00b9970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:32.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.423+0000 7f6fc53cd640 1 --2- 172.21.15.150:0/3254348899 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fc015de00 0x7f6fc00b9970 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0009870 tx=0x7f6fb002b980 comp rx=0 tx=0).stop 2024-03-17T23:47:32.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.423+0000 7f6fc53cd640 1 -- 172.21.15.150:0/3254348899 shutdown_connections 2024-03-17T23:47:32.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.423+0000 7f6fc53cd640 1 --2- 172.21.15.150:0/3254348899 >> v2:172.21.15.157:3300/0 conn(0x7f6fc006c690 0x7f6fc00b5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.423+0000 7f6fc53cd640 1 --2- 172.21.15.150:0/3254348899 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fc015de00 0x7f6fc00b9970 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.423+0000 7f6fc53cd640 1 -- 172.21.15.150:0/3254348899 >> 172.21.15.150:0/3254348899 conn(0x7f6fc00d0790 msgr2=0x7f6fc00d0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:32.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.423+0000 7f6fc53cd640 1 -- 172.21.15.150:0/3254348899 shutdown_connections 2024-03-17T23:47:32.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.423+0000 7f6fc53cd640 1 -- 172.21.15.150:0/3254348899 wait complete. 2024-03-17T23:47:32.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.424+0000 7f6fc53cd640 1 Processor -- start 2024-03-17T23:47:32.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.424+0000 7f6fc53cd640 1 -- start start 2024-03-17T23:47:32.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.425+0000 7f6fc53cd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6fc006c690 0x7f6fc015a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:32.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.425+0000 7f6fc53cd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fc00b5cd0 0x7f6fc00cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:32.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.425+0000 7f6fc53cd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fc015de00 0x7f6fc00b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:32.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.425+0000 7f6fc53cd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc015e1e0 con 0x7f6fc00b5cd0 2024-03-17T23:47:32.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.425+0000 7f6fc53cd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc00d2190 con 0x7f6fc006c690 2024-03-17T23:47:32.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.425+0000 7f6fc53cd640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6fc006cad0 con 0x7f6fc015de00 2024-03-17T23:47:32.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.425+0000 7f6fb7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fc00b5cd0 0x7f6fc00cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:32.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.425+0000 7f6fb7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fc00b5cd0 0x7f6fc00cdd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46080/0 (socket says 172.21.15.150:46080) 2024-03-17T23:47:32.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.425+0000 7f6fb7fff640 1 -- 172.21.15.150:0/112287859 learned_addr learned my addr 172.21.15.150:0/112287859 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:32.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.425+0000 7f6fbeffd640 1 --2- 172.21.15.150:0/112287859 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6fc006c690 0x7f6fc015a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:32.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.425+0000 7f6fbf7fe640 1 --2- 172.21.15.150:0/112287859 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fc015de00 0x7f6fc00b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:32.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.425+0000 7f6fb7fff640 1 -- 172.21.15.150:0/112287859 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fc015de00 msgr2=0x7f6fc00b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:32.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.425+0000 7f6fb7fff640 1 --2- 172.21.15.150:0/112287859 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fc015de00 0x7f6fc00b46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.425+0000 7f6fb7fff640 1 -- 172.21.15.150:0/112287859 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6fc006c690 msgr2=0x7f6fc015a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:32.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.425+0000 7f6fb7fff640 1 --2- 172.21.15.150:0/112287859 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6fc006c690 0x7f6fc015a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.425+0000 7f6fb7fff640 1 -- 172.21.15.150:0/112287859 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fc015a7a0 con 0x7f6fc00b5cd0 2024-03-17T23:47:32.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.425+0000 7f6fbeffd640 1 --2- 172.21.15.150:0/112287859 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6fc006c690 0x7f6fc015a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:32.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.425+0000 7f6fb7fff640 1 --2- 172.21.15.150:0/112287859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fc00b5cd0 0x7f6fc00cdd60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6fa800ebe0 tx=0x7f6fa800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:32.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.425+0000 7f6fb77fe640 1 -- 172.21.15.150:0/112287859 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8017070 con 0x7f6fc00b5cd0 2024-03-17T23:47:32.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.425+0000 7f6fb77fe640 1 -- 172.21.15.150:0/112287859 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fa800cc30 con 0x7f6fc00b5cd0 2024-03-17T23:47:32.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.425+0000 7f6fc53cd640 1 -- 172.21.15.150:0/112287859 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fc01601e0 con 0x7f6fc00b5cd0 2024-03-17T23:47:32.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.426+0000 7f6fb77fe640 1 -- 172.21.15.150:0/112287859 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8013470 con 0x7f6fc00b5cd0 2024-03-17T23:47:32.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.426+0000 7f6fc53cd640 1 -- 172.21.15.150:0/112287859 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fc01607a0 con 0x7f6fc00b5cd0 2024-03-17T23:47:32.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.426+0000 7f6fc53cd640 1 -- 172.21.15.150:0/112287859 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fc00681a0 con 0x7f6fc00b5cd0 2024-03-17T23:47:32.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.427+0000 7f6fb77fe640 1 -- 172.21.15.150:0/112287859 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6fa801a070 con 0x7f6fc00b5cd0 2024-03-17T23:47:32.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.427+0000 7f6fb77fe640 1 --2- 172.21.15.150:0/112287859 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6fa006bbd0 0x7f6fa006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:32.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.427+0000 7f6fbeffd640 1 --2- 172.21.15.150:0/112287859 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6fa006bbd0 0x7f6fa006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:32.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.427+0000 7f6fb77fe640 1 -- 172.21.15.150:0/112287859 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f6fa808ac80 con 0x7f6fc00b5cd0 2024-03-17T23:47:32.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.428+0000 7f6fbeffd640 1 --2- 172.21.15.150:0/112287859 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6fa006bbd0 0x7f6fa006e090 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0036000 tx=0x7f6fb003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:32.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.432+0000 7f6fb77fe640 1 -- 172.21.15.150:0/112287859 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6fa8058de0 con 0x7f6fc00b5cd0 2024-03-17T23:47:32.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.604+0000 7f6fc53cd640 1 -- 172.21.15.150:0/112287859 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6fc00d18e0 con 0x7f6fc00b5cd0 2024-03-17T23:47:32.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.604+0000 7f6fb77fe640 1 -- 172.21.15.150:0/112287859 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f6fa805ce80 con 0x7f6fc00b5cd0 2024-03-17T23:47:32.605 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:47:32.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.606+0000 7f6fc53cd640 1 -- 172.21.15.150:0/112287859 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6fa006bbd0 msgr2=0x7f6fa006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:32.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.606+0000 7f6fc53cd640 1 --2- 172.21.15.150:0/112287859 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6fa006bbd0 0x7f6fa006e090 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6fb0036000 tx=0x7f6fb003b040 comp rx=0 tx=0).stop 2024-03-17T23:47:32.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.606+0000 7f6fc53cd640 1 -- 172.21.15.150:0/112287859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fc00b5cd0 msgr2=0x7f6fc00cdd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:32.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.606+0000 7f6fc53cd640 1 --2- 172.21.15.150:0/112287859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fc00b5cd0 0x7f6fc00cdd60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6fa800ebe0 tx=0x7f6fa800c520 comp rx=0 tx=0).stop 2024-03-17T23:47:32.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.606+0000 7f6fc53cd640 1 -- 172.21.15.150:0/112287859 shutdown_connections 2024-03-17T23:47:32.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.606+0000 7f6fc53cd640 1 --2- 172.21.15.150:0/112287859 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6fc015de00 0x7f6fc00b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.606+0000 7f6fc53cd640 1 --2- 172.21.15.150:0/112287859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6fc00b5cd0 0x7f6fc00cdd60 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.606+0000 7f6fc53cd640 1 --2- 172.21.15.150:0/112287859 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6fa006bbd0 0x7f6fa006e090 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.606+0000 7f6fc53cd640 1 --2- 172.21.15.150:0/112287859 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6fc006c690 0x7f6fc015a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.606+0000 7f6fc53cd640 1 -- 172.21.15.150:0/112287859 >> 172.21.15.150:0/112287859 conn(0x7f6fc00d0790 msgr2=0x7f6fc015c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:32.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.606+0000 7f6fc53cd640 1 -- 172.21.15.150:0/112287859 shutdown_connections 2024-03-17T23:47:32.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.606+0000 7f6fc53cd640 1 -- 172.21.15.150:0/112287859 wait complete. 2024-03-17T23:47:32.618 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-17T23:47:32.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.722+0000 7f1abe19d640 1 Processor -- start 2024-03-17T23:47:32.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.722+0000 7f1abe19d640 1 -- start start 2024-03-17T23:47:32.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.723+0000 7f1abe19d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1ab8151c60 0x7f1ab80af7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:32.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.723+0000 7f1abe19d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1ab815bcd0 0x7f1ab80cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:32.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.723+0000 7f1abe19d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ab80cfc50 con 0x7f1ab806c1a0 2024-03-17T23:47:32.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.723+0000 7f1abe19d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f1ab80cff30 con 0x7f1ab8151c60 2024-03-17T23:47:32.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.723+0000 7f1abe19d640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f1ab815b1f0 con 0x7f1ab815bcd0 2024-03-17T23:47:32.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.723+0000 7f1abd19b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1ab8151c60 0x7f1ab80af7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:32.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.723+0000 7f1abd99c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f1ab806c1a0 0x7f1ab815e5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33706/0 (socket says 172.21.15.150:33706) 2024-03-17T23:47:32.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.723+0000 7f1abd99c640 1 -- 172.21.15.150:0/1357239437 learned_addr learned my addr 172.21.15.150:0/1357239437 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:32.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.723+0000 7f1abc99a640 1 --2- 172.21.15.150:0/1357239437 >> v2:172.21.15.157:3300/0 conn(0x7f1ab815bcd0 0x7f1ab80cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:32.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.723+0000 7f1abd19b640 1 -- 172.21.15.150:0/1357239437 >> v2:172.21.15.157:3300/0 conn(0x7f1ab815bcd0 msgr2=0x7f1ab80cf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:32.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.723+0000 7f1abd19b640 1 --2- 172.21.15.150:0/1357239437 >> v2:172.21.15.157:3300/0 conn(0x7f1ab815bcd0 0x7f1ab80cf3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.723+0000 7f1abd19b640 1 -- 172.21.15.150:0/1357239437 >> v1:172.21.15.150:6789/0 conn(0x7f1ab806c1a0 legacy=0x7f1ab815e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:32.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.723+0000 7f1aaffff640 1 -- 172.21.15.150:0/1357239437 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 152846323 0 0) 0x7f1ab80cfc50 con 0x7f1ab806c1a0 2024-03-17T23:47:32.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.723+0000 7f1abd19b640 1 -- 172.21.15.150:0/1357239437 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ab815b4d0 con 0x7f1ab8151c60 2024-03-17T23:47:32.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.724+0000 7f1abd19b640 1 --2- 172.21.15.150:0/1357239437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ab8151c60 0x7f1ab80af7c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1ab806cfb0 tx=0x7f1ab402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10526b32e4ae6823 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:32.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.724+0000 7f1aaffff640 1 -- 172.21.15.150:0/1357239437 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab4035020 con 0x7f1ab8151c60 2024-03-17T23:47:32.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.724+0000 7f1aaffff640 1 -- 172.21.15.150:0/1357239437 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ab4031830 con 0x7f1ab8151c60 2024-03-17T23:47:32.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.724+0000 7f1aaffff640 1 -- 172.21.15.150:0/1357239437 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab4031c00 con 0x7f1ab8151c60 2024-03-17T23:47:32.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.725+0000 7f1abe19d640 1 -- 172.21.15.150:0/1357239437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ab8151c60 msgr2=0x7f1ab80af7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:32.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.725+0000 7f1abe19d640 1 --2- 172.21.15.150:0/1357239437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ab8151c60 0x7f1ab80af7c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1ab806cfb0 tx=0x7f1ab402b8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.725+0000 7f1abe19d640 1 -- 172.21.15.150:0/1357239437 shutdown_connections 2024-03-17T23:47:32.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.725+0000 7f1abe19d640 1 --2- 172.21.15.150:0/1357239437 >> v2:172.21.15.157:3300/0 conn(0x7f1ab815bcd0 0x7f1ab80cf3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.725+0000 7f1abe19d640 1 --2- 172.21.15.150:0/1357239437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ab8151c60 0x7f1ab80af7c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.725+0000 7f1abe19d640 1 -- 172.21.15.150:0/1357239437 >> 172.21.15.150:0/1357239437 conn(0x7f1ab806f160 msgr2=0x7f1ab80722f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:32.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.725+0000 7f1abe19d640 1 -- 172.21.15.150:0/1357239437 shutdown_connections 2024-03-17T23:47:32.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.725+0000 7f1abe19d640 1 -- 172.21.15.150:0/1357239437 wait complete. 2024-03-17T23:47:32.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.726+0000 7f1abe19d640 1 Processor -- start 2024-03-17T23:47:32.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.726+0000 7f1abe19d640 1 -- start start 2024-03-17T23:47:32.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.727+0000 7f1abe19d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ab806c1a0 0x7f1ab80b3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:32.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.727+0000 7f1abe19d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ab8151c60 0x7f1ab814bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:32.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.727+0000 7f1abe19d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ab815bcd0 0x7f1ab814e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:32.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.727+0000 7f1abe19d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab806c630 con 0x7f1ab806c1a0 2024-03-17T23:47:32.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.727+0000 7f1abe19d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab80cf920 con 0x7f1ab815bcd0 2024-03-17T23:47:32.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.727+0000 7f1abe19d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1ab806c7b0 con 0x7f1ab8151c60 2024-03-17T23:47:32.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.727+0000 7f1abd19b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ab806c1a0 0x7f1ab80b3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:32.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.727+0000 7f1abd19b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ab806c1a0 0x7f1ab80b3e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46090/0 (socket says 172.21.15.150:46090) 2024-03-17T23:47:32.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.727+0000 7f1abd19b640 1 -- 172.21.15.150:0/1489283568 learned_addr learned my addr 172.21.15.150:0/1489283568 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:32.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.727+0000 7f1abd99c640 1 --2- 172.21.15.150:0/1489283568 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ab815bcd0 0x7f1ab814e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:32.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.727+0000 7f1abc99a640 1 --2- 172.21.15.150:0/1489283568 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ab8151c60 0x7f1ab814bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:32.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.727+0000 7f1abd19b640 1 -- 172.21.15.150:0/1489283568 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ab8151c60 msgr2=0x7f1ab814bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:32.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.727+0000 7f1abd19b640 1 --2- 172.21.15.150:0/1489283568 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ab8151c60 0x7f1ab814bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.727+0000 7f1abd19b640 1 -- 172.21.15.150:0/1489283568 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ab815bcd0 msgr2=0x7f1ab814e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:32.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.727+0000 7f1abd19b640 1 --2- 172.21.15.150:0/1489283568 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ab815bcd0 0x7f1ab814e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.727+0000 7f1abd19b640 1 -- 172.21.15.150:0/1489283568 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ab80b4340 con 0x7f1ab806c1a0 2024-03-17T23:47:32.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.727+0000 7f1abd19b640 1 --2- 172.21.15.150:0/1489283568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ab806c1a0 0x7f1ab80b3e00 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4009890 tx=0x7f1ab40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:32.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.727+0000 7f1aadffb640 1 -- 172.21.15.150:0/1489283568 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab403b8c0 con 0x7f1ab806c1a0 2024-03-17T23:47:32.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.728+0000 7f1aadffb640 1 -- 172.21.15.150:0/1489283568 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ab403baa0 con 0x7f1ab806c1a0 2024-03-17T23:47:32.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.728+0000 7f1abe19d640 1 -- 172.21.15.150:0/1489283568 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ab81f4000 con 0x7f1ab806c1a0 2024-03-17T23:47:32.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.728+0000 7f1aadffb640 1 -- 172.21.15.150:0/1489283568 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab40453f0 con 0x7f1ab806c1a0 2024-03-17T23:47:32.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.728+0000 7f1abe19d640 1 -- 172.21.15.150:0/1489283568 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ab80b3320 con 0x7f1ab806c1a0 2024-03-17T23:47:32.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.729+0000 7f1abe19d640 1 -- 172.21.15.150:0/1489283568 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ab8074c40 con 0x7f1ab806c1a0 2024-03-17T23:47:32.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.729+0000 7f1aadffb640 1 -- 172.21.15.150:0/1489283568 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1ab4038020 con 0x7f1ab806c1a0 2024-03-17T23:47:32.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.730+0000 7f1aadffb640 1 --2- 172.21.15.150:0/1489283568 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1a9c06bc00 0x7f1a9c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:32.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.730+0000 7f1abc99a640 1 --2- 172.21.15.150:0/1489283568 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1a9c06bc00 0x7f1a9c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:32.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.730+0000 7f1aadffb640 1 -- 172.21.15.150:0/1489283568 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f1ab40ac900 con 0x7f1ab806c1a0 2024-03-17T23:47:32.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.730+0000 7f1abc99a640 1 --2- 172.21.15.150:0/1489283568 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1a9c06bc00 0x7f1a9c06e0c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1ab814c960 tx=0x7f1aa8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:32.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.734+0000 7f1aadffb640 1 -- 172.21.15.150:0/1489283568 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1ab4040070 con 0x7f1ab806c1a0 2024-03-17T23:47:32.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.908+0000 7f1abe19d640 1 -- 172.21.15.150:0/1489283568 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f1ab806d340 con 0x7f1ab806c1a0 2024-03-17T23:47:32.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.909+0000 7f1aadffb640 1 -- 172.21.15.150:0/1489283568 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f1ab4031800 con 0x7f1ab806c1a0 2024-03-17T23:47:32.910 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:32.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.911+0000 7f1abe19d640 1 -- 172.21.15.150:0/1489283568 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1a9c06bc00 msgr2=0x7f1a9c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:32.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.911+0000 7f1abe19d640 1 --2- 172.21.15.150:0/1489283568 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1a9c06bc00 0x7f1a9c06e0c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1ab814c960 tx=0x7f1aa8009190 comp rx=0 tx=0).stop 2024-03-17T23:47:32.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.911+0000 7f1abe19d640 1 -- 172.21.15.150:0/1489283568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ab806c1a0 msgr2=0x7f1ab80b3e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:32.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.911+0000 7f1abe19d640 1 --2- 172.21.15.150:0/1489283568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ab806c1a0 0x7f1ab80b3e00 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4009890 tx=0x7f1ab40046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.911+0000 7f1abe19d640 1 -- 172.21.15.150:0/1489283568 shutdown_connections 2024-03-17T23:47:32.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.911+0000 7f1abe19d640 1 --2- 172.21.15.150:0/1489283568 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1a9c06bc00 0x7f1a9c06e0c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.911+0000 7f1abe19d640 1 --2- 172.21.15.150:0/1489283568 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ab815bcd0 0x7f1ab814e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.911+0000 7f1abe19d640 1 --2- 172.21.15.150:0/1489283568 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1ab8151c60 0x7f1ab814bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.911+0000 7f1abe19d640 1 --2- 172.21.15.150:0/1489283568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1ab806c1a0 0x7f1ab80b3e00 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:32.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.911+0000 7f1abe19d640 1 -- 172.21.15.150:0/1489283568 >> 172.21.15.150:0/1489283568 conn(0x7f1ab806f160 msgr2=0x7f1ab8157620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:32.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.911+0000 7f1abe19d640 1 -- 172.21.15.150:0/1489283568 shutdown_connections 2024-03-17T23:47:32.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:32.911+0000 7f1abe19d640 1 -- 172.21.15.150:0/1489283568 wait complete. 2024-03-17T23:47:32.923 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:33.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.024+0000 7f340100b640 1 Processor -- start 2024-03-17T23:47:33.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.024+0000 7f340100b640 1 -- start start 2024-03-17T23:47:33.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.025+0000 7f340100b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f33fc15bde0 0x7f33fc0ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:33.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.025+0000 7f340100b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f33fc0ceb40 0x7f33fc0d0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:33.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.025+0000 7f340100b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33fc0cfc70 con 0x7f33fc0cc660 2024-03-17T23:47:33.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.025+0000 7f340100b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f33fc0cefc0 con 0x7f33fc0ceb40 2024-03-17T23:47:33.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.025+0000 7f340100b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f33fc0cf2a0 con 0x7f33fc15bde0 2024-03-17T23:47:33.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.025+0000 7f33fb59e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f33fc0ceb40 0x7f33fc0d0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:33.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.025+0000 7f33f3fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f33fc0cc660 0x7f33fc06b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33716/0 (socket says 172.21.15.150:33716) 2024-03-17T23:47:33.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.025+0000 7f33f3fff640 1 -- 172.21.15.150:0/1539090516 learned_addr learned my addr 172.21.15.150:0/1539090516 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:33.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.025+0000 7f33fad9d640 1 --2- 172.21.15.150:0/1539090516 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f33fc15bde0 0x7f33fc0ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:33.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.025+0000 7f33fb59e640 1 -- 172.21.15.150:0/1539090516 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f33fc15bde0 msgr2=0x7f33fc0ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:33.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.025+0000 7f33fb59e640 1 --2- 172.21.15.150:0/1539090516 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f33fc15bde0 0x7f33fc0ce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.025+0000 7f33fb59e640 1 -- 172.21.15.150:0/1539090516 >> v1:172.21.15.150:6789/0 conn(0x7f33fc0cc660 legacy=0x7f33fc06b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:33.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.025+0000 7f33fa59c640 1 -- 172.21.15.150:0/1539090516 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1208114325 0 0) 0x7f33fc0cfc70 con 0x7f33fc0cc660 2024-03-17T23:47:33.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.025+0000 7f33fb59e640 1 -- 172.21.15.150:0/1539090516 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33fc0cf580 con 0x7f33fc0ceb40 2024-03-17T23:47:33.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.026+0000 7f33fb59e640 1 --2- 172.21.15.150:0/1539090516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33fc0ceb40 0x7f33fc0d0150 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f33f4009d90 tx=0x7f33f402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce38cb542c7a264d server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:33.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.026+0000 7f33fa59c640 1 -- 172.21.15.150:0/1539090516 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f4039070 con 0x7f33fc0ceb40 2024-03-17T23:47:33.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.026+0000 7f33fa59c640 1 -- 172.21.15.150:0/1539090516 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33f402cc50 con 0x7f33fc0ceb40 2024-03-17T23:47:33.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.026+0000 7f33fa59c640 1 -- 172.21.15.150:0/1539090516 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f4034440 con 0x7f33fc0ceb40 2024-03-17T23:47:33.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.027+0000 7f340100b640 1 -- 172.21.15.150:0/1539090516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33fc0ceb40 msgr2=0x7f33fc0d0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:33.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.027+0000 7f340100b640 1 --2- 172.21.15.150:0/1539090516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33fc0ceb40 0x7f33fc0d0150 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f33f4009d90 tx=0x7f33f402bea0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.027+0000 7f340100b640 1 -- 172.21.15.150:0/1539090516 shutdown_connections 2024-03-17T23:47:33.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.027+0000 7f340100b640 1 --2- 172.21.15.150:0/1539090516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33fc0ceb40 0x7f33fc0d0150 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.027+0000 7f340100b640 1 --2- 172.21.15.150:0/1539090516 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f33fc15bde0 0x7f33fc0ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.027+0000 7f340100b640 1 -- 172.21.15.150:0/1539090516 >> 172.21.15.150:0/1539090516 conn(0x7f33fc099a70 msgr2=0x7f33fc099e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:33.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.027+0000 7f340100b640 1 -- 172.21.15.150:0/1539090516 shutdown_connections 2024-03-17T23:47:33.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.027+0000 7f340100b640 1 -- 172.21.15.150:0/1539090516 wait complete. 2024-03-17T23:47:33.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.028+0000 7f340100b640 1 Processor -- start 2024-03-17T23:47:33.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.028+0000 7f340100b640 1 -- start start 2024-03-17T23:47:33.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.029+0000 7f340100b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f33fc0cc660 0x7f33fc0d0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:33.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.029+0000 7f340100b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33fc0ceb40 0x7f33fc15e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:33.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.029+0000 7f340100b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f33fc15bde0 0x7f33fc1559d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:33.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.029+0000 7f340100b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f33fc06ba20 con 0x7f33fc0ceb40 2024-03-17T23:47:33.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.029+0000 7f340100b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f33fc06b8a0 con 0x7f33fc0cc660 2024-03-17T23:47:33.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.029+0000 7f340100b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f33fc0d0690 con 0x7f33fc15bde0 2024-03-17T23:47:33.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.029+0000 7f33f3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33fc0ceb40 0x7f33fc15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:33.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.029+0000 7f33f3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33fc0ceb40 0x7f33fc15e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46108/0 (socket says 172.21.15.150:46108) 2024-03-17T23:47:33.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.029+0000 7f33f3fff640 1 -- 172.21.15.150:0/1398066470 learned_addr learned my addr 172.21.15.150:0/1398066470 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:33.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.029+0000 7f33fb59e640 1 --2- 172.21.15.150:0/1398066470 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f33fc15bde0 0x7f33fc1559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:33.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.029+0000 7f33fad9d640 1 --2- 172.21.15.150:0/1398066470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f33fc0cc660 0x7f33fc0d0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:33.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.029+0000 7f33f3fff640 1 -- 172.21.15.150:0/1398066470 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f33fc15bde0 msgr2=0x7f33fc1559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:33.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.029+0000 7f33f3fff640 1 --2- 172.21.15.150:0/1398066470 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f33fc15bde0 0x7f33fc1559d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.029+0000 7f33f3fff640 1 -- 172.21.15.150:0/1398066470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f33fc0cc660 msgr2=0x7f33fc0d0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:33.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.029+0000 7f33f3fff640 1 --2- 172.21.15.150:0/1398066470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f33fc0cc660 0x7f33fc0d0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.029+0000 7f33f3fff640 1 -- 172.21.15.150:0/1398066470 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33fc0c7cc0 con 0x7f33fc0ceb40 2024-03-17T23:47:33.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.029+0000 7f33f3fff640 1 --2- 172.21.15.150:0/1398066470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33fc0ceb40 0x7f33fc15e700 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f33e400c4d0 tx=0x7f33e400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:33.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.029+0000 7f33f37fe640 1 -- 172.21.15.150:0/1398066470 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e4018070 con 0x7f33fc0ceb40 2024-03-17T23:47:33.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.029+0000 7f33f37fe640 1 -- 172.21.15.150:0/1398066470 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33e4007500 con 0x7f33fc0ceb40 2024-03-17T23:47:33.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.029+0000 7f33f37fe640 1 -- 172.21.15.150:0/1398066470 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e4004a20 con 0x7f33fc0ceb40 2024-03-17T23:47:33.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.029+0000 7f340100b640 1 -- 172.21.15.150:0/1398066470 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33fc0c7fa0 con 0x7f33fc0ceb40 2024-03-17T23:47:33.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.030+0000 7f340100b640 1 -- 172.21.15.150:0/1398066470 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33fc0c8560 con 0x7f33fc0ceb40 2024-03-17T23:47:33.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.031+0000 7f340100b640 1 -- 172.21.15.150:0/1398066470 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33fc0cd220 con 0x7f33fc0ceb40 2024-03-17T23:47:33.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.031+0000 7f33f37fe640 1 -- 172.21.15.150:0/1398066470 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f33e401b070 con 0x7f33fc0ceb40 2024-03-17T23:47:33.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.031+0000 7f33f37fe640 1 --2- 172.21.15.150:0/1398066470 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f33dc06bbd0 0x7f33dc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:33.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.031+0000 7f33f37fe640 1 -- 172.21.15.150:0/1398066470 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f33e4089fa0 con 0x7f33fc0ceb40 2024-03-17T23:47:33.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.032+0000 7f33fad9d640 1 --2- 172.21.15.150:0/1398066470 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f33dc06bbd0 0x7f33dc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:33.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.032+0000 7f33fad9d640 1 --2- 172.21.15.150:0/1398066470 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f33dc06bbd0 0x7f33dc06e090 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f33ec009aa0 tx=0x7f33ec009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:33.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.036+0000 7f33f37fe640 1 -- 172.21.15.150:0/1398066470 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f33fc0cd220 con 0x7f33fc0ceb40 2024-03-17T23:47:33.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.207+0000 7f340100b640 1 -- 172.21.15.150:0/1398066470 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f33fc0688b0 con 0x7f33fc0ceb40 2024-03-17T23:47:33.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.209+0000 7f33f37fe640 1 -- 172.21.15.150:0/1398066470 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18550 (secure 0 0 0) 0x7f33e4058190 con 0x7f33fc0ceb40 2024-03-17T23:47:33.209 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:33.209 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:01.940019+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:33.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.210+0000 7f340100b640 1 -- 172.21.15.150:0/1398066470 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f33dc06bbd0 msgr2=0x7f33dc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:33.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.210+0000 7f340100b640 1 --2- 172.21.15.150:0/1398066470 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f33dc06bbd0 0x7f33dc06e090 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f33ec009aa0 tx=0x7f33ec009190 comp rx=0 tx=0).stop 2024-03-17T23:47:33.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.210+0000 7f340100b640 1 -- 172.21.15.150:0/1398066470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33fc0ceb40 msgr2=0x7f33fc15e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:33.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.210+0000 7f340100b640 1 --2- 172.21.15.150:0/1398066470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33fc0ceb40 0x7f33fc15e700 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f33e400c4d0 tx=0x7f33e400c990 comp rx=0 tx=0).stop 2024-03-17T23:47:33.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.211+0000 7f340100b640 1 -- 172.21.15.150:0/1398066470 shutdown_connections 2024-03-17T23:47:33.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.211+0000 7f340100b640 1 --2- 172.21.15.150:0/1398066470 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f33fc15bde0 0x7f33fc1559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.211+0000 7f340100b640 1 --2- 172.21.15.150:0/1398066470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33fc0ceb40 0x7f33fc15e700 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.211+0000 7f340100b640 1 --2- 172.21.15.150:0/1398066470 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f33dc06bbd0 0x7f33dc06e090 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.211+0000 7f340100b640 1 --2- 172.21.15.150:0/1398066470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f33fc0cc660 0x7f33fc0d0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.211+0000 7f340100b640 1 -- 172.21.15.150:0/1398066470 >> 172.21.15.150:0/1398066470 conn(0x7f33fc099a70 msgr2=0x7f33fc099e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:33.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.211+0000 7f340100b640 1 -- 172.21.15.150:0/1398066470 shutdown_connections 2024-03-17T23:47:33.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.211+0000 7f340100b640 1 -- 172.21.15.150:0/1398066470 wait complete. 2024-03-17T23:47:33.223 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:46:53.610678+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:46:54.227335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-17T23:46:57.949235+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-17T23:47:33.223 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:47:33.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.325+0000 7f36d08d1640 1 Processor -- start 2024-03-17T23:47:33.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.325+0000 7f36d08d1640 1 -- start start 2024-03-17T23:47:33.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.326+0000 7f36d08d1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f36cc1514f0 0x7f36cc15bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:33.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.326+0000 7f36d08d1640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f36cc15ab90 0x7f36cc06f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:33.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.326+0000 7f36d08d1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36cc0ce790 con 0x7f36cc06e5c0 2024-03-17T23:47:33.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.326+0000 7f36d08d1640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f36cc06fba0 con 0x7f36cc1514f0 2024-03-17T23:47:33.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.326+0000 7f36d08d1640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f36cc15b050 con 0x7f36cc15ab90 2024-03-17T23:47:33.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.326+0000 7f36ca59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f36cc1514f0 0x7f36cc15bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:33.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.326+0000 7f36cad9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f36cc06e5c0 0x7f36cc0cec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33730/0 (socket says 172.21.15.150:33730) 2024-03-17T23:47:33.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.326+0000 7f36cad9d640 1 -- 172.21.15.150:0/718202817 learned_addr learned my addr 172.21.15.150:0/718202817 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:33.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.326+0000 7f36c9d9b640 1 --2- 172.21.15.150:0/718202817 >> v2:172.21.15.157:3300/0 conn(0x7f36cc15ab90 0x7f36cc06f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:33.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.327+0000 7f36c9d9b640 1 -- 172.21.15.150:0/718202817 >> v2:172.21.15.150:3300/0 conn(0x7f36cc1514f0 msgr2=0x7f36cc15bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:33.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.327+0000 7f36c9d9b640 1 --2- 172.21.15.150:0/718202817 >> v2:172.21.15.150:3300/0 conn(0x7f36cc1514f0 0x7f36cc15bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.327+0000 7f36c9d9b640 1 -- 172.21.15.150:0/718202817 >> v1:172.21.15.150:6789/0 conn(0x7f36cc06e5c0 legacy=0x7f36cc0cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:33.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.327+0000 7f36c9d9b640 1 -- 172.21.15.150:0/718202817 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36cc15b330 con 0x7f36cc15ab90 2024-03-17T23:47:33.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.327+0000 7f36ca59c640 1 --2- 172.21.15.150:0/718202817 >> v2:172.21.15.150:3300/0 conn(0x7f36cc1514f0 0x7f36cc15bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:33.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.327+0000 7f36c9d9b640 1 --2- 172.21.15.150:0/718202817 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36cc15ab90 0x7f36cc06f660 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f36cc06f2d0 tx=0x7f36c002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=670654a78527e0d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:33.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.328+0000 7f36c959a640 1 -- 172.21.15.150:0/718202817 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c0035020 con 0x7f36cc15ab90 2024-03-17T23:47:33.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.328+0000 7f36c959a640 1 -- 172.21.15.150:0/718202817 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36c0031830 con 0x7f36cc15ab90 2024-03-17T23:47:33.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.328+0000 7f36c959a640 1 -- 172.21.15.150:0/718202817 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c0031c00 con 0x7f36cc15ab90 2024-03-17T23:47:33.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.328+0000 7f36d08d1640 1 -- 172.21.15.150:0/718202817 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36cc15ab90 msgr2=0x7f36cc06f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:33.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.328+0000 7f36d08d1640 1 --2- 172.21.15.150:0/718202817 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36cc15ab90 0x7f36cc06f660 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f36cc06f2d0 tx=0x7f36c002b8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.328+0000 7f36d08d1640 1 -- 172.21.15.150:0/718202817 shutdown_connections 2024-03-17T23:47:33.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.328+0000 7f36d08d1640 1 --2- 172.21.15.150:0/718202817 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36cc15ab90 0x7f36cc06f660 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.328+0000 7f36d08d1640 1 --2- 172.21.15.150:0/718202817 >> v2:172.21.15.150:3300/0 conn(0x7f36cc1514f0 0x7f36cc15bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.328+0000 7f36d08d1640 1 -- 172.21.15.150:0/718202817 >> 172.21.15.150:0/718202817 conn(0x7f36cc06b190 msgr2=0x7f36cc06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:33.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.328+0000 7f36d08d1640 1 -- 172.21.15.150:0/718202817 shutdown_connections 2024-03-17T23:47:33.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.328+0000 7f36d08d1640 1 -- 172.21.15.150:0/718202817 wait complete. 2024-03-17T23:47:33.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.330+0000 7f36d08d1640 1 Processor -- start 2024-03-17T23:47:33.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.330+0000 7f36d08d1640 1 -- start start 2024-03-17T23:47:33.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.330+0000 7f36d08d1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36cc06e5c0 0x7f36cc145d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:33.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.330+0000 7f36d08d1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f36cc1514f0 0x7f36cc14cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:33.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.330+0000 7f36d08d1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f36cc15ab90 0x7f36cc1419b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:33.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.330+0000 7f36d08d1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f36cc0af510 con 0x7f36cc15ab90 2024-03-17T23:47:33.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.330+0000 7f36d08d1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f36cc15b8f0 con 0x7f36cc06e5c0 2024-03-17T23:47:33.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.330+0000 7f36d08d1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f36cc1518d0 con 0x7f36cc1514f0 2024-03-17T23:47:33.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.330+0000 7f36cad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f36cc15ab90 0x7f36cc1419b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:33.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.330+0000 7f36cad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f36cc15ab90 0x7f36cc1419b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46122/0 (socket says 172.21.15.150:46122) 2024-03-17T23:47:33.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.331+0000 7f36cad9d640 1 -- 172.21.15.150:0/1112663463 learned_addr learned my addr 172.21.15.150:0/1112663463 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:33.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.331+0000 7f36ca59c640 1 --2- 172.21.15.150:0/1112663463 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36cc06e5c0 0x7f36cc145d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:33.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.331+0000 7f36c9d9b640 1 --2- 172.21.15.150:0/1112663463 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f36cc1514f0 0x7f36cc14cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:33.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.331+0000 7f36cad9d640 1 -- 172.21.15.150:0/1112663463 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f36cc1514f0 msgr2=0x7f36cc14cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:33.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.331+0000 7f36cad9d640 1 --2- 172.21.15.150:0/1112663463 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f36cc1514f0 0x7f36cc14cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.331+0000 7f36cad9d640 1 -- 172.21.15.150:0/1112663463 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36cc06e5c0 msgr2=0x7f36cc145d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:33.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.331+0000 7f36cad9d640 1 --2- 172.21.15.150:0/1112663463 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36cc06e5c0 0x7f36cc145d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.331+0000 7f36cad9d640 1 -- 172.21.15.150:0/1112663463 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36cc148050 con 0x7f36cc15ab90 2024-03-17T23:47:33.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.331+0000 7f36cad9d640 1 --2- 172.21.15.150:0/1112663463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f36cc15ab90 0x7f36cc1419b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f36c400c3a0 tx=0x7f36c400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:33.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.331+0000 7f36baffd640 1 -- 172.21.15.150:0/1112663463 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c4017070 con 0x7f36cc15ab90 2024-03-17T23:47:33.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.331+0000 7f36baffd640 1 -- 172.21.15.150:0/1112663463 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36c4005ce0 con 0x7f36cc15ab90 2024-03-17T23:47:33.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.331+0000 7f36d08d1640 1 -- 172.21.15.150:0/1112663463 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36cc154ab0 con 0x7f36cc15ab90 2024-03-17T23:47:33.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.331+0000 7f36baffd640 1 -- 172.21.15.150:0/1112663463 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c40072d0 con 0x7f36cc15ab90 2024-03-17T23:47:33.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.331+0000 7f36d08d1640 1 -- 172.21.15.150:0/1112663463 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36cc141ef0 con 0x7f36cc15ab90 2024-03-17T23:47:33.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.332+0000 7f36d08d1640 1 -- 172.21.15.150:0/1112663463 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36cc0681a0 con 0x7f36cc15ab90 2024-03-17T23:47:33.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.333+0000 7f36baffd640 1 -- 172.21.15.150:0/1112663463 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f36c401a070 con 0x7f36cc15ab90 2024-03-17T23:47:33.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.333+0000 7f36baffd640 1 --2- 172.21.15.150:0/1112663463 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f36ac06bbd0 0x7f36ac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:33.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.333+0000 7f36ca59c640 1 --2- 172.21.15.150:0/1112663463 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f36ac06bbd0 0x7f36ac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:33.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.333+0000 7f36baffd640 1 -- 172.21.15.150:0/1112663463 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f36c4089f30 con 0x7f36cc15ab90 2024-03-17T23:47:33.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.334+0000 7f36ca59c640 1 --2- 172.21.15.150:0/1112663463 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f36ac06bbd0 0x7f36ac06e090 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f36bc009bd0 tx=0x7f36bc00b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:33.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.337+0000 7f36baffd640 1 -- 172.21.15.150:0/1112663463 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f36cc0ce790 con 0x7f36cc15ab90 2024-03-17T23:47:33.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.509+0000 7f36d08d1640 1 -- 172.21.15.150:0/1112663463 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f36cc1438a0 con 0x7f36cc15ab90 2024-03-17T23:47:33.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.509+0000 7f36baffd640 1 -- 172.21.15.150:0/1112663463 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f36c4058120 con 0x7f36cc15ab90 2024-03-17T23:47:33.510 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:47:33.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.511+0000 7f36d08d1640 1 -- 172.21.15.150:0/1112663463 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f36ac06bbd0 msgr2=0x7f36ac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:33.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.511+0000 7f36d08d1640 1 --2- 172.21.15.150:0/1112663463 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f36ac06bbd0 0x7f36ac06e090 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f36bc009bd0 tx=0x7f36bc00b040 comp rx=0 tx=0).stop 2024-03-17T23:47:33.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.511+0000 7f36d08d1640 1 -- 172.21.15.150:0/1112663463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f36cc15ab90 msgr2=0x7f36cc1419b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:33.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.511+0000 7f36d08d1640 1 --2- 172.21.15.150:0/1112663463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f36cc15ab90 0x7f36cc1419b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f36c400c3a0 tx=0x7f36c400c860 comp rx=0 tx=0).stop 2024-03-17T23:47:33.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.511+0000 7f36d08d1640 1 -- 172.21.15.150:0/1112663463 shutdown_connections 2024-03-17T23:47:33.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.511+0000 7f36d08d1640 1 --2- 172.21.15.150:0/1112663463 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f36cc15ab90 0x7f36cc1419b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.511+0000 7f36d08d1640 1 --2- 172.21.15.150:0/1112663463 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f36cc1514f0 0x7f36cc14cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.511+0000 7f36d08d1640 1 --2- 172.21.15.150:0/1112663463 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f36ac06bbd0 0x7f36ac06e090 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.511+0000 7f36d08d1640 1 --2- 172.21.15.150:0/1112663463 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36cc06e5c0 0x7f36cc145d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.511+0000 7f36d08d1640 1 -- 172.21.15.150:0/1112663463 >> 172.21.15.150:0/1112663463 conn(0x7f36cc06b190 msgr2=0x7f36cc0aff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:33.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.511+0000 7f36d08d1640 1 -- 172.21.15.150:0/1112663463 shutdown_connections 2024-03-17T23:47:33.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.511+0000 7f36d08d1640 1 -- 172.21.15.150:0/1112663463 wait complete. 2024-03-17T23:47:33.523 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:47:33.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.626+0000 7f321ec85640 1 Processor -- start 2024-03-17T23:47:33.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.627+0000 7f321ec85640 1 -- start start 2024-03-17T23:47:33.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.627+0000 7f321ec85640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32180d0ce0 0x7f32180b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:33.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.627+0000 7f321ec85640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f32180b2bf0 0x7f32180d1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:33.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.627+0000 7f321ec85640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3218161d30 con 0x7f3218161520 2024-03-17T23:47:33.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.627+0000 7f321ec85640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f32180b9bf0 con 0x7f32180b2bf0 2024-03-17T23:47:33.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.627+0000 7f321ec85640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3218162940 con 0x7f32180d0ce0 2024-03-17T23:47:33.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.627+0000 7f321ca21640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f3218161520 0x7f32180d05d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33744/0 (socket says 172.21.15.150:33744) 2024-03-17T23:47:33.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.627+0000 7f321ca21640 1 -- 172.21.15.150:0/2369983347 learned_addr learned my addr 172.21.15.150:0/2369983347 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:33.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.627+0000 7f321d222640 1 --2- 172.21.15.150:0/2369983347 >> v2:172.21.15.150:3300/0 conn(0x7f32180b2bf0 0x7f32180d1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:33.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.628+0000 7f3217fff640 1 --2- 172.21.15.150:0/2369983347 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32180d0ce0 0x7f32180b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:33.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.628+0000 7f32177fe640 1 -- 172.21.15.150:0/2369983347 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 823423193 0 0) 0x7f3218161d30 con 0x7f3218161520 2024-03-17T23:47:33.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.628+0000 7f32177fe640 1 -- 172.21.15.150:0/2369983347 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3200003520 con 0x7f3218161520 2024-03-17T23:47:33.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.628+0000 7f321d222640 1 -- 172.21.15.150:0/2369983347 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32180d0ce0 msgr2=0x7f32180b96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:33.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.628+0000 7f321d222640 1 --2- 172.21.15.150:0/2369983347 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32180d0ce0 0x7f32180b96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.628+0000 7f321d222640 1 -- 172.21.15.150:0/2369983347 >> v1:172.21.15.150:6789/0 conn(0x7f3218161520 legacy=0x7f32180d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:33.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.628+0000 7f321d222640 1 -- 172.21.15.150:0/2369983347 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3218162010 con 0x7f32180b2bf0 2024-03-17T23:47:33.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.628+0000 7f3217fff640 1 --2- 172.21.15.150:0/2369983347 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32180d0ce0 0x7f32180b96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:33.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.629+0000 7f321d222640 1 --2- 172.21.15.150:0/2369983347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32180b2bf0 0x7f32180d1fa0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3210009d90 tx=0x7f321002bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aa74246cb43da84 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:33.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.629+0000 7f32177fe640 1 -- 172.21.15.150:0/2369983347 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3210039070 con 0x7f32180b2bf0 2024-03-17T23:47:33.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.629+0000 7f32177fe640 1 -- 172.21.15.150:0/2369983347 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f321002cb30 con 0x7f32180b2bf0 2024-03-17T23:47:33.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.629+0000 7f32177fe640 1 -- 172.21.15.150:0/2369983347 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3210034440 con 0x7f32180b2bf0 2024-03-17T23:47:33.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.629+0000 7f321ec85640 1 -- 172.21.15.150:0/2369983347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32180b2bf0 msgr2=0x7f32180d1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:33.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.629+0000 7f321ec85640 1 --2- 172.21.15.150:0/2369983347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32180b2bf0 0x7f32180d1fa0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3210009d90 tx=0x7f321002bd80 comp rx=0 tx=0).stop 2024-03-17T23:47:33.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.629+0000 7f321ec85640 1 -- 172.21.15.150:0/2369983347 shutdown_connections 2024-03-17T23:47:33.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.629+0000 7f321ec85640 1 --2- 172.21.15.150:0/2369983347 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32180d0ce0 0x7f32180b96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.629+0000 7f321ec85640 1 --2- 172.21.15.150:0/2369983347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32180b2bf0 0x7f32180d1fa0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.629+0000 7f321ec85640 1 -- 172.21.15.150:0/2369983347 >> 172.21.15.150:0/2369983347 conn(0x7f321806ca00 msgr2=0x7f321806ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:33.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.629+0000 7f321ec85640 1 -- 172.21.15.150:0/2369983347 shutdown_connections 2024-03-17T23:47:33.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.629+0000 7f321ec85640 1 -- 172.21.15.150:0/2369983347 wait complete. 2024-03-17T23:47:33.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.630+0000 7f321ec85640 1 Processor -- start 2024-03-17T23:47:33.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.631+0000 7f321ec85640 1 -- start start 2024-03-17T23:47:33.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.631+0000 7f321ec85640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32180b2bf0 0x7f3218154b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:33.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.631+0000 7f321ec85640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32180d0ce0 0x7f32180cb800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:33.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.631+0000 7f321ec85640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3218161520 0x7f32180cd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:33.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.631+0000 7f321ec85640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3218161900 con 0x7f32180b2bf0 2024-03-17T23:47:33.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.631+0000 7f321ec85640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f32181625d0 con 0x7f3218161520 2024-03-17T23:47:33.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.631+0000 7f321ec85640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f32180d24e0 con 0x7f32180d0ce0 2024-03-17T23:47:33.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.632+0000 7f321ca21640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32180b2bf0 0x7f3218154b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:33.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.632+0000 7f321ca21640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32180b2bf0 0x7f3218154b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46146/0 (socket says 172.21.15.150:46146) 2024-03-17T23:47:33.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.632+0000 7f321ca21640 1 -- 172.21.15.150:0/2786216173 learned_addr learned my addr 172.21.15.150:0/2786216173 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:33.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.632+0000 7f321d222640 1 --2- 172.21.15.150:0/2786216173 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3218161520 0x7f32180cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:33.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.632+0000 7f3217fff640 1 --2- 172.21.15.150:0/2786216173 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32180d0ce0 0x7f32180cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:33.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.632+0000 7f321ca21640 1 -- 172.21.15.150:0/2786216173 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32180d0ce0 msgr2=0x7f32180cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:33.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.632+0000 7f321ca21640 1 --2- 172.21.15.150:0/2786216173 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32180d0ce0 0x7f32180cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.632+0000 7f321ca21640 1 -- 172.21.15.150:0/2786216173 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3218161520 msgr2=0x7f32180cd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:33.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.632+0000 7f321ca21640 1 --2- 172.21.15.150:0/2786216173 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3218161520 0x7f32180cd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.632+0000 7f321ca21640 1 -- 172.21.15.150:0/2786216173 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32180cbf70 con 0x7f32180b2bf0 2024-03-17T23:47:33.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.632+0000 7f321d222640 1 --2- 172.21.15.150:0/2786216173 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3218161520 0x7f32180cd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:33.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.632+0000 7f321ca21640 1 --2- 172.21.15.150:0/2786216173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32180b2bf0 0x7f3218154b90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f320800c3c0 tx=0x7f320800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:33.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.632+0000 7f32157fa640 1 -- 172.21.15.150:0/2786216173 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3208017070 con 0x7f32180b2bf0 2024-03-17T23:47:33.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.632+0000 7f32157fa640 1 -- 172.21.15.150:0/2786216173 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3208005ce0 con 0x7f32180b2bf0 2024-03-17T23:47:33.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.632+0000 7f321ec85640 1 -- 172.21.15.150:0/2786216173 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32180cc250 con 0x7f32180b2bf0 2024-03-17T23:47:33.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.632+0000 7f32157fa640 1 -- 172.21.15.150:0/2786216173 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32080072f0 con 0x7f32180b2bf0 2024-03-17T23:47:33.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.632+0000 7f321ec85640 1 -- 172.21.15.150:0/2786216173 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32180cc810 con 0x7f32180b2bf0 2024-03-17T23:47:33.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.633+0000 7f321ec85640 1 -- 172.21.15.150:0/2786216173 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32180681a0 con 0x7f32180b2bf0 2024-03-17T23:47:33.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.634+0000 7f32157fa640 1 -- 172.21.15.150:0/2786216173 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f320801a070 con 0x7f32180b2bf0 2024-03-17T23:47:33.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.634+0000 7f32157fa640 1 --2- 172.21.15.150:0/2786216173 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f320406bbd0 0x7f320406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:33.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.634+0000 7f32157fa640 1 -- 172.21.15.150:0/2786216173 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f320808a0f0 con 0x7f32180b2bf0 2024-03-17T23:47:33.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.634+0000 7f3217fff640 1 --2- 172.21.15.150:0/2786216173 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f320406bbd0 0x7f320406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:33.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.634+0000 7f3217fff640 1 --2- 172.21.15.150:0/2786216173 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f320406bbd0 0x7f320406e090 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f320c009560 tx=0x7f320c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:33.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.638+0000 7f32157fa640 1 -- 172.21.15.150:0/2786216173 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3208058250 con 0x7f32180b2bf0 2024-03-17T23:47:33.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.810+0000 7f321ec85640 1 -- 172.21.15.150:0/2786216173 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f32180d0a60 con 0x7f32180b2bf0 2024-03-17T23:47:33.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.810+0000 7f32157fa640 1 -- 172.21.15.150:0/2786216173 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f320805c2f0 con 0x7f32180b2bf0 2024-03-17T23:47:33.811 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:33.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.812+0000 7f321ec85640 1 -- 172.21.15.150:0/2786216173 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f320406bbd0 msgr2=0x7f320406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:33.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.812+0000 7f321ec85640 1 --2- 172.21.15.150:0/2786216173 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f320406bbd0 0x7f320406e090 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f320c009560 tx=0x7f320c009190 comp rx=0 tx=0).stop 2024-03-17T23:47:33.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.812+0000 7f321ec85640 1 -- 172.21.15.150:0/2786216173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32180b2bf0 msgr2=0x7f3218154b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:33.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.812+0000 7f321ec85640 1 --2- 172.21.15.150:0/2786216173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32180b2bf0 0x7f3218154b90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f320800c3c0 tx=0x7f320800c880 comp rx=0 tx=0).stop 2024-03-17T23:47:33.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.812+0000 7f321ec85640 1 -- 172.21.15.150:0/2786216173 shutdown_connections 2024-03-17T23:47:33.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.812+0000 7f321ec85640 1 --2- 172.21.15.150:0/2786216173 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f320406bbd0 0x7f320406e090 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.812+0000 7f321ec85640 1 --2- 172.21.15.150:0/2786216173 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3218161520 0x7f32180cd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.812+0000 7f321ec85640 1 --2- 172.21.15.150:0/2786216173 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32180d0ce0 0x7f32180cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.812+0000 7f321ec85640 1 --2- 172.21.15.150:0/2786216173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32180b2bf0 0x7f3218154b90 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.812+0000 7f321ec85640 1 -- 172.21.15.150:0/2786216173 >> 172.21.15.150:0/2786216173 conn(0x7f321806ca00 msgr2=0x7f32180b6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:33.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.812+0000 7f321ec85640 1 -- 172.21.15.150:0/2786216173 shutdown_connections 2024-03-17T23:47:33.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.812+0000 7f321ec85640 1 -- 172.21.15.150:0/2786216173 wait complete. 2024-03-17T23:47:33.825 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:47:33.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.928+0000 7f2715510640 1 Processor -- start 2024-03-17T23:47:33.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.928+0000 7f2715510640 1 -- start start 2024-03-17T23:47:33.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.929+0000 7f2715510640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2710161520 0x7f27100b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:33.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.929+0000 7f2715510640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f27100b2bf0 0x7f27100d05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:33.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.929+0000 7f2715510640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27100d0b10 0x7f27100b57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:33.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.929+0000 7f2715510640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f27100b9bf0 con 0x7f27100b2bf0 2024-03-17T23:47:33.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.929+0000 7f2715510640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f2710162940 con 0x7f2710161520 2024-03-17T23:47:33.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.929+0000 7f2715510640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f27100d0f40 con 0x7f27100d0b10 2024-03-17T23:47:33.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.929+0000 7f270e7fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f27100b2bf0 0x7f27100d05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:33.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.929+0000 7f270e7fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f27100b2bf0 0x7f27100d05d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46154/0 (socket says 172.21.15.150:46154) 2024-03-17T23:47:33.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.929+0000 7f270e7fc640 1 -- 172.21.15.150:0/2360133539 learned_addr learned my addr 172.21.15.150:0/2360133539 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:33.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.929+0000 7f270effd640 1 --2- 172.21.15.150:0/2360133539 >> v2:172.21.15.157:3300/0 conn(0x7f2710161520 0x7f27100b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:33.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.929+0000 7f270f7fe640 1 --2- 172.21.15.150:0/2360133539 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27100d0b10 0x7f27100b57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:33.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.929+0000 7f270effd640 1 -- 172.21.15.150:0/2360133539 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27100d0b10 msgr2=0x7f27100b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:33.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.929+0000 7f270effd640 1 --2- 172.21.15.150:0/2360133539 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27100d0b10 0x7f27100b57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.929+0000 7f270effd640 1 -- 172.21.15.150:0/2360133539 >> v2:172.21.15.150:3300/0 conn(0x7f27100b2bf0 msgr2=0x7f27100d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:33.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.929+0000 7f270effd640 1 --2- 172.21.15.150:0/2360133539 >> v2:172.21.15.150:3300/0 conn(0x7f27100b2bf0 0x7f27100d05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.929+0000 7f270effd640 1 -- 172.21.15.150:0/2360133539 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2710161d30 con 0x7f2710161520 2024-03-17T23:47:33.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.929+0000 7f270e7fc640 1 --2- 172.21.15.150:0/2360133539 >> v2:172.21.15.150:3300/0 conn(0x7f27100b2bf0 0x7f27100d05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:33.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.930+0000 7f270effd640 1 --2- 172.21.15.150:0/2360133539 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2710161520 0x7f27100b96b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f271008c940 tx=0x7f270002bd00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=feedc85a5e46114a server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.930+0000 7f270dffb640 1 -- 172.21.15.150:0/2360133539 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2700039070 con 0x7f2710161520 2024-03-17T23:47:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.930+0000 7f270dffb640 1 -- 172.21.15.150:0/2360133539 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f270002cab0 con 0x7f2710161520 2024-03-17T23:47:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.930+0000 7f270dffb640 1 -- 172.21.15.150:0/2360133539 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2700034440 con 0x7f2710161520 2024-03-17T23:47:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.931+0000 7f2715510640 1 -- 172.21.15.150:0/2360133539 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2710161520 msgr2=0x7f27100b96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.931+0000 7f2715510640 1 --2- 172.21.15.150:0/2360133539 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2710161520 0x7f27100b96b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f271008c940 tx=0x7f270002bd00 comp rx=0 tx=0).stop 2024-03-17T23:47:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.931+0000 7f2715510640 1 -- 172.21.15.150:0/2360133539 shutdown_connections 2024-03-17T23:47:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.931+0000 7f2715510640 1 --2- 172.21.15.150:0/2360133539 >> v2:172.21.15.150:3300/0 conn(0x7f27100b2bf0 0x7f27100d05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.931+0000 7f2715510640 1 --2- 172.21.15.150:0/2360133539 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27100d0b10 0x7f27100b57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.931+0000 7f2715510640 1 --2- 172.21.15.150:0/2360133539 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2710161520 0x7f27100b96b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.931+0000 7f2715510640 1 -- 172.21.15.150:0/2360133539 >> 172.21.15.150:0/2360133539 conn(0x7f271006ca00 msgr2=0x7f271006ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.931+0000 7f2715510640 1 -- 172.21.15.150:0/2360133539 shutdown_connections 2024-03-17T23:47:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.931+0000 7f2715510640 1 -- 172.21.15.150:0/2360133539 wait complete. 2024-03-17T23:47:33.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.932+0000 7f2715510640 1 Processor -- start 2024-03-17T23:47:33.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.932+0000 7f2715510640 1 -- start start 2024-03-17T23:47:33.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.933+0000 7f2715510640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27100b2bf0 0x7f27100b60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:33.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.933+0000 7f2715510640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27100d0b10 0x7f27101525c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:33.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.933+0000 7f2715510640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2710161520 0x7f27100cd150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:33.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.933+0000 7f2715510640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f27101625d0 con 0x7f2710161520 2024-03-17T23:47:33.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.933+0000 7f2715510640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f27101622f0 con 0x7f27100b2bf0 2024-03-17T23:47:33.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.933+0000 7f2715510640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2710161900 con 0x7f27100d0b10 2024-03-17T23:47:33.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.933+0000 7f270f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2710161520 0x7f27100cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:33.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.933+0000 7f270f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2710161520 0x7f27100cd150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46164/0 (socket says 172.21.15.150:46164) 2024-03-17T23:47:33.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.933+0000 7f270f7fe640 1 -- 172.21.15.150:0/1366651153 learned_addr learned my addr 172.21.15.150:0/1366651153 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:33.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.933+0000 7f270effd640 1 --2- 172.21.15.150:0/1366651153 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27100b2bf0 0x7f27100b60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:33.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.933+0000 7f270e7fc640 1 --2- 172.21.15.150:0/1366651153 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27100d0b10 0x7f27101525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:33.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.933+0000 7f270effd640 1 -- 172.21.15.150:0/1366651153 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27100d0b10 msgr2=0x7f27101525c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:33.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.933+0000 7f270effd640 1 --2- 172.21.15.150:0/1366651153 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27100d0b10 0x7f27101525c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.933+0000 7f270effd640 1 -- 172.21.15.150:0/1366651153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2710161520 msgr2=0x7f27100cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:33.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.933+0000 7f270effd640 1 --2- 172.21.15.150:0/1366651153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2710161520 0x7f27100cd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:33.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.933+0000 7f270effd640 1 -- 172.21.15.150:0/1366651153 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27100cb9a0 con 0x7f27100b2bf0 2024-03-17T23:47:33.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.933+0000 7f270f7fe640 1 --2- 172.21.15.150:0/1366651153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2710161520 0x7f27100cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:33.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.934+0000 7f270effd640 1 --2- 172.21.15.150:0/1366651153 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27100b2bf0 0x7f27100b60b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2700033000 tx=0x7f2700004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:33.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.934+0000 7f26f77fe640 1 -- 172.21.15.150:0/1366651153 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2700039050 con 0x7f27100b2bf0 2024-03-17T23:47:33.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.934+0000 7f26f77fe640 1 -- 172.21.15.150:0/1366651153 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2700045070 con 0x7f27100b2bf0 2024-03-17T23:47:33.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.934+0000 7f2715510640 1 -- 172.21.15.150:0/1366651153 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27100cbc80 con 0x7f27100b2bf0 2024-03-17T23:47:33.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.934+0000 7f26f77fe640 1 -- 172.21.15.150:0/1366651153 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270003e850 con 0x7f27100b2bf0 2024-03-17T23:47:33.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.934+0000 7f2715510640 1 -- 172.21.15.150:0/1366651153 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27100b3890 con 0x7f27100b2bf0 2024-03-17T23:47:33.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.935+0000 7f2715510640 1 -- 172.21.15.150:0/1366651153 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27100681a0 con 0x7f27100b2bf0 2024-03-17T23:47:33.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.935+0000 7f26f77fe640 1 -- 172.21.15.150:0/1366651153 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f270003e9f0 con 0x7f27100b2bf0 2024-03-17T23:47:33.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.935+0000 7f26f77fe640 1 --2- 172.21.15.150:0/1366651153 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f26fc06bbd0 0x7f26fc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:33.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.936+0000 7f270e7fc640 1 --2- 172.21.15.150:0/1366651153 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f26fc06bbd0 0x7f26fc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:33.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.936+0000 7f26f77fe640 1 -- 172.21.15.150:0/1366651153 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f27000ad900 con 0x7f27100b2bf0 2024-03-17T23:47:33.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.936+0000 7f270e7fc640 1 --2- 172.21.15.150:0/1366651153 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f26fc06bbd0 0x7f26fc06e090 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f27040096d0 tx=0x7f2704009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:33.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:33.940+0000 7f26f77fe640 1 -- 172.21.15.150:0/1366651153 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f270007c260 con 0x7f27100b2bf0 2024-03-17T23:47:34.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.115+0000 7f2715510640 1 -- 172.21.15.150:0/1366651153 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f27100d19c0 con 0x7f27100b2bf0 2024-03-17T23:47:34.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.115+0000 7f26f77fe640 1 -- 172.21.15.150:0/1366651153 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f270007bb60 con 0x7f27100b2bf0 2024-03-17T23:47:34.116 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:47:34.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.117+0000 7f2715510640 1 -- 172.21.15.150:0/1366651153 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f26fc06bbd0 msgr2=0x7f26fc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:34.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.117+0000 7f2715510640 1 --2- 172.21.15.150:0/1366651153 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f26fc06bbd0 0x7f26fc06e090 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f27040096d0 tx=0x7f2704009240 comp rx=0 tx=0).stop 2024-03-17T23:47:34.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.117+0000 7f2715510640 1 -- 172.21.15.150:0/1366651153 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27100b2bf0 msgr2=0x7f27100b60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:34.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.117+0000 7f2715510640 1 --2- 172.21.15.150:0/1366651153 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27100b2bf0 0x7f27100b60b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2700033000 tx=0x7f2700004a40 comp rx=0 tx=0).stop 2024-03-17T23:47:34.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.117+0000 7f2715510640 1 -- 172.21.15.150:0/1366651153 shutdown_connections 2024-03-17T23:47:34.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.117+0000 7f2715510640 1 --2- 172.21.15.150:0/1366651153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2710161520 0x7f27100cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.117+0000 7f2715510640 1 --2- 172.21.15.150:0/1366651153 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f27100d0b10 0x7f27101525c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.117+0000 7f2715510640 1 --2- 172.21.15.150:0/1366651153 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f26fc06bbd0 0x7f26fc06e090 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.117+0000 7f2715510640 1 --2- 172.21.15.150:0/1366651153 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27100b2bf0 0x7f27100b60b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.117+0000 7f2715510640 1 -- 172.21.15.150:0/1366651153 >> 172.21.15.150:0/1366651153 conn(0x7f271006ca00 msgr2=0x7f27100d45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:34.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.117+0000 7f2715510640 1 -- 172.21.15.150:0/1366651153 shutdown_connections 2024-03-17T23:47:34.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.117+0000 7f2715510640 1 -- 172.21.15.150:0/1366651153 wait complete. 2024-03-17T23:47:34.129 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-17T23:47:34.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.240+0000 7f3a9864b640 1 Processor -- start 2024-03-17T23:47:34.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.240+0000 7f3a9864b640 1 -- start start 2024-03-17T23:47:34.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.241+0000 7f3a9864b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3a90144bb0 0x7f3a9015ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:34.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.241+0000 7f3a9864b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a900ce6a0 0x7f3a90165a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:34.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.241+0000 7f3a9864b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a90145170 con 0x7f3a9015d390 2024-03-17T23:47:34.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.241+0000 7f3a9864b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3a90145450 con 0x7f3a90144bb0 2024-03-17T23:47:34.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.241+0000 7f3a9864b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3a90145730 con 0x7f3a900ce6a0 2024-03-17T23:47:34.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.241+0000 7f3a95be6640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f3a9015d390 0x7f3a900cdf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:32998/0 (socket says 172.21.15.150:32998) 2024-03-17T23:47:34.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.241+0000 7f3a963e7640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3a90144bb0 0x7f3a9015ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:34.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.241+0000 7f3a95be6640 1 -- 172.21.15.150:0/29656974 learned_addr learned my addr 172.21.15.150:0/29656974 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:34.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.241+0000 7f3a96be8640 1 --2- 172.21.15.150:0/29656974 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a900ce6a0 0x7f3a90165a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:34.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.241+0000 7f3a953e5640 1 -- 172.21.15.150:0/29656974 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2938494084 0 0) 0x7f3a90145170 con 0x7f3a9015d390 2024-03-17T23:47:34.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.241+0000 7f3a953e5640 1 -- 172.21.15.150:0/29656974 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a7c003520 con 0x7f3a9015d390 2024-03-17T23:47:34.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.242+0000 7f3a963e7640 1 -- 172.21.15.150:0/29656974 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a900ce6a0 msgr2=0x7f3a90165a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:34.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.242+0000 7f3a963e7640 1 --2- 172.21.15.150:0/29656974 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a900ce6a0 0x7f3a90165a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.242+0000 7f3a963e7640 1 -- 172.21.15.150:0/29656974 >> v1:172.21.15.157:6789/0 conn(0x7f3a9015d390 legacy=0x7f3a900cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:34.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.242+0000 7f3a963e7640 1 -- 172.21.15.150:0/29656974 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a9006bd60 con 0x7f3a90144bb0 2024-03-17T23:47:34.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.242+0000 7f3a96be8640 1 --2- 172.21.15.150:0/29656974 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a900ce6a0 0x7f3a90165a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:34.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.242+0000 7f3a963e7640 1 --2- 172.21.15.150:0/29656974 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a90144bb0 0x7f3a9015ce50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c0097a0 tx=0x7f3a8c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa1e0a638c02a6f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:34.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.242+0000 7f3a953e5640 1 -- 172.21.15.150:0/29656974 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c0315a0 con 0x7f3a90144bb0 2024-03-17T23:47:34.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.243+0000 7f3a953e5640 1 -- 172.21.15.150:0/29656974 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a8c037070 con 0x7f3a90144bb0 2024-03-17T23:47:34.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.243+0000 7f3a953e5640 1 -- 172.21.15.150:0/29656974 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c031960 con 0x7f3a90144bb0 2024-03-17T23:47:34.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.243+0000 7f3a9864b640 1 -- 172.21.15.150:0/29656974 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a90144bb0 msgr2=0x7f3a9015ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:34.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.243+0000 7f3a9864b640 1 --2- 172.21.15.150:0/29656974 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a90144bb0 0x7f3a9015ce50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c0097a0 tx=0x7f3a8c02b840 comp rx=0 tx=0).stop 2024-03-17T23:47:34.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.243+0000 7f3a9864b640 1 -- 172.21.15.150:0/29656974 shutdown_connections 2024-03-17T23:47:34.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.243+0000 7f3a9864b640 1 --2- 172.21.15.150:0/29656974 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a900ce6a0 0x7f3a90165a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.243+0000 7f3a9864b640 1 --2- 172.21.15.150:0/29656974 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a90144bb0 0x7f3a9015ce50 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.243+0000 7f3a9864b640 1 -- 172.21.15.150:0/29656974 >> 172.21.15.150:0/29656974 conn(0x7f3a9006b190 msgr2=0x7f3a9006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:34.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.243+0000 7f3a9864b640 1 -- 172.21.15.150:0/29656974 shutdown_connections 2024-03-17T23:47:34.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.243+0000 7f3a9864b640 1 -- 172.21.15.150:0/29656974 wait complete. 2024-03-17T23:47:34.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.244+0000 7f3a9864b640 1 Processor -- start 2024-03-17T23:47:34.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.244+0000 7f3a9864b640 1 -- start start 2024-03-17T23:47:34.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.245+0000 7f3a9864b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a900ce6a0 0x7f3a900a9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:34.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.245+0000 7f3a9864b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a90144bb0 0x7f3a9013bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:34.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.245+0000 7f3a9864b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a9015d390 0x7f3a90142880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:34.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.245+0000 7f3a9864b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3a90165f70 con 0x7f3a9015d390 2024-03-17T23:47:34.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.245+0000 7f3a9864b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9006f610 con 0x7f3a900ce6a0 2024-03-17T23:47:34.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.245+0000 7f3a9864b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3a901660f0 con 0x7f3a90144bb0 2024-03-17T23:47:34.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.245+0000 7f3a96be8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a9015d390 0x7f3a90142880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:34.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.245+0000 7f3a96be8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a9015d390 0x7f3a90142880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46176/0 (socket says 172.21.15.150:46176) 2024-03-17T23:47:34.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.245+0000 7f3a96be8640 1 -- 172.21.15.150:0/1752615562 learned_addr learned my addr 172.21.15.150:0/1752615562 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:34.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.245+0000 7f3a95be6640 1 --2- 172.21.15.150:0/1752615562 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a90144bb0 0x7f3a9013bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:34.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.245+0000 7f3a963e7640 1 --2- 172.21.15.150:0/1752615562 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a900ce6a0 0x7f3a900a9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:34.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.246+0000 7f3a96be8640 1 -- 172.21.15.150:0/1752615562 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a90144bb0 msgr2=0x7f3a9013bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:34.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.246+0000 7f3a96be8640 1 --2- 172.21.15.150:0/1752615562 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a90144bb0 0x7f3a9013bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.246+0000 7f3a96be8640 1 -- 172.21.15.150:0/1752615562 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a900ce6a0 msgr2=0x7f3a900a9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:34.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.246+0000 7f3a96be8640 1 --2- 172.21.15.150:0/1752615562 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a900ce6a0 0x7f3a900a9b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.246+0000 7f3a96be8640 1 -- 172.21.15.150:0/1752615562 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a900ad560 con 0x7f3a9015d390 2024-03-17T23:47:34.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.246+0000 7f3a96be8640 1 --2- 172.21.15.150:0/1752615562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a9015d390 0x7f3a90142880 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3a8800efc0 tx=0x7f3a8800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:34.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.246+0000 7f3a86ffd640 1 -- 172.21.15.150:0/1752615562 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a88018030 con 0x7f3a9015d390 2024-03-17T23:47:34.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.246+0000 7f3a86ffd640 1 -- 172.21.15.150:0/1752615562 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a8800cdf0 con 0x7f3a9015d390 2024-03-17T23:47:34.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.246+0000 7f3a9864b640 1 -- 172.21.15.150:0/1752615562 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a901434d0 con 0x7f3a9015d390 2024-03-17T23:47:34.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.246+0000 7f3a86ffd640 1 -- 172.21.15.150:0/1752615562 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a88003bb0 con 0x7f3a9015d390 2024-03-17T23:47:34.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.246+0000 7f3a9864b640 1 -- 172.21.15.150:0/1752615562 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a901437b0 con 0x7f3a9015d390 2024-03-17T23:47:34.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.247+0000 7f3a9864b640 1 -- 172.21.15.150:0/1752615562 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a900681a0 con 0x7f3a9015d390 2024-03-17T23:47:34.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.247+0000 7f3a86ffd640 1 -- 172.21.15.150:0/1752615562 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3a88005d10 con 0x7f3a9015d390 2024-03-17T23:47:34.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.248+0000 7f3a86ffd640 1 --2- 172.21.15.150:0/1752615562 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3a7406bbd0 0x7f3a7406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:34.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.248+0000 7f3a86ffd640 1 -- 172.21.15.150:0/1752615562 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f3a8808b630 con 0x7f3a9015d390 2024-03-17T23:47:34.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.248+0000 7f3a963e7640 1 --2- 172.21.15.150:0/1752615562 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3a7406bbd0 0x7f3a7406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:34.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.248+0000 7f3a963e7640 1 --2- 172.21.15.150:0/1752615562 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3a7406bbd0 0x7f3a7406e090 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c036000 tx=0x7f3a8c03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:34.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.252+0000 7f3a86ffd640 1 -- 172.21.15.150:0/1752615562 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3a88022070 con 0x7f3a9015d390 2024-03-17T23:47:34.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.423+0000 7f3a9864b640 1 -- 172.21.15.150:0/1752615562 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3a9013dd90 con 0x7f3a9015d390 2024-03-17T23:47:34.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.423+0000 7f3a86ffd640 1 -- 172.21.15.150:0/1752615562 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f3a88010040 con 0x7f3a9015d390 2024-03-17T23:47:34.424 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:34.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.425+0000 7f3a9864b640 1 -- 172.21.15.150:0/1752615562 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3a7406bbd0 msgr2=0x7f3a7406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:34.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.425+0000 7f3a9864b640 1 --2- 172.21.15.150:0/1752615562 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3a7406bbd0 0x7f3a7406e090 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c036000 tx=0x7f3a8c03d040 comp rx=0 tx=0).stop 2024-03-17T23:47:34.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.425+0000 7f3a9864b640 1 -- 172.21.15.150:0/1752615562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a9015d390 msgr2=0x7f3a90142880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:34.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.425+0000 7f3a9864b640 1 --2- 172.21.15.150:0/1752615562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a9015d390 0x7f3a90142880 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3a8800efc0 tx=0x7f3a8800c490 comp rx=0 tx=0).stop 2024-03-17T23:47:34.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.425+0000 7f3a9864b640 1 -- 172.21.15.150:0/1752615562 shutdown_connections 2024-03-17T23:47:34.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.425+0000 7f3a9864b640 1 --2- 172.21.15.150:0/1752615562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a9015d390 0x7f3a90142880 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.425+0000 7f3a9864b640 1 --2- 172.21.15.150:0/1752615562 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a90144bb0 0x7f3a9013bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.425+0000 7f3a9864b640 1 --2- 172.21.15.150:0/1752615562 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3a7406bbd0 0x7f3a7406e090 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.425+0000 7f3a9864b640 1 --2- 172.21.15.150:0/1752615562 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a900ce6a0 0x7f3a900a9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.425+0000 7f3a9864b640 1 -- 172.21.15.150:0/1752615562 >> 172.21.15.150:0/1752615562 conn(0x7f3a9006b190 msgr2=0x7f3a9015c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:34.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.425+0000 7f3a9864b640 1 -- 172.21.15.150:0/1752615562 shutdown_connections 2024-03-17T23:47:34.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.425+0000 7f3a9864b640 1 -- 172.21.15.150:0/1752615562 wait complete. 2024-03-17T23:47:34.437 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:34.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.543+0000 7f1105af1640 1 Processor -- start 2024-03-17T23:47:34.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.543+0000 7f1105af1640 1 -- start start 2024-03-17T23:47:34.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.543+0000 7f1105af1640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f11000b31c0 0x7f11000b4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:34.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.543+0000 7f1105af1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f110006ccb0 0x7f110006d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:34.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.543+0000 7f1105af1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f110009a0c0 0x7f110009a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:34.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.543+0000 7f1105af1640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f110006d650 con 0x7f110009a0c0 2024-03-17T23:47:34.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.543+0000 7f1105af1640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f11000cc220 con 0x7f11000b31c0 2024-03-17T23:47:34.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.543+0000 7f1105af1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f11000cc500 con 0x7f110006ccb0 2024-03-17T23:47:34.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.544+0000 7f10fffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f110009a0c0 0x7f110009a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:34.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.544+0000 7f10ff7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f11000b31c0 0x7f11000b4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:34.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.544+0000 7f10feffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f110006ccb0 0x7f110006d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:34.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.544+0000 7f10fffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f110009a0c0 0x7f110009a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46188/0 (socket says 172.21.15.150:46188) 2024-03-17T23:47:34.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.544+0000 7f10fffff640 1 -- 172.21.15.150:0/2955979477 learned_addr learned my addr 172.21.15.150:0/2955979477 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:34.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.544+0000 7f10fffff640 1 -- 172.21.15.150:0/2955979477 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f110006ccb0 msgr2=0x7f110006d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:34.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.544+0000 7f10fffff640 1 --2- 172.21.15.150:0/2955979477 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f110006ccb0 0x7f110006d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.544+0000 7f10fffff640 1 -- 172.21.15.150:0/2955979477 >> v2:172.21.15.157:3300/0 conn(0x7f11000b31c0 msgr2=0x7f11000b4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:34.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.544+0000 7f10fffff640 1 --2- 172.21.15.150:0/2955979477 >> v2:172.21.15.157:3300/0 conn(0x7f11000b31c0 0x7f11000b4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.544+0000 7f10fffff640 1 -- 172.21.15.150:0/2955979477 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11000cc7e0 con 0x7f110009a0c0 2024-03-17T23:47:34.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.544+0000 7f10ff7fe640 1 --2- 172.21.15.150:0/2955979477 >> v2:172.21.15.157:3300/0 conn(0x7f11000b31c0 0x7f11000b4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:34.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.545+0000 7f10fffff640 1 --2- 172.21.15.150:0/2955979477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110009a0c0 0x7f110009a550 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1100072de0 tx=0x7f10f802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=499f8c6d6cfbf245 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:34.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.545+0000 7f10fe7fc640 1 -- 172.21.15.150:0/2955979477 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f802ca70 con 0x7f110009a0c0 2024-03-17T23:47:34.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.545+0000 7f10fe7fc640 1 -- 172.21.15.150:0/2955979477 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10f802cc10 con 0x7f110009a0c0 2024-03-17T23:47:34.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.545+0000 7f10fe7fc640 1 -- 172.21.15.150:0/2955979477 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f80343f0 con 0x7f110009a0c0 2024-03-17T23:47:34.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.545+0000 7f1105af1640 1 -- 172.21.15.150:0/2955979477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110009a0c0 msgr2=0x7f110009a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:34.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.545+0000 7f1105af1640 1 --2- 172.21.15.150:0/2955979477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110009a0c0 0x7f110009a550 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1100072de0 tx=0x7f10f802bed0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.545+0000 7f1105af1640 1 -- 172.21.15.150:0/2955979477 shutdown_connections 2024-03-17T23:47:34.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.545+0000 7f1105af1640 1 --2- 172.21.15.150:0/2955979477 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110009a0c0 0x7f110009a550 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.545+0000 7f1105af1640 1 --2- 172.21.15.150:0/2955979477 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f110006ccb0 0x7f110006d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.545+0000 7f1105af1640 1 --2- 172.21.15.150:0/2955979477 >> v2:172.21.15.157:3300/0 conn(0x7f11000b31c0 0x7f11000b4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.545+0000 7f1105af1640 1 -- 172.21.15.150:0/2955979477 >> 172.21.15.150:0/2955979477 conn(0x7f11000b3620 msgr2=0x7f11000b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:34.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.545+0000 7f1105af1640 1 -- 172.21.15.150:0/2955979477 shutdown_connections 2024-03-17T23:47:34.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.545+0000 7f1105af1640 1 -- 172.21.15.150:0/2955979477 wait complete. 2024-03-17T23:47:34.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.546+0000 7f1105af1640 1 Processor -- start 2024-03-17T23:47:34.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.547+0000 7f1105af1640 1 -- start start 2024-03-17T23:47:34.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.547+0000 7f1105af1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f110006ccb0 0x7f110009f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:34.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.547+0000 7f1105af1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110009a0c0 0x7f110015d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:34.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.547+0000 7f1105af1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11000b31c0 0x7f11000ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:34.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.547+0000 7f1105af1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f110009aa90 con 0x7f110009a0c0 2024-03-17T23:47:34.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.547+0000 7f1105af1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f11000ccda0 con 0x7f110006ccb0 2024-03-17T23:47:34.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.547+0000 7f1105af1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f11000c6480 con 0x7f11000b31c0 2024-03-17T23:47:34.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.547+0000 7f10feffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110009a0c0 0x7f110015d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:34.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.548+0000 7f10feffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110009a0c0 0x7f110015d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46194/0 (socket says 172.21.15.150:46194) 2024-03-17T23:47:34.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.548+0000 7f10feffd640 1 -- 172.21.15.150:0/66154656 learned_addr learned my addr 172.21.15.150:0/66154656 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:34.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.548+0000 7f10fffff640 1 --2- 172.21.15.150:0/66154656 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11000b31c0 0x7f11000ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:34.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.548+0000 7f10ff7fe640 1 --2- 172.21.15.150:0/66154656 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f110006ccb0 0x7f110009f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:34.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.548+0000 7f10feffd640 1 -- 172.21.15.150:0/66154656 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11000b31c0 msgr2=0x7f11000ca0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:34.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.548+0000 7f10feffd640 1 --2- 172.21.15.150:0/66154656 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11000b31c0 0x7f11000ca0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.548+0000 7f10feffd640 1 -- 172.21.15.150:0/66154656 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f110006ccb0 msgr2=0x7f110009f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:34.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.548+0000 7f10feffd640 1 --2- 172.21.15.150:0/66154656 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f110006ccb0 0x7f110009f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.548+0000 7f10feffd640 1 -- 172.21.15.150:0/66154656 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11000c6760 con 0x7f110009a0c0 2024-03-17T23:47:34.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.548+0000 7f10feffd640 1 --2- 172.21.15.150:0/66154656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110009a0c0 0x7f110015d7f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f10f000e870 tx=0x7f10f000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:34.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.548+0000 7f10e7fff640 1 -- 172.21.15.150:0/66154656 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f0017070 con 0x7f110009a0c0 2024-03-17T23:47:34.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.548+0000 7f10e7fff640 1 -- 172.21.15.150:0/66154656 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10f000c940 con 0x7f110009a0c0 2024-03-17T23:47:34.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.548+0000 7f10e7fff640 1 -- 172.21.15.150:0/66154656 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f000cd30 con 0x7f110009a0c0 2024-03-17T23:47:34.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.548+0000 7f1105af1640 1 -- 172.21.15.150:0/66154656 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11000c6a40 con 0x7f110009a0c0 2024-03-17T23:47:34.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.548+0000 7f1105af1640 1 -- 172.21.15.150:0/66154656 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f110015e490 con 0x7f110009a0c0 2024-03-17T23:47:34.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.549+0000 7f1105af1640 1 -- 172.21.15.150:0/66154656 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11000688b0 con 0x7f110009a0c0 2024-03-17T23:47:34.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.550+0000 7f10e7fff640 1 -- 172.21.15.150:0/66154656 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f10f001a070 con 0x7f110009a0c0 2024-03-17T23:47:34.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.550+0000 7f10e7fff640 1 --2- 172.21.15.150:0/66154656 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f10ec06bbd0 0x7f10ec06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:34.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.550+0000 7f10e7fff640 1 -- 172.21.15.150:0/66154656 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f10f008af10 con 0x7f110009a0c0 2024-03-17T23:47:34.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.550+0000 7f10ff7fe640 1 --2- 172.21.15.150:0/66154656 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f10ec06bbd0 0x7f10ec06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:34.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.550+0000 7f10ff7fe640 1 --2- 172.21.15.150:0/66154656 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f10ec06bbd0 0x7f10ec06e090 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f10f4009560 tx=0x7f10f4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:34.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.554+0000 7f10e7fff640 1 -- 172.21.15.150:0/66154656 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f10f0059070 con 0x7f110009a0c0 2024-03-17T23:47:34.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.729+0000 7f1105af1640 1 -- 172.21.15.150:0/66154656 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f110015e890 con 0x7f110009a0c0 2024-03-17T23:47:34.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.730+0000 7f10e7fff640 1 -- 172.21.15.150:0/66154656 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18550 (secure 0 0 0) 0x7f10f005d110 con 0x7f110009a0c0 2024-03-17T23:47:34.731 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:34.731 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:01.940019+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:34.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.733+0000 7f1105af1640 1 -- 172.21.15.150:0/66154656 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f10ec06bbd0 msgr2=0x7f10ec06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:34.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.733+0000 7f1105af1640 1 --2- 172.21.15.150:0/66154656 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f10ec06bbd0 0x7f10ec06e090 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f10f4009560 tx=0x7f10f4009190 comp rx=0 tx=0).stop 2024-03-17T23:47:34.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.733+0000 7f1105af1640 1 -- 172.21.15.150:0/66154656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110009a0c0 msgr2=0x7f110015d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:34.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.733+0000 7f1105af1640 1 --2- 172.21.15.150:0/66154656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110009a0c0 0x7f110015d7f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f10f000e870 tx=0x7f10f000ed30 comp rx=0 tx=0).stop 2024-03-17T23:47:34.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.733+0000 7f1105af1640 1 -- 172.21.15.150:0/66154656 shutdown_connections 2024-03-17T23:47:34.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.733+0000 7f1105af1640 1 --2- 172.21.15.150:0/66154656 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11000b31c0 0x7f11000ca0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.733+0000 7f1105af1640 1 --2- 172.21.15.150:0/66154656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110009a0c0 0x7f110015d7f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.733+0000 7f1105af1640 1 --2- 172.21.15.150:0/66154656 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f10ec06bbd0 0x7f10ec06e090 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.733+0000 7f1105af1640 1 --2- 172.21.15.150:0/66154656 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f110006ccb0 0x7f110009f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.733+0000 7f1105af1640 1 -- 172.21.15.150:0/66154656 >> 172.21.15.150:0/66154656 conn(0x7f11000b3620 msgr2=0x7f1100070f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:34.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.733+0000 7f1105af1640 1 -- 172.21.15.150:0/66154656 shutdown_connections 2024-03-17T23:47:34.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.733+0000 7f1105af1640 1 -- 172.21.15.150:0/66154656 wait complete. 2024-03-17T23:47:34.745 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:46:53.610678+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:46:54.227335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-17T23:46:57.949235+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-17T23:47:34.745 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:47:34.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.849+0000 7f969cc6c640 1 Processor -- start 2024-03-17T23:47:34.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.849+0000 7f969cc6c640 1 -- start start 2024-03-17T23:47:34.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.850+0000 7f969cc6c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f969815ab90 0x7f969815bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:34.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.850+0000 7f969cc6c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f969806ecd0 0x7f969806f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:34.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.850+0000 7f969cc6c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96980ce790 con 0x7f96981514f0 2024-03-17T23:47:34.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.850+0000 7f969cc6c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f969806fba0 con 0x7f969815ab90 2024-03-17T23:47:34.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.850+0000 7f969cc6c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f969815b050 con 0x7f969806ecd0 2024-03-17T23:47:34.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.850+0000 7f968ffff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f969815ab90 0x7f969815bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:34.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.850+0000 7f969659c640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f96981514f0 0x7f969806e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:33012/0 (socket says 172.21.15.150:33012) 2024-03-17T23:47:34.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.850+0000 7f9696d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f969806ecd0 0x7f969806f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:34.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.850+0000 7f969659c640 1 -- 172.21.15.150:0/78619024 learned_addr learned my addr 172.21.15.150:0/78619024 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:34.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.850+0000 7f9696d9d640 1 -- 172.21.15.150:0/78619024 >> v2:172.21.15.157:3300/0 conn(0x7f969815ab90 msgr2=0x7f969815bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:34.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.850+0000 7f9696d9d640 1 --2- 172.21.15.150:0/78619024 >> v2:172.21.15.157:3300/0 conn(0x7f969815ab90 0x7f969815bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.850+0000 7f9696d9d640 1 -- 172.21.15.150:0/78619024 >> v1:172.21.15.157:6789/0 conn(0x7f96981514f0 legacy=0x7f969806e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:34.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.850+0000 7f9696d9d640 1 -- 172.21.15.150:0/78619024 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f969815b330 con 0x7f969806ecd0 2024-03-17T23:47:34.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.850+0000 7f9695d9b640 1 -- 172.21.15.150:0/78619024 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1793374606 0 0) 0x7f9688001110 con 0x7f96981514f0 2024-03-17T23:47:34.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.850+0000 7f968ffff640 1 --2- 172.21.15.150:0/78619024 >> v2:172.21.15.157:3300/0 conn(0x7f969815ab90 0x7f969815bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:34.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.851+0000 7f9696d9d640 1 --2- 172.21.15.150:0/78619024 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f969806ecd0 0x7f969806f660 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9690009d90 tx=0x7f969002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32b384a884ca9f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:34.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.851+0000 7f9695d9b640 1 -- 172.21.15.150:0/78619024 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9690039070 con 0x7f969806ecd0 2024-03-17T23:47:34.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.851+0000 7f9695d9b640 1 -- 172.21.15.150:0/78619024 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f969002cc50 con 0x7f969806ecd0 2024-03-17T23:47:34.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.851+0000 7f9695d9b640 1 -- 172.21.15.150:0/78619024 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9690034440 con 0x7f969806ecd0 2024-03-17T23:47:34.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.852+0000 7f969cc6c640 1 -- 172.21.15.150:0/78619024 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f969806ecd0 msgr2=0x7f969806f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:34.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.852+0000 7f969cc6c640 1 --2- 172.21.15.150:0/78619024 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f969806ecd0 0x7f969806f660 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9690009d90 tx=0x7f969002bea0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.852+0000 7f969cc6c640 1 -- 172.21.15.150:0/78619024 shutdown_connections 2024-03-17T23:47:34.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.852+0000 7f969cc6c640 1 --2- 172.21.15.150:0/78619024 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f969806ecd0 0x7f969806f660 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.852+0000 7f969cc6c640 1 --2- 172.21.15.150:0/78619024 >> v2:172.21.15.157:3300/0 conn(0x7f969815ab90 0x7f969815bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.852+0000 7f969cc6c640 1 -- 172.21.15.150:0/78619024 >> 172.21.15.150:0/78619024 conn(0x7f969806b190 msgr2=0x7f969806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:34.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.852+0000 7f969cc6c640 1 -- 172.21.15.150:0/78619024 shutdown_connections 2024-03-17T23:47:34.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.852+0000 7f969cc6c640 1 -- 172.21.15.150:0/78619024 wait complete. 2024-03-17T23:47:34.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.853+0000 7f969cc6c640 1 Processor -- start 2024-03-17T23:47:34.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.853+0000 7f969cc6c640 1 -- start start 2024-03-17T23:47:34.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.854+0000 7f969cc6c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f969806ecd0 0x7f96981f1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:34.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.854+0000 7f969cc6c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96981514f0 0x7f969806e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:34.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.854+0000 7f969cc6c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f969815ab90 0x7f9698147ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:34.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.854+0000 7f969cc6c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f96980af510 con 0x7f969815ab90 2024-03-17T23:47:34.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.854+0000 7f969cc6c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f969815b8f0 con 0x7f969806ecd0 2024-03-17T23:47:34.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.854+0000 7f969cc6c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f96981518d0 con 0x7f96981514f0 2024-03-17T23:47:34.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.854+0000 7f9696d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f969815ab90 0x7f9698147ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:34.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.854+0000 7f9696d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f969815ab90 0x7f9698147ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46202/0 (socket says 172.21.15.150:46202) 2024-03-17T23:47:34.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.854+0000 7f9696d9d640 1 -- 172.21.15.150:0/3063486611 learned_addr learned my addr 172.21.15.150:0/3063486611 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:34.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.854+0000 7f969659c640 1 --2- 172.21.15.150:0/3063486611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f969806ecd0 0x7f96981f1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:34.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.854+0000 7f968ffff640 1 --2- 172.21.15.150:0/3063486611 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96981514f0 0x7f969806e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:34.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.854+0000 7f9696d9d640 1 -- 172.21.15.150:0/3063486611 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96981514f0 msgr2=0x7f969806e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:34.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.854+0000 7f9696d9d640 1 --2- 172.21.15.150:0/3063486611 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96981514f0 0x7f969806e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.854+0000 7f9696d9d640 1 -- 172.21.15.150:0/3063486611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f969806ecd0 msgr2=0x7f96981f1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:34.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.854+0000 7f9696d9d640 1 --2- 172.21.15.150:0/3063486611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f969806ecd0 0x7f96981f1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:34.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.854+0000 7f9696d9d640 1 -- 172.21.15.150:0/3063486611 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96981f2740 con 0x7f969815ab90 2024-03-17T23:47:34.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.854+0000 7f9696d9d640 1 --2- 172.21.15.150:0/3063486611 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f969815ab90 0x7f9698147ee0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9690037040 tx=0x7f9690004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:34.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.854+0000 7f968effd640 1 -- 172.21.15.150:0/3063486611 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9690039050 con 0x7f969815ab90 2024-03-17T23:47:34.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.854+0000 7f968effd640 1 -- 172.21.15.150:0/3063486611 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f969003a040 con 0x7f969815ab90 2024-03-17T23:47:34.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.855+0000 7f969cc6c640 1 -- 172.21.15.150:0/3063486611 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96981f2a20 con 0x7f969815ab90 2024-03-17T23:47:34.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.855+0000 7f968effd640 1 -- 172.21.15.150:0/3063486611 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9690009450 con 0x7f969815ab90 2024-03-17T23:47:34.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.855+0000 7f969cc6c640 1 -- 172.21.15.150:0/3063486611 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9698149050 con 0x7f969815ab90 2024-03-17T23:47:34.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.855+0000 7f969cc6c640 1 -- 172.21.15.150:0/3063486611 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96980681a0 con 0x7f969815ab90 2024-03-17T23:47:34.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.856+0000 7f968effd640 1 -- 172.21.15.150:0/3063486611 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9690047030 con 0x7f969815ab90 2024-03-17T23:47:34.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.856+0000 7f968effd640 1 --2- 172.21.15.150:0/3063486611 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f967806bbd0 0x7f967806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:34.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.856+0000 7f968effd640 1 -- 172.21.15.150:0/3063486611 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f96900acb80 con 0x7f969815ab90 2024-03-17T23:47:34.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.856+0000 7f969659c640 1 --2- 172.21.15.150:0/3063486611 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f967806bbd0 0x7f967806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:34.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.857+0000 7f969659c640 1 --2- 172.21.15.150:0/3063486611 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f967806bbd0 0x7f967806e090 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9688000f60 tx=0x7f9688007b50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:34.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:34.861+0000 7f968effd640 1 -- 172.21.15.150:0/3063486611 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f969007acf0 con 0x7f969815ab90 2024-03-17T23:47:35.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.036+0000 7f969cc6c640 1 -- 172.21.15.150:0/3063486611 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f96981f2460 con 0x7f969815ab90 2024-03-17T23:47:35.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.036+0000 7f968effd640 1 -- 172.21.15.150:0/3063486611 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f969007ed90 con 0x7f969815ab90 2024-03-17T23:47:35.037 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:47:35.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.038+0000 7f969cc6c640 1 -- 172.21.15.150:0/3063486611 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f967806bbd0 msgr2=0x7f967806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:35.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.038+0000 7f969cc6c640 1 --2- 172.21.15.150:0/3063486611 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f967806bbd0 0x7f967806e090 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9688000f60 tx=0x7f9688007b50 comp rx=0 tx=0).stop 2024-03-17T23:47:35.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.038+0000 7f969cc6c640 1 -- 172.21.15.150:0/3063486611 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f969815ab90 msgr2=0x7f9698147ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:35.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.038+0000 7f969cc6c640 1 --2- 172.21.15.150:0/3063486611 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f969815ab90 0x7f9698147ee0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9690037040 tx=0x7f9690004a40 comp rx=0 tx=0).stop 2024-03-17T23:47:35.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.038+0000 7f969cc6c640 1 -- 172.21.15.150:0/3063486611 shutdown_connections 2024-03-17T23:47:35.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.038+0000 7f969cc6c640 1 --2- 172.21.15.150:0/3063486611 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f969815ab90 0x7f9698147ee0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.038+0000 7f969cc6c640 1 --2- 172.21.15.150:0/3063486611 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f96981514f0 0x7f969806e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.038+0000 7f969cc6c640 1 --2- 172.21.15.150:0/3063486611 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f967806bbd0 0x7f967806e090 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.038+0000 7f969cc6c640 1 --2- 172.21.15.150:0/3063486611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f969806ecd0 0x7f96981f1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.038+0000 7f969cc6c640 1 -- 172.21.15.150:0/3063486611 >> 172.21.15.150:0/3063486611 conn(0x7f969806b190 msgr2=0x7f969815c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:35.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.038+0000 7f969cc6c640 1 -- 172.21.15.150:0/3063486611 shutdown_connections 2024-03-17T23:47:35.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.038+0000 7f969cc6c640 1 -- 172.21.15.150:0/3063486611 wait complete. 2024-03-17T23:47:35.051 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:47:35.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.154+0000 7f5b701d3640 1 Processor -- start 2024-03-17T23:47:35.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.155+0000 7f5b701d3640 1 -- start start 2024-03-17T23:47:35.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.155+0000 7f5b701d3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5b680b2bf0 0x7f5b680b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:35.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.155+0000 7f5b701d3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b68161d30 con 0x7f5b680d0ce0 2024-03-17T23:47:35.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.155+0000 7f5b701d3640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b68162010 con 0x7f5b68161520 2024-03-17T23:47:35.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.155+0000 7f5b701d3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f5b680b9bf0 con 0x7f5b680b2bf0 2024-03-17T23:47:35.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.156+0000 7f5b6e770640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5b680b2bf0 0x7f5b680b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:35.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.156+0000 7f5b6d76e640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f5b680d0ce0 0x7f5b680d1fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33752/0 (socket says 172.21.15.150:33752) 2024-03-17T23:47:35.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.156+0000 7f5b6d76e640 1 -- 172.21.15.150:0/710424189 learned_addr learned my addr 172.21.15.150:0/710424189 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:35.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.156+0000 7f5b6e770640 1 -- 172.21.15.150:0/710424189 >> v1:172.21.15.157:6789/0 conn(0x7f5b68161520 legacy=0x7f5b680d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:35.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.156+0000 7f5b6e770640 1 -- 172.21.15.150:0/710424189 >> v1:172.21.15.150:6789/0 conn(0x7f5b680d0ce0 legacy=0x7f5b680d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:35.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.156+0000 7f5b6e770640 1 -- 172.21.15.150:0/710424189 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b681622f0 con 0x7f5b680b2bf0 2024-03-17T23:47:35.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.156+0000 7f5b6cf6d640 1 -- 172.21.15.150:0/710424189 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3069777707 0 0) 0x7f5b68161d30 con 0x7f5b680d0ce0 2024-03-17T23:47:35.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.157+0000 7f5b6e770640 1 --2- 172.21.15.150:0/710424189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b680b2bf0 0x7f5b680b96b0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5b60009d90 tx=0x7f5b6002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a5d4abadc35a7c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:35.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.157+0000 7f5b6cf6d640 1 -- 172.21.15.150:0/710424189 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b60039070 con 0x7f5b680b2bf0 2024-03-17T23:47:35.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.157+0000 7f5b6cf6d640 1 -- 172.21.15.150:0/710424189 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b6002cbd0 con 0x7f5b680b2bf0 2024-03-17T23:47:35.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.157+0000 7f5b6cf6d640 1 -- 172.21.15.150:0/710424189 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b60034440 con 0x7f5b680b2bf0 2024-03-17T23:47:35.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.157+0000 7f5b701d3640 1 -- 172.21.15.150:0/710424189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b680b2bf0 msgr2=0x7f5b680b96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:35.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.157+0000 7f5b701d3640 1 --2- 172.21.15.150:0/710424189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b680b2bf0 0x7f5b680b96b0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5b60009d90 tx=0x7f5b6002be20 comp rx=0 tx=0).stop 2024-03-17T23:47:35.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.157+0000 7f5b701d3640 1 -- 172.21.15.150:0/710424189 shutdown_connections 2024-03-17T23:47:35.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.157+0000 7f5b701d3640 1 --2- 172.21.15.150:0/710424189 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b680b2bf0 0x7f5b680b96b0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.157+0000 7f5b701d3640 1 -- 172.21.15.150:0/710424189 >> 172.21.15.150:0/710424189 conn(0x7f5b6806ca00 msgr2=0x7f5b6806ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:35.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.157+0000 7f5b701d3640 1 -- 172.21.15.150:0/710424189 shutdown_connections 2024-03-17T23:47:35.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.157+0000 7f5b701d3640 1 -- 172.21.15.150:0/710424189 wait complete. 2024-03-17T23:47:35.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.158+0000 7f5b701d3640 1 Processor -- start 2024-03-17T23:47:35.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.159+0000 7f5b701d3640 1 -- start start 2024-03-17T23:47:35.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.159+0000 7f5b701d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b680b2bf0 0x7f5b6815b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:35.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.159+0000 7f5b701d3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b680d0ce0 0x7f5b680b7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:35.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.159+0000 7f5b701d3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b68161520 0x7f5b680b81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:35.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.159+0000 7f5b701d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5b68161900 con 0x7f5b680b2bf0 2024-03-17T23:47:35.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.159+0000 7f5b701d3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5b68162940 con 0x7f5b68161520 2024-03-17T23:47:35.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.159+0000 7f5b701d3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5b680d26b0 con 0x7f5b680d0ce0 2024-03-17T23:47:35.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.159+0000 7f5b6df6f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b680b2bf0 0x7f5b6815b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:35.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.159+0000 7f5b6df6f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b680b2bf0 0x7f5b6815b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46220/0 (socket says 172.21.15.150:46220) 2024-03-17T23:47:35.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.159+0000 7f5b6df6f640 1 -- 172.21.15.150:0/1071113963 learned_addr learned my addr 172.21.15.150:0/1071113963 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:35.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.159+0000 7f5b6d76e640 1 --2- 172.21.15.150:0/1071113963 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b680d0ce0 0x7f5b680b7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:35.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.159+0000 7f5b6e770640 1 --2- 172.21.15.150:0/1071113963 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b68161520 0x7f5b680b81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:35.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.160+0000 7f5b6df6f640 1 -- 172.21.15.150:0/1071113963 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b680d0ce0 msgr2=0x7f5b680b7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:35.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.160+0000 7f5b6df6f640 1 --2- 172.21.15.150:0/1071113963 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b680d0ce0 0x7f5b680b7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.160+0000 7f5b6df6f640 1 -- 172.21.15.150:0/1071113963 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b68161520 msgr2=0x7f5b680b81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:35.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.160+0000 7f5b6df6f640 1 --2- 172.21.15.150:0/1071113963 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b68161520 0x7f5b680b81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.160+0000 7f5b6df6f640 1 -- 172.21.15.150:0/1071113963 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b6815b7e0 con 0x7f5b680b2bf0 2024-03-17T23:47:35.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.160+0000 7f5b6df6f640 1 --2- 172.21.15.150:0/1071113963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b680b2bf0 0x7f5b6815b2a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5b5800c430 tx=0x7f5b5800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:35.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.160+0000 7f5b5e7fc640 1 -- 172.21.15.150:0/1071113963 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b58017070 con 0x7f5b680b2bf0 2024-03-17T23:47:35.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.160+0000 7f5b5e7fc640 1 -- 172.21.15.150:0/1071113963 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b580052d0 con 0x7f5b680b2bf0 2024-03-17T23:47:35.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.160+0000 7f5b5e7fc640 1 -- 172.21.15.150:0/1071113963 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b58003830 con 0x7f5b680b2bf0 2024-03-17T23:47:35.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.160+0000 7f5b701d3640 1 -- 172.21.15.150:0/1071113963 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b680b4370 con 0x7f5b680b2bf0 2024-03-17T23:47:35.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.160+0000 7f5b701d3640 1 -- 172.21.15.150:0/1071113963 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b680b4930 con 0x7f5b680b2bf0 2024-03-17T23:47:35.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.161+0000 7f5b701d3640 1 -- 172.21.15.150:0/1071113963 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b680681a0 con 0x7f5b680b2bf0 2024-03-17T23:47:35.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.162+0000 7f5b5e7fc640 1 -- 172.21.15.150:0/1071113963 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5b5801a070 con 0x7f5b680b2bf0 2024-03-17T23:47:35.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.162+0000 7f5b5e7fc640 1 --2- 172.21.15.150:0/1071113963 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5b4c06bbd0 0x7f5b4c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:35.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.162+0000 7f5b5e7fc640 1 -- 172.21.15.150:0/1071113963 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f5b58089fa0 con 0x7f5b680b2bf0 2024-03-17T23:47:35.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.162+0000 7f5b6d76e640 1 --2- 172.21.15.150:0/1071113963 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5b4c06bbd0 0x7f5b4c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:35.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.162+0000 7f5b6d76e640 1 --2- 172.21.15.150:0/1071113963 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5b4c06bbd0 0x7f5b4c06e090 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5b680b2770 tx=0x7f5b64007f00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:35.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.166+0000 7f5b5e7fc640 1 -- 172.21.15.150:0/1071113963 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5b6815b7e0 con 0x7f5b680b2bf0 2024-03-17T23:47:35.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.337+0000 7f5b701d3640 1 -- 172.21.15.150:0/1071113963 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5b680b4b00 con 0x7f5b680b2bf0 2024-03-17T23:47:35.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.337+0000 7f5b5e7fc640 1 -- 172.21.15.150:0/1071113963 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f5b58058190 con 0x7f5b680b2bf0 2024-03-17T23:47:35.338 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:35.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.339+0000 7f5b701d3640 1 -- 172.21.15.150:0/1071113963 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5b4c06bbd0 msgr2=0x7f5b4c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:35.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.339+0000 7f5b701d3640 1 --2- 172.21.15.150:0/1071113963 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5b4c06bbd0 0x7f5b4c06e090 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5b680b2770 tx=0x7f5b64007f00 comp rx=0 tx=0).stop 2024-03-17T23:47:35.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.339+0000 7f5b701d3640 1 -- 172.21.15.150:0/1071113963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b680b2bf0 msgr2=0x7f5b6815b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:35.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.339+0000 7f5b701d3640 1 --2- 172.21.15.150:0/1071113963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b680b2bf0 0x7f5b6815b2a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5b5800c430 tx=0x7f5b5800c8f0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.339+0000 7f5b701d3640 1 -- 172.21.15.150:0/1071113963 shutdown_connections 2024-03-17T23:47:35.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.339+0000 7f5b701d3640 1 --2- 172.21.15.150:0/1071113963 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5b4c06bbd0 0x7f5b4c06e090 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.339+0000 7f5b701d3640 1 --2- 172.21.15.150:0/1071113963 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b68161520 0x7f5b680b81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.339+0000 7f5b701d3640 1 --2- 172.21.15.150:0/1071113963 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b680d0ce0 0x7f5b680b7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.339+0000 7f5b701d3640 1 --2- 172.21.15.150:0/1071113963 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b680b2bf0 0x7f5b6815b2a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.339+0000 7f5b701d3640 1 -- 172.21.15.150:0/1071113963 >> 172.21.15.150:0/1071113963 conn(0x7f5b6806ca00 msgr2=0x7f5b680c7050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:35.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.339+0000 7f5b701d3640 1 -- 172.21.15.150:0/1071113963 shutdown_connections 2024-03-17T23:47:35.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.339+0000 7f5b701d3640 1 -- 172.21.15.150:0/1071113963 wait complete. 2024-03-17T23:47:35.351 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:47:35.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.454+0000 7fee83137640 1 Processor -- start 2024-03-17T23:47:35.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.454+0000 7fee83137640 1 -- start start 2024-03-17T23:47:35.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.455+0000 7fee83137640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fee7c155320 0x7fee7c15bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:35.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.455+0000 7fee83137640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fee7c06dd30 0x7fee7c0ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:35.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.455+0000 7fee83137640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee7c0cf3b0 con 0x7fee7c0cd250 2024-03-17T23:47:35.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.455+0000 7fee83137640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fee7c0cd6c0 con 0x7fee7c06dd30 2024-03-17T23:47:35.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.455+0000 7fee83137640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fee7c15c550 con 0x7fee7c155320 2024-03-17T23:47:35.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.455+0000 7fee7bfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fee7c06dd30 0x7fee7c0ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:35.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.455+0000 7fee7bfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fee7c06dd30 0x7fee7c0ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46018/0 (socket says 172.21.15.150:46018) 2024-03-17T23:47:35.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.455+0000 7fee7bfff640 1 -- 172.21.15.150:0/1921620938 learned_addr learned my addr 172.21.15.150:0/1921620938 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:35.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.455+0000 7fee80ed3640 1 --2- 172.21.15.150:0/1921620938 >> v2:172.21.15.157:3300/0 conn(0x7fee7c155320 0x7fee7c15bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:35.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.455+0000 7fee7bfff640 1 -- 172.21.15.150:0/1921620938 >> v2:172.21.15.157:3300/0 conn(0x7fee7c155320 msgr2=0x7fee7c15bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:35.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.455+0000 7fee7bfff640 1 --2- 172.21.15.150:0/1921620938 >> v2:172.21.15.157:3300/0 conn(0x7fee7c155320 0x7fee7c15bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.455+0000 7fee7bfff640 1 -- 172.21.15.150:0/1921620938 >> v1:172.21.15.157:6789/0 conn(0x7fee7c0cd250 legacy=0x7fee7c15e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:35.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.455+0000 7fee7bfff640 1 -- 172.21.15.150:0/1921620938 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee7c0cf690 con 0x7fee7c06dd30 2024-03-17T23:47:35.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.456+0000 7fee7bfff640 1 --2- 172.21.15.150:0/1921620938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee7c06dd30 0x7fee7c0ccd10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fee7c06d5a0 tx=0x7fee6802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4159a52bc3a2cf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:35.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.456+0000 7fee7b7fe640 1 -- 172.21.15.150:0/1921620938 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee68034070 con 0x7fee7c06dd30 2024-03-17T23:47:35.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.456+0000 7fee7b7fe640 1 -- 172.21.15.150:0/1921620938 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee68030860 con 0x7fee7c06dd30 2024-03-17T23:47:35.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.456+0000 7fee7b7fe640 1 -- 172.21.15.150:0/1921620938 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee68030c30 con 0x7fee7c06dd30 2024-03-17T23:47:35.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.457+0000 7fee83137640 1 -- 172.21.15.150:0/1921620938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee7c06dd30 msgr2=0x7fee7c0ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:35.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.457+0000 7fee83137640 1 --2- 172.21.15.150:0/1921620938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee7c06dd30 0x7fee7c0ccd10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fee7c06d5a0 tx=0x7fee6802b9a0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.457+0000 7fee83137640 1 -- 172.21.15.150:0/1921620938 shutdown_connections 2024-03-17T23:47:35.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.457+0000 7fee83137640 1 --2- 172.21.15.150:0/1921620938 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee7c06dd30 0x7fee7c0ccd10 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.457+0000 7fee83137640 1 --2- 172.21.15.150:0/1921620938 >> v2:172.21.15.157:3300/0 conn(0x7fee7c155320 0x7fee7c15bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.457+0000 7fee83137640 1 -- 172.21.15.150:0/1921620938 >> 172.21.15.150:0/1921620938 conn(0x7fee7c06b190 msgr2=0x7fee7c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:35.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.457+0000 7fee83137640 1 -- 172.21.15.150:0/1921620938 shutdown_connections 2024-03-17T23:47:35.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.457+0000 7fee83137640 1 -- 172.21.15.150:0/1921620938 wait complete. 2024-03-17T23:47:35.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.458+0000 7fee83137640 1 Processor -- start 2024-03-17T23:47:35.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.458+0000 7fee83137640 1 -- start start 2024-03-17T23:47:35.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.459+0000 7fee83137640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee7c06dd30 0x7fee7c149720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:35.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.459+0000 7fee83137640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee7c0cd250 0x7fee7c157cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:35.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.459+0000 7fee83137640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee7c155320 0x7fee7c14ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:35.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.459+0000 7fee83137640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fee7c06e190 con 0x7fee7c155320 2024-03-17T23:47:35.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.459+0000 7fee83137640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fee7c0cf060 con 0x7fee7c06dd30 2024-03-17T23:47:35.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.459+0000 7fee83137640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fee7c155700 con 0x7fee7c0cd250 2024-03-17T23:47:35.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.459+0000 7fee816d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee7c155320 0x7fee7c14ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:35.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.459+0000 7fee80ed3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee7c06dd30 0x7fee7c149720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:35.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.459+0000 7fee80ed3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee7c06dd30 0x7fee7c149720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:34942/0 (socket says 172.21.15.150:34942) 2024-03-17T23:47:35.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.459+0000 7fee80ed3640 1 -- 172.21.15.150:0/1562071035 learned_addr learned my addr 172.21.15.150:0/1562071035 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:35.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.459+0000 7fee816d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee7c155320 0x7fee7c14ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46026/0 (socket says 172.21.15.150:46026) 2024-03-17T23:47:35.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.459+0000 7fee7bfff640 1 --2- 172.21.15.150:0/1562071035 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee7c0cd250 0x7fee7c157cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:35.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.459+0000 7fee816d4640 1 -- 172.21.15.150:0/1562071035 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee7c0cd250 msgr2=0x7fee7c157cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:35.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.459+0000 7fee816d4640 1 --2- 172.21.15.150:0/1562071035 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee7c0cd250 0x7fee7c157cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.459+0000 7fee816d4640 1 -- 172.21.15.150:0/1562071035 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee7c06dd30 msgr2=0x7fee7c149720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:35.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.459+0000 7fee816d4640 1 --2- 172.21.15.150:0/1562071035 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee7c06dd30 0x7fee7c149720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.459+0000 7fee816d4640 1 -- 172.21.15.150:0/1562071035 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee7c1587b0 con 0x7fee7c155320 2024-03-17T23:47:35.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.459+0000 7fee80ed3640 1 --2- 172.21.15.150:0/1562071035 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee7c06dd30 0x7fee7c149720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:35.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.459+0000 7fee816d4640 1 --2- 172.21.15.150:0/1562071035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee7c155320 0x7fee7c14ba30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fee7400c690 tx=0x7fee7400cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:35.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.460+0000 7fee797fa640 1 -- 172.21.15.150:0/1562071035 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee74003b70 con 0x7fee7c155320 2024-03-17T23:47:35.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.460+0000 7fee797fa640 1 -- 172.21.15.150:0/1562071035 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee74003d10 con 0x7fee7c155320 2024-03-17T23:47:35.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.460+0000 7fee83137640 1 -- 172.21.15.150:0/1562071035 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee7c149f00 con 0x7fee7c155320 2024-03-17T23:47:35.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.460+0000 7fee797fa640 1 -- 172.21.15.150:0/1562071035 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee74005310 con 0x7fee7c155320 2024-03-17T23:47:35.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.460+0000 7fee83137640 1 -- 172.21.15.150:0/1562071035 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee7c14a4c0 con 0x7fee7c155320 2024-03-17T23:47:35.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.461+0000 7fee83137640 1 -- 172.21.15.150:0/1562071035 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee7c0681a0 con 0x7fee7c155320 2024-03-17T23:47:35.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.461+0000 7fee797fa640 1 -- 172.21.15.150:0/1562071035 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fee740054b0 con 0x7fee7c155320 2024-03-17T23:47:35.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.461+0000 7fee797fa640 1 --2- 172.21.15.150:0/1562071035 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fee6006bbd0 0x7fee6006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:35.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.462+0000 7fee80ed3640 1 --2- 172.21.15.150:0/1562071035 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fee6006bbd0 0x7fee6006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:35.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.462+0000 7fee797fa640 1 -- 172.21.15.150:0/1562071035 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fee7408aa50 con 0x7fee7c155320 2024-03-17T23:47:35.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.462+0000 7fee80ed3640 1 --2- 172.21.15.150:0/1562071035 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fee6006bbd0 0x7fee6006e090 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fee70009f50 tx=0x7fee70009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:35.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.466+0000 7fee797fa640 1 -- 172.21.15.150:0/1562071035 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fee74058bb0 con 0x7fee7c155320 2024-03-17T23:47:35.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.638+0000 7fee83137640 1 -- 172.21.15.150:0/1562071035 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fee7c15e6c0 con 0x7fee7c155320 2024-03-17T23:47:35.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.638+0000 7fee797fa640 1 -- 172.21.15.150:0/1562071035 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fee7405cc50 con 0x7fee7c155320 2024-03-17T23:47:35.639 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:47:35.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.640+0000 7fee83137640 1 -- 172.21.15.150:0/1562071035 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fee6006bbd0 msgr2=0x7fee6006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:35.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.640+0000 7fee83137640 1 --2- 172.21.15.150:0/1562071035 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fee6006bbd0 0x7fee6006e090 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fee70009f50 tx=0x7fee70009190 comp rx=0 tx=0).stop 2024-03-17T23:47:35.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.640+0000 7fee83137640 1 -- 172.21.15.150:0/1562071035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee7c155320 msgr2=0x7fee7c14ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:35.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.640+0000 7fee83137640 1 --2- 172.21.15.150:0/1562071035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee7c155320 0x7fee7c14ba30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fee7400c690 tx=0x7fee7400cb50 comp rx=0 tx=0).stop 2024-03-17T23:47:35.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.641+0000 7fee83137640 1 -- 172.21.15.150:0/1562071035 shutdown_connections 2024-03-17T23:47:35.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.641+0000 7fee83137640 1 --2- 172.21.15.150:0/1562071035 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee7c155320 0x7fee7c14ba30 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.641+0000 7fee83137640 1 --2- 172.21.15.150:0/1562071035 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee7c0cd250 0x7fee7c157cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.641+0000 7fee83137640 1 --2- 172.21.15.150:0/1562071035 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fee6006bbd0 0x7fee6006e090 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.641+0000 7fee83137640 1 --2- 172.21.15.150:0/1562071035 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee7c06dd30 0x7fee7c149720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.641+0000 7fee83137640 1 -- 172.21.15.150:0/1562071035 >> 172.21.15.150:0/1562071035 conn(0x7fee7c06b190 msgr2=0x7fee7c0b4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:35.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.641+0000 7fee83137640 1 -- 172.21.15.150:0/1562071035 shutdown_connections 2024-03-17T23:47:35.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.641+0000 7fee83137640 1 -- 172.21.15.150:0/1562071035 wait complete. 2024-03-17T23:47:35.652 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-17T23:47:35.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.755+0000 7f2fae537640 1 Processor -- start 2024-03-17T23:47:35.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.755+0000 7f2fae537640 1 -- start start 2024-03-17T23:47:35.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.756+0000 7f2fae537640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2fa809a470 0x7f2fa809a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:35.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.756+0000 7f2fae537640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fa809ad90 0x7f2fa815f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:35.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.756+0000 7f2fae537640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2fa815fcd0 0x7f2fa815c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:35.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.756+0000 7f2fae537640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f2fa80cfad0 con 0x7f2fa809a470 2024-03-17T23:47:35.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.756+0000 7f2fae537640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f2fa80cfdb0 con 0x7f2fa815fcd0 2024-03-17T23:47:35.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.756+0000 7f2fae537640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2fa80d0090 con 0x7f2fa809ad90 2024-03-17T23:47:35.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.756+0000 7f2fa7fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2fa809a470 0x7f2fa809a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:35.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.756+0000 7f2fa7fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2fa809a470 0x7f2fa809a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46034/0 (socket says 172.21.15.150:46034) 2024-03-17T23:47:35.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.756+0000 7f2fa7fff640 1 -- 172.21.15.150:0/1100278451 learned_addr learned my addr 172.21.15.150:0/1100278451 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:35.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.756+0000 7f2facad4640 1 --2- 172.21.15.150:0/1100278451 >> v2:172.21.15.157:3300/0 conn(0x7f2fa815fcd0 0x7f2fa815c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:35.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.756+0000 7f2fa77fe640 1 --2- 172.21.15.150:0/1100278451 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fa809ad90 0x7f2fa815f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:35.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.756+0000 7f2fa7fff640 1 -- 172.21.15.150:0/1100278451 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fa809ad90 msgr2=0x7f2fa815f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:35.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.756+0000 7f2fa7fff640 1 --2- 172.21.15.150:0/1100278451 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fa809ad90 0x7f2fa815f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.756+0000 7f2fa7fff640 1 -- 172.21.15.150:0/1100278451 >> v2:172.21.15.157:3300/0 conn(0x7f2fa815fcd0 msgr2=0x7f2fa815c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:35.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.756+0000 7f2fa7fff640 1 --2- 172.21.15.150:0/1100278451 >> v2:172.21.15.157:3300/0 conn(0x7f2fa815fcd0 0x7f2fa815c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.756+0000 7f2fa7fff640 1 -- 172.21.15.150:0/1100278451 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa80d0370 con 0x7f2fa809a470 2024-03-17T23:47:35.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.757+0000 7f2fa7fff640 1 --2- 172.21.15.150:0/1100278451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fa809a470 0x7f2fa809a850 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2fa80cc330 tx=0x7f2f9802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d648ec95423b8bb server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:35.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.757+0000 7f2fa6ffd640 1 -- 172.21.15.150:0/1100278451 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f98035020 con 0x7f2fa809a470 2024-03-17T23:47:35.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.757+0000 7f2fa6ffd640 1 -- 172.21.15.150:0/1100278451 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f98031870 con 0x7f2fa809a470 2024-03-17T23:47:35.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.757+0000 7f2fa6ffd640 1 -- 172.21.15.150:0/1100278451 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f98031c40 con 0x7f2fa809a470 2024-03-17T23:47:35.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.758+0000 7f2fae537640 1 -- 172.21.15.150:0/1100278451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fa809a470 msgr2=0x7f2fa809a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:35.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.758+0000 7f2fae537640 1 --2- 172.21.15.150:0/1100278451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fa809a470 0x7f2fa809a850 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2fa80cc330 tx=0x7f2f9802b910 comp rx=0 tx=0).stop 2024-03-17T23:47:35.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.758+0000 7f2fae537640 1 -- 172.21.15.150:0/1100278451 shutdown_connections 2024-03-17T23:47:35.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.758+0000 7f2fae537640 1 --2- 172.21.15.150:0/1100278451 >> v2:172.21.15.157:3300/0 conn(0x7f2fa815fcd0 0x7f2fa815c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.758+0000 7f2fae537640 1 --2- 172.21.15.150:0/1100278451 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fa809ad90 0x7f2fa815f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.758+0000 7f2fae537640 1 --2- 172.21.15.150:0/1100278451 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fa809a470 0x7f2fa809a850 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.758+0000 7f2fae537640 1 -- 172.21.15.150:0/1100278451 >> 172.21.15.150:0/1100278451 conn(0x7f2fa80b3630 msgr2=0x7f2fa80b3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:35.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.758+0000 7f2fae537640 1 -- 172.21.15.150:0/1100278451 shutdown_connections 2024-03-17T23:47:35.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.758+0000 7f2fae537640 1 -- 172.21.15.150:0/1100278451 wait complete. 2024-03-17T23:47:35.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.759+0000 7f2fae537640 1 Processor -- start 2024-03-17T23:47:35.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.759+0000 7f2fae537640 1 -- start start 2024-03-17T23:47:35.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.760+0000 7f2fae537640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fa809a470 0x7f2fa80c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:35.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.760+0000 7f2fae537640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fa809ad90 0x7f2fa80ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:35.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.760+0000 7f2fae537640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2fa815fcd0 0x7f2fa815df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:35.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.760+0000 7f2fae537640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa80cf890 con 0x7f2fa809a470 2024-03-17T23:47:35.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.760+0000 7f2fae537640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa80cf710 con 0x7f2fa815fcd0 2024-03-17T23:47:35.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.760+0000 7f2fae537640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2fa815c690 con 0x7f2fa809ad90 2024-03-17T23:47:35.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.760+0000 7f2fa7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fa809a470 0x7f2fa80c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:35.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.760+0000 7f2fa7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fa809a470 0x7f2fa80c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46044/0 (socket says 172.21.15.150:46044) 2024-03-17T23:47:35.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.760+0000 7f2fa7fff640 1 -- 172.21.15.150:0/4210993038 learned_addr learned my addr 172.21.15.150:0/4210993038 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:35.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.760+0000 7f2fa77fe640 1 --2- 172.21.15.150:0/4210993038 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fa809ad90 0x7f2fa80ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:35.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.760+0000 7f2facad4640 1 --2- 172.21.15.150:0/4210993038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2fa815fcd0 0x7f2fa815df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:35.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.760+0000 7f2fa7fff640 1 -- 172.21.15.150:0/4210993038 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fa809ad90 msgr2=0x7f2fa80ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:35.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.760+0000 7f2fa7fff640 1 --2- 172.21.15.150:0/4210993038 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fa809ad90 0x7f2fa80ca450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.760+0000 7f2fa7fff640 1 -- 172.21.15.150:0/4210993038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2fa815fcd0 msgr2=0x7f2fa815df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:35.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.760+0000 7f2fa7fff640 1 --2- 172.21.15.150:0/4210993038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2fa815fcd0 0x7f2fa815df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.760+0000 7f2fa7fff640 1 -- 172.21.15.150:0/4210993038 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa80c93b0 con 0x7f2fa809a470 2024-03-17T23:47:35.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.760+0000 7f2fa7fff640 1 --2- 172.21.15.150:0/4210993038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fa809a470 0x7f2fa80c8c10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2f9802b950 tx=0x7f2f980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:35.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.760+0000 7f2fa4ff9640 1 -- 172.21.15.150:0/4210993038 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f98040070 con 0x7f2fa809a470 2024-03-17T23:47:35.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.760+0000 7f2fa4ff9640 1 -- 172.21.15.150:0/4210993038 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f98031e10 con 0x7f2fa809a470 2024-03-17T23:47:35.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.760+0000 7f2fae537640 1 -- 172.21.15.150:0/4210993038 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fa80c9690 con 0x7f2fa809a470 2024-03-17T23:47:35.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.760+0000 7f2fa4ff9640 1 -- 172.21.15.150:0/4210993038 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f98035040 con 0x7f2fa809a470 2024-03-17T23:47:35.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.761+0000 7f2fae537640 1 -- 172.21.15.150:0/4210993038 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fa80c9c50 con 0x7f2fa809a470 2024-03-17T23:47:35.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.762+0000 7f2fae537640 1 -- 172.21.15.150:0/4210993038 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fa80681a0 con 0x7f2fa809a470 2024-03-17T23:47:35.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.762+0000 7f2fa4ff9640 1 -- 172.21.15.150:0/4210993038 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2f9804b060 con 0x7f2fa809a470 2024-03-17T23:47:35.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.762+0000 7f2fa4ff9640 1 --2- 172.21.15.150:0/4210993038 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2f9406bbd0 0x7f2f9406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:35.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.762+0000 7f2fa4ff9640 1 -- 172.21.15.150:0/4210993038 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f2f980ac860 con 0x7f2fa809a470 2024-03-17T23:47:35.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.762+0000 7f2fa77fe640 1 --2- 172.21.15.150:0/4210993038 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2f9406bbd0 0x7f2f9406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:35.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.763+0000 7f2fa77fe640 1 --2- 172.21.15.150:0/4210993038 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2f9406bbd0 0x7f2f9406e090 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2fa809b4d0 tx=0x7f2f9c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:35.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.767+0000 7f2fa4ff9640 1 -- 172.21.15.150:0/4210993038 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2f9807aa50 con 0x7f2fa809a470 2024-03-17T23:47:35.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.943+0000 7f2fae537640 1 -- 172.21.15.150:0/4210993038 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f2fa8156370 con 0x7f2fa809a470 2024-03-17T23:47:35.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.944+0000 7f2fa4ff9640 1 -- 172.21.15.150:0/4210993038 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f2f9807eaf0 con 0x7f2fa809a470 2024-03-17T23:47:35.944 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:35.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.946+0000 7f2fae537640 1 -- 172.21.15.150:0/4210993038 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2f9406bbd0 msgr2=0x7f2f9406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:35.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.946+0000 7f2fae537640 1 --2- 172.21.15.150:0/4210993038 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2f9406bbd0 0x7f2f9406e090 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2fa809b4d0 tx=0x7f2f9c009190 comp rx=0 tx=0).stop 2024-03-17T23:47:35.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.946+0000 7f2fae537640 1 -- 172.21.15.150:0/4210993038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fa809a470 msgr2=0x7f2fa80c8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:35.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.946+0000 7f2fae537640 1 --2- 172.21.15.150:0/4210993038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fa809a470 0x7f2fa80c8c10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2f9802b950 tx=0x7f2f980046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.946+0000 7f2fae537640 1 -- 172.21.15.150:0/4210993038 shutdown_connections 2024-03-17T23:47:35.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.946+0000 7f2fae537640 1 --2- 172.21.15.150:0/4210993038 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2f9406bbd0 0x7f2f9406e090 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.946+0000 7f2fae537640 1 --2- 172.21.15.150:0/4210993038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2fa815fcd0 0x7f2fa815df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.946+0000 7f2fae537640 1 --2- 172.21.15.150:0/4210993038 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fa809ad90 0x7f2fa80ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.946+0000 7f2fae537640 1 --2- 172.21.15.150:0/4210993038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fa809a470 0x7f2fa80c8c10 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:35.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.946+0000 7f2fae537640 1 -- 172.21.15.150:0/4210993038 >> 172.21.15.150:0/4210993038 conn(0x7f2fa80b3630 msgr2=0x7f2fa80cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:35.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.946+0000 7f2fae537640 1 -- 172.21.15.150:0/4210993038 shutdown_connections 2024-03-17T23:47:35.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:35.946+0000 7f2fae537640 1 -- 172.21.15.150:0/4210993038 wait complete. 2024-03-17T23:47:35.958 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:36.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.064+0000 7f9379a8a640 1 Processor -- start 2024-03-17T23:47:36.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.064+0000 7f9379a8a640 1 -- start start 2024-03-17T23:47:36.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.065+0000 7f9379a8a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f937415bfd0 0x7f93740ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:36.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.065+0000 7f9379a8a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93740cf3b0 con 0x7f9374155320 2024-03-17T23:47:36.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.065+0000 7f9379a8a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93740cf690 con 0x7f93740cd250 2024-03-17T23:47:36.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.065+0000 7f9379a8a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f937415c550 con 0x7f937415bfd0 2024-03-17T23:47:36.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.065+0000 7f93737fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f9374155320 0x7f937406dd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59174/0 (socket says 172.21.15.150:59174) 2024-03-17T23:47:36.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.065+0000 7f93737fe640 1 -- 172.21.15.150:0/1692698675 learned_addr learned my addr 172.21.15.150:0/1692698675 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:36.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.065+0000 7f9372ffd640 1 --2- 172.21.15.150:0/1692698675 >> v2:172.21.15.157:3300/0 conn(0x7f937415bfd0 0x7f93740ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:36.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.065+0000 7f93727fc640 1 -- 172.21.15.150:0/1692698675 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2435703233 0 0) 0x7f93740cf3b0 con 0x7f9374155320 2024-03-17T23:47:36.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.065+0000 7f93727fc640 1 -- 172.21.15.150:0/1692698675 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f935c003520 con 0x7f9374155320 2024-03-17T23:47:36.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.065+0000 7f93727fc640 1 -- 172.21.15.150:0/1692698675 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 569382887 0 0) 0x7f93740cf690 con 0x7f93740cd250 2024-03-17T23:47:36.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.065+0000 7f93727fc640 1 -- 172.21.15.150:0/1692698675 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93740cf3b0 con 0x7f93740cd250 2024-03-17T23:47:36.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.065+0000 7f9372ffd640 1 -- 172.21.15.150:0/1692698675 >> v1:172.21.15.157:6789/0 conn(0x7f93740cd250 legacy=0x7f937415ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:36.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.065+0000 7f9372ffd640 1 -- 172.21.15.150:0/1692698675 >> v1:172.21.15.150:6789/0 conn(0x7f9374155320 legacy=0x7f937406dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:36.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.065+0000 7f93727fc640 1 -- 172.21.15.150:0/1692698675 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1834640279 0 0) 0x7f935c003520 con 0x7f9374155320 2024-03-17T23:47:36.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.065+0000 7f9372ffd640 1 -- 172.21.15.150:0/1692698675 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f937415c7a0 con 0x7f937415bfd0 2024-03-17T23:47:36.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.066+0000 7f9372ffd640 1 --2- 172.21.15.150:0/1692698675 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f937415bfd0 0x7f93740ccd10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f93680097a0 tx=0x7f936802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=965616e6029cb63c server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:36.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.066+0000 7f93727fc640 1 -- 172.21.15.150:0/1692698675 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9368038070 con 0x7f937415bfd0 2024-03-17T23:47:36.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.066+0000 7f93727fc640 1 -- 172.21.15.150:0/1692698675 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9368031500 con 0x7f937415bfd0 2024-03-17T23:47:36.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.066+0000 7f93727fc640 1 -- 172.21.15.150:0/1692698675 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9364001670 con 0x7f937415bfd0 2024-03-17T23:47:36.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.067+0000 7f9379a8a640 1 -- 172.21.15.150:0/1692698675 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f937415bfd0 msgr2=0x7f93740ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:36.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.067+0000 7f9379a8a640 1 --2- 172.21.15.150:0/1692698675 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f937415bfd0 0x7f93740ccd10 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f93680097a0 tx=0x7f936802b820 comp rx=0 tx=0).stop 2024-03-17T23:47:36.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.067+0000 7f9379a8a640 1 -- 172.21.15.150:0/1692698675 shutdown_connections 2024-03-17T23:47:36.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.067+0000 7f9379a8a640 1 --2- 172.21.15.150:0/1692698675 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f937415bfd0 0x7f93740ccd10 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.067+0000 7f9379a8a640 1 -- 172.21.15.150:0/1692698675 >> 172.21.15.150:0/1692698675 conn(0x7f937406b190 msgr2=0x7f937406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:36.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.067+0000 7f9379a8a640 1 -- 172.21.15.150:0/1692698675 shutdown_connections 2024-03-17T23:47:36.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.067+0000 7f9379a8a640 1 -- 172.21.15.150:0/1692698675 wait complete. 2024-03-17T23:47:36.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.068+0000 7f9379a8a640 1 Processor -- start 2024-03-17T23:47:36.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.068+0000 7f9379a8a640 1 -- start start 2024-03-17T23:47:36.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.069+0000 7f9379a8a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f93740cd250 0x7f937415af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:36.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.069+0000 7f9379a8a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9374155320 0x7f9374148360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:36.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.069+0000 7f9379a8a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f937415bfd0 0x7f937415d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:36.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.069+0000 7f9379a8a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9374155880 con 0x7f9374155320 2024-03-17T23:47:36.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.069+0000 7f9379a8a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9374155700 con 0x7f93740cd250 2024-03-17T23:47:36.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.069+0000 7f9379a8a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f937415d3d0 con 0x7f937415bfd0 2024-03-17T23:47:36.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.069+0000 7f9372ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9374155320 0x7f9374148360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:36.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.069+0000 7f9372ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9374155320 0x7f9374148360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46048/0 (socket says 172.21.15.150:46048) 2024-03-17T23:47:36.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.069+0000 7f9372ffd640 1 -- 172.21.15.150:0/1405282905 learned_addr learned my addr 172.21.15.150:0/1405282905 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:36.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.069+0000 7f93737fe640 1 --2- 172.21.15.150:0/1405282905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f93740cd250 0x7f937415af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:36.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.069+0000 7f9373fff640 1 --2- 172.21.15.150:0/1405282905 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f937415bfd0 0x7f937415d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:36.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.069+0000 7f9372ffd640 1 -- 172.21.15.150:0/1405282905 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f937415bfd0 msgr2=0x7f937415d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:36.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.069+0000 7f9372ffd640 1 --2- 172.21.15.150:0/1405282905 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f937415bfd0 0x7f937415d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.069+0000 7f9372ffd640 1 -- 172.21.15.150:0/1405282905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f93740cd250 msgr2=0x7f937415af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:36.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.069+0000 7f9372ffd640 1 --2- 172.21.15.150:0/1405282905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f93740cd250 0x7f937415af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.069+0000 7f9372ffd640 1 -- 172.21.15.150:0/1405282905 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93741575a0 con 0x7f9374155320 2024-03-17T23:47:36.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.070+0000 7f9372ffd640 1 --2- 172.21.15.150:0/1405282905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9374155320 0x7f9374148360 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f936802b860 tx=0x7f93680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:36.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.070+0000 7f935bfff640 1 -- 172.21.15.150:0/1405282905 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9368043020 con 0x7f9374155320 2024-03-17T23:47:36.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.070+0000 7f935bfff640 1 -- 172.21.15.150:0/1405282905 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93680336f0 con 0x7f9374155320 2024-03-17T23:47:36.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.070+0000 7f935bfff640 1 -- 172.21.15.150:0/1405282905 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9368038040 con 0x7f9374155320 2024-03-17T23:47:36.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.070+0000 7f9379a8a640 1 -- 172.21.15.150:0/1405282905 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9374157880 con 0x7f9374155320 2024-03-17T23:47:36.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.070+0000 7f9379a8a640 1 -- 172.21.15.150:0/1405282905 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9374157e40 con 0x7f9374155320 2024-03-17T23:47:36.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.071+0000 7f9379a8a640 1 -- 172.21.15.150:0/1405282905 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93740681a0 con 0x7f9374155320 2024-03-17T23:47:36.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.071+0000 7f935bfff640 1 -- 172.21.15.150:0/1405282905 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9368046020 con 0x7f9374155320 2024-03-17T23:47:36.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.072+0000 7f935bfff640 1 --2- 172.21.15.150:0/1405282905 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f936006bbd0 0x7f936006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:36.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.072+0000 7f935bfff640 1 -- 172.21.15.150:0/1405282905 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f93680ac9f0 con 0x7f9374155320 2024-03-17T23:47:36.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.072+0000 7f93737fe640 1 --2- 172.21.15.150:0/1405282905 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f936006bbd0 0x7f936006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:36.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.072+0000 7f93737fe640 1 --2- 172.21.15.150:0/1405282905 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f936006bbd0 0x7f936006e090 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9364000fd0 tx=0x7f9364001150 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:36.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.076+0000 7f935bfff640 1 -- 172.21.15.150:0/1405282905 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f936804d030 con 0x7f9374155320 2024-03-17T23:47:36.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.250+0000 7f9379a8a640 1 -- 172.21.15.150:0/1405282905 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9374073d10 con 0x7f9374155320 2024-03-17T23:47:36.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.251+0000 7f935bfff640 1 -- 172.21.15.150:0/1405282905 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18881 (secure 0 0 0) 0x7f9368030070 con 0x7f9374155320 2024-03-17T23:47:36.251 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:36.252 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":27,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:36.026096+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.e","mappings":[{"from":7,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:36.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.253+0000 7f9379a8a640 1 -- 172.21.15.150:0/1405282905 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f936006bbd0 msgr2=0x7f936006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:36.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.253+0000 7f9379a8a640 1 --2- 172.21.15.150:0/1405282905 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f936006bbd0 0x7f936006e090 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9364000fd0 tx=0x7f9364001150 comp rx=0 tx=0).stop 2024-03-17T23:47:36.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.253+0000 7f9379a8a640 1 -- 172.21.15.150:0/1405282905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9374155320 msgr2=0x7f9374148360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:36.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.253+0000 7f9379a8a640 1 --2- 172.21.15.150:0/1405282905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9374155320 0x7f9374148360 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f936802b860 tx=0x7f93680046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.253+0000 7f9379a8a640 1 -- 172.21.15.150:0/1405282905 shutdown_connections 2024-03-17T23:47:36.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.253+0000 7f9379a8a640 1 --2- 172.21.15.150:0/1405282905 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f937415bfd0 0x7f937415d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.253+0000 7f9379a8a640 1 --2- 172.21.15.150:0/1405282905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9374155320 0x7f9374148360 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.253+0000 7f9379a8a640 1 --2- 172.21.15.150:0/1405282905 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f936006bbd0 0x7f936006e090 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.253+0000 7f9379a8a640 1 --2- 172.21.15.150:0/1405282905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f93740cd250 0x7f937415af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.253+0000 7f9379a8a640 1 -- 172.21.15.150:0/1405282905 >> 172.21.15.150:0/1405282905 conn(0x7f937406b190 msgr2=0x7f937415ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:36.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.253+0000 7f9379a8a640 1 -- 172.21.15.150:0/1405282905 shutdown_connections 2024-03-17T23:47:36.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.253+0000 7f9379a8a640 1 -- 172.21.15.150:0/1405282905 wait complete. 2024-03-17T23:47:36.265 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:46:53.610678+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:46:54.227335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-17T23:46:57.949235+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-17T23:47:36.265 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:47:36.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.368+0000 7fd5a699d640 1 Processor -- start 2024-03-17T23:47:36.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.369+0000 7fd5a699d640 1 -- start start 2024-03-17T23:47:36.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.369+0000 7fd5a699d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd5a015deb0 0x7fd5a0161a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:36.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.369+0000 7fd5a699d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd5a00d0fd0 0x7fd5a00d2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:36.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.369+0000 7fd5a699d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5a0162110 con 0x7fd5a00b9610 2024-03-17T23:47:36.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.369+0000 7fd5a699d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fd5a01623f0 con 0x7fd5a015deb0 2024-03-17T23:47:36.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.369+0000 7fd5a699d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd5a01626d0 con 0x7fd5a00d0fd0 2024-03-17T23:47:36.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.370+0000 7fd5a599b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd5a015deb0 0x7fd5a0161a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:36.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.370+0000 7fd5a519a640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd5a00b9610 0x7fd5a00d08c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59186/0 (socket says 172.21.15.150:59186) 2024-03-17T23:47:36.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.370+0000 7fd5a519a640 1 -- 172.21.15.150:0/4262125207 learned_addr learned my addr 172.21.15.150:0/4262125207 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:36.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.370+0000 7fd5a619c640 1 --2- 172.21.15.150:0/4262125207 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd5a00d0fd0 0x7fd5a00d2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:36.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.370+0000 7fd5a599b640 1 -- 172.21.15.150:0/4262125207 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd5a00d0fd0 msgr2=0x7fd5a00d2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:36.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.370+0000 7fd5a599b640 1 --2- 172.21.15.150:0/4262125207 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd5a00d0fd0 0x7fd5a00d2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.370+0000 7fd5a599b640 1 -- 172.21.15.150:0/4262125207 >> v1:172.21.15.150:6789/0 conn(0x7fd5a00b9610 legacy=0x7fd5a00d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:36.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.370+0000 7fd5a599b640 1 -- 172.21.15.150:0/4262125207 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a00d2110 con 0x7fd5a015deb0 2024-03-17T23:47:36.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.370+0000 7fd5a4999640 1 -- 172.21.15.150:0/4262125207 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1790532958 0 0) 0x7fd5a0162110 con 0x7fd5a00b9610 2024-03-17T23:47:36.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.370+0000 7fd5a619c640 1 --2- 172.21.15.150:0/4262125207 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd5a00d0fd0 0x7fd5a00d2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:36.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.371+0000 7fd5a599b640 1 --2- 172.21.15.150:0/4262125207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a015deb0 0x7fd5a0161a10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd598009d90 tx=0x7fd59802bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5da157a3a07da80c server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:36.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.371+0000 7fd5a4999640 1 -- 172.21.15.150:0/4262125207 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd598039070 con 0x7fd5a015deb0 2024-03-17T23:47:36.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.371+0000 7fd5a4999640 1 -- 172.21.15.150:0/4262125207 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd59802cce0 con 0x7fd5a015deb0 2024-03-17T23:47:36.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.371+0000 7fd5a4999640 1 -- 172.21.15.150:0/4262125207 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd598034660 con 0x7fd5a015deb0 2024-03-17T23:47:36.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.371+0000 7fd5a699d640 1 -- 172.21.15.150:0/4262125207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a015deb0 msgr2=0x7fd5a0161a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:36.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.371+0000 7fd5a699d640 1 --2- 172.21.15.150:0/4262125207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a015deb0 0x7fd5a0161a10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd598009d90 tx=0x7fd59802bf30 comp rx=0 tx=0).stop 2024-03-17T23:47:36.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.371+0000 7fd5a699d640 1 -- 172.21.15.150:0/4262125207 shutdown_connections 2024-03-17T23:47:36.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.371+0000 7fd5a699d640 1 --2- 172.21.15.150:0/4262125207 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd5a00d0fd0 0x7fd5a00d2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.371+0000 7fd5a699d640 1 --2- 172.21.15.150:0/4262125207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a015deb0 0x7fd5a0161a10 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.371+0000 7fd5a699d640 1 -- 172.21.15.150:0/4262125207 >> 172.21.15.150:0/4262125207 conn(0x7fd5a006ca70 msgr2=0x7fd5a006ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:36.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.371+0000 7fd5a699d640 1 -- 172.21.15.150:0/4262125207 shutdown_connections 2024-03-17T23:47:36.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.372+0000 7fd5a699d640 1 -- 172.21.15.150:0/4262125207 wait complete. 2024-03-17T23:47:36.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.373+0000 7fd5a699d640 1 Processor -- start 2024-03-17T23:47:36.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.373+0000 7fd5a699d640 1 -- start start 2024-03-17T23:47:36.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.373+0000 7fd5a699d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd5a00b9610 0x7fd5a0146c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:36.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.373+0000 7fd5a699d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd5a00d0fd0 0x7fd5a0149590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:36.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.373+0000 7fd5a699d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a015deb0 0x7fd5a0149ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:36.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.373+0000 7fd5a699d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a00d2b70 con 0x7fd5a015deb0 2024-03-17T23:47:36.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.373+0000 7fd5a699d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a015e290 con 0x7fd5a00b9610 2024-03-17T23:47:36.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.373+0000 7fd5a699d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd5a00d2cf0 con 0x7fd5a00d0fd0 2024-03-17T23:47:36.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.373+0000 7fd5a619c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a015deb0 0x7fd5a0149ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:36.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.373+0000 7fd5a619c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a015deb0 0x7fd5a0149ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46072/0 (socket says 172.21.15.150:46072) 2024-03-17T23:47:36.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.373+0000 7fd5a619c640 1 -- 172.21.15.150:0/2059757112 learned_addr learned my addr 172.21.15.150:0/2059757112 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:36.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.373+0000 7fd5a599b640 1 --2- 172.21.15.150:0/2059757112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd5a00b9610 0x7fd5a0146c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:36.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.373+0000 7fd5a519a640 1 --2- 172.21.15.150:0/2059757112 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd5a00d0fd0 0x7fd5a0149590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:36.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.374+0000 7fd5a619c640 1 -- 172.21.15.150:0/2059757112 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd5a00d0fd0 msgr2=0x7fd5a0149590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:36.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.374+0000 7fd5a619c640 1 --2- 172.21.15.150:0/2059757112 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd5a00d0fd0 0x7fd5a0149590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.374+0000 7fd5a619c640 1 -- 172.21.15.150:0/2059757112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd5a00b9610 msgr2=0x7fd5a0146c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:36.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.374+0000 7fd5a619c640 1 --2- 172.21.15.150:0/2059757112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd5a00b9610 0x7fd5a0146c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.374+0000 7fd5a619c640 1 -- 172.21.15.150:0/2059757112 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a015ae70 con 0x7fd5a015deb0 2024-03-17T23:47:36.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.374+0000 7fd5a619c640 1 --2- 172.21.15.150:0/2059757112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a015deb0 0x7fd5a0149ad0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd59000e880 tx=0x7fd59000ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:36.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.374+0000 7fd59e7fc640 1 -- 172.21.15.150:0/2059757112 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd590017070 con 0x7fd5a015deb0 2024-03-17T23:47:36.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.374+0000 7fd59e7fc640 1 -- 172.21.15.150:0/2059757112 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd590004020 con 0x7fd5a015deb0 2024-03-17T23:47:36.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.374+0000 7fd59e7fc640 1 -- 172.21.15.150:0/2059757112 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5900043f0 con 0x7fd5a015deb0 2024-03-17T23:47:36.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.374+0000 7fd5a699d640 1 -- 172.21.15.150:0/2059757112 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5a0160a50 con 0x7fd5a015deb0 2024-03-17T23:47:36.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.374+0000 7fd5a699d640 1 -- 172.21.15.150:0/2059757112 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5a01475a0 con 0x7fd5a015deb0 2024-03-17T23:47:36.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.375+0000 7fd5a699d640 1 -- 172.21.15.150:0/2059757112 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5a0074c40 con 0x7fd5a015deb0 2024-03-17T23:47:36.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.375+0000 7fd59e7fc640 1 -- 172.21.15.150:0/2059757112 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd59000c940 con 0x7fd5a015deb0 2024-03-17T23:47:36.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.376+0000 7fd59e7fc640 1 --2- 172.21.15.150:0/2059757112 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd58c06bbd0 0x7fd58c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:36.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.376+0000 7fd59e7fc640 1 -- 172.21.15.150:0/2059757112 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fd59008ac30 con 0x7fd5a015deb0 2024-03-17T23:47:36.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.376+0000 7fd5a599b640 1 --2- 172.21.15.150:0/2059757112 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd58c06bbd0 0x7fd58c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:36.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.376+0000 7fd5a599b640 1 --2- 172.21.15.150:0/2059757112 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd58c06bbd0 0x7fd58c06e090 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd598036000 tx=0x7fd598030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:36.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.380+0000 7fd59e7fc640 1 -- 172.21.15.150:0/2059757112 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd590058c40 con 0x7fd5a015deb0 2024-03-17T23:47:36.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.552+0000 7fd5a699d640 1 -- 172.21.15.150:0/2059757112 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd5a00d1840 con 0x7fd5a015deb0 2024-03-17T23:47:36.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.552+0000 7fd59e7fc640 1 -- 172.21.15.150:0/2059757112 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fd59005cce0 con 0x7fd5a015deb0 2024-03-17T23:47:36.553 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:47:36.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.554+0000 7fd5a699d640 1 -- 172.21.15.150:0/2059757112 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd58c06bbd0 msgr2=0x7fd58c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:36.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.554+0000 7fd5a699d640 1 --2- 172.21.15.150:0/2059757112 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd58c06bbd0 0x7fd58c06e090 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd598036000 tx=0x7fd598030000 comp rx=0 tx=0).stop 2024-03-17T23:47:36.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.554+0000 7fd5a699d640 1 -- 172.21.15.150:0/2059757112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a015deb0 msgr2=0x7fd5a0149ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:36.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.554+0000 7fd5a699d640 1 --2- 172.21.15.150:0/2059757112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a015deb0 0x7fd5a0149ad0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd59000e880 tx=0x7fd59000ed40 comp rx=0 tx=0).stop 2024-03-17T23:47:36.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.554+0000 7fd5a699d640 1 -- 172.21.15.150:0/2059757112 shutdown_connections 2024-03-17T23:47:36.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.554+0000 7fd5a699d640 1 --2- 172.21.15.150:0/2059757112 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5a015deb0 0x7fd5a0149ad0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.554+0000 7fd5a699d640 1 --2- 172.21.15.150:0/2059757112 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd5a00d0fd0 0x7fd5a0149590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.554+0000 7fd5a699d640 1 --2- 172.21.15.150:0/2059757112 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd58c06bbd0 0x7fd58c06e090 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.554+0000 7fd5a699d640 1 --2- 172.21.15.150:0/2059757112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd5a00b9610 0x7fd5a0146c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.554+0000 7fd5a699d640 1 -- 172.21.15.150:0/2059757112 >> 172.21.15.150:0/2059757112 conn(0x7fd5a006ca70 msgr2=0x7fd5a015d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:36.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.554+0000 7fd5a699d640 1 -- 172.21.15.150:0/2059757112 shutdown_connections 2024-03-17T23:47:36.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.554+0000 7fd5a699d640 1 -- 172.21.15.150:0/2059757112 wait complete. 2024-03-17T23:47:36.566 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:47:36.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.670+0000 7f6189adc640 1 Processor -- start 2024-03-17T23:47:36.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.670+0000 7f6189adc640 1 -- start start 2024-03-17T23:47:36.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.671+0000 7f6189adc640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6184155320 0x7f618415bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:36.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.671+0000 7f6189adc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f618406dd30 0x7f61840ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:36.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.671+0000 7f6189adc640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61840cf3b0 con 0x7f61840cd250 2024-03-17T23:47:36.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.671+0000 7f6189adc640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f618415c550 con 0x7f6184155320 2024-03-17T23:47:36.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.671+0000 7f6189adc640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f61840cf690 con 0x7f618406dd30 2024-03-17T23:47:36.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.671+0000 7f61837fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6184155320 0x7f618415bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:36.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.671+0000 7f6182ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f618406dd30 0x7f61840ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:36.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.671+0000 7f6183fff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f61840cd250 0x7f618415e980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:46752/0 (socket says 172.21.15.150:46752) 2024-03-17T23:47:36.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.671+0000 7f61837fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6184155320 0x7f618415bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:35006/0 (socket says 172.21.15.150:35006) 2024-03-17T23:47:36.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.671+0000 7f6183fff640 1 -- 172.21.15.150:0/2517267081 learned_addr learned my addr 172.21.15.150:0/2517267081 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:36.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.671+0000 7f61837fe640 1 -- 172.21.15.150:0/2517267081 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f618406dd30 msgr2=0x7f61840ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:36.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.671+0000 7f61837fe640 1 --2- 172.21.15.150:0/2517267081 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f618406dd30 0x7f61840ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.671+0000 7f61837fe640 1 -- 172.21.15.150:0/2517267081 >> v1:172.21.15.157:6789/0 conn(0x7f61840cd250 legacy=0x7f618415e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:36.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.671+0000 7f61837fe640 1 -- 172.21.15.150:0/2517267081 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618415c7a0 con 0x7f6184155320 2024-03-17T23:47:36.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.671+0000 7f61827fc640 1 -- 172.21.15.150:0/2517267081 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3384200300 0 0) 0x7f61840cf3b0 con 0x7f61840cd250 2024-03-17T23:47:36.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.672+0000 7f61837fe640 1 --2- 172.21.15.150:0/2517267081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6184155320 0x7f618415bfd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f61740097a0 tx=0x7f617402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf9454a0e8323ef4 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:36.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.672+0000 7f61827fc640 1 -- 172.21.15.150:0/2517267081 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174035020 con 0x7f6184155320 2024-03-17T23:47:36.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.672+0000 7f61827fc640 1 -- 172.21.15.150:0/2517267081 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6174031830 con 0x7f6184155320 2024-03-17T23:47:36.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.672+0000 7f61827fc640 1 -- 172.21.15.150:0/2517267081 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174031c00 con 0x7f6184155320 2024-03-17T23:47:36.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.673+0000 7f6189adc640 1 -- 172.21.15.150:0/2517267081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6184155320 msgr2=0x7f618415bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:36.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.673+0000 7f6189adc640 1 --2- 172.21.15.150:0/2517267081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6184155320 0x7f618415bfd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f61740097a0 tx=0x7f617402b8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.673+0000 7f6189adc640 1 -- 172.21.15.150:0/2517267081 shutdown_connections 2024-03-17T23:47:36.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.673+0000 7f6189adc640 1 --2- 172.21.15.150:0/2517267081 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f618406dd30 0x7f61840ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.673+0000 7f6189adc640 1 --2- 172.21.15.150:0/2517267081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6184155320 0x7f618415bfd0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.673+0000 7f6189adc640 1 -- 172.21.15.150:0/2517267081 >> 172.21.15.150:0/2517267081 conn(0x7f618406b190 msgr2=0x7f618406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:36.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.673+0000 7f6189adc640 1 -- 172.21.15.150:0/2517267081 shutdown_connections 2024-03-17T23:47:36.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.673+0000 7f6189adc640 1 -- 172.21.15.150:0/2517267081 wait complete. 2024-03-17T23:47:36.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.674+0000 7f6189adc640 1 Processor -- start 2024-03-17T23:47:36.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.674+0000 7f6189adc640 1 -- start start 2024-03-17T23:47:36.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.675+0000 7f6189adc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f618406dd30 0x7f61841f2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:36.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.675+0000 7f6189adc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61840cd250 0x7f618415ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:36.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.675+0000 7f6189adc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6184155320 0x7f61840c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:36.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.675+0000 7f6189adc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6184155700 con 0x7f61840cd250 2024-03-17T23:47:36.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.675+0000 7f6189adc640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f618406e190 con 0x7f618406dd30 2024-03-17T23:47:36.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.675+0000 7f6189adc640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6184155880 con 0x7f6184155320 2024-03-17T23:47:36.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.675+0000 7f6182ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61840cd250 0x7f618415ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:36.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.675+0000 7f6182ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61840cd250 0x7f618415ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46076/0 (socket says 172.21.15.150:46076) 2024-03-17T23:47:36.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.675+0000 7f6182ffd640 1 -- 172.21.15.150:0/3126469843 learned_addr learned my addr 172.21.15.150:0/3126469843 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:36.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.675+0000 7f6183fff640 1 --2- 172.21.15.150:0/3126469843 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6184155320 0x7f61840c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:36.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.675+0000 7f61837fe640 1 --2- 172.21.15.150:0/3126469843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f618406dd30 0x7f61841f2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:36.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.675+0000 7f6182ffd640 1 -- 172.21.15.150:0/3126469843 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6184155320 msgr2=0x7f61840c70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:36.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.675+0000 7f6182ffd640 1 --2- 172.21.15.150:0/3126469843 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6184155320 0x7f61840c70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.675+0000 7f6182ffd640 1 -- 172.21.15.150:0/3126469843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f618406dd30 msgr2=0x7f61841f2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:36.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.675+0000 7f6182ffd640 1 --2- 172.21.15.150:0/3126469843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f618406dd30 0x7f61841f2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.675+0000 7f6182ffd640 1 -- 172.21.15.150:0/3126469843 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618415d6a0 con 0x7f61840cd250 2024-03-17T23:47:36.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.675+0000 7f61837fe640 1 --2- 172.21.15.150:0/3126469843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f618406dd30 0x7f61841f2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:36.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.676+0000 7f6182ffd640 1 --2- 172.21.15.150:0/3126469843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61840cd250 0x7f618415ccc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f617800e870 tx=0x7f617800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:36.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.676+0000 7f616bfff640 1 -- 172.21.15.150:0/3126469843 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6178017070 con 0x7f61840cd250 2024-03-17T23:47:36.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.676+0000 7f616bfff640 1 -- 172.21.15.150:0/3126469843 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f617800c940 con 0x7f61840cd250 2024-03-17T23:47:36.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.676+0000 7f6189adc640 1 -- 172.21.15.150:0/3126469843 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f618415d980 con 0x7f61840cd250 2024-03-17T23:47:36.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.676+0000 7f616bfff640 1 -- 172.21.15.150:0/3126469843 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617800cd30 con 0x7f61840cd250 2024-03-17T23:47:36.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.676+0000 7f6189adc640 1 -- 172.21.15.150:0/3126469843 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f618415df40 con 0x7f61840cd250 2024-03-17T23:47:36.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.677+0000 7f6189adc640 1 -- 172.21.15.150:0/3126469843 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61840681a0 con 0x7f61840cd250 2024-03-17T23:47:36.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.677+0000 7f616bfff640 1 -- 172.21.15.150:0/3126469843 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f617801a070 con 0x7f61840cd250 2024-03-17T23:47:36.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.678+0000 7f616bfff640 1 --2- 172.21.15.150:0/3126469843 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f617006bbd0 0x7f617006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:36.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.678+0000 7f616bfff640 1 -- 172.21.15.150:0/3126469843 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f617808afd0 con 0x7f61840cd250 2024-03-17T23:47:36.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.678+0000 7f61837fe640 1 --2- 172.21.15.150:0/3126469843 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f617006bbd0 0x7f617006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:36.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.678+0000 7f61837fe640 1 --2- 172.21.15.150:0/3126469843 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f617006bbd0 0x7f617006e090 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6174036000 tx=0x7f617403e040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:36.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.682+0000 7f616bfff640 1 -- 172.21.15.150:0/3126469843 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6178059100 con 0x7f61840cd250 2024-03-17T23:47:36.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.854+0000 7f6189adc640 1 -- 172.21.15.150:0/3126469843 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f618406b960 con 0x7f61840cd250 2024-03-17T23:47:36.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.854+0000 7f616bfff640 1 -- 172.21.15.150:0/3126469843 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f617805d1a0 con 0x7f61840cd250 2024-03-17T23:47:36.855 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:36.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.856+0000 7f6189adc640 1 -- 172.21.15.150:0/3126469843 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f617006bbd0 msgr2=0x7f617006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:36.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.856+0000 7f6189adc640 1 --2- 172.21.15.150:0/3126469843 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f617006bbd0 0x7f617006e090 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6174036000 tx=0x7f617403e040 comp rx=0 tx=0).stop 2024-03-17T23:47:36.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.856+0000 7f6189adc640 1 -- 172.21.15.150:0/3126469843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61840cd250 msgr2=0x7f618415ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:36.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.856+0000 7f6189adc640 1 --2- 172.21.15.150:0/3126469843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61840cd250 0x7f618415ccc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f617800e870 tx=0x7f617800ed30 comp rx=0 tx=0).stop 2024-03-17T23:47:36.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.856+0000 7f6189adc640 1 -- 172.21.15.150:0/3126469843 shutdown_connections 2024-03-17T23:47:36.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.856+0000 7f6189adc640 1 --2- 172.21.15.150:0/3126469843 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6184155320 0x7f61840c70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.856+0000 7f6189adc640 1 --2- 172.21.15.150:0/3126469843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61840cd250 0x7f618415ccc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.856+0000 7f6189adc640 1 --2- 172.21.15.150:0/3126469843 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f617006bbd0 0x7f617006e090 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.856+0000 7f6189adc640 1 --2- 172.21.15.150:0/3126469843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f618406dd30 0x7f61841f2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.856+0000 7f6189adc640 1 -- 172.21.15.150:0/3126469843 >> 172.21.15.150:0/3126469843 conn(0x7f618406b190 msgr2=0x7f61840b2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:36.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.856+0000 7f6189adc640 1 -- 172.21.15.150:0/3126469843 shutdown_connections 2024-03-17T23:47:36.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.856+0000 7f6189adc640 1 -- 172.21.15.150:0/3126469843 wait complete. 2024-03-17T23:47:36.869 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:47:36.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.971+0000 7f410ba7e640 1 Processor -- start 2024-03-17T23:47:36.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.972+0000 7f410ba7e640 1 -- start start 2024-03-17T23:47:36.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.972+0000 7f410ba7e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f410415f3d0 0x7f41040cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:36.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.972+0000 7f410ba7e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f41040cc770 0x7f41041603c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:36.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.972+0000 7f410ba7e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41040ccc50 con 0x7f41040d0170 2024-03-17T23:47:36.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.972+0000 7f410ba7e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f410415f7b0 con 0x7f410415f3d0 2024-03-17T23:47:36.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.972+0000 7f410ba7e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f4104160900 con 0x7f41040cc770 2024-03-17T23:47:36.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.972+0000 7f410981a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f410415f3d0 0x7f41040cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:36.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.973+0000 7f410981a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f410415f3d0 0x7f41040cdcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46084/0 (socket says 172.21.15.150:46084) 2024-03-17T23:47:36.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.973+0000 7f410981a640 1 -- 172.21.15.150:0/1317935036 learned_addr learned my addr 172.21.15.150:0/1317935036 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:36.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.973+0000 7f4109019640 1 --2- 172.21.15.150:0/1317935036 >> v2:172.21.15.157:3300/0 conn(0x7f41040cc770 0x7f41041603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:36.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.973+0000 7f410981a640 1 -- 172.21.15.150:0/1317935036 >> v2:172.21.15.157:3300/0 conn(0x7f41040cc770 msgr2=0x7f41041603c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:36.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.973+0000 7f410981a640 1 --2- 172.21.15.150:0/1317935036 >> v2:172.21.15.157:3300/0 conn(0x7f41040cc770 0x7f41041603c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.973+0000 7f410981a640 1 -- 172.21.15.150:0/1317935036 >> v1:172.21.15.157:6789/0 conn(0x7f41040d0170 legacy=0x7f41040d0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:36.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.973+0000 7f410981a640 1 -- 172.21.15.150:0/1317935036 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41040ccf30 con 0x7f410415f3d0 2024-03-17T23:47:36.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.974+0000 7f410981a640 1 --2- 172.21.15.150:0/1317935036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f410415f3d0 0x7f41040cdcf0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4104072b70 tx=0x7f410002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaf8073ed76b185e server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:36.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.974+0000 7f4108818640 1 -- 172.21.15.150:0/1317935036 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410002cb20 con 0x7f410415f3d0 2024-03-17T23:47:36.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.974+0000 7f4108818640 1 -- 172.21.15.150:0/1317935036 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f410002ccc0 con 0x7f410415f3d0 2024-03-17T23:47:36.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.974+0000 7f4108818640 1 -- 172.21.15.150:0/1317935036 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4100034610 con 0x7f410415f3d0 2024-03-17T23:47:36.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.974+0000 7f410ba7e640 1 -- 172.21.15.150:0/1317935036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f410415f3d0 msgr2=0x7f41040cdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:36.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.974+0000 7f410ba7e640 1 --2- 172.21.15.150:0/1317935036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f410415f3d0 0x7f41040cdcf0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4104072b70 tx=0x7f410002bf80 comp rx=0 tx=0).stop 2024-03-17T23:47:36.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.974+0000 7f410ba7e640 1 -- 172.21.15.150:0/1317935036 shutdown_connections 2024-03-17T23:47:36.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.974+0000 7f410ba7e640 1 --2- 172.21.15.150:0/1317935036 >> v2:172.21.15.157:3300/0 conn(0x7f41040cc770 0x7f41041603c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.974+0000 7f410ba7e640 1 --2- 172.21.15.150:0/1317935036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f410415f3d0 0x7f41040cdcf0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.974+0000 7f410ba7e640 1 -- 172.21.15.150:0/1317935036 >> 172.21.15.150:0/1317935036 conn(0x7f41040b36c0 msgr2=0x7f41040b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:36.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.974+0000 7f410ba7e640 1 -- 172.21.15.150:0/1317935036 shutdown_connections 2024-03-17T23:47:36.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.974+0000 7f410ba7e640 1 -- 172.21.15.150:0/1317935036 wait complete. 2024-03-17T23:47:36.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.975+0000 7f410ba7e640 1 Processor -- start 2024-03-17T23:47:36.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.975+0000 7f410ba7e640 1 -- start start 2024-03-17T23:47:36.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.976+0000 7f410ba7e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41040cc770 0x7f41040cb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:36.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.976+0000 7f410ba7e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41040d0170 0x7f41040b0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:36.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.976+0000 7f410ba7e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f410415f3d0 0x7f4104150070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:36.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.976+0000 7f410ba7e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f41040cd7d0 con 0x7f41040d0170 2024-03-17T23:47:36.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.976+0000 7f410ba7e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f41040cd4f0 con 0x7f410415f3d0 2024-03-17T23:47:36.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.976+0000 7f410ba7e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f41040cfa30 con 0x7f41040cc770 2024-03-17T23:47:36.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.976+0000 7f4109019640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41040d0170 0x7f41040b0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:36.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.976+0000 7f410981a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41040cc770 0x7f41040cb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:36.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.976+0000 7f4109019640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41040d0170 0x7f41040b0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46086/0 (socket says 172.21.15.150:46086) 2024-03-17T23:47:36.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.976+0000 7f4109019640 1 -- 172.21.15.150:0/4241861710 learned_addr learned my addr 172.21.15.150:0/4241861710 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:36.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.976+0000 7f410a01b640 1 --2- 172.21.15.150:0/4241861710 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f410415f3d0 0x7f4104150070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:36.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.977+0000 7f4109019640 1 -- 172.21.15.150:0/4241861710 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41040cc770 msgr2=0x7f41040cb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:36.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.977+0000 7f4109019640 1 --2- 172.21.15.150:0/4241861710 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41040cc770 0x7f41040cb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.977+0000 7f4109019640 1 -- 172.21.15.150:0/4241861710 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f410415f3d0 msgr2=0x7f4104150070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:36.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.977+0000 7f4109019640 1 --2- 172.21.15.150:0/4241861710 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f410415f3d0 0x7f4104150070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:36.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.977+0000 7f4109019640 1 -- 172.21.15.150:0/4241861710 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41040c9750 con 0x7f41040d0170 2024-03-17T23:47:36.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.977+0000 7f4109019640 1 --2- 172.21.15.150:0/4241861710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41040d0170 0x7f41040b0f00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f40f400e9e0 tx=0x7f40f400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:36.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.977+0000 7f40fa7fc640 1 -- 172.21.15.150:0/4241861710 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f4017070 con 0x7f41040d0170 2024-03-17T23:47:36.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.977+0000 7f40fa7fc640 1 -- 172.21.15.150:0/4241861710 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40f400c9d0 con 0x7f41040d0170 2024-03-17T23:47:36.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.977+0000 7f40fa7fc640 1 -- 172.21.15.150:0/4241861710 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f400cdc0 con 0x7f41040d0170 2024-03-17T23:47:36.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.977+0000 7f410ba7e640 1 -- 172.21.15.150:0/4241861710 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41040c99f0 con 0x7f41040d0170 2024-03-17T23:47:36.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.977+0000 7f410ba7e640 1 -- 172.21.15.150:0/4241861710 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41040c9fb0 con 0x7f41040d0170 2024-03-17T23:47:36.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.978+0000 7f410ba7e640 1 -- 172.21.15.150:0/4241861710 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41040688b0 con 0x7f41040d0170 2024-03-17T23:47:36.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.978+0000 7f40fa7fc640 1 -- 172.21.15.150:0/4241861710 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f40f401a070 con 0x7f41040d0170 2024-03-17T23:47:36.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.979+0000 7f40fa7fc640 1 --2- 172.21.15.150:0/4241861710 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f40e806bbd0 0x7f40e806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:36.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.979+0000 7f410981a640 1 --2- 172.21.15.150:0/4241861710 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f40e806bbd0 0x7f40e806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:36.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.979+0000 7f40fa7fc640 1 -- 172.21.15.150:0/4241861710 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f40f408ae50 con 0x7f41040d0170 2024-03-17T23:47:36.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.979+0000 7f410981a640 1 --2- 172.21.15.150:0/4241861710 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f40e806bbd0 0x7f40e806e090 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4100035000 tx=0x7f4100030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:36.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:36.983+0000 7f40fa7fc640 1 -- 172.21.15.150:0/4241861710 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f40f4058ef0 con 0x7f41040d0170 2024-03-17T23:47:37.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.164+0000 7f410ba7e640 1 -- 172.21.15.150:0/4241861710 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f41040c8320 con 0x7f41040d0170 2024-03-17T23:47:37.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.165+0000 7f40fa7fc640 1 -- 172.21.15.150:0/4241861710 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7f40f405cf90 con 0x7f41040d0170 2024-03-17T23:47:37.165 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:47:37.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.167+0000 7f410ba7e640 1 -- 172.21.15.150:0/4241861710 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f40e806bbd0 msgr2=0x7f40e806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:37.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.167+0000 7f410ba7e640 1 --2- 172.21.15.150:0/4241861710 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f40e806bbd0 0x7f40e806e090 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4100035000 tx=0x7f4100030000 comp rx=0 tx=0).stop 2024-03-17T23:47:37.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.167+0000 7f410ba7e640 1 -- 172.21.15.150:0/4241861710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41040d0170 msgr2=0x7f41040b0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:37.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.167+0000 7f410ba7e640 1 --2- 172.21.15.150:0/4241861710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41040d0170 0x7f41040b0f00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f40f400e9e0 tx=0x7f40f400eea0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.167+0000 7f410ba7e640 1 -- 172.21.15.150:0/4241861710 shutdown_connections 2024-03-17T23:47:37.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.167+0000 7f410ba7e640 1 --2- 172.21.15.150:0/4241861710 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f40e806bbd0 0x7f40e806e090 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.167+0000 7f410ba7e640 1 --2- 172.21.15.150:0/4241861710 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f410415f3d0 0x7f4104150070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.167+0000 7f410ba7e640 1 --2- 172.21.15.150:0/4241861710 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41040d0170 0x7f41040b0f00 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.167+0000 7f410ba7e640 1 --2- 172.21.15.150:0/4241861710 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f41040cc770 0x7f41040cb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.167+0000 7f410ba7e640 1 -- 172.21.15.150:0/4241861710 >> 172.21.15.150:0/4241861710 conn(0x7f41040b36c0 msgr2=0x7f41040c5890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:37.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.167+0000 7f410ba7e640 1 -- 172.21.15.150:0/4241861710 shutdown_connections 2024-03-17T23:47:37.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.167+0000 7f410ba7e640 1 -- 172.21.15.150:0/4241861710 wait complete. 2024-03-17T23:47:37.178 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-17T23:47:37.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.282+0000 7f17292bf640 1 Processor -- start 2024-03-17T23:47:37.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.282+0000 7f17292bf640 1 -- start start 2024-03-17T23:47:37.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.282+0000 7f17292bf640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f17240d0ce0 0x7f17240b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:37.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.282+0000 7f17292bf640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f17240b2bf0 0x7f17240d1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:37.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.282+0000 7f17292bf640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1724161d30 con 0x7f1724161520 2024-03-17T23:47:37.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.282+0000 7f17292bf640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f17240b9bf0 con 0x7f17240d0ce0 2024-03-17T23:47:37.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.282+0000 7f17292bf640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f1724162940 con 0x7f17240b2bf0 2024-03-17T23:47:37.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.283+0000 7f17227fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f17240d0ce0 0x7f17240b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:37.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.283+0000 7f17227fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f17240d0ce0 0x7f17240b96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46104/0 (socket says 172.21.15.150:46104) 2024-03-17T23:47:37.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.283+0000 7f17227fc640 1 -- 172.21.15.150:0/1938561110 learned_addr learned my addr 172.21.15.150:0/1938561110 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:37.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.283+0000 7f17237fe640 1 --2- 172.21.15.150:0/1938561110 >> v2:172.21.15.157:3300/0 conn(0x7f17240b2bf0 0x7f17240d1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:37.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.283+0000 7f17227fc640 1 -- 172.21.15.150:0/1938561110 >> v2:172.21.15.157:3300/0 conn(0x7f17240b2bf0 msgr2=0x7f17240d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:37.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.283+0000 7f17227fc640 1 --2- 172.21.15.150:0/1938561110 >> v2:172.21.15.157:3300/0 conn(0x7f17240b2bf0 0x7f17240d1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.283+0000 7f17227fc640 1 -- 172.21.15.150:0/1938561110 >> v1:172.21.15.157:6789/0 conn(0x7f1724161520 legacy=0x7f17240d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:37.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.283+0000 7f17227fc640 1 -- 172.21.15.150:0/1938561110 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1724162010 con 0x7f17240d0ce0 2024-03-17T23:47:37.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.283+0000 7f17227fc640 1 --2- 172.21.15.150:0/1938561110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17240d0ce0 0x7f17240b96b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f17240ce6a0 tx=0x7f171802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2ff8c3ae2a32ccb server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:37.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.284+0000 7f1721ffb640 1 -- 172.21.15.150:0/1938561110 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f171802e070 con 0x7f17240d0ce0 2024-03-17T23:47:37.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.284+0000 7f1721ffb640 1 -- 172.21.15.150:0/1938561110 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1718037070 con 0x7f17240d0ce0 2024-03-17T23:47:37.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.284+0000 7f1721ffb640 1 -- 172.21.15.150:0/1938561110 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1718031a00 con 0x7f17240d0ce0 2024-03-17T23:47:37.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.284+0000 7f17292bf640 1 -- 172.21.15.150:0/1938561110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17240d0ce0 msgr2=0x7f17240b96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:37.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.284+0000 7f17292bf640 1 --2- 172.21.15.150:0/1938561110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17240d0ce0 0x7f17240b96b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f17240ce6a0 tx=0x7f171802b880 comp rx=0 tx=0).stop 2024-03-17T23:47:37.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.284+0000 7f17292bf640 1 -- 172.21.15.150:0/1938561110 shutdown_connections 2024-03-17T23:47:37.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.284+0000 7f17292bf640 1 --2- 172.21.15.150:0/1938561110 >> v2:172.21.15.157:3300/0 conn(0x7f17240b2bf0 0x7f17240d1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.284+0000 7f17292bf640 1 --2- 172.21.15.150:0/1938561110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f17240d0ce0 0x7f17240b96b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.284+0000 7f17292bf640 1 -- 172.21.15.150:0/1938561110 >> 172.21.15.150:0/1938561110 conn(0x7f172406ca00 msgr2=0x7f172406ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:37.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.284+0000 7f17292bf640 1 -- 172.21.15.150:0/1938561110 shutdown_connections 2024-03-17T23:47:37.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.284+0000 7f17292bf640 1 -- 172.21.15.150:0/1938561110 wait complete. 2024-03-17T23:47:37.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.286+0000 7f17292bf640 1 Processor -- start 2024-03-17T23:47:37.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.286+0000 7f17292bf640 1 -- start start 2024-03-17T23:47:37.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.286+0000 7f17292bf640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17240b2bf0 0x7f17240c7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:37.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.286+0000 7f17292bf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17240d0ce0 0x7f17240cd790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:37.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.286+0000 7f17292bf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1724161520 0x7f17240c9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:37.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.286+0000 7f17292bf640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1724161900 con 0x7f1724161520 2024-03-17T23:47:37.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.286+0000 7f17292bf640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f17241625d0 con 0x7f17240b2bf0 2024-03-17T23:47:37.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.286+0000 7f17292bf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f17240d24e0 con 0x7f17240d0ce0 2024-03-17T23:47:37.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.286+0000 7f17237fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1724161520 0x7f17240c9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:37.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.286+0000 7f17237fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1724161520 0x7f17240c9bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46106/0 (socket says 172.21.15.150:46106) 2024-03-17T23:47:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.286+0000 7f17237fe640 1 -- 172.21.15.150:0/296051527 learned_addr learned my addr 172.21.15.150:0/296051527 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.286+0000 7f1722ffd640 1 --2- 172.21.15.150:0/296051527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17240b2bf0 0x7f17240c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.287+0000 7f17227fc640 1 --2- 172.21.15.150:0/296051527 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17240d0ce0 0x7f17240cd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.287+0000 7f17237fe640 1 -- 172.21.15.150:0/296051527 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17240d0ce0 msgr2=0x7f17240cd790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.287+0000 7f17237fe640 1 --2- 172.21.15.150:0/296051527 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17240d0ce0 0x7f17240cd790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.287+0000 7f17237fe640 1 -- 172.21.15.150:0/296051527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17240b2bf0 msgr2=0x7f17240c7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.287+0000 7f17237fe640 1 --2- 172.21.15.150:0/296051527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17240b2bf0 0x7f17240c7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.287+0000 7f17237fe640 1 -- 172.21.15.150:0/296051527 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17240b3680 con 0x7f1724161520 2024-03-17T23:47:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.287+0000 7f17237fe640 1 --2- 172.21.15.150:0/296051527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1724161520 0x7f17240c9bd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f171c00e930 tx=0x7f171c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.287+0000 7f170b7fe640 1 -- 172.21.15.150:0/296051527 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f171c017070 con 0x7f1724161520 2024-03-17T23:47:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.287+0000 7f170b7fe640 1 -- 172.21.15.150:0/296051527 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f171c00c980 con 0x7f1724161520 2024-03-17T23:47:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.287+0000 7f170b7fe640 1 -- 172.21.15.150:0/296051527 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f171c00cd70 con 0x7f1724161520 2024-03-17T23:47:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.287+0000 7f17292bf640 1 -- 172.21.15.150:0/296051527 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17240cdcd0 con 0x7f1724161520 2024-03-17T23:47:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.287+0000 7f17292bf640 1 -- 172.21.15.150:0/296051527 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17240cd4a0 con 0x7f1724161520 2024-03-17T23:47:37.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.288+0000 7f17292bf640 1 -- 172.21.15.150:0/296051527 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17240681a0 con 0x7f1724161520 2024-03-17T23:47:37.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.289+0000 7f170b7fe640 1 -- 172.21.15.150:0/296051527 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f171c01a070 con 0x7f1724161520 2024-03-17T23:47:37.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.289+0000 7f170b7fe640 1 --2- 172.21.15.150:0/296051527 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f171006bbd0 0x7f171006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:37.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.289+0000 7f170b7fe640 1 -- 172.21.15.150:0/296051527 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f171c08adb0 con 0x7f1724161520 2024-03-17T23:47:37.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.289+0000 7f1722ffd640 1 --2- 172.21.15.150:0/296051527 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f171006bbd0 0x7f171006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:37.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.289+0000 7f1722ffd640 1 --2- 172.21.15.150:0/296051527 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f171006bbd0 0x7f171006e090 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1714001e70 tx=0x7f1714009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:37.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.293+0000 7f170b7fe640 1 -- 172.21.15.150:0/296051527 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f17240cdcd0 con 0x7f1724161520 2024-03-17T23:47:37.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.470+0000 7f17292bf640 1 -- 172.21.15.150:0/296051527 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f17240d19c0 con 0x7f1724161520 2024-03-17T23:47:37.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.471+0000 7f170b7fe640 1 -- 172.21.15.150:0/296051527 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7f17240b3680 con 0x7f1724161520 2024-03-17T23:47:37.472 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:37.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.473+0000 7f17292bf640 1 -- 172.21.15.150:0/296051527 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f171006bbd0 msgr2=0x7f171006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:37.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.473+0000 7f17292bf640 1 --2- 172.21.15.150:0/296051527 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f171006bbd0 0x7f171006e090 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1714001e70 tx=0x7f1714009040 comp rx=0 tx=0).stop 2024-03-17T23:47:37.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.473+0000 7f17292bf640 1 -- 172.21.15.150:0/296051527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1724161520 msgr2=0x7f17240c9bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:37.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.473+0000 7f17292bf640 1 --2- 172.21.15.150:0/296051527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1724161520 0x7f17240c9bd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f171c00e930 tx=0x7f171c00edf0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.473+0000 7f17292bf640 1 -- 172.21.15.150:0/296051527 shutdown_connections 2024-03-17T23:47:37.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.473+0000 7f17292bf640 1 --2- 172.21.15.150:0/296051527 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1724161520 0x7f17240c9bd0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.473+0000 7f17292bf640 1 --2- 172.21.15.150:0/296051527 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f17240d0ce0 0x7f17240cd790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.473+0000 7f17292bf640 1 --2- 172.21.15.150:0/296051527 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f171006bbd0 0x7f171006e090 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.473+0000 7f17292bf640 1 --2- 172.21.15.150:0/296051527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17240b2bf0 0x7f17240c7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.473+0000 7f17292bf640 1 -- 172.21.15.150:0/296051527 >> 172.21.15.150:0/296051527 conn(0x7f172406ca00 msgr2=0x7f172415bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:37.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.473+0000 7f17292bf640 1 -- 172.21.15.150:0/296051527 shutdown_connections 2024-03-17T23:47:37.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.473+0000 7f17292bf640 1 -- 172.21.15.150:0/296051527 wait complete. 2024-03-17T23:47:37.489 INFO:tasks.cephfs_test_runner:Starting test: test_forbidden_modification (tasks.cephfs.test_pool_perm.TestPoolPerm) That a client who does not have the capability for setting 2024-03-17T23:47:37.489 INFO:tasks.cephfs_test_runner:test_forbidden_modification (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-17T23:47:37.490 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:37.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.590+0000 7fda5da0b640 1 Processor -- start 2024-03-17T23:47:37.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.590+0000 7fda5da0b640 1 -- start start 2024-03-17T23:47:37.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.591+0000 7fda5da0b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fda5815f460 0x7fda58099e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:37.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.591+0000 7fda5da0b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fda580b2dc0 0x7fda5809a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:37.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.591+0000 7fda5da0b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda580cc4e0 con 0x7fda580b0a20 2024-03-17T23:47:37.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.591+0000 7fda5da0b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fda580d09c0 con 0x7fda580b2dc0 2024-03-17T23:47:37.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.591+0000 7fda5da0b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fda580b3250 con 0x7fda5815f460 2024-03-17T23:47:37.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.591+0000 7fda567fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fda580b2dc0 0x7fda5809a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:37.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.591+0000 7fda567fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fda580b2dc0 0x7fda5809a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46118/0 (socket says 172.21.15.150:46118) 2024-03-17T23:47:37.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.591+0000 7fda567fc640 1 -- 172.21.15.150:0/2841479779 learned_addr learned my addr 172.21.15.150:0/2841479779 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:37.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.591+0000 7fda56ffd640 1 --2- 172.21.15.150:0/2841479779 >> v2:172.21.15.157:3300/0 conn(0x7fda5815f460 0x7fda58099e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:37.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.591+0000 7fda567fc640 1 -- 172.21.15.150:0/2841479779 >> v2:172.21.15.157:3300/0 conn(0x7fda5815f460 msgr2=0x7fda58099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:37.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.591+0000 7fda567fc640 1 --2- 172.21.15.150:0/2841479779 >> v2:172.21.15.157:3300/0 conn(0x7fda5815f460 0x7fda58099e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.591+0000 7fda567fc640 1 -- 172.21.15.150:0/2841479779 >> v1:172.21.15.157:6789/0 conn(0x7fda580b0a20 legacy=0x7fda580b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:37.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.591+0000 7fda567fc640 1 -- 172.21.15.150:0/2841479779 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda580d0120 con 0x7fda580b2dc0 2024-03-17T23:47:37.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.592+0000 7fda567fc640 1 --2- 172.21.15.150:0/2841479779 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda580b2dc0 0x7fda5809a360 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fda580733d0 tx=0x7fda4c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3393439f442ce2cf server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:37.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.592+0000 7fda55ffb640 1 -- 172.21.15.150:0/2841479779 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4c035020 con 0x7fda580b2dc0 2024-03-17T23:47:37.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.592+0000 7fda55ffb640 1 -- 172.21.15.150:0/2841479779 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda4c031870 con 0x7fda580b2dc0 2024-03-17T23:47:37.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.592+0000 7fda55ffb640 1 -- 172.21.15.150:0/2841479779 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4c031c40 con 0x7fda580b2dc0 2024-03-17T23:47:37.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.593+0000 7fda5da0b640 1 -- 172.21.15.150:0/2841479779 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda580b2dc0 msgr2=0x7fda5809a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:37.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.593+0000 7fda5da0b640 1 --2- 172.21.15.150:0/2841479779 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda580b2dc0 0x7fda5809a360 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fda580733d0 tx=0x7fda4c02b910 comp rx=0 tx=0).stop 2024-03-17T23:47:37.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.593+0000 7fda5da0b640 1 -- 172.21.15.150:0/2841479779 shutdown_connections 2024-03-17T23:47:37.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.593+0000 7fda5da0b640 1 --2- 172.21.15.150:0/2841479779 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda580b2dc0 0x7fda5809a360 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.593+0000 7fda5da0b640 1 --2- 172.21.15.150:0/2841479779 >> v2:172.21.15.157:3300/0 conn(0x7fda5815f460 0x7fda58099e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.593+0000 7fda5da0b640 1 -- 172.21.15.150:0/2841479779 >> 172.21.15.150:0/2841479779 conn(0x7fda580b3630 msgr2=0x7fda580b3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:37.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.593+0000 7fda5da0b640 1 -- 172.21.15.150:0/2841479779 shutdown_connections 2024-03-17T23:47:37.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.593+0000 7fda5da0b640 1 -- 172.21.15.150:0/2841479779 wait complete. 2024-03-17T23:47:37.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.594+0000 7fda5da0b640 1 Processor -- start 2024-03-17T23:47:37.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.594+0000 7fda5da0b640 1 -- start start 2024-03-17T23:47:37.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.595+0000 7fda5da0b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fda580b0a20 0x7fda5815bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:37.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.595+0000 7fda5da0b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda5815f460 0x7fda580b5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:37.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.595+0000 7fda5da0b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda580b5750 0x7fda580ca800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:37.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.595+0000 7fda5da0b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fda5815f840 con 0x7fda580b5750 2024-03-17T23:47:37.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.595+0000 7fda5da0b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fda580d06e0 con 0x7fda580b0a20 2024-03-17T23:47:37.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.595+0000 7fda5da0b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fda580c63c0 con 0x7fda5815f460 2024-03-17T23:47:37.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.595+0000 7fda577fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda580b5750 0x7fda580ca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:37.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.595+0000 7fda577fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda580b5750 0x7fda580ca800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46124/0 (socket says 172.21.15.150:46124) 2024-03-17T23:47:37.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.595+0000 7fda577fe640 1 -- 172.21.15.150:0/1883609958 learned_addr learned my addr 172.21.15.150:0/1883609958 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:37.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.595+0000 7fda567fc640 1 --2- 172.21.15.150:0/1883609958 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda5815f460 0x7fda580b5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:37.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.595+0000 7fda56ffd640 1 --2- 172.21.15.150:0/1883609958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fda580b0a20 0x7fda5815bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:37.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.595+0000 7fda577fe640 1 -- 172.21.15.150:0/1883609958 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda5815f460 msgr2=0x7fda580b5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:37.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.595+0000 7fda577fe640 1 --2- 172.21.15.150:0/1883609958 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda5815f460 0x7fda580b5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.595+0000 7fda577fe640 1 -- 172.21.15.150:0/1883609958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fda580b0a20 msgr2=0x7fda5815bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:37.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.595+0000 7fda577fe640 1 --2- 172.21.15.150:0/1883609958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fda580b0a20 0x7fda5815bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.595+0000 7fda577fe640 1 -- 172.21.15.150:0/1883609958 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda580cb560 con 0x7fda580b5750 2024-03-17T23:47:37.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.595+0000 7fda577fe640 1 --2- 172.21.15.150:0/1883609958 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda580b5750 0x7fda580ca800 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fda5000c730 tx=0x7fda5000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:37.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.596+0000 7fda3ffff640 1 -- 172.21.15.150:0/1883609958 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda50003b90 con 0x7fda580b5750 2024-03-17T23:47:37.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.596+0000 7fda3ffff640 1 -- 172.21.15.150:0/1883609958 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda50003d50 con 0x7fda580b5750 2024-03-17T23:47:37.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.596+0000 7fda5da0b640 1 -- 172.21.15.150:0/1883609958 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda580c6610 con 0x7fda580b5750 2024-03-17T23:47:37.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.596+0000 7fda3ffff640 1 -- 172.21.15.150:0/1883609958 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda5000aa90 con 0x7fda580b5750 2024-03-17T23:47:37.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.596+0000 7fda5da0b640 1 -- 172.21.15.150:0/1883609958 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda5815c300 con 0x7fda580b5750 2024-03-17T23:47:37.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.597+0000 7fda5da0b640 1 -- 172.21.15.150:0/1883609958 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda580681a0 con 0x7fda580b5750 2024-03-17T23:47:37.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.597+0000 7fda3ffff640 1 -- 172.21.15.150:0/1883609958 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fda5000ac30 con 0x7fda580b5750 2024-03-17T23:47:37.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.597+0000 7fda3ffff640 1 --2- 172.21.15.150:0/1883609958 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fda4406bbd0 0x7fda4406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:37.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.597+0000 7fda3ffff640 1 -- 172.21.15.150:0/1883609958 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fda5008ab80 con 0x7fda580b5750 2024-03-17T23:47:37.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.597+0000 7fda56ffd640 1 --2- 172.21.15.150:0/1883609958 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fda4406bbd0 0x7fda4406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:37.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.598+0000 7fda56ffd640 1 --2- 172.21.15.150:0/1883609958 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fda4406bbd0 0x7fda4406e090 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fda48009aa0 tx=0x7fda48009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:37.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.602+0000 7fda3ffff640 1 -- 172.21.15.150:0/1883609958 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fda50058cb0 con 0x7fda580b5750 2024-03-17T23:47:37.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.778+0000 7fda5da0b640 1 -- 172.21.15.150:0/1883609958 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fda580b21a0 con 0x7fda580b5750 2024-03-17T23:47:37.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.779+0000 7fda3ffff640 1 -- 172.21.15.150:0/1883609958 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18881 (secure 0 0 0) 0x7fda5005cd50 con 0x7fda580b5750 2024-03-17T23:47:37.780 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:37.780 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":28,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:37.031864+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.e","mappings":[{"from":7,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:37.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.781+0000 7fda5da0b640 1 -- 172.21.15.150:0/1883609958 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fda4406bbd0 msgr2=0x7fda4406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:37.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.781+0000 7fda5da0b640 1 --2- 172.21.15.150:0/1883609958 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fda4406bbd0 0x7fda4406e090 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fda48009aa0 tx=0x7fda48009190 comp rx=0 tx=0).stop 2024-03-17T23:47:37.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.781+0000 7fda5da0b640 1 -- 172.21.15.150:0/1883609958 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda580b5750 msgr2=0x7fda580ca800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:37.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.781+0000 7fda5da0b640 1 --2- 172.21.15.150:0/1883609958 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda580b5750 0x7fda580ca800 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fda5000c730 tx=0x7fda5000cbf0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.781+0000 7fda5da0b640 1 -- 172.21.15.150:0/1883609958 shutdown_connections 2024-03-17T23:47:37.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.781+0000 7fda5da0b640 1 --2- 172.21.15.150:0/1883609958 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fda580b5750 0x7fda580ca800 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.781+0000 7fda5da0b640 1 --2- 172.21.15.150:0/1883609958 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fda5815f460 0x7fda580b5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.781+0000 7fda5da0b640 1 --2- 172.21.15.150:0/1883609958 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fda4406bbd0 0x7fda4406e090 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.781+0000 7fda5da0b640 1 --2- 172.21.15.150:0/1883609958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fda580b0a20 0x7fda5815bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.781+0000 7fda5da0b640 1 -- 172.21.15.150:0/1883609958 >> 172.21.15.150:0/1883609958 conn(0x7fda580b3630 msgr2=0x7fda580cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:37.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.781+0000 7fda5da0b640 1 -- 172.21.15.150:0/1883609958 shutdown_connections 2024-03-17T23:47:37.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.782+0000 7fda5da0b640 1 -- 172.21.15.150:0/1883609958 wait complete. 2024-03-17T23:47:37.795 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:46:53.610678+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:46:54.227335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-17T23:46:57.949235+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-17T23:47:37.796 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:47:37.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.900+0000 7fac21953640 1 Processor -- start 2024-03-17T23:47:37.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.900+0000 7fac21953640 1 -- start start 2024-03-17T23:47:37.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.900+0000 7fac21953640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fac1c161520 0x7fac1c0b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:37.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.901+0000 7fac21953640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fac1c0d0ce0 0x7fac1c0d1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:37.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.901+0000 7fac21953640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac1c161d30 con 0x7fac1c0b2bf0 2024-03-17T23:47:37.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.901+0000 7fac21953640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fac1c0b9bf0 con 0x7fac1c161520 2024-03-17T23:47:37.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.901+0000 7fac21953640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fac1c162940 con 0x7fac1c0d0ce0 2024-03-17T23:47:37.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.901+0000 7fac1affd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fac1c161520 0x7fac1c0b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:37.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.901+0000 7fac1affd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fac1c161520 0x7fac1c0b96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46128/0 (socket says 172.21.15.150:46128) 2024-03-17T23:47:37.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.901+0000 7fac1affd640 1 -- 172.21.15.150:0/1490656472 learned_addr learned my addr 172.21.15.150:0/1490656472 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:37.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.901+0000 7fac1b7fe640 1 --2- 172.21.15.150:0/1490656472 >> v2:172.21.15.157:3300/0 conn(0x7fac1c0d0ce0 0x7fac1c0d1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:37.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.901+0000 7fac1affd640 1 -- 172.21.15.150:0/1490656472 >> v2:172.21.15.157:3300/0 conn(0x7fac1c0d0ce0 msgr2=0x7fac1c0d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:37.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.901+0000 7fac1affd640 1 --2- 172.21.15.150:0/1490656472 >> v2:172.21.15.157:3300/0 conn(0x7fac1c0d0ce0 0x7fac1c0d1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.901+0000 7fac1affd640 1 -- 172.21.15.150:0/1490656472 >> v1:172.21.15.157:6789/0 conn(0x7fac1c0b2bf0 legacy=0x7fac1c0d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:37.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.901+0000 7fac1affd640 1 -- 172.21.15.150:0/1490656472 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac1c162010 con 0x7fac1c161520 2024-03-17T23:47:37.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.902+0000 7fac1affd640 1 --2- 172.21.15.150:0/1490656472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac1c161520 0x7fac1c0b96b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fac1c0d16a0 tx=0x7fac0c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3567ee479f20eb server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:37.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.902+0000 7fac19ffb640 1 -- 172.21.15.150:0/1490656472 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0c02e070 con 0x7fac1c161520 2024-03-17T23:47:37.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.902+0000 7fac19ffb640 1 -- 172.21.15.150:0/1490656472 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac0c037070 con 0x7fac1c161520 2024-03-17T23:47:37.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.902+0000 7fac19ffb640 1 -- 172.21.15.150:0/1490656472 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0c031a00 con 0x7fac1c161520 2024-03-17T23:47:37.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.902+0000 7fac21953640 1 -- 172.21.15.150:0/1490656472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac1c161520 msgr2=0x7fac1c0b96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:37.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.902+0000 7fac21953640 1 --2- 172.21.15.150:0/1490656472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac1c161520 0x7fac1c0b96b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fac1c0d16a0 tx=0x7fac0c02b880 comp rx=0 tx=0).stop 2024-03-17T23:47:37.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.902+0000 7fac21953640 1 -- 172.21.15.150:0/1490656472 shutdown_connections 2024-03-17T23:47:37.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.902+0000 7fac21953640 1 --2- 172.21.15.150:0/1490656472 >> v2:172.21.15.157:3300/0 conn(0x7fac1c0d0ce0 0x7fac1c0d1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.902+0000 7fac21953640 1 --2- 172.21.15.150:0/1490656472 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac1c161520 0x7fac1c0b96b0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.902+0000 7fac21953640 1 -- 172.21.15.150:0/1490656472 >> 172.21.15.150:0/1490656472 conn(0x7fac1c06ca00 msgr2=0x7fac1c06ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:37.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.902+0000 7fac21953640 1 -- 172.21.15.150:0/1490656472 shutdown_connections 2024-03-17T23:47:37.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.902+0000 7fac21953640 1 -- 172.21.15.150:0/1490656472 wait complete. 2024-03-17T23:47:37.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.904+0000 7fac21953640 1 Processor -- start 2024-03-17T23:47:37.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.904+0000 7fac21953640 1 -- start start 2024-03-17T23:47:37.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.904+0000 7fac21953640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac1c0b2bf0 0x7fac1c1521a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:37.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.904+0000 7fac21953640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac1c0d0ce0 0x7fac1c0cc130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:37.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.905+0000 7fac21953640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac1c161520 0x7fac1c0cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:37.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.905+0000 7fac21953640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fac1c161900 con 0x7fac1c0b2bf0 2024-03-17T23:47:37.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.905+0000 7fac21953640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fac1c1625d0 con 0x7fac1c161520 2024-03-17T23:47:37.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.905+0000 7fac21953640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fac1c0d24e0 con 0x7fac1c0d0ce0 2024-03-17T23:47:37.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.905+0000 7fac1affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac1c0b2bf0 0x7fac1c1521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:37.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.905+0000 7fac1affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac1c0b2bf0 0x7fac1c1521a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46142/0 (socket says 172.21.15.150:46142) 2024-03-17T23:47:37.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.905+0000 7fac1affd640 1 -- 172.21.15.150:0/971685052 learned_addr learned my addr 172.21.15.150:0/971685052 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:37.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.905+0000 7fac1a7fc640 1 --2- 172.21.15.150:0/971685052 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac1c0d0ce0 0x7fac1c0cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:37.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.905+0000 7fac1b7fe640 1 --2- 172.21.15.150:0/971685052 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac1c161520 0x7fac1c0cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:37.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.905+0000 7fac1affd640 1 -- 172.21.15.150:0/971685052 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac1c0d0ce0 msgr2=0x7fac1c0cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:37.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.905+0000 7fac1affd640 1 --2- 172.21.15.150:0/971685052 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac1c0d0ce0 0x7fac1c0cc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.905+0000 7fac1affd640 1 -- 172.21.15.150:0/971685052 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac1c161520 msgr2=0x7fac1c0cd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:37.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.905+0000 7fac1affd640 1 --2- 172.21.15.150:0/971685052 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac1c161520 0x7fac1c0cd6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:37.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.905+0000 7fac1affd640 1 -- 172.21.15.150:0/971685052 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac1c0b4ea0 con 0x7fac1c0b2bf0 2024-03-17T23:47:37.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.905+0000 7fac1affd640 1 --2- 172.21.15.150:0/971685052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac1c0b2bf0 0x7fac1c1521a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fac0c000c00 tx=0x7fac0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:37.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.905+0000 7fac03fff640 1 -- 172.21.15.150:0/971685052 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0c032670 con 0x7fac1c0b2bf0 2024-03-17T23:47:37.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.905+0000 7fac03fff640 1 -- 172.21.15.150:0/971685052 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac0c032850 con 0x7fac1c0b2bf0 2024-03-17T23:47:37.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.905+0000 7fac21953640 1 -- 172.21.15.150:0/971685052 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac1c160980 con 0x7fac1c0b2bf0 2024-03-17T23:47:37.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.905+0000 7fac03fff640 1 -- 172.21.15.150:0/971685052 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0c032c40 con 0x7fac1c0b2bf0 2024-03-17T23:47:37.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.905+0000 7fac21953640 1 -- 172.21.15.150:0/971685052 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac1c0b4bc0 con 0x7fac1c0b2bf0 2024-03-17T23:47:37.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.906+0000 7fac21953640 1 -- 172.21.15.150:0/971685052 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac1c0681a0 con 0x7fac1c0b2bf0 2024-03-17T23:47:37.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.907+0000 7fac03fff640 1 -- 172.21.15.150:0/971685052 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fac0c0453f0 con 0x7fac1c0b2bf0 2024-03-17T23:47:37.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.907+0000 7fac03fff640 1 --2- 172.21.15.150:0/971685052 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fac0806bbd0 0x7fac0806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:37.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.907+0000 7fac03fff640 1 -- 172.21.15.150:0/971685052 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fac0c0aca50 con 0x7fac1c0b2bf0 2024-03-17T23:47:37.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.907+0000 7fac1a7fc640 1 --2- 172.21.15.150:0/971685052 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fac0806bbd0 0x7fac0806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:37.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.908+0000 7fac1a7fc640 1 --2- 172.21.15.150:0/971685052 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fac0806bbd0 0x7fac0806e090 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fac1c1526e0 tx=0x7fac10009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:37.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:37.911+0000 7fac03fff640 1 -- 172.21.15.150:0/971685052 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fac1c0b4ea0 con 0x7fac1c0b2bf0 2024-03-17T23:47:38.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.085+0000 7fac21953640 1 -- 172.21.15.150:0/971685052 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fac1c0c75b0 con 0x7fac1c0b2bf0 2024-03-17T23:47:38.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.086+0000 7fac03fff640 1 -- 172.21.15.150:0/971685052 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7fac0c040070 con 0x7fac1c0b2bf0 2024-03-17T23:47:38.086 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:47:38.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.088+0000 7fac21953640 1 -- 172.21.15.150:0/971685052 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fac0806bbd0 msgr2=0x7fac0806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:38.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.088+0000 7fac21953640 1 --2- 172.21.15.150:0/971685052 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fac0806bbd0 0x7fac0806e090 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fac1c1526e0 tx=0x7fac10009040 comp rx=0 tx=0).stop 2024-03-17T23:47:38.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.088+0000 7fac21953640 1 -- 172.21.15.150:0/971685052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac1c0b2bf0 msgr2=0x7fac1c1521a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:38.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.088+0000 7fac21953640 1 --2- 172.21.15.150:0/971685052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac1c0b2bf0 0x7fac1c1521a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fac0c000c00 tx=0x7fac0c0046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.088+0000 7fac21953640 1 -- 172.21.15.150:0/971685052 shutdown_connections 2024-03-17T23:47:38.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.088+0000 7fac21953640 1 --2- 172.21.15.150:0/971685052 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fac0806bbd0 0x7fac0806e090 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.088+0000 7fac21953640 1 --2- 172.21.15.150:0/971685052 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac1c161520 0x7fac1c0cd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.088+0000 7fac21953640 1 --2- 172.21.15.150:0/971685052 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fac1c0d0ce0 0x7fac1c0cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.088+0000 7fac21953640 1 --2- 172.21.15.150:0/971685052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac1c0b2bf0 0x7fac1c1521a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.088+0000 7fac21953640 1 -- 172.21.15.150:0/971685052 >> 172.21.15.150:0/971685052 conn(0x7fac1c06ca00 msgr2=0x7fac1c0b93b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:38.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.088+0000 7fac21953640 1 -- 172.21.15.150:0/971685052 shutdown_connections 2024-03-17T23:47:38.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.088+0000 7fac21953640 1 -- 172.21.15.150:0/971685052 wait complete. 2024-03-17T23:47:38.099 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:47:38.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.201+0000 7fa335aa1640 1 Processor -- start 2024-03-17T23:47:38.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.202+0000 7fa335aa1640 1 -- start start 2024-03-17T23:47:38.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.202+0000 7fa335aa1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa330161520 0x7fa3300b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:38.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.202+0000 7fa335aa1640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa3300b2bf0 0x7fa3300d05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:38.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.202+0000 7fa335aa1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3300d0f20 con 0x7fa3300d0b10 2024-03-17T23:47:38.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.202+0000 7fa335aa1640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fa3300b9bf0 con 0x7fa330161520 2024-03-17T23:47:38.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.202+0000 7fa335aa1640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fa330162940 con 0x7fa3300b2bf0 2024-03-17T23:47:38.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.202+0000 7fa32f7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa330161520 0x7fa3300b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:38.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.202+0000 7fa32ffff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa3300d0b10 0x7fa3300b57a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59194/0 (socket says 172.21.15.150:59194) 2024-03-17T23:47:38.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.203+0000 7fa32ffff640 1 -- 172.21.15.150:0/471814806 learned_addr learned my addr 172.21.15.150:0/471814806 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:38.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.203+0000 7fa326ffd640 1 --2- 172.21.15.150:0/471814806 >> v2:172.21.15.157:3300/0 conn(0x7fa3300b2bf0 0x7fa3300d05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:38.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.203+0000 7fa32f7fe640 1 -- 172.21.15.150:0/471814806 >> v2:172.21.15.157:3300/0 conn(0x7fa3300b2bf0 msgr2=0x7fa3300d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:38.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.203+0000 7fa32f7fe640 1 --2- 172.21.15.150:0/471814806 >> v2:172.21.15.157:3300/0 conn(0x7fa3300b2bf0 0x7fa3300d05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.203+0000 7fa32f7fe640 1 -- 172.21.15.150:0/471814806 >> v1:172.21.15.150:6789/0 conn(0x7fa3300d0b10 legacy=0x7fa3300b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:38.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.203+0000 7fa32f7fe640 1 -- 172.21.15.150:0/471814806 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa330161d30 con 0x7fa330161520 2024-03-17T23:47:38.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.203+0000 7fa32effd640 1 -- 172.21.15.150:0/471814806 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 192450640 0 0) 0x7fa3300d0f20 con 0x7fa3300d0b10 2024-03-17T23:47:38.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.203+0000 7fa326ffd640 1 --2- 172.21.15.150:0/471814806 >> v2:172.21.15.157:3300/0 conn(0x7fa3300b2bf0 0x7fa3300d05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:38.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.204+0000 7fa32f7fe640 1 --2- 172.21.15.150:0/471814806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa330161520 0x7fa3300b96b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa3300ce7a0 tx=0x7fa32002b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92e52bd13bad4148 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:38.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.204+0000 7fa32effd640 1 -- 172.21.15.150:0/471814806 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa320038070 con 0x7fa330161520 2024-03-17T23:47:38.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.204+0000 7fa32effd640 1 -- 172.21.15.150:0/471814806 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa320031710 con 0x7fa330161520 2024-03-17T23:47:38.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.204+0000 7fa32effd640 1 -- 172.21.15.150:0/471814806 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa320031ae0 con 0x7fa330161520 2024-03-17T23:47:38.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.204+0000 7fa335aa1640 1 -- 172.21.15.150:0/471814806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa330161520 msgr2=0x7fa3300b96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:38.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.204+0000 7fa335aa1640 1 --2- 172.21.15.150:0/471814806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa330161520 0x7fa3300b96b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa3300ce7a0 tx=0x7fa32002b7b0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.204+0000 7fa335aa1640 1 -- 172.21.15.150:0/471814806 shutdown_connections 2024-03-17T23:47:38.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.204+0000 7fa335aa1640 1 --2- 172.21.15.150:0/471814806 >> v2:172.21.15.157:3300/0 conn(0x7fa3300b2bf0 0x7fa3300d05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.204+0000 7fa335aa1640 1 --2- 172.21.15.150:0/471814806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa330161520 0x7fa3300b96b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.204+0000 7fa335aa1640 1 -- 172.21.15.150:0/471814806 >> 172.21.15.150:0/471814806 conn(0x7fa33006ca00 msgr2=0x7fa33006ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:38.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.204+0000 7fa335aa1640 1 -- 172.21.15.150:0/471814806 shutdown_connections 2024-03-17T23:47:38.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.204+0000 7fa335aa1640 1 -- 172.21.15.150:0/471814806 wait complete. 2024-03-17T23:47:38.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.205+0000 7fa335aa1640 1 Processor -- start 2024-03-17T23:47:38.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.206+0000 7fa335aa1640 1 -- start start 2024-03-17T23:47:38.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.206+0000 7fa335aa1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3300b2bf0 0x7fa3301f4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:38.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.206+0000 7fa335aa1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3300d0b10 0x7fa3301523c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:38.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.206+0000 7fa335aa1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa330161520 0x7fa33015a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:38.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.206+0000 7fa335aa1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa3301625d0 con 0x7fa330161520 2024-03-17T23:47:38.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.206+0000 7fa335aa1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa3301622f0 con 0x7fa3300b2bf0 2024-03-17T23:47:38.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.206+0000 7fa335aa1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa330161900 con 0x7fa3300d0b10 2024-03-17T23:47:38.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.207+0000 7fa32ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa330161520 0x7fa33015a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:38.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.207+0000 7fa32ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa330161520 0x7fa33015a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46158/0 (socket says 172.21.15.150:46158) 2024-03-17T23:47:38.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.207+0000 7fa32ffff640 1 -- 172.21.15.150:0/1987587582 learned_addr learned my addr 172.21.15.150:0/1987587582 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:38.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.207+0000 7fa32f7fe640 1 --2- 172.21.15.150:0/1987587582 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3300b2bf0 0x7fa3301f4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:38.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.207+0000 7fa326ffd640 1 --2- 172.21.15.150:0/1987587582 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3300d0b10 0x7fa3301523c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:38.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.207+0000 7fa32ffff640 1 -- 172.21.15.150:0/1987587582 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3300d0b10 msgr2=0x7fa3301523c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:38.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.207+0000 7fa32ffff640 1 --2- 172.21.15.150:0/1987587582 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3300d0b10 0x7fa3301523c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.207+0000 7fa32ffff640 1 -- 172.21.15.150:0/1987587582 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3300b2bf0 msgr2=0x7fa3301f4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:38.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.207+0000 7fa32ffff640 1 --2- 172.21.15.150:0/1987587582 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3300b2bf0 0x7fa3301f4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.207+0000 7fa32ffff640 1 -- 172.21.15.150:0/1987587582 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3301f4ab0 con 0x7fa330161520 2024-03-17T23:47:38.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.207+0000 7fa32ffff640 1 --2- 172.21.15.150:0/1987587582 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa330161520 0x7fa33015a840 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa328002230 tx=0x7fa32800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:38.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.207+0000 7fa32cff9640 1 -- 172.21.15.150:0/1987587582 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa328017070 con 0x7fa330161520 2024-03-17T23:47:38.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.207+0000 7fa32cff9640 1 -- 172.21.15.150:0/1987587582 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa32800ac30 con 0x7fa330161520 2024-03-17T23:47:38.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.207+0000 7fa335aa1640 1 -- 172.21.15.150:0/1987587582 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa33015d630 con 0x7fa330161520 2024-03-17T23:47:38.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.207+0000 7fa32cff9640 1 -- 172.21.15.150:0/1987587582 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa328012400 con 0x7fa330161520 2024-03-17T23:47:38.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.207+0000 7fa335aa1640 1 -- 172.21.15.150:0/1987587582 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa33015ded0 con 0x7fa330161520 2024-03-17T23:47:38.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.208+0000 7fa335aa1640 1 -- 172.21.15.150:0/1987587582 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3300681a0 con 0x7fa330161520 2024-03-17T23:47:38.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.208+0000 7fa32cff9640 1 -- 172.21.15.150:0/1987587582 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa32801a070 con 0x7fa330161520 2024-03-17T23:47:38.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.209+0000 7fa32cff9640 1 --2- 172.21.15.150:0/1987587582 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa31006bbd0 0x7fa31006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:38.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.209+0000 7fa32cff9640 1 -- 172.21.15.150:0/1987587582 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fa32808ab50 con 0x7fa330161520 2024-03-17T23:47:38.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.209+0000 7fa32f7fe640 1 --2- 172.21.15.150:0/1987587582 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa31006bbd0 0x7fa31006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:38.211 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.209+0000 7fa32f7fe640 1 --2- 172.21.15.150:0/1987587582 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa31006bbd0 0x7fa31006e090 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa320035000 tx=0x7fa320008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:38.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.213+0000 7fa32cff9640 1 -- 172.21.15.150:0/1987587582 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa3301f4ab0 con 0x7fa330161520 2024-03-17T23:47:38.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.383+0000 7fa335aa1640 1 -- 172.21.15.150:0/1987587582 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa3300d19c0 con 0x7fa330161520 2024-03-17T23:47:38.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.383+0000 7fa32cff9640 1 -- 172.21.15.150:0/1987587582 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7fa328058be0 con 0x7fa330161520 2024-03-17T23:47:38.384 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:38.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.385+0000 7fa335aa1640 1 -- 172.21.15.150:0/1987587582 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa31006bbd0 msgr2=0x7fa31006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:38.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.385+0000 7fa335aa1640 1 --2- 172.21.15.150:0/1987587582 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa31006bbd0 0x7fa31006e090 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa320035000 tx=0x7fa320008030 comp rx=0 tx=0).stop 2024-03-17T23:47:38.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.385+0000 7fa335aa1640 1 -- 172.21.15.150:0/1987587582 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa330161520 msgr2=0x7fa33015a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:38.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.385+0000 7fa335aa1640 1 --2- 172.21.15.150:0/1987587582 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa330161520 0x7fa33015a840 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa328002230 tx=0x7fa32800cf50 comp rx=0 tx=0).stop 2024-03-17T23:47:38.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.385+0000 7fa335aa1640 1 -- 172.21.15.150:0/1987587582 shutdown_connections 2024-03-17T23:47:38.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.385+0000 7fa335aa1640 1 --2- 172.21.15.150:0/1987587582 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa330161520 0x7fa33015a840 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.385+0000 7fa335aa1640 1 --2- 172.21.15.150:0/1987587582 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3300d0b10 0x7fa3301523c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.385+0000 7fa335aa1640 1 --2- 172.21.15.150:0/1987587582 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa31006bbd0 0x7fa31006e090 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.385+0000 7fa335aa1640 1 --2- 172.21.15.150:0/1987587582 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3300b2bf0 0x7fa3301f4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.385+0000 7fa335aa1640 1 -- 172.21.15.150:0/1987587582 >> 172.21.15.150:0/1987587582 conn(0x7fa33006ca00 msgr2=0x7fa3300b9300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:38.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.385+0000 7fa335aa1640 1 -- 172.21.15.150:0/1987587582 shutdown_connections 2024-03-17T23:47:38.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.385+0000 7fa335aa1640 1 -- 172.21.15.150:0/1987587582 wait complete. 2024-03-17T23:47:38.397 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:47:38.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.500+0000 7f3470377640 1 Processor -- start 2024-03-17T23:47:38.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.500+0000 7f3470377640 1 -- start start 2024-03-17T23:47:38.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.501+0000 7f3470377640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34680d0ce0 0x7f34680b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:38.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.501+0000 7f3470377640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f34680b2bf0 0x7f34680d1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:38.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.501+0000 7f3470377640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3468161d30 con 0x7f3468161520 2024-03-17T23:47:38.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.501+0000 7f3470377640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f34680b9bf0 con 0x7f34680b2bf0 2024-03-17T23:47:38.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.501+0000 7f3470377640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3468162940 con 0x7f34680d0ce0 2024-03-17T23:47:38.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.501+0000 7f346e914640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f34680b2bf0 0x7f34680d1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:38.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.501+0000 7f346e113640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f3468161520 0x7f34680d05d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59198/0 (socket says 172.21.15.150:59198) 2024-03-17T23:47:38.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.501+0000 7f346e113640 1 -- 172.21.15.150:0/1791462682 learned_addr learned my addr 172.21.15.150:0/1791462682 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:38.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.502+0000 7f346d912640 1 --2- 172.21.15.150:0/1791462682 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34680d0ce0 0x7f34680b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:38.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.502+0000 7f346e914640 1 -- 172.21.15.150:0/1791462682 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34680d0ce0 msgr2=0x7f34680b96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:38.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.502+0000 7f346e914640 1 --2- 172.21.15.150:0/1791462682 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34680d0ce0 0x7f34680b96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.502+0000 7f346e914640 1 -- 172.21.15.150:0/1791462682 >> v1:172.21.15.150:6789/0 conn(0x7f3468161520 legacy=0x7f34680d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:38.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.502+0000 7f346e914640 1 -- 172.21.15.150:0/1791462682 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3468162010 con 0x7f34680b2bf0 2024-03-17T23:47:38.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.502+0000 7f346d111640 1 -- 172.21.15.150:0/1791462682 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4000533415 0 0) 0x7f3468161d30 con 0x7f3468161520 2024-03-17T23:47:38.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.502+0000 7f346e914640 1 --2- 172.21.15.150:0/1791462682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34680b2bf0 0x7f34680d1fa0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f34600097a0 tx=0x7f346002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=383eb1bd57141645 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:38.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.503+0000 7f346d111640 1 -- 172.21.15.150:0/1791462682 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3460038070 con 0x7f34680b2bf0 2024-03-17T23:47:38.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.503+0000 7f346d111640 1 -- 172.21.15.150:0/1791462682 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3460031700 con 0x7f34680b2bf0 2024-03-17T23:47:38.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.503+0000 7f346d111640 1 -- 172.21.15.150:0/1791462682 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3460031ad0 con 0x7f34680b2bf0 2024-03-17T23:47:38.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.503+0000 7f3470377640 1 -- 172.21.15.150:0/1791462682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34680b2bf0 msgr2=0x7f34680d1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:38.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.503+0000 7f3470377640 1 --2- 172.21.15.150:0/1791462682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34680b2bf0 0x7f34680d1fa0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f34600097a0 tx=0x7f346002b7a0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.503+0000 7f3470377640 1 -- 172.21.15.150:0/1791462682 shutdown_connections 2024-03-17T23:47:38.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.503+0000 7f3470377640 1 --2- 172.21.15.150:0/1791462682 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34680d0ce0 0x7f34680b96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.503+0000 7f3470377640 1 --2- 172.21.15.150:0/1791462682 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34680b2bf0 0x7f34680d1fa0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.503+0000 7f3470377640 1 -- 172.21.15.150:0/1791462682 >> 172.21.15.150:0/1791462682 conn(0x7f346806ca00 msgr2=0x7f346806ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:38.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.503+0000 7f3470377640 1 -- 172.21.15.150:0/1791462682 shutdown_connections 2024-03-17T23:47:38.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.503+0000 7f3470377640 1 -- 172.21.15.150:0/1791462682 wait complete. 2024-03-17T23:47:38.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.504+0000 7f3470377640 1 Processor -- start 2024-03-17T23:47:38.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.504+0000 7f3470377640 1 -- start start 2024-03-17T23:47:38.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.505+0000 7f3470377640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34680b2bf0 0x7f3468152050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:38.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.505+0000 7f3470377640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34680d0ce0 0x7f34680ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:38.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.505+0000 7f3470377640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3468161520 0x7f34680c93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:38.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.505+0000 7f3470377640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3468161900 con 0x7f34680b2bf0 2024-03-17T23:47:38.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.505+0000 7f3470377640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f34681625d0 con 0x7f3468161520 2024-03-17T23:47:38.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.505+0000 7f3470377640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f34680d24e0 con 0x7f34680d0ce0 2024-03-17T23:47:38.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.505+0000 7f346e113640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34680b2bf0 0x7f3468152050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:38.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.505+0000 7f346e113640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34680b2bf0 0x7f3468152050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46178/0 (socket says 172.21.15.150:46178) 2024-03-17T23:47:38.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.505+0000 7f346e113640 1 -- 172.21.15.150:0/2695009824 learned_addr learned my addr 172.21.15.150:0/2695009824 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:38.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.505+0000 7f346d912640 1 --2- 172.21.15.150:0/2695009824 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34680d0ce0 0x7f34680ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:38.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.505+0000 7f346e914640 1 --2- 172.21.15.150:0/2695009824 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3468161520 0x7f34680c93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:38.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.505+0000 7f346e113640 1 -- 172.21.15.150:0/2695009824 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34680d0ce0 msgr2=0x7f34680ccbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:38.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.505+0000 7f346e113640 1 --2- 172.21.15.150:0/2695009824 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34680d0ce0 0x7f34680ccbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.505+0000 7f346e113640 1 -- 172.21.15.150:0/2695009824 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3468161520 msgr2=0x7f34680c93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:38.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.505+0000 7f346e113640 1 --2- 172.21.15.150:0/2695009824 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3468161520 0x7f34680c93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.506+0000 7f346e113640 1 -- 172.21.15.150:0/2695009824 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34680b4720 con 0x7f34680b2bf0 2024-03-17T23:47:38.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.506+0000 7f346e113640 1 --2- 172.21.15.150:0/2695009824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34680b2bf0 0x7f3468152050 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f346400c7c0 tx=0x7f346400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:38.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.506+0000 7f345effd640 1 -- 172.21.15.150:0/2695009824 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3464018070 con 0x7f34680b2bf0 2024-03-17T23:47:38.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.506+0000 7f345effd640 1 -- 172.21.15.150:0/2695009824 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f346400a640 con 0x7f34680b2bf0 2024-03-17T23:47:38.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.506+0000 7f3470377640 1 -- 172.21.15.150:0/2695009824 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34680b4a00 con 0x7f34680b2bf0 2024-03-17T23:47:38.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.506+0000 7f345effd640 1 -- 172.21.15.150:0/2695009824 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346400aa30 con 0x7f34680b2bf0 2024-03-17T23:47:38.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.506+0000 7f3470377640 1 -- 172.21.15.150:0/2695009824 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34680cd0f0 con 0x7f34680b2bf0 2024-03-17T23:47:38.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.507+0000 7f3470377640 1 -- 172.21.15.150:0/2695009824 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34680681a0 con 0x7f34680b2bf0 2024-03-17T23:47:38.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.507+0000 7f345effd640 1 -- 172.21.15.150:0/2695009824 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f346401b070 con 0x7f34680b2bf0 2024-03-17T23:47:38.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.508+0000 7f345effd640 1 --2- 172.21.15.150:0/2695009824 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f344c06bbd0 0x7f344c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:38.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.508+0000 7f345effd640 1 -- 172.21.15.150:0/2695009824 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f346408a420 con 0x7f34680b2bf0 2024-03-17T23:47:38.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.508+0000 7f346d912640 1 --2- 172.21.15.150:0/2695009824 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f344c06bbd0 0x7f344c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:38.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.508+0000 7f346d912640 1 --2- 172.21.15.150:0/2695009824 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f344c06bbd0 0x7f344c06e090 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f34680c72b0 tx=0x7f3458009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:38.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.512+0000 7f345effd640 1 -- 172.21.15.150:0/2695009824 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f34640584c0 con 0x7f34680b2bf0 2024-03-17T23:47:38.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.687+0000 7f3470377640 1 -- 172.21.15.150:0/2695009824 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f34680c75b0 con 0x7f34680b2bf0 2024-03-17T23:47:38.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.688+0000 7f345effd640 1 -- 172.21.15.150:0/2695009824 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7f346405c560 con 0x7f34680b2bf0 2024-03-17T23:47:38.689 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:47:38.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.690+0000 7f3470377640 1 -- 172.21.15.150:0/2695009824 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f344c06bbd0 msgr2=0x7f344c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:38.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.690+0000 7f3470377640 1 --2- 172.21.15.150:0/2695009824 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f344c06bbd0 0x7f344c06e090 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f34680c72b0 tx=0x7f3458009190 comp rx=0 tx=0).stop 2024-03-17T23:47:38.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.690+0000 7f3470377640 1 -- 172.21.15.150:0/2695009824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34680b2bf0 msgr2=0x7f3468152050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:38.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.690+0000 7f3470377640 1 --2- 172.21.15.150:0/2695009824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34680b2bf0 0x7f3468152050 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f346400c7c0 tx=0x7f346400cc80 comp rx=0 tx=0).stop 2024-03-17T23:47:38.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.690+0000 7f3470377640 1 -- 172.21.15.150:0/2695009824 shutdown_connections 2024-03-17T23:47:38.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.690+0000 7f3470377640 1 --2- 172.21.15.150:0/2695009824 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f344c06bbd0 0x7f344c06e090 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.690+0000 7f3470377640 1 --2- 172.21.15.150:0/2695009824 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3468161520 0x7f34680c93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.690+0000 7f3470377640 1 --2- 172.21.15.150:0/2695009824 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f34680d0ce0 0x7f34680ccbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.690+0000 7f3470377640 1 --2- 172.21.15.150:0/2695009824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f34680b2bf0 0x7f3468152050 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.690+0000 7f3470377640 1 -- 172.21.15.150:0/2695009824 >> 172.21.15.150:0/2695009824 conn(0x7f346806ca00 msgr2=0x7f34680b9340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:38.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.690+0000 7f3470377640 1 -- 172.21.15.150:0/2695009824 shutdown_connections 2024-03-17T23:47:38.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.690+0000 7f3470377640 1 -- 172.21.15.150:0/2695009824 wait complete. 2024-03-17T23:47:38.702 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-17T23:47:38.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.805+0000 7f8b15ff3640 1 Processor -- start 2024-03-17T23:47:38.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.805+0000 7f8b15ff3640 1 -- start start 2024-03-17T23:47:38.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.806+0000 7f8b15ff3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b100d3910 0x7f8b100d0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:38.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.806+0000 7f8b15ff3640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8b100b9660 0x7f8b100d0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:38.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.806+0000 7f8b15ff3640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b100d2870 con 0x7f8b101615d0 2024-03-17T23:47:38.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.806+0000 7f8b15ff3640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f8b101619b0 con 0x7f8b100b9660 2024-03-17T23:47:38.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.806+0000 7f8b15ff3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8b100b7e80 con 0x7f8b100d3910 2024-03-17T23:47:38.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.806+0000 7f8b0f7fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f8b101615d0 0x7f8b100d2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:46800/0 (socket says 172.21.15.150:46800) 2024-03-17T23:47:38.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.806+0000 7f8b0ffff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8b100b9660 0x7f8b100d0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:38.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.806+0000 7f8b0f7fe640 1 -- 172.21.15.150:0/3146841999 learned_addr learned my addr 172.21.15.150:0/3146841999 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:38.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.806+0000 7f8b0effd640 1 --2- 172.21.15.150:0/3146841999 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b100d3910 0x7f8b100d0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:38.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.807+0000 7f8b0effd640 1 -- 172.21.15.150:0/3146841999 >> v2:172.21.15.157:3300/0 conn(0x7f8b100b9660 msgr2=0x7f8b100d0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:38.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.807+0000 7f8b0effd640 1 --2- 172.21.15.150:0/3146841999 >> v2:172.21.15.157:3300/0 conn(0x7f8b100b9660 0x7f8b100d0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.807+0000 7f8b0e7fc640 1 -- 172.21.15.150:0/3146841999 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1365040708 0 0) 0x7f8b100d2870 con 0x7f8b101615d0 2024-03-17T23:47:38.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.807+0000 7f8b0effd640 1 -- 172.21.15.150:0/3146841999 >> v1:172.21.15.157:6789/0 conn(0x7f8b101615d0 legacy=0x7f8b100d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:38.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.807+0000 7f8b0effd640 1 -- 172.21.15.150:0/3146841999 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b100d2b50 con 0x7f8b100d3910 2024-03-17T23:47:38.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.807+0000 7f8b0ffff640 1 --2- 172.21.15.150:0/3146841999 >> v2:172.21.15.157:3300/0 conn(0x7f8b100b9660 0x7f8b100d0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:38.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.807+0000 7f8b0effd640 1 --2- 172.21.15.150:0/3146841999 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b100d3910 0x7f8b100d0720 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8b040097a0 tx=0x7f8b0402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3719c7289e5e0706 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:38.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.807+0000 7f8b0e7fc640 1 -- 172.21.15.150:0/3146841999 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b04035020 con 0x7f8b100d3910 2024-03-17T23:47:38.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.807+0000 7f8b0e7fc640 1 -- 172.21.15.150:0/3146841999 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b04031830 con 0x7f8b100d3910 2024-03-17T23:47:38.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.808+0000 7f8b0e7fc640 1 -- 172.21.15.150:0/3146841999 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b04031c00 con 0x7f8b100d3910 2024-03-17T23:47:38.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.808+0000 7f8b15ff3640 1 -- 172.21.15.150:0/3146841999 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b100d3910 msgr2=0x7f8b100d0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:38.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.808+0000 7f8b15ff3640 1 --2- 172.21.15.150:0/3146841999 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b100d3910 0x7f8b100d0720 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8b040097a0 tx=0x7f8b0402b8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.808+0000 7f8b15ff3640 1 -- 172.21.15.150:0/3146841999 shutdown_connections 2024-03-17T23:47:38.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.808+0000 7f8b15ff3640 1 --2- 172.21.15.150:0/3146841999 >> v2:172.21.15.157:3300/0 conn(0x7f8b100b9660 0x7f8b100d0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.808+0000 7f8b15ff3640 1 --2- 172.21.15.150:0/3146841999 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b100d3910 0x7f8b100d0720 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.808+0000 7f8b15ff3640 1 -- 172.21.15.150:0/3146841999 >> 172.21.15.150:0/3146841999 conn(0x7f8b100730b0 msgr2=0x7f8b100b5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:38.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.808+0000 7f8b15ff3640 1 -- 172.21.15.150:0/3146841999 shutdown_connections 2024-03-17T23:47:38.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.808+0000 7f8b15ff3640 1 -- 172.21.15.150:0/3146841999 wait complete. 2024-03-17T23:47:38.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.809+0000 7f8b15ff3640 1 Processor -- start 2024-03-17T23:47:38.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.809+0000 7f8b15ff3640 1 -- start start 2024-03-17T23:47:38.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.810+0000 7f8b15ff3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b100b9660 0x7f8b1014c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:38.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.810+0000 7f8b15ff3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b100d3910 0x7f8b100b4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:38.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.810+0000 7f8b15ff3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b101615d0 0x7f8b1015e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:38.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.810+0000 7f8b15ff3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8b100d33f0 con 0x7f8b100b9660 2024-03-17T23:47:38.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.810+0000 7f8b15ff3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8b100d3110 con 0x7f8b101615d0 2024-03-17T23:47:38.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.810+0000 7f8b15ff3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8b100b6e70 con 0x7f8b100d3910 2024-03-17T23:47:38.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.810+0000 7f8b0f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b100b9660 0x7f8b1014c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:38.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.810+0000 7f8b0f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b100b9660 0x7f8b1014c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46190/0 (socket says 172.21.15.150:46190) 2024-03-17T23:47:38.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.810+0000 7f8b0f7fe640 1 -- 172.21.15.150:0/37850468 learned_addr learned my addr 172.21.15.150:0/37850468 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:38.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.810+0000 7f8b0effd640 1 --2- 172.21.15.150:0/37850468 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b100d3910 0x7f8b100b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:38.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.810+0000 7f8b0ffff640 1 --2- 172.21.15.150:0/37850468 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b101615d0 0x7f8b1015e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:38.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.811+0000 7f8b0f7fe640 1 -- 172.21.15.150:0/37850468 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b100d3910 msgr2=0x7f8b100b4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:38.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.811+0000 7f8b0f7fe640 1 --2- 172.21.15.150:0/37850468 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b100d3910 0x7f8b100b4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.811+0000 7f8b0f7fe640 1 -- 172.21.15.150:0/37850468 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b101615d0 msgr2=0x7f8b1015e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:38.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.811+0000 7f8b0f7fe640 1 --2- 172.21.15.150:0/37850468 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b101615d0 0x7f8b1015e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.811+0000 7f8b0f7fe640 1 -- 172.21.15.150:0/37850468 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b100cb060 con 0x7f8b100b9660 2024-03-17T23:47:38.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.811+0000 7f8b0f7fe640 1 --2- 172.21.15.150:0/37850468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b100b9660 0x7f8b1014c580 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8b0000c460 tx=0x7f8b0000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:38.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.811+0000 7f8af7fff640 1 -- 172.21.15.150:0/37850468 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b000074e0 con 0x7f8b100b9660 2024-03-17T23:47:38.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.811+0000 7f8b15ff3640 1 -- 172.21.15.150:0/37850468 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b100cb340 con 0x7f8b100b9660 2024-03-17T23:47:38.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.811+0000 7f8af7fff640 1 -- 172.21.15.150:0/37850468 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b000047e0 con 0x7f8b100b9660 2024-03-17T23:47:38.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.811+0000 7f8af7fff640 1 -- 172.21.15.150:0/37850468 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b00004bd0 con 0x7f8b100b9660 2024-03-17T23:47:38.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.811+0000 7f8b15ff3640 1 -- 172.21.15.150:0/37850468 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b1014cac0 con 0x7f8b100b9660 2024-03-17T23:47:38.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.812+0000 7f8b15ff3640 1 -- 172.21.15.150:0/37850468 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b100cadf0 con 0x7f8b100b9660 2024-03-17T23:47:38.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.812+0000 7f8af7fff640 1 -- 172.21.15.150:0/37850468 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8b00011070 con 0x7f8b100b9660 2024-03-17T23:47:38.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.813+0000 7f8af7fff640 1 --2- 172.21.15.150:0/37850468 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8afc06bc00 0x7f8afc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:38.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.813+0000 7f8af7fff640 1 -- 172.21.15.150:0/37850468 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f8b0008a920 con 0x7f8b100b9660 2024-03-17T23:47:38.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.813+0000 7f8b0effd640 1 --2- 172.21.15.150:0/37850468 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8afc06bc00 0x7f8afc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:38.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.813+0000 7f8b0effd640 1 --2- 172.21.15.150:0/37850468 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8afc06bc00 0x7f8afc06e0c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8b04036000 tx=0x7f8b0403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:38.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.817+0000 7f8af7fff640 1 -- 172.21.15.150:0/37850468 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8b00059020 con 0x7f8b100b9660 2024-03-17T23:47:38.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.988+0000 7f8b15ff3640 1 -- 172.21.15.150:0/37850468 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8b100d2460 con 0x7f8b100b9660 2024-03-17T23:47:38.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.988+0000 7f8af7fff640 1 -- 172.21.15.150:0/37850468 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7f8b00058920 con 0x7f8b100b9660 2024-03-17T23:47:38.989 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.990+0000 7f8b15ff3640 1 -- 172.21.15.150:0/37850468 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8afc06bc00 msgr2=0x7f8afc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.990+0000 7f8b15ff3640 1 --2- 172.21.15.150:0/37850468 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8afc06bc00 0x7f8afc06e0c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8b04036000 tx=0x7f8b0403d040 comp rx=0 tx=0).stop 2024-03-17T23:47:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.990+0000 7f8b15ff3640 1 -- 172.21.15.150:0/37850468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b100b9660 msgr2=0x7f8b1014c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.990+0000 7f8b15ff3640 1 --2- 172.21.15.150:0/37850468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b100b9660 0x7f8b1014c580 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8b0000c460 tx=0x7f8b0000c860 comp rx=0 tx=0).stop 2024-03-17T23:47:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.990+0000 7f8b15ff3640 1 -- 172.21.15.150:0/37850468 shutdown_connections 2024-03-17T23:47:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.990+0000 7f8b15ff3640 1 --2- 172.21.15.150:0/37850468 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8afc06bc00 0x7f8afc06e0c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.990+0000 7f8b15ff3640 1 --2- 172.21.15.150:0/37850468 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b101615d0 0x7f8b1015e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.990+0000 7f8b15ff3640 1 --2- 172.21.15.150:0/37850468 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8b100d3910 0x7f8b100b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.990+0000 7f8b15ff3640 1 --2- 172.21.15.150:0/37850468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b100b9660 0x7f8b1014c580 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.990+0000 7f8b15ff3640 1 -- 172.21.15.150:0/37850468 >> 172.21.15.150:0/37850468 conn(0x7f8b100730b0 msgr2=0x7f8b100ce640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:38.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.990+0000 7f8b15ff3640 1 -- 172.21.15.150:0/37850468 shutdown_connections 2024-03-17T23:47:38.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:38.990+0000 7f8b15ff3640 1 -- 172.21.15.150:0/37850468 wait complete. 2024-03-17T23:47:39.002 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification' 2024-03-17T23:47:39.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.103+0000 7fee24adc640 1 Processor -- start 2024-03-17T23:47:39.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.103+0000 7fee24adc640 1 -- start start 2024-03-17T23:47:39.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.104+0000 7fee24adc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fee2015be40 0x7fee2009a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:39.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.104+0000 7fee24adc640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fee200c8210 0x7fee200cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:39.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.104+0000 7fee24adc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee200d0070 0x7fee200d0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:39.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.104+0000 7fee24adc640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fee200c5d70 con 0x7fee2015be40 2024-03-17T23:47:39.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.104+0000 7fee24adc640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fee2009ad00 con 0x7fee200c8210 2024-03-17T23:47:39.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.104+0000 7fee24adc640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fee200d0a40 con 0x7fee200d0070 2024-03-17T23:47:39.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.104+0000 7fee1e59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fee2015be40 0x7fee2009a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:39.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.104+0000 7fee1e59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fee2015be40 0x7fee2009a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46192/0 (socket says 172.21.15.150:46192) 2024-03-17T23:47:39.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.104+0000 7fee1e59c640 1 -- 172.21.15.150:0/1585428379 learned_addr learned my addr 172.21.15.150:0/1585428379 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:39.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.104+0000 7fee1ed9d640 1 --2- 172.21.15.150:0/1585428379 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee200d0070 0x7fee200d0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:39.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.104+0000 7fee1dd9b640 1 --2- 172.21.15.150:0/1585428379 >> v2:172.21.15.157:3300/0 conn(0x7fee200c8210 0x7fee200cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:39.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.104+0000 7fee1e59c640 1 -- 172.21.15.150:0/1585428379 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee200d0070 msgr2=0x7fee200d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:39.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.104+0000 7fee1e59c640 1 --2- 172.21.15.150:0/1585428379 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee200d0070 0x7fee200d0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:39.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.104+0000 7fee1e59c640 1 -- 172.21.15.150:0/1585428379 >> v2:172.21.15.157:3300/0 conn(0x7fee200c8210 msgr2=0x7fee200cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:39.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.104+0000 7fee1e59c640 1 --2- 172.21.15.150:0/1585428379 >> v2:172.21.15.157:3300/0 conn(0x7fee200c8210 0x7fee200cfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:39.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.104+0000 7fee1e59c640 1 -- 172.21.15.150:0/1585428379 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee2009b260 con 0x7fee2015be40 2024-03-17T23:47:39.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.105+0000 7fee1e59c640 1 --2- 172.21.15.150:0/1585428379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee2015be40 0x7fee2009a7c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fee200b4720 tx=0x7fee1002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29ae00181658fcfd server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:39.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.105+0000 7fee1d59a640 1 -- 172.21.15.150:0/1585428379 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1002ca70 con 0x7fee2015be40 2024-03-17T23:47:39.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.105+0000 7fee1d59a640 1 -- 172.21.15.150:0/1585428379 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee1002cc10 con 0x7fee2015be40 2024-03-17T23:47:39.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.105+0000 7fee1d59a640 1 -- 172.21.15.150:0/1585428379 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee100343f0 con 0x7fee2015be40 2024-03-17T23:47:39.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.106+0000 7fee24adc640 1 -- 172.21.15.150:0/1585428379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee2015be40 msgr2=0x7fee2009a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:39.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.106+0000 7fee24adc640 1 --2- 172.21.15.150:0/1585428379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee2015be40 0x7fee2009a7c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fee200b4720 tx=0x7fee1002bed0 comp rx=0 tx=0).stop 2024-03-17T23:47:39.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.106+0000 7fee24adc640 1 -- 172.21.15.150:0/1585428379 shutdown_connections 2024-03-17T23:47:39.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.106+0000 7fee24adc640 1 --2- 172.21.15.150:0/1585428379 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee200d0070 0x7fee200d0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:39.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.106+0000 7fee24adc640 1 --2- 172.21.15.150:0/1585428379 >> v2:172.21.15.157:3300/0 conn(0x7fee200c8210 0x7fee200cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:39.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.106+0000 7fee24adc640 1 --2- 172.21.15.150:0/1585428379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee2015be40 0x7fee2009a7c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:39.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.106+0000 7fee24adc640 1 -- 172.21.15.150:0/1585428379 >> 172.21.15.150:0/1585428379 conn(0x7fee200b3620 msgr2=0x7fee200b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:39.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.106+0000 7fee24adc640 1 -- 172.21.15.150:0/1585428379 shutdown_connections 2024-03-17T23:47:39.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.106+0000 7fee24adc640 1 -- 172.21.15.150:0/1585428379 wait complete. 2024-03-17T23:47:39.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.107+0000 7fee24adc640 1 Processor -- start 2024-03-17T23:47:39.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.107+0000 7fee24adc640 1 -- start start 2024-03-17T23:47:39.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.108+0000 7fee24adc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee200c8210 0x7fee20154f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:39.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.108+0000 7fee24adc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee200d0070 0x7fee20157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:39.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.108+0000 7fee24adc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee2015be40 0x7fee20149920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:39.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.108+0000 7fee24adc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fee200c5a00 con 0x7fee2015be40 2024-03-17T23:47:39.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.108+0000 7fee24adc640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fee200c5720 con 0x7fee200d0070 2024-03-17T23:47:39.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.108+0000 7fee24adc640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fee2015c220 con 0x7fee200c8210 2024-03-17T23:47:39.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.108+0000 7fee1ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee2015be40 0x7fee20149920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:39.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.108+0000 7fee1ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee2015be40 0x7fee20149920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46198/0 (socket says 172.21.15.150:46198) 2024-03-17T23:47:39.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.108+0000 7fee1ed9d640 1 -- 172.21.15.150:0/1331941535 learned_addr learned my addr 172.21.15.150:0/1331941535 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:39.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.108+0000 7fee1e59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee200c8210 0x7fee20154f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:39.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.108+0000 7fee1dd9b640 1 --2- 172.21.15.150:0/1331941535 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee200d0070 0x7fee20157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:39.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.108+0000 7fee1ed9d640 1 -- 172.21.15.150:0/1331941535 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee200c8210 msgr2=0x7fee20154f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:39.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.108+0000 7fee1ed9d640 1 --2- 172.21.15.150:0/1331941535 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee200c8210 0x7fee20154f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:39.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.108+0000 7fee1ed9d640 1 -- 172.21.15.150:0/1331941535 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee200d0070 msgr2=0x7fee20157600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:39.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.108+0000 7fee1ed9d640 1 --2- 172.21.15.150:0/1331941535 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee200d0070 0x7fee20157600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:39.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.108+0000 7fee1ed9d640 1 -- 172.21.15.150:0/1331941535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee20155860 con 0x7fee2015be40 2024-03-17T23:47:39.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.108+0000 7fee1ed9d640 1 --2- 172.21.15.150:0/1331941535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee2015be40 0x7fee20149920 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fee1800e7f0 tx=0x7fee1800ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:39.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.109+0000 7fee0effd640 1 -- 172.21.15.150:0/1331941535 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee18017070 con 0x7fee2015be40 2024-03-17T23:47:39.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.109+0000 7fee0effd640 1 -- 172.21.15.150:0/1331941535 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee1800ee40 con 0x7fee2015be40 2024-03-17T23:47:39.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.109+0000 7fee24adc640 1 -- 172.21.15.150:0/1331941535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee20155b40 con 0x7fee2015be40 2024-03-17T23:47:39.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.109+0000 7fee0effd640 1 -- 172.21.15.150:0/1331941535 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1800cb80 con 0x7fee2015be40 2024-03-17T23:47:39.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.109+0000 7fee24adc640 1 -- 172.21.15.150:0/1331941535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee20156100 con 0x7fee2015be40 2024-03-17T23:47:39.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.110+0000 7fee24adc640 1 -- 172.21.15.150:0/1331941535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee200688b0 con 0x7fee2015be40 2024-03-17T23:47:39.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.110+0000 7fee0effd640 1 -- 172.21.15.150:0/1331941535 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fee1801a070 con 0x7fee2015be40 2024-03-17T23:47:39.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.111+0000 7fee0effd640 1 --2- 172.21.15.150:0/1331941535 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fee0006bbd0 0x7fee0006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:39.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.111+0000 7fee1e59c640 1 --2- 172.21.15.150:0/1331941535 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fee0006bbd0 0x7fee0006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:39.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.111+0000 7fee0effd640 1 -- 172.21.15.150:0/1331941535 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fee1808aff0 con 0x7fee2015be40 2024-03-17T23:47:39.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.111+0000 7fee1e59c640 1 --2- 172.21.15.150:0/1331941535 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fee0006bbd0 0x7fee0006e090 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fee10035000 tx=0x7fee10030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:39.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.115+0000 7fee0effd640 1 -- 172.21.15.150:0/1331941535 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fee18059090 con 0x7fee2015be40 2024-03-17T23:47:39.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:39.354+0000 7fee24adc640 1 -- 172.21.15.150:0/1331941535 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]} v 0) -- 0x7fee200cfad0 con 0x7fee2015be40 2024-03-17T23:47:40.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.050+0000 7fee0effd640 1 -- 172.21.15.150:0/1331941535 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]}]=0 v68) ==== 150+0+0 (secure 0 0 0) 0x7fee1805d130 con 0x7fee2015be40 2024-03-17T23:47:40.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.052+0000 7fee24adc640 1 -- 172.21.15.150:0/1331941535 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fee0006bbd0 msgr2=0x7fee0006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:40.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.052+0000 7fee24adc640 1 --2- 172.21.15.150:0/1331941535 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fee0006bbd0 0x7fee0006e090 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fee10035000 tx=0x7fee10030000 comp rx=0 tx=0).stop 2024-03-17T23:47:40.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.052+0000 7fee24adc640 1 -- 172.21.15.150:0/1331941535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee2015be40 msgr2=0x7fee20149920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:40.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.052+0000 7fee24adc640 1 --2- 172.21.15.150:0/1331941535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee2015be40 0x7fee20149920 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fee1800e7f0 tx=0x7fee1800ebf0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.052+0000 7fee24adc640 1 -- 172.21.15.150:0/1331941535 shutdown_connections 2024-03-17T23:47:40.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.052+0000 7fee24adc640 1 --2- 172.21.15.150:0/1331941535 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee2015be40 0x7fee20149920 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.052+0000 7fee24adc640 1 --2- 172.21.15.150:0/1331941535 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fee0006bbd0 0x7fee0006e090 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.052+0000 7fee24adc640 1 --2- 172.21.15.150:0/1331941535 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee200d0070 0x7fee20157600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.052+0000 7fee24adc640 1 --2- 172.21.15.150:0/1331941535 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fee200c8210 0x7fee20154f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.052+0000 7fee24adc640 1 -- 172.21.15.150:0/1331941535 >> 172.21.15.150:0/1331941535 conn(0x7fee200b3620 msgr2=0x7fee200c7090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:40.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.052+0000 7fee24adc640 1 -- 172.21.15.150:0/1331941535 shutdown_connections 2024-03-17T23:47:40.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.052+0000 7fee24adc640 1 -- 172.21.15.150:0/1331941535 wait complete. 2024-03-17T23:47:40.065 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-17T23:47:40.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.171+0000 7f2d36340640 1 Processor -- start 2024-03-17T23:47:40.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.171+0000 7f2d36340640 1 -- start start 2024-03-17T23:47:40.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.171+0000 7f2d36340640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2d30161520 0x7f2d300b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:40.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.171+0000 7f2d36340640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d300d0ce0 0x7f2d300d1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:40.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.171+0000 7f2d36340640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d30161d30 con 0x7f2d300b2bf0 2024-03-17T23:47:40.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.171+0000 7f2d36340640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f2d300b9bf0 con 0x7f2d30161520 2024-03-17T23:47:40.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.171+0000 7f2d36340640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2d30162940 con 0x7f2d300d0ce0 2024-03-17T23:47:40.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.171+0000 7f2d2f7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f2d300b2bf0 0x7f2d300d05d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59204/0 (socket says 172.21.15.150:59204) 2024-03-17T23:47:40.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.171+0000 7f2d2f7fe640 1 -- 172.21.15.150:0/3901180714 learned_addr learned my addr 172.21.15.150:0/3901180714 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:40.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.172+0000 7f2d2ffff640 1 --2- 172.21.15.150:0/3901180714 >> v2:172.21.15.157:3300/0 conn(0x7f2d30161520 0x7f2d300b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:40.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.172+0000 7f2d348dd640 1 --2- 172.21.15.150:0/3901180714 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d300d0ce0 0x7f2d300d1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:40.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.172+0000 7f2d2effd640 1 -- 172.21.15.150:0/3901180714 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3888934019 0 0) 0x7f2d30161d30 con 0x7f2d300b2bf0 2024-03-17T23:47:40.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.172+0000 7f2d2effd640 1 -- 172.21.15.150:0/3901180714 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d18003520 con 0x7f2d300b2bf0 2024-03-17T23:47:40.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.172+0000 7f2d2ffff640 1 -- 172.21.15.150:0/3901180714 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d300d0ce0 msgr2=0x7f2d300d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:40.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.172+0000 7f2d2ffff640 1 --2- 172.21.15.150:0/3901180714 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d300d0ce0 0x7f2d300d1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.172+0000 7f2d2ffff640 1 -- 172.21.15.150:0/3901180714 >> v1:172.21.15.150:6789/0 conn(0x7f2d300b2bf0 legacy=0x7f2d300d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:40.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.172+0000 7f2d2ffff640 1 -- 172.21.15.150:0/3901180714 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d30162010 con 0x7f2d30161520 2024-03-17T23:47:40.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.173+0000 7f2d2ffff640 1 --2- 172.21.15.150:0/3901180714 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d30161520 0x7f2d300b96b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2d200097a0 tx=0x7f2d2002b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ebd3e2c998615ba server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:40.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.173+0000 7f2d2effd640 1 -- 172.21.15.150:0/3901180714 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d20038070 con 0x7f2d30161520 2024-03-17T23:47:40.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.173+0000 7f2d2effd640 1 -- 172.21.15.150:0/3901180714 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d20031700 con 0x7f2d30161520 2024-03-17T23:47:40.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.173+0000 7f2d2effd640 1 -- 172.21.15.150:0/3901180714 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d20031ab0 con 0x7f2d30161520 2024-03-17T23:47:40.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.173+0000 7f2d36340640 1 -- 172.21.15.150:0/3901180714 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d30161520 msgr2=0x7f2d300b96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:40.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.173+0000 7f2d36340640 1 --2- 172.21.15.150:0/3901180714 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d30161520 0x7f2d300b96b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2d200097a0 tx=0x7f2d2002b7a0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.173+0000 7f2d36340640 1 -- 172.21.15.150:0/3901180714 shutdown_connections 2024-03-17T23:47:40.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.173+0000 7f2d36340640 1 --2- 172.21.15.150:0/3901180714 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d300d0ce0 0x7f2d300d1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.173+0000 7f2d36340640 1 --2- 172.21.15.150:0/3901180714 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d30161520 0x7f2d300b96b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.173+0000 7f2d36340640 1 -- 172.21.15.150:0/3901180714 >> 172.21.15.150:0/3901180714 conn(0x7f2d3006ca00 msgr2=0x7f2d3006ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:40.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.173+0000 7f2d36340640 1 -- 172.21.15.150:0/3901180714 shutdown_connections 2024-03-17T23:47:40.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.173+0000 7f2d36340640 1 -- 172.21.15.150:0/3901180714 wait complete. 2024-03-17T23:47:40.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.174+0000 7f2d36340640 1 Processor -- start 2024-03-17T23:47:40.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.175+0000 7f2d36340640 1 -- start start 2024-03-17T23:47:40.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.175+0000 7f2d36340640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d300b2bf0 0x7f2d30157c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:40.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.175+0000 7f2d36340640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d300d0ce0 0x7f2d301566f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:40.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.175+0000 7f2d36340640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d30161520 0x7f2d30156c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:40.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.175+0000 7f2d36340640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2d30161900 con 0x7f2d30161520 2024-03-17T23:47:40.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.175+0000 7f2d36340640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2d301625d0 con 0x7f2d300b2bf0 2024-03-17T23:47:40.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.175+0000 7f2d36340640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2d300d24e0 con 0x7f2d300d0ce0 2024-03-17T23:47:40.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.175+0000 7f2d348dd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d30161520 0x7f2d30156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:40.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.175+0000 7f2d348dd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d30161520 0x7f2d30156c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46214/0 (socket says 172.21.15.150:46214) 2024-03-17T23:47:40.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.175+0000 7f2d348dd640 1 -- 172.21.15.150:0/2262980738 learned_addr learned my addr 172.21.15.150:0/2262980738 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:40.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.176+0000 7f2d2ffff640 1 --2- 172.21.15.150:0/2262980738 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d300b2bf0 0x7f2d30157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:40.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.176+0000 7f2d2f7fe640 1 --2- 172.21.15.150:0/2262980738 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d300d0ce0 0x7f2d301566f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:40.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.176+0000 7f2d348dd640 1 -- 172.21.15.150:0/2262980738 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d300d0ce0 msgr2=0x7f2d301566f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:40.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.176+0000 7f2d348dd640 1 --2- 172.21.15.150:0/2262980738 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d300d0ce0 0x7f2d301566f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.176+0000 7f2d348dd640 1 -- 172.21.15.150:0/2262980738 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d300b2bf0 msgr2=0x7f2d30157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:40.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.176+0000 7f2d348dd640 1 --2- 172.21.15.150:0/2262980738 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d300b2bf0 0x7f2d30157c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.176+0000 7f2d348dd640 1 -- 172.21.15.150:0/2262980738 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d30158ee0 con 0x7f2d30161520 2024-03-17T23:47:40.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.176+0000 7f2d348dd640 1 --2- 172.21.15.150:0/2262980738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d30161520 0x7f2d30156c30 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2d2800efc0 tx=0x7f2d2800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:40.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.176+0000 7f2d2cff9640 1 -- 172.21.15.150:0/2262980738 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d28018020 con 0x7f2d30161520 2024-03-17T23:47:40.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.176+0000 7f2d2cff9640 1 -- 172.21.15.150:0/2262980738 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d28014480 con 0x7f2d30161520 2024-03-17T23:47:40.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.176+0000 7f2d2cff9640 1 -- 172.21.15.150:0/2262980738 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d28014870 con 0x7f2d30161520 2024-03-17T23:47:40.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.176+0000 7f2d36340640 1 -- 172.21.15.150:0/2262980738 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d300b6370 con 0x7f2d30161520 2024-03-17T23:47:40.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.176+0000 7f2d36340640 1 -- 172.21.15.150:0/2262980738 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d301586e0 con 0x7f2d30161520 2024-03-17T23:47:40.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.177+0000 7f2d36340640 1 -- 172.21.15.150:0/2262980738 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d300681a0 con 0x7f2d30161520 2024-03-17T23:47:40.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.177+0000 7f2d2cff9640 1 -- 172.21.15.150:0/2262980738 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2d28005d10 con 0x7f2d30161520 2024-03-17T23:47:40.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.178+0000 7f2d2cff9640 1 --2- 172.21.15.150:0/2262980738 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2d1c06bbd0 0x7f2d1c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:40.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.178+0000 7f2d2cff9640 1 -- 172.21.15.150:0/2262980738 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f2d2808bbc0 con 0x7f2d30161520 2024-03-17T23:47:40.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.178+0000 7f2d2ffff640 1 --2- 172.21.15.150:0/2262980738 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2d1c06bbd0 0x7f2d1c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:40.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.178+0000 7f2d2ffff640 1 --2- 172.21.15.150:0/2262980738 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2d1c06bbd0 0x7f2d1c06e090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2d20035000 tx=0x7f2d20008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:40.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.182+0000 7f2d2cff9640 1 -- 172.21.15.150:0/2262980738 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2d28022030 con 0x7f2d30161520 2024-03-17T23:47:40.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.354+0000 7f2d36340640 1 -- 172.21.15.150:0/2262980738 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f2d300c75b0 con 0x7f2d30161520 2024-03-17T23:47:40.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.357+0000 7f2d2cff9640 1 -- 172.21.15.150:0/2262980738 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7f2d28010040 con 0x7f2d30161520 2024-03-17T23:47:40.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.359+0000 7f2d36340640 1 -- 172.21.15.150:0/2262980738 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2d1c06bbd0 msgr2=0x7f2d1c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:40.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.359+0000 7f2d36340640 1 --2- 172.21.15.150:0/2262980738 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2d1c06bbd0 0x7f2d1c06e090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2d20035000 tx=0x7f2d20008030 comp rx=0 tx=0).stop 2024-03-17T23:47:40.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.359+0000 7f2d36340640 1 -- 172.21.15.150:0/2262980738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d30161520 msgr2=0x7f2d30156c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:40.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.359+0000 7f2d36340640 1 --2- 172.21.15.150:0/2262980738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d30161520 0x7f2d30156c30 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2d2800efc0 tx=0x7f2d2800c490 comp rx=0 tx=0).stop 2024-03-17T23:47:40.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.359+0000 7f2d36340640 1 -- 172.21.15.150:0/2262980738 shutdown_connections 2024-03-17T23:47:40.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.359+0000 7f2d36340640 1 --2- 172.21.15.150:0/2262980738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d30161520 0x7f2d30156c30 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.359+0000 7f2d36340640 1 --2- 172.21.15.150:0/2262980738 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d300d0ce0 0x7f2d301566f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.359+0000 7f2d36340640 1 --2- 172.21.15.150:0/2262980738 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2d1c06bbd0 0x7f2d1c06e090 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.359+0000 7f2d36340640 1 --2- 172.21.15.150:0/2262980738 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d300b2bf0 0x7f2d30157c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.359+0000 7f2d36340640 1 -- 172.21.15.150:0/2262980738 >> 172.21.15.150:0/2262980738 conn(0x7f2d3006ca00 msgr2=0x7f2d300d2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:40.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.359+0000 7f2d36340640 1 -- 172.21.15.150:0/2262980738 shutdown_connections 2024-03-17T23:47:40.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.359+0000 7f2d36340640 1 -- 172.21.15.150:0/2262980738 wait complete. 2024-03-17T23:47:40.371 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-17T23:47:40.371 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-17T23:47:40.416 DEBUG:teuthology.orchestra.run.smithi157:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-17T23:47:40.460 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-17T23:47:40.460 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:47:40.460 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:47:40.476 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:47:40.476 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:47:40.531 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:40.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.672+0000 7f154f587640 1 Processor -- start 2024-03-17T23:47:40.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.672+0000 7f154f587640 1 -- start start 2024-03-17T23:47:40.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.673+0000 7f154f587640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f154815bfd0 0x7f15480ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:40.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.673+0000 7f154f587640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f15480cd250 0x7f154815ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:40.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.673+0000 7f154f587640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15480cf3b0 con 0x7f1548155320 2024-03-17T23:47:40.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.673+0000 7f154f587640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f154815c550 con 0x7f154815bfd0 2024-03-17T23:47:40.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.673+0000 7f154f587640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f15480cf690 con 0x7f15480cd250 2024-03-17T23:47:40.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.673+0000 7f154cb22640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f154815bfd0 0x7f15480ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:40.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.673+0000 7f154cb22640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f154815bfd0 0x7f15480ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46230/0 (socket says 172.21.15.150:46230) 2024-03-17T23:47:40.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.673+0000 7f154cb22640 1 -- 172.21.15.150:0/528988018 learned_addr learned my addr 172.21.15.150:0/528988018 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:40.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.673+0000 7f154db24640 1 --2- 172.21.15.150:0/528988018 >> v2:172.21.15.157:3300/0 conn(0x7f15480cd250 0x7f154815ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:40.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.673+0000 7f154cb22640 1 -- 172.21.15.150:0/528988018 >> v2:172.21.15.157:3300/0 conn(0x7f15480cd250 msgr2=0x7f154815ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:40.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.673+0000 7f154cb22640 1 --2- 172.21.15.150:0/528988018 >> v2:172.21.15.157:3300/0 conn(0x7f15480cd250 0x7f154815ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.673+0000 7f154cb22640 1 -- 172.21.15.150:0/528988018 >> v1:172.21.15.157:6789/0 conn(0x7f1548155320 legacy=0x7f154806dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:40.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.673+0000 7f154cb22640 1 -- 172.21.15.150:0/528988018 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f154815c7a0 con 0x7f154815bfd0 2024-03-17T23:47:40.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.674+0000 7f154cb22640 1 --2- 172.21.15.150:0/528988018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f154815bfd0 0x7f15480ccd10 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f154806c0f0 tx=0x7f153802e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbac4326e62c3290 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:40.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.674+0000 7f153ffff640 1 -- 172.21.15.150:0/528988018 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153803b070 con 0x7f154815bfd0 2024-03-17T23:47:40.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.674+0000 7f153ffff640 1 -- 172.21.15.150:0/528988018 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1538036440 con 0x7f154815bfd0 2024-03-17T23:47:40.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.674+0000 7f153ffff640 1 -- 172.21.15.150:0/528988018 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1538036810 con 0x7f154815bfd0 2024-03-17T23:47:40.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.675+0000 7f154f587640 1 -- 172.21.15.150:0/528988018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f154815bfd0 msgr2=0x7f15480ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:40.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.675+0000 7f154f587640 1 --2- 172.21.15.150:0/528988018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f154815bfd0 0x7f15480ccd10 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f154806c0f0 tx=0x7f153802e170 comp rx=0 tx=0).stop 2024-03-17T23:47:40.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.675+0000 7f154f587640 1 -- 172.21.15.150:0/528988018 shutdown_connections 2024-03-17T23:47:40.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.675+0000 7f154f587640 1 --2- 172.21.15.150:0/528988018 >> v2:172.21.15.157:3300/0 conn(0x7f15480cd250 0x7f154815ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.675+0000 7f154f587640 1 --2- 172.21.15.150:0/528988018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f154815bfd0 0x7f15480ccd10 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.675+0000 7f154f587640 1 -- 172.21.15.150:0/528988018 >> 172.21.15.150:0/528988018 conn(0x7f154806b190 msgr2=0x7f154806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:40.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.675+0000 7f154f587640 1 -- 172.21.15.150:0/528988018 shutdown_connections 2024-03-17T23:47:40.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.675+0000 7f154f587640 1 -- 172.21.15.150:0/528988018 wait complete. 2024-03-17T23:47:40.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.676+0000 7f154f587640 1 Processor -- start 2024-03-17T23:47:40.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.676+0000 7f154f587640 1 -- start start 2024-03-17T23:47:40.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.677+0000 7f154f587640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15480cd250 0x7f154813d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:40.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.677+0000 7f154f587640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1548155320 0x7f15480b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:40.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.677+0000 7f154f587640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f154815bfd0 0x7f1548159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:40.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.677+0000 7f154f587640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1548155880 con 0x7f1548155320 2024-03-17T23:47:40.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.677+0000 7f154f587640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1548155700 con 0x7f15480cd250 2024-03-17T23:47:40.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.677+0000 7f154f587640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f154815d200 con 0x7f154815bfd0 2024-03-17T23:47:40.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.677+0000 7f154cb22640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1548155320 0x7f15480b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:40.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.677+0000 7f154cb22640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1548155320 0x7f15480b47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46242/0 (socket says 172.21.15.150:46242) 2024-03-17T23:47:40.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.677+0000 7f154cb22640 1 -- 172.21.15.150:0/2720843311 learned_addr learned my addr 172.21.15.150:0/2720843311 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:40.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.677+0000 7f154d323640 1 --2- 172.21.15.150:0/2720843311 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15480cd250 0x7f154813d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:40.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.677+0000 7f154db24640 1 --2- 172.21.15.150:0/2720843311 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f154815bfd0 0x7f1548159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:40.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.677+0000 7f154cb22640 1 -- 172.21.15.150:0/2720843311 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f154815bfd0 msgr2=0x7f1548159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:40.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.677+0000 7f154cb22640 1 --2- 172.21.15.150:0/2720843311 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f154815bfd0 0x7f1548159bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.677+0000 7f154cb22640 1 -- 172.21.15.150:0/2720843311 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15480cd250 msgr2=0x7f154813d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:40.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.677+0000 7f154cb22640 1 --2- 172.21.15.150:0/2720843311 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15480cd250 0x7f154813d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.677+0000 7f154cb22640 1 -- 172.21.15.150:0/2720843311 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1548155f70 con 0x7f1548155320 2024-03-17T23:47:40.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.677+0000 7f154db24640 1 --2- 172.21.15.150:0/2720843311 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f154815bfd0 0x7f1548159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:47:40.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.677+0000 7f154cb22640 1 --2- 172.21.15.150:0/2720843311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1548155320 0x7f15480b47a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f153802e1b0 tx=0x7f15380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:40.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.677+0000 7f153dffb640 1 -- 172.21.15.150:0/2720843311 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153803b040 con 0x7f1548155320 2024-03-17T23:47:40.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.677+0000 7f153dffb640 1 -- 172.21.15.150:0/2720843311 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1538046070 con 0x7f1548155320 2024-03-17T23:47:40.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.677+0000 7f154f587640 1 -- 172.21.15.150:0/2720843311 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1548156250 con 0x7f1548155320 2024-03-17T23:47:40.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.677+0000 7f153dffb640 1 -- 172.21.15.150:0/2720843311 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1538034880 con 0x7f1548155320 2024-03-17T23:47:40.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.678+0000 7f154f587640 1 -- 172.21.15.150:0/2720843311 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1548156810 con 0x7f1548155320 2024-03-17T23:47:40.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.679+0000 7f154f587640 1 -- 172.21.15.150:0/2720843311 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15480681a0 con 0x7f1548155320 2024-03-17T23:47:40.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.679+0000 7f153dffb640 1 -- 172.21.15.150:0/2720843311 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1538034a20 con 0x7f1548155320 2024-03-17T23:47:40.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.679+0000 7f153dffb640 1 --2- 172.21.15.150:0/2720843311 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f152c06bbd0 0x7f152c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:40.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.680+0000 7f153dffb640 1 -- 172.21.15.150:0/2720843311 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f15380af990 con 0x7f1548155320 2024-03-17T23:47:40.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.680+0000 7f154d323640 1 --2- 172.21.15.150:0/2720843311 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f152c06bbd0 0x7f152c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:40.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.680+0000 7f154d323640 1 --2- 172.21.15.150:0/2720843311 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f152c06bbd0 0x7f152c06e090 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1544001780 tx=0x7f1544009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:40.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.684+0000 7f153dffb640 1 -- 172.21.15.150:0/2720843311 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f153807dff0 con 0x7f1548155320 2024-03-17T23:47:40.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.856+0000 7f154f587640 1 -- 172.21.15.150:0/2720843311 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f154806fb50 con 0x7f1548155320 2024-03-17T23:47:40.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.857+0000 7f153dffb640 1 -- 172.21.15.150:0/2720843311 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18881 (secure 0 0 0) 0x7f153807d8f0 con 0x7f1548155320 2024-03-17T23:47:40.857 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:40.858 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":28,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:37.031864+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.e","mappings":[{"from":7,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:40.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.859+0000 7f154f587640 1 -- 172.21.15.150:0/2720843311 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f152c06bbd0 msgr2=0x7f152c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:40.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.859+0000 7f154f587640 1 --2- 172.21.15.150:0/2720843311 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f152c06bbd0 0x7f152c06e090 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1544001780 tx=0x7f1544009040 comp rx=0 tx=0).stop 2024-03-17T23:47:40.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.859+0000 7f154f587640 1 -- 172.21.15.150:0/2720843311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1548155320 msgr2=0x7f15480b47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:40.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.859+0000 7f154f587640 1 --2- 172.21.15.150:0/2720843311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1548155320 0x7f15480b47a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f153802e1b0 tx=0x7f15380046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.859+0000 7f154f587640 1 -- 172.21.15.150:0/2720843311 shutdown_connections 2024-03-17T23:47:40.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.859+0000 7f154f587640 1 --2- 172.21.15.150:0/2720843311 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f154815bfd0 0x7f1548159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.859+0000 7f154f587640 1 --2- 172.21.15.150:0/2720843311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1548155320 0x7f15480b47a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.859+0000 7f154f587640 1 --2- 172.21.15.150:0/2720843311 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f152c06bbd0 0x7f152c06e090 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.859+0000 7f154f587640 1 --2- 172.21.15.150:0/2720843311 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15480cd250 0x7f154813d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.859+0000 7f154f587640 1 -- 172.21.15.150:0/2720843311 >> 172.21.15.150:0/2720843311 conn(0x7f154806b190 msgr2=0x7f15480cc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:40.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.859+0000 7f154f587640 1 -- 172.21.15.150:0/2720843311 shutdown_connections 2024-03-17T23:47:40.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.859+0000 7f154f587640 1 -- 172.21.15.150:0/2720843311 wait complete. 2024-03-17T23:47:40.871 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:46:53.610678+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:46:54.227335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-17T23:46:57.949235+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-17T23:47:40.871 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:47:40.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.974+0000 7effbd162640 1 Processor -- start 2024-03-17T23:47:40.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.974+0000 7effbd162640 1 -- start start 2024-03-17T23:47:40.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.975+0000 7effbd162640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effb815f7e0 0x7effb8072270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:40.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.975+0000 7effbd162640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effb815fe00 con 0x7effb80727b0 2024-03-17T23:47:40.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.975+0000 7effbd162640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effb81600e0 con 0x7effb8156460 2024-03-17T23:47:40.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.975+0000 7effbd162640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7effb81603c0 con 0x7effb815f7e0 2024-03-17T23:47:40.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.975+0000 7effb759e640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7effb80727b0 0x7effb815d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59208/0 (socket says 172.21.15.150:59208) 2024-03-17T23:47:40.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.975+0000 7effb759e640 1 -- 172.21.15.150:0/593872735 learned_addr learned my addr 172.21.15.150:0/593872735 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:40.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.975+0000 7effb659c640 1 --2- 172.21.15.150:0/593872735 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effb815f7e0 0x7effb8072270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:40.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.975+0000 7effb5d9b640 1 -- 172.21.15.150:0/593872735 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 318063773 0 0) 0x7effb815fe00 con 0x7effb80727b0 2024-03-17T23:47:40.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.975+0000 7effb5d9b640 1 -- 172.21.15.150:0/593872735 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7effa0003520 con 0x7effb80727b0 2024-03-17T23:47:40.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.975+0000 7effb5d9b640 1 -- 172.21.15.150:0/593872735 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4286682027 0 0) 0x7effb81600e0 con 0x7effb8156460 2024-03-17T23:47:40.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.975+0000 7effb5d9b640 1 -- 172.21.15.150:0/593872735 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7effb815fe00 con 0x7effb8156460 2024-03-17T23:47:40.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.976+0000 7effb659c640 1 -- 172.21.15.150:0/593872735 >> v1:172.21.15.157:6789/0 conn(0x7effb8156460 legacy=0x7effb815cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:40.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.976+0000 7effb659c640 1 -- 172.21.15.150:0/593872735 >> v1:172.21.15.150:6789/0 conn(0x7effb80727b0 legacy=0x7effb815d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:40.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.976+0000 7effb659c640 1 -- 172.21.15.150:0/593872735 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effb81606a0 con 0x7effb815f7e0 2024-03-17T23:47:40.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.976+0000 7effb659c640 1 --2- 172.21.15.150:0/593872735 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effb815f7e0 0x7effb8072270 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7effac0097d0 tx=0x7effac02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6541089f0c378b11 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:40.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.976+0000 7effb5d9b640 1 -- 172.21.15.150:0/593872735 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effac035020 con 0x7effb815f7e0 2024-03-17T23:47:40.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.977+0000 7effb5d9b640 1 -- 172.21.15.150:0/593872735 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effac031890 con 0x7effb815f7e0 2024-03-17T23:47:40.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.977+0000 7effb5d9b640 1 -- 172.21.15.150:0/593872735 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effac031c40 con 0x7effb815f7e0 2024-03-17T23:47:40.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.977+0000 7effbd162640 1 -- 172.21.15.150:0/593872735 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effb815f7e0 msgr2=0x7effb8072270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:40.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.977+0000 7effbd162640 1 --2- 172.21.15.150:0/593872735 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effb815f7e0 0x7effb8072270 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7effac0097d0 tx=0x7effac02b930 comp rx=0 tx=0).stop 2024-03-17T23:47:40.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.977+0000 7effbd162640 1 -- 172.21.15.150:0/593872735 shutdown_connections 2024-03-17T23:47:40.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.977+0000 7effbd162640 1 --2- 172.21.15.150:0/593872735 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effb815f7e0 0x7effb8072270 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.977+0000 7effbd162640 1 -- 172.21.15.150:0/593872735 >> 172.21.15.150:0/593872735 conn(0x7effb806b370 msgr2=0x7effb806b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:40.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.977+0000 7effbd162640 1 -- 172.21.15.150:0/593872735 shutdown_connections 2024-03-17T23:47:40.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.977+0000 7effbd162640 1 -- 172.21.15.150:0/593872735 wait complete. 2024-03-17T23:47:40.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.979+0000 7effbd162640 1 Processor -- start 2024-03-17T23:47:40.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.979+0000 7effbd162640 1 -- start start 2024-03-17T23:47:40.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.979+0000 7effbd162640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effb80727b0 0x7effb8149850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:40.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.979+0000 7effbd162640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effb8156460 0x7effb80b1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:40.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.979+0000 7effbd162640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7effb815f7e0 0x7effb80adcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:40.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.979+0000 7effbd162640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7effb815e200 con 0x7effb8156460 2024-03-17T23:47:40.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.979+0000 7effbd162640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7effb815e080 con 0x7effb815f7e0 2024-03-17T23:47:40.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.979+0000 7effbd162640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7effb815e380 con 0x7effb80727b0 2024-03-17T23:47:40.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.979+0000 7effb659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effb8156460 0x7effb80b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:40.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.979+0000 7effb659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effb8156460 0x7effb80b1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46256/0 (socket says 172.21.15.150:46256) 2024-03-17T23:47:40.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.979+0000 7effb659c640 1 -- 172.21.15.150:0/3161252025 learned_addr learned my addr 172.21.15.150:0/3161252025 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:40.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.980+0000 7effb759e640 1 --2- 172.21.15.150:0/3161252025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7effb815f7e0 0x7effb80adcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:40.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.980+0000 7effb6d9d640 1 --2- 172.21.15.150:0/3161252025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effb80727b0 0x7effb8149850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:40.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.980+0000 7effb659c640 1 -- 172.21.15.150:0/3161252025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effb80727b0 msgr2=0x7effb8149850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:40.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.980+0000 7effb659c640 1 --2- 172.21.15.150:0/3161252025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effb80727b0 0x7effb8149850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.980+0000 7effb659c640 1 -- 172.21.15.150:0/3161252025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7effb815f7e0 msgr2=0x7effb80adcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:40.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.980+0000 7effb659c640 1 --2- 172.21.15.150:0/3161252025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7effb815f7e0 0x7effb80adcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:40.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.980+0000 7effb659c640 1 -- 172.21.15.150:0/3161252025 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effb80b3b60 con 0x7effb8156460 2024-03-17T23:47:40.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.980+0000 7effb759e640 1 --2- 172.21.15.150:0/3161252025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7effb815f7e0 0x7effb80adcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:40.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.980+0000 7effb659c640 1 --2- 172.21.15.150:0/3161252025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effb8156460 0x7effb80b1170 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7effac034040 tx=0x7effac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:40.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.980+0000 7eff9f7fe640 1 -- 172.21.15.150:0/3161252025 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effac035040 con 0x7effb8156460 2024-03-17T23:47:40.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.980+0000 7eff9f7fe640 1 -- 172.21.15.150:0/3161252025 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effac030d20 con 0x7effb8156460 2024-03-17T23:47:40.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.980+0000 7effbd162640 1 -- 172.21.15.150:0/3161252025 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effb80b3e40 con 0x7effb8156460 2024-03-17T23:47:40.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.980+0000 7eff9f7fe640 1 -- 172.21.15.150:0/3161252025 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effac0456a0 con 0x7effb8156460 2024-03-17T23:47:40.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.980+0000 7effbd162640 1 -- 172.21.15.150:0/3161252025 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effb80b4400 con 0x7effb8156460 2024-03-17T23:47:40.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.981+0000 7effbd162640 1 -- 172.21.15.150:0/3161252025 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effb80b4640 con 0x7effb8156460 2024-03-17T23:47:40.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.982+0000 7eff9f7fe640 1 -- 172.21.15.150:0/3161252025 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7effac038020 con 0x7effb8156460 2024-03-17T23:47:40.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.982+0000 7eff9f7fe640 1 --2- 172.21.15.150:0/3161252025 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7effa406bbd0 0x7effa406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:40.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.982+0000 7effb6d9d640 1 --2- 172.21.15.150:0/3161252025 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7effa406bbd0 0x7effa406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:40.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.982+0000 7eff9f7fe640 1 -- 172.21.15.150:0/3161252025 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7effac0ad060 con 0x7effb8156460 2024-03-17T23:47:40.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.982+0000 7effb6d9d640 1 --2- 172.21.15.150:0/3161252025 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7effa406bbd0 0x7effa406e090 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7effa8001ea0 tx=0x7effa8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:40.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:40.987+0000 7eff9f7fe640 1 -- 172.21.15.150:0/3161252025 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7effac02d040 con 0x7effb8156460 2024-03-17T23:47:41.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.158+0000 7effbd162640 1 -- 172.21.15.150:0/3161252025 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7effb806f140 con 0x7effb8156460 2024-03-17T23:47:41.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.159+0000 7eff9f7fe640 1 -- 172.21.15.150:0/3161252025 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7effac07b100 con 0x7effb8156460 2024-03-17T23:47:41.159 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:47:41.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.161+0000 7effbd162640 1 -- 172.21.15.150:0/3161252025 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7effa406bbd0 msgr2=0x7effa406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:41.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.161+0000 7effbd162640 1 --2- 172.21.15.150:0/3161252025 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7effa406bbd0 0x7effa406e090 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7effa8001ea0 tx=0x7effa8009040 comp rx=0 tx=0).stop 2024-03-17T23:47:41.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.161+0000 7effbd162640 1 -- 172.21.15.150:0/3161252025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effb8156460 msgr2=0x7effb80b1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:41.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.161+0000 7effbd162640 1 --2- 172.21.15.150:0/3161252025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effb8156460 0x7effb80b1170 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7effac034040 tx=0x7effac0046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.161+0000 7effbd162640 1 -- 172.21.15.150:0/3161252025 shutdown_connections 2024-03-17T23:47:41.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.161+0000 7effbd162640 1 --2- 172.21.15.150:0/3161252025 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7effa406bbd0 0x7effa406e090 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.161+0000 7effbd162640 1 --2- 172.21.15.150:0/3161252025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7effb815f7e0 0x7effb80adcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.161+0000 7effbd162640 1 --2- 172.21.15.150:0/3161252025 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effb8156460 0x7effb80b1170 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.161+0000 7effbd162640 1 --2- 172.21.15.150:0/3161252025 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effb80727b0 0x7effb8149850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.161+0000 7effbd162640 1 -- 172.21.15.150:0/3161252025 >> 172.21.15.150:0/3161252025 conn(0x7effb806b370 msgr2=0x7effb806b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:41.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.161+0000 7effbd162640 1 -- 172.21.15.150:0/3161252025 shutdown_connections 2024-03-17T23:47:41.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.161+0000 7effbd162640 1 -- 172.21.15.150:0/3161252025 wait complete. 2024-03-17T23:47:41.172 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:47:41.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.276+0000 7f8bdef1b640 1 Processor -- start 2024-03-17T23:47:41.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.276+0000 7f8bdef1b640 1 -- start start 2024-03-17T23:47:41.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.277+0000 7f8bdef1b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8bd815bfd0 0x7f8bd80ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:41.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.277+0000 7f8bdef1b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8bd80cd250 0x7f8bd815ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:41.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.277+0000 7f8bdef1b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bd80cf3b0 con 0x7f8bd8155320 2024-03-17T23:47:41.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.277+0000 7f8bdef1b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f8bd815c550 con 0x7f8bd815bfd0 2024-03-17T23:47:41.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.277+0000 7f8bdef1b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f8bd80cf690 con 0x7f8bd80cd250 2024-03-17T23:47:41.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.277+0000 7f8bdccb7640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f8bd8155320 0x7f8bd806dd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59210/0 (socket says 172.21.15.150:59210) 2024-03-17T23:47:41.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.277+0000 7f8bd7fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8bd815bfd0 0x7f8bd80ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:41.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.277+0000 7f8bdccb7640 1 -- 172.21.15.150:0/105638755 learned_addr learned my addr 172.21.15.150:0/105638755 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:41.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.277+0000 7f8bdd4b8640 1 --2- 172.21.15.150:0/105638755 >> v2:172.21.15.157:3300/0 conn(0x7f8bd80cd250 0x7f8bd815ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:41.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.277+0000 7f8bd77fe640 1 -- 172.21.15.150:0/105638755 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2375025114 0 0) 0x7f8bd80cf3b0 con 0x7f8bd8155320 2024-03-17T23:47:41.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.277+0000 7f8bd77fe640 1 -- 172.21.15.150:0/105638755 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bc8003520 con 0x7f8bd8155320 2024-03-17T23:47:41.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.277+0000 7f8bd7fff640 1 -- 172.21.15.150:0/105638755 >> v2:172.21.15.157:3300/0 conn(0x7f8bd80cd250 msgr2=0x7f8bd815ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:41.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.277+0000 7f8bd7fff640 1 --2- 172.21.15.150:0/105638755 >> v2:172.21.15.157:3300/0 conn(0x7f8bd80cd250 0x7f8bd815ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.277+0000 7f8bd7fff640 1 -- 172.21.15.150:0/105638755 >> v1:172.21.15.150:6789/0 conn(0x7f8bd8155320 legacy=0x7f8bd806dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:41.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.277+0000 7f8bd7fff640 1 -- 172.21.15.150:0/105638755 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd815c7a0 con 0x7f8bd815bfd0 2024-03-17T23:47:41.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.278+0000 7f8bd7fff640 1 --2- 172.21.15.150:0/105638755 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bd815bfd0 0x7f8bd80ccd10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8bd806c0f0 tx=0x7f8bc402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=248e86377fe1db57 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:41.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.278+0000 7f8bd77fe640 1 -- 172.21.15.150:0/105638755 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4035020 con 0x7f8bd815bfd0 2024-03-17T23:47:41.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.278+0000 7f8bd77fe640 1 -- 172.21.15.150:0/105638755 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bc4031830 con 0x7f8bd815bfd0 2024-03-17T23:47:41.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.278+0000 7f8bd77fe640 1 -- 172.21.15.150:0/105638755 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4031c00 con 0x7f8bd815bfd0 2024-03-17T23:47:41.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.279+0000 7f8bdef1b640 1 -- 172.21.15.150:0/105638755 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bd815bfd0 msgr2=0x7f8bd80ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:41.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.279+0000 7f8bdef1b640 1 --2- 172.21.15.150:0/105638755 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bd815bfd0 0x7f8bd80ccd10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8bd806c0f0 tx=0x7f8bc402b8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.279+0000 7f8bdef1b640 1 -- 172.21.15.150:0/105638755 shutdown_connections 2024-03-17T23:47:41.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.279+0000 7f8bdef1b640 1 --2- 172.21.15.150:0/105638755 >> v2:172.21.15.157:3300/0 conn(0x7f8bd80cd250 0x7f8bd815ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.279+0000 7f8bdef1b640 1 --2- 172.21.15.150:0/105638755 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bd815bfd0 0x7f8bd80ccd10 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.279+0000 7f8bdef1b640 1 -- 172.21.15.150:0/105638755 >> 172.21.15.150:0/105638755 conn(0x7f8bd806b190 msgr2=0x7f8bd806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:41.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.279+0000 7f8bdef1b640 1 -- 172.21.15.150:0/105638755 shutdown_connections 2024-03-17T23:47:41.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.279+0000 7f8bdef1b640 1 -- 172.21.15.150:0/105638755 wait complete. 2024-03-17T23:47:41.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.280+0000 7f8bdef1b640 1 Processor -- start 2024-03-17T23:47:41.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.280+0000 7f8bdef1b640 1 -- start start 2024-03-17T23:47:41.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.281+0000 7f8bdef1b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8bd80cd250 0x7f8bd814dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:41.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.281+0000 7f8bdef1b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bd8155320 0x7f8bd8151260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:41.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.281+0000 7f8bdef1b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8bd815bfd0 0x7f8bd81f0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:41.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.281+0000 7f8bdef1b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd8155880 con 0x7f8bd8155320 2024-03-17T23:47:41.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.281+0000 7f8bdef1b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd8155700 con 0x7f8bd815bfd0 2024-03-17T23:47:41.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.281+0000 7f8bdef1b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8bd815d200 con 0x7f8bd80cd250 2024-03-17T23:47:41.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.281+0000 7f8bd7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bd8155320 0x7f8bd8151260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:41.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.281+0000 7f8bd7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bd8155320 0x7f8bd8151260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46274/0 (socket says 172.21.15.150:46274) 2024-03-17T23:47:41.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.281+0000 7f8bd7fff640 1 -- 172.21.15.150:0/1519579363 learned_addr learned my addr 172.21.15.150:0/1519579363 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:41.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.281+0000 7f8bdccb7640 1 --2- 172.21.15.150:0/1519579363 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8bd80cd250 0x7f8bd814dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:41.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.281+0000 7f8bdd4b8640 1 --2- 172.21.15.150:0/1519579363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8bd815bfd0 0x7f8bd81f0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:41.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.281+0000 7f8bd7fff640 1 -- 172.21.15.150:0/1519579363 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8bd80cd250 msgr2=0x7f8bd814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:41.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.281+0000 7f8bd7fff640 1 --2- 172.21.15.150:0/1519579363 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8bd80cd250 0x7f8bd814dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.281+0000 7f8bd7fff640 1 -- 172.21.15.150:0/1519579363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8bd815bfd0 msgr2=0x7f8bd81f0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:41.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.281+0000 7f8bd7fff640 1 --2- 172.21.15.150:0/1519579363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8bd815bfd0 0x7f8bd81f0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.281+0000 7f8bd7fff640 1 -- 172.21.15.150:0/1519579363 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd8157590 con 0x7f8bd8155320 2024-03-17T23:47:41.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.281+0000 7f8bd7fff640 1 --2- 172.21.15.150:0/1519579363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bd8155320 0x7f8bd8151260 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4000c00 tx=0x7f8bc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:41.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.282+0000 7f8bd57fa640 1 -- 172.21.15.150:0/1519579363 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4040070 con 0x7f8bd8155320 2024-03-17T23:47:41.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.282+0000 7f8bd57fa640 1 -- 172.21.15.150:0/1519579363 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bc403a830 con 0x7f8bd8155320 2024-03-17T23:47:41.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.282+0000 7f8bdef1b640 1 -- 172.21.15.150:0/1519579363 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bd8157870 con 0x7f8bd8155320 2024-03-17T23:47:41.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.282+0000 7f8bd57fa640 1 -- 172.21.15.150:0/1519579363 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4035040 con 0x7f8bd8155320 2024-03-17T23:47:41.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.282+0000 7f8bdef1b640 1 -- 172.21.15.150:0/1519579363 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bd81517a0 con 0x7f8bd8155320 2024-03-17T23:47:41.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.283+0000 7f8bdef1b640 1 -- 172.21.15.150:0/1519579363 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bd80681a0 con 0x7f8bd8155320 2024-03-17T23:47:41.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.283+0000 7f8bd57fa640 1 -- 172.21.15.150:0/1519579363 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8bc404b060 con 0x7f8bd8155320 2024-03-17T23:47:41.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.283+0000 7f8bd57fa640 1 --2- 172.21.15.150:0/1519579363 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8bbc06bbd0 0x7f8bbc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:41.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.284+0000 7f8bdccb7640 1 --2- 172.21.15.150:0/1519579363 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8bbc06bbd0 0x7f8bbc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:41.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.284+0000 7f8bd57fa640 1 -- 172.21.15.150:0/1519579363 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f8bc40ac450 con 0x7f8bd8155320 2024-03-17T23:47:41.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.284+0000 7f8bdccb7640 1 --2- 172.21.15.150:0/1519579363 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8bbc06bbd0 0x7f8bbc06e090 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc005880 tx=0x7f8bcc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:41.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.288+0000 7f8bd57fa640 1 -- 172.21.15.150:0/1519579363 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8bc407a580 con 0x7f8bd8155320 2024-03-17T23:47:41.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.462+0000 7f8bdef1b640 1 -- 172.21.15.150:0/1519579363 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8bd806e900 con 0x7f8bd8155320 2024-03-17T23:47:41.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.463+0000 7f8bd57fa640 1 -- 172.21.15.150:0/1519579363 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7f8bc407e620 con 0x7f8bd8155320 2024-03-17T23:47:41.463 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:41.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.465+0000 7f8bdef1b640 1 -- 172.21.15.150:0/1519579363 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8bbc06bbd0 msgr2=0x7f8bbc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:41.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.465+0000 7f8bdef1b640 1 --2- 172.21.15.150:0/1519579363 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8bbc06bbd0 0x7f8bbc06e090 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc005880 tx=0x7f8bcc009040 comp rx=0 tx=0).stop 2024-03-17T23:47:41.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.465+0000 7f8bdef1b640 1 -- 172.21.15.150:0/1519579363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bd8155320 msgr2=0x7f8bd8151260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:41.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.465+0000 7f8bdef1b640 1 --2- 172.21.15.150:0/1519579363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bd8155320 0x7f8bd8151260 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4000c00 tx=0x7f8bc40046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.465+0000 7f8bdef1b640 1 -- 172.21.15.150:0/1519579363 shutdown_connections 2024-03-17T23:47:41.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.465+0000 7f8bdef1b640 1 --2- 172.21.15.150:0/1519579363 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8bbc06bbd0 0x7f8bbc06e090 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.465+0000 7f8bdef1b640 1 --2- 172.21.15.150:0/1519579363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8bd815bfd0 0x7f8bd81f0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.465+0000 7f8bdef1b640 1 --2- 172.21.15.150:0/1519579363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8bd8155320 0x7f8bd8151260 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.465+0000 7f8bdef1b640 1 --2- 172.21.15.150:0/1519579363 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8bd80cd250 0x7f8bd814dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.465+0000 7f8bdef1b640 1 -- 172.21.15.150:0/1519579363 >> 172.21.15.150:0/1519579363 conn(0x7f8bd806b190 msgr2=0x7f8bd815d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:41.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.465+0000 7f8bdef1b640 1 -- 172.21.15.150:0/1519579363 shutdown_connections 2024-03-17T23:47:41.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.465+0000 7f8bdef1b640 1 -- 172.21.15.150:0/1519579363 wait complete. 2024-03-17T23:47:41.476 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:47:41.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.580+0000 7fbbfcc12640 1 Processor -- start 2024-03-17T23:47:41.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.580+0000 7fbbfcc12640 1 -- start start 2024-03-17T23:47:41.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.580+0000 7fbbfcc12640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbbf81615d0 0x7fbbf80d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:41.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.580+0000 7fbbfcc12640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbbf80d3910 0x7fbbf80d0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:41.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.580+0000 7fbbfcc12640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbf80d2870 con 0x7fbbf80b9660 2024-03-17T23:47:41.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.580+0000 7fbbfcc12640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fbbf81619b0 con 0x7fbbf80d3910 2024-03-17T23:47:41.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.580+0000 7fbbfcc12640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbbf80b7e80 con 0x7fbbf81615d0 2024-03-17T23:47:41.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.581+0000 7fbbeffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbbf80d3910 0x7fbbf80d0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:41.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.581+0000 7fbbeffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbbf80d3910 0x7fbbf80d0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46290/0 (socket says 172.21.15.150:46290) 2024-03-17T23:47:41.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.581+0000 7fbbeffff640 1 -- 172.21.15.150:0/1771051179 learned_addr learned my addr 172.21.15.150:0/1771051179 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:41.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.581+0000 7fbbf659c640 1 --2- 172.21.15.150:0/1771051179 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbbf81615d0 0x7fbbf80d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:41.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.581+0000 7fbbeffff640 1 -- 172.21.15.150:0/1771051179 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbbf81615d0 msgr2=0x7fbbf80d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:41.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.581+0000 7fbbeffff640 1 --2- 172.21.15.150:0/1771051179 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbbf81615d0 0x7fbbf80d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.581+0000 7fbbeffff640 1 -- 172.21.15.150:0/1771051179 >> v1:172.21.15.157:6789/0 conn(0x7fbbf80b9660 legacy=0x7fbbf806c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:41.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.581+0000 7fbbeffff640 1 -- 172.21.15.150:0/1771051179 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbf80d2b50 con 0x7fbbf80d3910 2024-03-17T23:47:41.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.582+0000 7fbbeffff640 1 --2- 172.21.15.150:0/1771051179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbf80d3910 0x7fbbf80d0720 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0009870 tx=0x7fbbe002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8302acc1bd2f465c server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:41.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.582+0000 7fbbf5d9b640 1 -- 172.21.15.150:0/1771051179 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0034070 con 0x7fbbf80d3910 2024-03-17T23:47:41.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.582+0000 7fbbf5d9b640 1 -- 172.21.15.150:0/1771051179 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbe0030840 con 0x7fbbf80d3910 2024-03-17T23:47:41.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.582+0000 7fbbf5d9b640 1 -- 172.21.15.150:0/1771051179 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0030c10 con 0x7fbbf80d3910 2024-03-17T23:47:41.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.582+0000 7fbbfcc12640 1 -- 172.21.15.150:0/1771051179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbf80d3910 msgr2=0x7fbbf80d0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:41.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.582+0000 7fbbfcc12640 1 --2- 172.21.15.150:0/1771051179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbf80d3910 0x7fbbf80d0720 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0009870 tx=0x7fbbe002b980 comp rx=0 tx=0).stop 2024-03-17T23:47:41.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.582+0000 7fbbfcc12640 1 -- 172.21.15.150:0/1771051179 shutdown_connections 2024-03-17T23:47:41.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.582+0000 7fbbfcc12640 1 --2- 172.21.15.150:0/1771051179 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbf80d3910 0x7fbbf80d0720 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.582+0000 7fbbfcc12640 1 --2- 172.21.15.150:0/1771051179 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbbf81615d0 0x7fbbf80d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.582+0000 7fbbfcc12640 1 -- 172.21.15.150:0/1771051179 >> 172.21.15.150:0/1771051179 conn(0x7fbbf80730b0 msgr2=0x7fbbf80b5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:41.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.582+0000 7fbbfcc12640 1 -- 172.21.15.150:0/1771051179 shutdown_connections 2024-03-17T23:47:41.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.582+0000 7fbbfcc12640 1 -- 172.21.15.150:0/1771051179 wait complete. 2024-03-17T23:47:41.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.584+0000 7fbbfcc12640 1 Processor -- start 2024-03-17T23:47:41.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.584+0000 7fbbfcc12640 1 -- start start 2024-03-17T23:47:41.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.584+0000 7fbbfcc12640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbbf80b9660 0x7fbbf8152360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:41.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.584+0000 7fbbfcc12640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbbf80d3910 0x7fbbf814c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:41.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.584+0000 7fbbfcc12640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbf81615d0 0x7fbbf80b8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:41.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.584+0000 7fbbfcc12640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf80d33f0 con 0x7fbbf81615d0 2024-03-17T23:47:41.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.584+0000 7fbbfcc12640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf80d3110 con 0x7fbbf80b9660 2024-03-17T23:47:41.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.584+0000 7fbbfcc12640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbbf80d2560 con 0x7fbbf80d3910 2024-03-17T23:47:41.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.585+0000 7fbbf6d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbf81615d0 0x7fbbf80b8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:41.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.585+0000 7fbbf659c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbbf80b9660 0x7fbbf8152360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:41.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.585+0000 7fbbf6d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbf81615d0 0x7fbbf80b8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46302/0 (socket says 172.21.15.150:46302) 2024-03-17T23:47:41.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.585+0000 7fbbf6d9d640 1 -- 172.21.15.150:0/3146863492 learned_addr learned my addr 172.21.15.150:0/3146863492 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:41.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.585+0000 7fbbeffff640 1 --2- 172.21.15.150:0/3146863492 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbbf80d3910 0x7fbbf814c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:41.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.585+0000 7fbbf6d9d640 1 -- 172.21.15.150:0/3146863492 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbbf80d3910 msgr2=0x7fbbf814c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:41.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.585+0000 7fbbf6d9d640 1 --2- 172.21.15.150:0/3146863492 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbbf80d3910 0x7fbbf814c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.585+0000 7fbbf6d9d640 1 -- 172.21.15.150:0/3146863492 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbbf80b9660 msgr2=0x7fbbf8152360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:41.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.585+0000 7fbbf6d9d640 1 --2- 172.21.15.150:0/3146863492 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbbf80b9660 0x7fbbf8152360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.585+0000 7fbbf6d9d640 1 -- 172.21.15.150:0/3146863492 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbf814c7a0 con 0x7fbbf81615d0 2024-03-17T23:47:41.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.585+0000 7fbbf6d9d640 1 --2- 172.21.15.150:0/3146863492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbf81615d0 0x7fbbf80b8540 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fbbf000c7a0 tx=0x7fbbf000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:41.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.585+0000 7fbbeeffd640 1 -- 172.21.15.150:0/3146863492 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0003b70 con 0x7fbbf81615d0 2024-03-17T23:47:41.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.585+0000 7fbbeeffd640 1 -- 172.21.15.150:0/3146863492 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbf0003d30 con 0x7fbbf81615d0 2024-03-17T23:47:41.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.585+0000 7fbbfcc12640 1 -- 172.21.15.150:0/3146863492 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbf80cb310 con 0x7fbbf81615d0 2024-03-17T23:47:41.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.585+0000 7fbbeeffd640 1 -- 172.21.15.150:0/3146863492 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf000aa90 con 0x7fbbf81615d0 2024-03-17T23:47:41.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.585+0000 7fbbfcc12640 1 -- 172.21.15.150:0/3146863492 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbf80cb8d0 con 0x7fbbf81615d0 2024-03-17T23:47:41.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.586+0000 7fbbfcc12640 1 -- 172.21.15.150:0/3146863492 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbf80688b0 con 0x7fbbf81615d0 2024-03-17T23:47:41.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.586+0000 7fbbeeffd640 1 -- 172.21.15.150:0/3146863492 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbbf000ac30 con 0x7fbbf81615d0 2024-03-17T23:47:41.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.587+0000 7fbbeeffd640 1 --2- 172.21.15.150:0/3146863492 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbbd806bc00 0x7fbbd806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:41.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.587+0000 7fbbeeffd640 1 -- 172.21.15.150:0/3146863492 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fbbf008ac00 con 0x7fbbf81615d0 2024-03-17T23:47:41.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.587+0000 7fbbf659c640 1 --2- 172.21.15.150:0/3146863492 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbbd806bc00 0x7fbbd806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:41.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.587+0000 7fbbf659c640 1 --2- 172.21.15.150:0/3146863492 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbbd806bc00 0x7fbbd806e0c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8009aa0 tx=0x7fbbe8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:41.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.591+0000 7fbbeeffd640 1 -- 172.21.15.150:0/3146863492 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbbf0091080 con 0x7fbbf81615d0 2024-03-17T23:47:41.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.766+0000 7fbbfcc12640 1 -- 172.21.15.150:0/3146863492 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbbf815e540 con 0x7fbbf81615d0 2024-03-17T23:47:41.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.766+0000 7fbbeeffd640 1 -- 172.21.15.150:0/3146863492 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7fbbf814c7a0 con 0x7fbbf81615d0 2024-03-17T23:47:41.767 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:47:41.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.768+0000 7fbbfcc12640 1 -- 172.21.15.150:0/3146863492 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbbd806bc00 msgr2=0x7fbbd806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:41.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.768+0000 7fbbfcc12640 1 --2- 172.21.15.150:0/3146863492 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbbd806bc00 0x7fbbd806e0c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8009aa0 tx=0x7fbbe8009190 comp rx=0 tx=0).stop 2024-03-17T23:47:41.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.768+0000 7fbbfcc12640 1 -- 172.21.15.150:0/3146863492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbf81615d0 msgr2=0x7fbbf80b8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:41.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.768+0000 7fbbfcc12640 1 --2- 172.21.15.150:0/3146863492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbf81615d0 0x7fbbf80b8540 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fbbf000c7a0 tx=0x7fbbf000cc60 comp rx=0 tx=0).stop 2024-03-17T23:47:41.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.768+0000 7fbbfcc12640 1 -- 172.21.15.150:0/3146863492 shutdown_connections 2024-03-17T23:47:41.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.768+0000 7fbbfcc12640 1 --2- 172.21.15.150:0/3146863492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbf81615d0 0x7fbbf80b8540 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.768+0000 7fbbfcc12640 1 --2- 172.21.15.150:0/3146863492 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbbf80d3910 0x7fbbf814c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.768+0000 7fbbfcc12640 1 --2- 172.21.15.150:0/3146863492 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbbd806bc00 0x7fbbd806e0c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.768+0000 7fbbfcc12640 1 --2- 172.21.15.150:0/3146863492 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbbf80b9660 0x7fbbf8152360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.768+0000 7fbbfcc12640 1 -- 172.21.15.150:0/3146863492 >> 172.21.15.150:0/3146863492 conn(0x7fbbf80730b0 msgr2=0x7fbbf80d3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:41.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.768+0000 7fbbfcc12640 1 -- 172.21.15.150:0/3146863492 shutdown_connections 2024-03-17T23:47:41.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.768+0000 7fbbfcc12640 1 -- 172.21.15.150:0/3146863492 wait complete. 2024-03-17T23:47:41.780 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-17T23:47:41.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.883+0000 7f9347adc640 1 Processor -- start 2024-03-17T23:47:41.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.883+0000 7f9347adc640 1 -- start start 2024-03-17T23:47:41.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.884+0000 7f9347adc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9340144bb0 0x7f934015ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:41.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.884+0000 7f9347adc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93400ce6a0 0x7f9340165a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:41.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.884+0000 7f9347adc640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9340145170 con 0x7f934015d390 2024-03-17T23:47:41.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.884+0000 7f9347adc640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f9340145450 con 0x7f9340144bb0 2024-03-17T23:47:41.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.884+0000 7f9347adc640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9340145730 con 0x7f93400ce6a0 2024-03-17T23:47:41.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.884+0000 7f9345878640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9340144bb0 0x7f934015ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:41.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.884+0000 7f9345077640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f934015d390 0x7f93400cdf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59216/0 (socket says 172.21.15.150:59216) 2024-03-17T23:47:41.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.884+0000 7f9345077640 1 -- 172.21.15.150:0/2369056022 learned_addr learned my addr 172.21.15.150:0/2369056022 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:41.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.884+0000 7f9346079640 1 --2- 172.21.15.150:0/2369056022 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93400ce6a0 0x7f9340165a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:41.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.884+0000 7f9345878640 1 -- 172.21.15.150:0/2369056022 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93400ce6a0 msgr2=0x7f9340165a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:41.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.884+0000 7f9345878640 1 --2- 172.21.15.150:0/2369056022 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93400ce6a0 0x7f9340165a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.884+0000 7f9345878640 1 -- 172.21.15.150:0/2369056022 >> v1:172.21.15.150:6789/0 conn(0x7f934015d390 legacy=0x7f93400cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:41.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.884+0000 7f9345878640 1 -- 172.21.15.150:0/2369056022 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f934006bd60 con 0x7f9340144bb0 2024-03-17T23:47:41.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.884+0000 7f9344876640 1 -- 172.21.15.150:0/2369056022 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 33645225 0 0) 0x7f9340145170 con 0x7f934015d390 2024-03-17T23:47:41.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.885+0000 7f9345878640 1 --2- 172.21.15.150:0/2369056022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9340144bb0 0x7f934015ce50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f933c0097a0 tx=0x7f933c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1359975c708d323 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:41.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.885+0000 7f9344876640 1 -- 172.21.15.150:0/2369056022 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933c0315a0 con 0x7f9340144bb0 2024-03-17T23:47:41.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.885+0000 7f9344876640 1 -- 172.21.15.150:0/2369056022 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f933c037070 con 0x7f9340144bb0 2024-03-17T23:47:41.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.885+0000 7f9344876640 1 -- 172.21.15.150:0/2369056022 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933c031960 con 0x7f9340144bb0 2024-03-17T23:47:41.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.886+0000 7f9347adc640 1 -- 172.21.15.150:0/2369056022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9340144bb0 msgr2=0x7f934015ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:41.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.886+0000 7f9347adc640 1 --2- 172.21.15.150:0/2369056022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9340144bb0 0x7f934015ce50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f933c0097a0 tx=0x7f933c02b840 comp rx=0 tx=0).stop 2024-03-17T23:47:41.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.886+0000 7f9347adc640 1 -- 172.21.15.150:0/2369056022 shutdown_connections 2024-03-17T23:47:41.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.886+0000 7f9347adc640 1 --2- 172.21.15.150:0/2369056022 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f93400ce6a0 0x7f9340165a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.886+0000 7f9347adc640 1 --2- 172.21.15.150:0/2369056022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9340144bb0 0x7f934015ce50 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.886+0000 7f9347adc640 1 -- 172.21.15.150:0/2369056022 >> 172.21.15.150:0/2369056022 conn(0x7f934006b190 msgr2=0x7f934006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:41.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.886+0000 7f9347adc640 1 -- 172.21.15.150:0/2369056022 shutdown_connections 2024-03-17T23:47:41.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.886+0000 7f9347adc640 1 -- 172.21.15.150:0/2369056022 wait complete. 2024-03-17T23:47:41.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.887+0000 7f9347adc640 1 Processor -- start 2024-03-17T23:47:41.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.887+0000 7f9347adc640 1 -- start start 2024-03-17T23:47:41.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.887+0000 7f9347adc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93400ce6a0 0x7f93400aeb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:41.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.887+0000 7f9347adc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9340144bb0 0x7f93400adfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:41.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.887+0000 7f9347adc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f934015d390 0x7f93401f28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:41.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.887+0000 7f9347adc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9340165f70 con 0x7f93400ce6a0 2024-03-17T23:47:41.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.887+0000 7f9347adc640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f934006f610 con 0x7f9340144bb0 2024-03-17T23:47:41.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.887+0000 7f9347adc640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f93401660f0 con 0x7f934015d390 2024-03-17T23:47:41.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.888+0000 7f9345878640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93400ce6a0 0x7f93400aeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:41.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.888+0000 7f9345878640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93400ce6a0 0x7f93400aeb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46318/0 (socket says 172.21.15.150:46318) 2024-03-17T23:47:41.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.888+0000 7f9345878640 1 -- 172.21.15.150:0/3295480658 learned_addr learned my addr 172.21.15.150:0/3295480658 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:41.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.888+0000 7f9346079640 1 --2- 172.21.15.150:0/3295480658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f934015d390 0x7f93401f28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:41.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.888+0000 7f9345077640 1 --2- 172.21.15.150:0/3295480658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9340144bb0 0x7f93400adfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:41.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.888+0000 7f9345878640 1 -- 172.21.15.150:0/3295480658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f934015d390 msgr2=0x7f93401f28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:41.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.888+0000 7f9345878640 1 --2- 172.21.15.150:0/3295480658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f934015d390 0x7f93401f28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.888+0000 7f9345878640 1 -- 172.21.15.150:0/3295480658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9340144bb0 msgr2=0x7f93400adfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:41.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.888+0000 7f9345878640 1 --2- 172.21.15.150:0/3295480658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9340144bb0 0x7f93400adfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:41.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.888+0000 7f9345878640 1 -- 172.21.15.150:0/3295480658 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f934013f620 con 0x7f93400ce6a0 2024-03-17T23:47:41.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.888+0000 7f9345077640 1 --2- 172.21.15.150:0/3295480658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9340144bb0 0x7f93400adfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:47:41.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.888+0000 7f9345878640 1 --2- 172.21.15.150:0/3295480658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93400ce6a0 0x7f93400aeb70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f933c0098c0 tx=0x7f933c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:41.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.888+0000 7f93367fc640 1 -- 172.21.15.150:0/3295480658 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933c032630 con 0x7f93400ce6a0 2024-03-17T23:47:41.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.888+0000 7f93367fc640 1 -- 172.21.15.150:0/3295480658 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f933c0327d0 con 0x7f93400ce6a0 2024-03-17T23:47:41.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.888+0000 7f9347adc640 1 -- 172.21.15.150:0/3295480658 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f934013f8e0 con 0x7f93400ce6a0 2024-03-17T23:47:41.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.888+0000 7f93367fc640 1 -- 172.21.15.150:0/3295480658 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933c032bc0 con 0x7f93400ce6a0 2024-03-17T23:47:41.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.888+0000 7f9347adc640 1 -- 172.21.15.150:0/3295480658 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f934013d1b0 con 0x7f93400ce6a0 2024-03-17T23:47:41.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.889+0000 7f9347adc640 1 -- 172.21.15.150:0/3295480658 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f934013fe00 con 0x7f93400ce6a0 2024-03-17T23:47:41.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.889+0000 7f93367fc640 1 -- 172.21.15.150:0/3295480658 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f933c04e430 con 0x7f93400ce6a0 2024-03-17T23:47:41.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.890+0000 7f93367fc640 1 --2- 172.21.15.150:0/3295480658 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f932406bbd0 0x7f932406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:41.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.890+0000 7f93367fc640 1 -- 172.21.15.150:0/3295480658 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f933c02f030 con 0x7f93400ce6a0 2024-03-17T23:47:41.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.890+0000 7f9345077640 1 --2- 172.21.15.150:0/3295480658 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f932406bbd0 0x7f932406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:41.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.890+0000 7f9345077640 1 --2- 172.21.15.150:0/3295480658 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f932406bbd0 0x7f932406e090 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f93400b13a0 tx=0x7f9330007ca0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:41.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:41.894+0000 7f93367fc640 1 -- 172.21.15.150:0/3295480658 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f934013fe00 con 0x7f93400ce6a0 2024-03-17T23:47:42.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.069+0000 7f9347adc640 1 -- 172.21.15.150:0/3295480658 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f93400681a0 con 0x7f93400ce6a0 2024-03-17T23:47:42.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.069+0000 7f93367fc640 1 -- 172.21.15.150:0/3295480658 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7f933c0ac020 con 0x7f93400ce6a0 2024-03-17T23:47:42.070 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:42.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.071+0000 7f9347adc640 1 -- 172.21.15.150:0/3295480658 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f932406bbd0 msgr2=0x7f932406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:42.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.071+0000 7f9347adc640 1 --2- 172.21.15.150:0/3295480658 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f932406bbd0 0x7f932406e090 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f93400b13a0 tx=0x7f9330007ca0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.071+0000 7f9347adc640 1 -- 172.21.15.150:0/3295480658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93400ce6a0 msgr2=0x7f93400aeb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:42.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.071+0000 7f9347adc640 1 --2- 172.21.15.150:0/3295480658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93400ce6a0 0x7f93400aeb70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f933c0098c0 tx=0x7f933c0046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.071+0000 7f9347adc640 1 -- 172.21.15.150:0/3295480658 shutdown_connections 2024-03-17T23:47:42.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.071+0000 7f9347adc640 1 --2- 172.21.15.150:0/3295480658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f934015d390 0x7f93401f28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.071+0000 7f9347adc640 1 --2- 172.21.15.150:0/3295480658 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f932406bbd0 0x7f932406e090 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.071+0000 7f9347adc640 1 --2- 172.21.15.150:0/3295480658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9340144bb0 0x7f93400adfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.071+0000 7f9347adc640 1 --2- 172.21.15.150:0/3295480658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93400ce6a0 0x7f93400aeb70 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.071+0000 7f9347adc640 1 -- 172.21.15.150:0/3295480658 >> 172.21.15.150:0/3295480658 conn(0x7f934006b190 msgr2=0x7f934015c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:42.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.071+0000 7f9347adc640 1 -- 172.21.15.150:0/3295480658 shutdown_connections 2024-03-17T23:47:42.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.071+0000 7f9347adc640 1 -- 172.21.15.150:0/3295480658 wait complete. 2024-03-17T23:47:42.083 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-17T23:47:42.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.183+0000 7f69106a1640 1 Processor -- start 2024-03-17T23:47:42.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.184+0000 7f69106a1640 1 -- start start 2024-03-17T23:47:42.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.184+0000 7f69106a1640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f69080cf100 0x7f690815bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:42.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.184+0000 7f69106a1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f690815cac0 0x7f690815cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:42.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.184+0000 7f69106a1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f690806de60 con 0x7f69080ccf90 2024-03-17T23:47:42.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.184+0000 7f69106a1640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f690806e420 con 0x7f690815cac0 2024-03-17T23:47:42.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.184+0000 7f69106a1640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f69080cf8a0 con 0x7f69080cf100 2024-03-17T23:47:42.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.185+0000 7f690ec3e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f690815cac0 0x7f690815cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:42.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.185+0000 7f690e43d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f69080ccf90 0x7f69080cd370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59230/0 (socket says 172.21.15.150:59230) 2024-03-17T23:47:42.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.185+0000 7f690e43d640 1 -- 172.21.15.150:0/1288557761 learned_addr learned my addr 172.21.15.150:0/1288557761 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:42.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.185+0000 7f690dc3c640 1 --2- 172.21.15.150:0/1288557761 >> v2:172.21.15.157:3300/0 conn(0x7f69080cf100 0x7f690815bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:42.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.185+0000 7f690ec3e640 1 -- 172.21.15.150:0/1288557761 >> v2:172.21.15.157:3300/0 conn(0x7f69080cf100 msgr2=0x7f690815bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:42.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.185+0000 7f690ec3e640 1 --2- 172.21.15.150:0/1288557761 >> v2:172.21.15.157:3300/0 conn(0x7f69080cf100 0x7f690815bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.185+0000 7f690d43b640 1 -- 172.21.15.150:0/1288557761 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3817798616 0 0) 0x7f690806de60 con 0x7f69080ccf90 2024-03-17T23:47:42.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.185+0000 7f690ec3e640 1 -- 172.21.15.150:0/1288557761 >> v1:172.21.15.150:6789/0 conn(0x7f69080ccf90 legacy=0x7f69080cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:42.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.185+0000 7f690ec3e640 1 -- 172.21.15.150:0/1288557761 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f690806e140 con 0x7f690815cac0 2024-03-17T23:47:42.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.186+0000 7f690ec3e640 1 --2- 172.21.15.150:0/1288557761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f690815cac0 0x7f690815cf50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f69080691e0 tx=0x7f690002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4868eb8e9f16760 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:42.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.186+0000 7f690d43b640 1 -- 172.21.15.150:0/1288557761 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6900039070 con 0x7f690815cac0 2024-03-17T23:47:42.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.186+0000 7f690d43b640 1 -- 172.21.15.150:0/1288557761 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f690002cbf0 con 0x7f690815cac0 2024-03-17T23:47:42.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.186+0000 7f690d43b640 1 -- 172.21.15.150:0/1288557761 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6900034440 con 0x7f690815cac0 2024-03-17T23:47:42.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.186+0000 7f69106a1640 1 -- 172.21.15.150:0/1288557761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f690815cac0 msgr2=0x7f690815cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:42.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.186+0000 7f69106a1640 1 --2- 172.21.15.150:0/1288557761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f690815cac0 0x7f690815cf50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f69080691e0 tx=0x7f690002be40 comp rx=0 tx=0).stop 2024-03-17T23:47:42.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.186+0000 7f69106a1640 1 -- 172.21.15.150:0/1288557761 shutdown_connections 2024-03-17T23:47:42.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.186+0000 7f69106a1640 1 --2- 172.21.15.150:0/1288557761 >> v2:172.21.15.157:3300/0 conn(0x7f69080cf100 0x7f690815bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.186+0000 7f69106a1640 1 --2- 172.21.15.150:0/1288557761 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f690815cac0 0x7f690815cf50 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.186+0000 7f69106a1640 1 -- 172.21.15.150:0/1288557761 >> 172.21.15.150:0/1288557761 conn(0x7f690806b190 msgr2=0x7f690806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:42.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.186+0000 7f69106a1640 1 -- 172.21.15.150:0/1288557761 shutdown_connections 2024-03-17T23:47:42.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.186+0000 7f69106a1640 1 -- 172.21.15.150:0/1288557761 wait complete. 2024-03-17T23:47:42.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.188+0000 7f69106a1640 1 Processor -- start 2024-03-17T23:47:42.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.188+0000 7f69106a1640 1 -- start start 2024-03-17T23:47:42.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.188+0000 7f69106a1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69080ccf90 0x7f6908157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:42.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.188+0000 7f69106a1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69080cf100 0x7f6908158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:42.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.188+0000 7f69106a1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f690815cac0 0x7f6908151f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:42.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.188+0000 7f69106a1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f69080cf5b0 con 0x7f69080cf100 2024-03-17T23:47:42.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.188+0000 7f69106a1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f690815c220 con 0x7f69080ccf90 2024-03-17T23:47:42.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.188+0000 7f69106a1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6908155530 con 0x7f690815cac0 2024-03-17T23:47:42.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.188+0000 7f690dc3c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69080cf100 0x7f6908158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:42.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.188+0000 7f690dc3c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69080cf100 0x7f6908158130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46336/0 (socket says 172.21.15.150:46336) 2024-03-17T23:47:42.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.188+0000 7f690dc3c640 1 -- 172.21.15.150:0/1391582554 learned_addr learned my addr 172.21.15.150:0/1391582554 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:42.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.188+0000 7f690e43d640 1 --2- 172.21.15.150:0/1391582554 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69080ccf90 0x7f6908157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:42.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.188+0000 7f690ec3e640 1 --2- 172.21.15.150:0/1391582554 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f690815cac0 0x7f6908151f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:42.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.189+0000 7f690dc3c640 1 -- 172.21.15.150:0/1391582554 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f690815cac0 msgr2=0x7f6908151f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:42.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.189+0000 7f690dc3c640 1 --2- 172.21.15.150:0/1391582554 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f690815cac0 0x7f6908151f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.189+0000 7f690dc3c640 1 -- 172.21.15.150:0/1391582554 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69080ccf90 msgr2=0x7f6908157bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:42.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.189+0000 7f690dc3c640 1 --2- 172.21.15.150:0/1391582554 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69080ccf90 0x7f6908157bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.189+0000 7f690dc3c640 1 -- 172.21.15.150:0/1391582554 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6908150eb0 con 0x7f69080cf100 2024-03-17T23:47:42.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.189+0000 7f690dc3c640 1 --2- 172.21.15.150:0/1391582554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69080cf100 0x7f6908158130 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f68f800e8d0 tx=0x7f68f800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:42.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.189+0000 7f68feffd640 1 -- 172.21.15.150:0/1391582554 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f8017070 con 0x7f69080cf100 2024-03-17T23:47:42.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.189+0000 7f68feffd640 1 -- 172.21.15.150:0/1391582554 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68f800c8e0 con 0x7f69080cf100 2024-03-17T23:47:42.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.189+0000 7f69106a1640 1 -- 172.21.15.150:0/1391582554 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6908151190 con 0x7f69080cf100 2024-03-17T23:47:42.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.189+0000 7f68feffd640 1 -- 172.21.15.150:0/1391582554 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f800ccd0 con 0x7f69080cf100 2024-03-17T23:47:42.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.189+0000 7f69106a1640 1 -- 172.21.15.150:0/1391582554 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6908151750 con 0x7f69080cf100 2024-03-17T23:47:42.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.190+0000 7f69106a1640 1 -- 172.21.15.150:0/1391582554 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69080681a0 con 0x7f69080cf100 2024-03-17T23:47:42.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.190+0000 7f68feffd640 1 -- 172.21.15.150:0/1391582554 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f68f801a070 con 0x7f69080cf100 2024-03-17T23:47:42.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.191+0000 7f68feffd640 1 --2- 172.21.15.150:0/1391582554 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f68ec06bbd0 0x7f68ec06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:42.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.191+0000 7f68feffd640 1 -- 172.21.15.150:0/1391582554 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f68f808aff0 con 0x7f69080cf100 2024-03-17T23:47:42.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.191+0000 7f690e43d640 1 --2- 172.21.15.150:0/1391582554 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f68ec06bbd0 0x7f68ec06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:42.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.191+0000 7f690e43d640 1 --2- 172.21.15.150:0/1391582554 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f68ec06bbd0 0x7f68ec06e090 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6904007fd0 tx=0x7f6904007de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:42.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.195+0000 7f68feffd640 1 -- 172.21.15.150:0/1391582554 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f68f8059000 con 0x7f69080cf100 2024-03-17T23:47:42.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.381+0000 7f69106a1640 1 -- 172.21.15.150:0/1391582554 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f690806fff0 con 0x7f69080cf100 2024-03-17T23:47:42.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.382+0000 7f68feffd640 1 -- 172.21.15.150:0/1391582554 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) ==== 80+0+0 (secure 0 0 0) 0x7f68f805d0a0 con 0x7f69080cf100 2024-03-17T23:47:42.382 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-03-17T23:47:42.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.384+0000 7f69106a1640 1 -- 172.21.15.150:0/1391582554 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f68ec06bbd0 msgr2=0x7f68ec06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:42.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.384+0000 7f69106a1640 1 --2- 172.21.15.150:0/1391582554 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f68ec06bbd0 0x7f68ec06e090 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6904007fd0 tx=0x7f6904007de0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.384+0000 7f69106a1640 1 -- 172.21.15.150:0/1391582554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69080cf100 msgr2=0x7f6908158130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:42.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.384+0000 7f69106a1640 1 --2- 172.21.15.150:0/1391582554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69080cf100 0x7f6908158130 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f68f800e8d0 tx=0x7f68f800ecd0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.384+0000 7f69106a1640 1 -- 172.21.15.150:0/1391582554 shutdown_connections 2024-03-17T23:47:42.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.384+0000 7f69106a1640 1 --2- 172.21.15.150:0/1391582554 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f690815cac0 0x7f6908151f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.384+0000 7f69106a1640 1 --2- 172.21.15.150:0/1391582554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69080cf100 0x7f6908158130 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.384+0000 7f69106a1640 1 --2- 172.21.15.150:0/1391582554 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f68ec06bbd0 0x7f68ec06e090 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.384+0000 7f69106a1640 1 --2- 172.21.15.150:0/1391582554 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69080ccf90 0x7f6908157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.384+0000 7f69106a1640 1 -- 172.21.15.150:0/1391582554 >> 172.21.15.150:0/1391582554 conn(0x7f690806b190 msgr2=0x7f690815ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:42.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.384+0000 7f69106a1640 1 -- 172.21.15.150:0/1391582554 shutdown_connections 2024-03-17T23:47:42.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.384+0000 7f69106a1640 1 -- 172.21.15.150:0/1391582554 wait complete. 2024-03-17T23:47:42.395 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:47:42.396 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:47:42.411 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:42.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.551+0000 7fad26cae640 1 Processor -- start 2024-03-17T23:47:42.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.551+0000 7fad26cae640 1 -- start start 2024-03-17T23:47:42.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.552+0000 7fad26cae640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fad20155320 0x7fad2015bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:42.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.552+0000 7fad26cae640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fad2006dd30 0x7fad200ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:42.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.552+0000 7fad26cae640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fad200cd250 0x7fad2015e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:42.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.552+0000 7fad26cae640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fad2015c550 con 0x7fad2006dd30 2024-03-17T23:47:42.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.552+0000 7fad26cae640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fad200cf3b0 con 0x7fad200cd250 2024-03-17T23:47:42.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.552+0000 7fad26cae640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fad200cf690 con 0x7fad20155320 2024-03-17T23:47:42.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.552+0000 7fad1ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fad2006dd30 0x7fad200ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:42.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.552+0000 7fad1ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fad2006dd30 0x7fad200ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46338/0 (socket says 172.21.15.150:46338) 2024-03-17T23:47:42.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.552+0000 7fad1ffff640 1 -- 172.21.15.150:0/1454846756 learned_addr learned my addr 172.21.15.150:0/1454846756 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:42.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.552+0000 7fad2524b640 1 --2- 172.21.15.150:0/1454846756 >> v2:172.21.15.157:3300/0 conn(0x7fad200cd250 0x7fad2015e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:42.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.552+0000 7fad24a4a640 1 --2- 172.21.15.150:0/1454846756 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fad20155320 0x7fad2015bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:42.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.552+0000 7fad1ffff640 1 -- 172.21.15.150:0/1454846756 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fad20155320 msgr2=0x7fad2015bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:42.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.552+0000 7fad1ffff640 1 --2- 172.21.15.150:0/1454846756 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fad20155320 0x7fad2015bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.552+0000 7fad1ffff640 1 -- 172.21.15.150:0/1454846756 >> v2:172.21.15.157:3300/0 conn(0x7fad200cd250 msgr2=0x7fad2015e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:42.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.552+0000 7fad1ffff640 1 --2- 172.21.15.150:0/1454846756 >> v2:172.21.15.157:3300/0 conn(0x7fad200cd250 0x7fad2015e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.552+0000 7fad1ffff640 1 -- 172.21.15.150:0/1454846756 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad2015c7a0 con 0x7fad2006dd30 2024-03-17T23:47:42.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.553+0000 7fad1ffff640 1 --2- 172.21.15.150:0/1454846756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad2006dd30 0x7fad200ccd10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fad20071c60 tx=0x7fad0c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afd953e40ac2ad79 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:42.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.553+0000 7fad1f7fe640 1 -- 172.21.15.150:0/1454846756 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad0c035020 con 0x7fad2006dd30 2024-03-17T23:47:42.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.553+0000 7fad1f7fe640 1 -- 172.21.15.150:0/1454846756 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad0c031870 con 0x7fad2006dd30 2024-03-17T23:47:42.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.553+0000 7fad1f7fe640 1 -- 172.21.15.150:0/1454846756 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad0c031c40 con 0x7fad2006dd30 2024-03-17T23:47:42.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.554+0000 7fad26cae640 1 -- 172.21.15.150:0/1454846756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad2006dd30 msgr2=0x7fad200ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:42.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.554+0000 7fad26cae640 1 --2- 172.21.15.150:0/1454846756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad2006dd30 0x7fad200ccd10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fad20071c60 tx=0x7fad0c02b910 comp rx=0 tx=0).stop 2024-03-17T23:47:42.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.554+0000 7fad26cae640 1 -- 172.21.15.150:0/1454846756 shutdown_connections 2024-03-17T23:47:42.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.554+0000 7fad26cae640 1 --2- 172.21.15.150:0/1454846756 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad2006dd30 0x7fad200ccd10 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.554+0000 7fad26cae640 1 --2- 172.21.15.150:0/1454846756 >> v2:172.21.15.157:3300/0 conn(0x7fad200cd250 0x7fad2015e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.554+0000 7fad26cae640 1 --2- 172.21.15.150:0/1454846756 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fad20155320 0x7fad2015bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.554+0000 7fad26cae640 1 -- 172.21.15.150:0/1454846756 >> 172.21.15.150:0/1454846756 conn(0x7fad2006b190 msgr2=0x7fad2006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:42.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.554+0000 7fad26cae640 1 -- 172.21.15.150:0/1454846756 shutdown_connections 2024-03-17T23:47:42.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.554+0000 7fad26cae640 1 -- 172.21.15.150:0/1454846756 wait complete. 2024-03-17T23:47:42.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.555+0000 7fad26cae640 1 Processor -- start 2024-03-17T23:47:42.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.555+0000 7fad26cae640 1 -- start start 2024-03-17T23:47:42.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.556+0000 7fad26cae640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad2006dd30 0x7fad20141f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:42.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.556+0000 7fad26cae640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fad200cd250 0x7fad200c98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:42.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.556+0000 7fad26cae640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad20155320 0x7fad201445d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:42.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.556+0000 7fad26cae640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fad20155700 con 0x7fad20155320 2024-03-17T23:47:42.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.556+0000 7fad26cae640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fad2006e190 con 0x7fad2006dd30 2024-03-17T23:47:42.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.556+0000 7fad26cae640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fad20155880 con 0x7fad200cd250 2024-03-17T23:47:42.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.556+0000 7fad2524b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad20155320 0x7fad201445d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:42.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.556+0000 7fad2524b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad20155320 0x7fad201445d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46348/0 (socket says 172.21.15.150:46348) 2024-03-17T23:47:42.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.556+0000 7fad2524b640 1 -- 172.21.15.150:0/3269633273 learned_addr learned my addr 172.21.15.150:0/3269633273 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:42.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.556+0000 7fad24a4a640 1 --2- 172.21.15.150:0/3269633273 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad2006dd30 0x7fad20141f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:42.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.556+0000 7fad1ffff640 1 --2- 172.21.15.150:0/3269633273 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fad200cd250 0x7fad200c98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:42.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.556+0000 7fad2524b640 1 -- 172.21.15.150:0/3269633273 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fad200cd250 msgr2=0x7fad200c98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:42.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.556+0000 7fad2524b640 1 --2- 172.21.15.150:0/3269633273 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fad200cd250 0x7fad200c98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.556+0000 7fad2524b640 1 -- 172.21.15.150:0/3269633273 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad2006dd30 msgr2=0x7fad20141f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:42.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.556+0000 7fad2524b640 1 --2- 172.21.15.150:0/3269633273 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad2006dd30 0x7fad20141f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.556+0000 7fad2524b640 1 -- 172.21.15.150:0/3269633273 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad200cb590 con 0x7fad20155320 2024-03-17T23:47:42.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.556+0000 7fad2524b640 1 --2- 172.21.15.150:0/3269633273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad20155320 0x7fad201445d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fad1800eed0 tx=0x7fad1800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:42.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.556+0000 7fad1d7fa640 1 -- 172.21.15.150:0/3269633273 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad18018020 con 0x7fad20155320 2024-03-17T23:47:42.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.556+0000 7fad1d7fa640 1 -- 172.21.15.150:0/3269633273 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad18014400 con 0x7fad20155320 2024-03-17T23:47:42.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.556+0000 7fad26cae640 1 -- 172.21.15.150:0/3269633273 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad20144ca0 con 0x7fad20155320 2024-03-17T23:47:42.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.556+0000 7fad1d7fa640 1 -- 172.21.15.150:0/3269633273 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad180147f0 con 0x7fad20155320 2024-03-17T23:47:42.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.557+0000 7fad26cae640 1 -- 172.21.15.150:0/3269633273 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad20145260 con 0x7fad20155320 2024-03-17T23:47:42.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.557+0000 7fad26cae640 1 -- 172.21.15.150:0/3269633273 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad200681a0 con 0x7fad20155320 2024-03-17T23:47:42.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.558+0000 7fad1d7fa640 1 -- 172.21.15.150:0/3269633273 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fad18005d10 con 0x7fad20155320 2024-03-17T23:47:42.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.558+0000 7fad1d7fa640 1 --2- 172.21.15.150:0/3269633273 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fad0406bbd0 0x7fad0406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:42.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.558+0000 7fad1d7fa640 1 -- 172.21.15.150:0/3269633273 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fad1808b3a0 con 0x7fad20155320 2024-03-17T23:47:42.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.558+0000 7fad24a4a640 1 --2- 172.21.15.150:0/3269633273 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fad0406bbd0 0x7fad0406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:42.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.559+0000 7fad24a4a640 1 --2- 172.21.15.150:0/3269633273 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fad0406bbd0 0x7fad0406e090 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fad14009b40 tx=0x7fad14009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:42.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.563+0000 7fad1d7fa640 1 -- 172.21.15.150:0/3269633273 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fad18022070 con 0x7fad20155320 2024-03-17T23:47:42.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.740+0000 7fad26cae640 1 -- 172.21.15.150:0/3269633273 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fad201592c0 con 0x7fad20155320 2024-03-17T23:47:42.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.741+0000 7fad1d7fa640 1 -- 172.21.15.150:0/3269633273 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18881 (secure 0 0 0) 0x7fad200cb590 con 0x7fad20155320 2024-03-17T23:47:42.742 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:42.742 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":28,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:37.031864+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.e","mappings":[{"from":7,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:42.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.743+0000 7fad26cae640 1 -- 172.21.15.150:0/3269633273 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fad0406bbd0 msgr2=0x7fad0406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:42.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.743+0000 7fad26cae640 1 --2- 172.21.15.150:0/3269633273 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fad0406bbd0 0x7fad0406e090 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fad14009b40 tx=0x7fad14009190 comp rx=0 tx=0).stop 2024-03-17T23:47:42.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.743+0000 7fad26cae640 1 -- 172.21.15.150:0/3269633273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad20155320 msgr2=0x7fad201445d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:42.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.743+0000 7fad26cae640 1 --2- 172.21.15.150:0/3269633273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad20155320 0x7fad201445d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fad1800eed0 tx=0x7fad1800b480 comp rx=0 tx=0).stop 2024-03-17T23:47:42.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.743+0000 7fad26cae640 1 -- 172.21.15.150:0/3269633273 shutdown_connections 2024-03-17T23:47:42.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.743+0000 7fad26cae640 1 --2- 172.21.15.150:0/3269633273 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad20155320 0x7fad201445d0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.743+0000 7fad26cae640 1 --2- 172.21.15.150:0/3269633273 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fad200cd250 0x7fad200c98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.743+0000 7fad26cae640 1 --2- 172.21.15.150:0/3269633273 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fad0406bbd0 0x7fad0406e090 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.743+0000 7fad26cae640 1 --2- 172.21.15.150:0/3269633273 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad2006dd30 0x7fad20141f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.743+0000 7fad26cae640 1 -- 172.21.15.150:0/3269633273 >> 172.21.15.150:0/3269633273 conn(0x7fad2006b190 msgr2=0x7fad20073b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:42.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.743+0000 7fad26cae640 1 -- 172.21.15.150:0/3269633273 shutdown_connections 2024-03-17T23:47:42.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.743+0000 7fad26cae640 1 -- 172.21.15.150:0/3269633273 wait complete. 2024-03-17T23:47:42.755 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:46:53.610678+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:46:54.227335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-17T23:46:57.949235+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-17T23:47:42.756 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:47:42.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.859+0000 7fb27dfbc640 1 Processor -- start 2024-03-17T23:47:42.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.859+0000 7fb27dfbc640 1 -- start start 2024-03-17T23:47:42.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.859+0000 7fb27dfbc640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb2780d3910 0x7fb2780d0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:42.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.859+0000 7fb27dfbc640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2780d2870 con 0x7fb2781615d0 2024-03-17T23:47:42.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.859+0000 7fb27dfbc640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2780d2b50 con 0x7fb2780b9660 2024-03-17T23:47:42.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.859+0000 7fb27dfbc640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fb2781619b0 con 0x7fb2780d3910 2024-03-17T23:47:42.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.860+0000 7fb2777fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb2781615d0 0x7fb2780d2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59242/0 (socket says 172.21.15.150:59242) 2024-03-17T23:47:42.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.860+0000 7fb2777fe640 1 -- 172.21.15.150:0/3411942430 learned_addr learned my addr 172.21.15.150:0/3411942430 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:42.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.860+0000 7fb276ffd640 1 --2- 172.21.15.150:0/3411942430 >> v2:172.21.15.157:3300/0 conn(0x7fb2780d3910 0x7fb2780d0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:42.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.860+0000 7fb2767fc640 1 -- 172.21.15.150:0/3411942430 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2225250369 0 0) 0x7fb2780d2870 con 0x7fb2781615d0 2024-03-17T23:47:42.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.860+0000 7fb2767fc640 1 -- 172.21.15.150:0/3411942430 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb260003520 con 0x7fb2781615d0 2024-03-17T23:47:42.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.860+0000 7fb2767fc640 1 -- 172.21.15.150:0/3411942430 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1472352999 0 0) 0x7fb2780d2b50 con 0x7fb2780b9660 2024-03-17T23:47:42.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.860+0000 7fb2767fc640 1 -- 172.21.15.150:0/3411942430 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2780d2870 con 0x7fb2780b9660 2024-03-17T23:47:42.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.860+0000 7fb276ffd640 1 -- 172.21.15.150:0/3411942430 >> v1:172.21.15.157:6789/0 conn(0x7fb2780b9660 legacy=0x7fb2780b6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:42.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.860+0000 7fb276ffd640 1 -- 172.21.15.150:0/3411942430 >> v1:172.21.15.150:6789/0 conn(0x7fb2781615d0 legacy=0x7fb2780d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:42.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.860+0000 7fb276ffd640 1 -- 172.21.15.150:0/3411942430 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2780d2e30 con 0x7fb2780d3910 2024-03-17T23:47:42.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.861+0000 7fb276ffd640 1 --2- 172.21.15.150:0/3411942430 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2780d3910 0x7fb2780d0720 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb26c0097a0 tx=0x7fb26c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5c4cc3d7f69ee0a server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:42.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.861+0000 7fb2767fc640 1 -- 172.21.15.150:0/3411942430 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26c034070 con 0x7fb2780d3910 2024-03-17T23:47:42.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.861+0000 7fb2767fc640 1 -- 172.21.15.150:0/3411942430 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb26c030610 con 0x7fb2780d3910 2024-03-17T23:47:42.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.861+0000 7fb2767fc640 1 -- 172.21.15.150:0/3411942430 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26c0309e0 con 0x7fb2780d3910 2024-03-17T23:47:42.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.861+0000 7fb27dfbc640 1 -- 172.21.15.150:0/3411942430 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2780d3910 msgr2=0x7fb2780d0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:42.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.861+0000 7fb27dfbc640 1 --2- 172.21.15.150:0/3411942430 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2780d3910 0x7fb2780d0720 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb26c0097a0 tx=0x7fb26c02b960 comp rx=0 tx=0).stop 2024-03-17T23:47:42.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.861+0000 7fb27dfbc640 1 -- 172.21.15.150:0/3411942430 shutdown_connections 2024-03-17T23:47:42.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.861+0000 7fb27dfbc640 1 --2- 172.21.15.150:0/3411942430 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2780d3910 0x7fb2780d0720 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.861+0000 7fb27dfbc640 1 -- 172.21.15.150:0/3411942430 >> 172.21.15.150:0/3411942430 conn(0x7fb2780730b0 msgr2=0x7fb2780b5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:42.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.861+0000 7fb27dfbc640 1 -- 172.21.15.150:0/3411942430 shutdown_connections 2024-03-17T23:47:42.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.861+0000 7fb27dfbc640 1 -- 172.21.15.150:0/3411942430 wait complete. 2024-03-17T23:47:42.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.863+0000 7fb27dfbc640 1 Processor -- start 2024-03-17T23:47:42.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.863+0000 7fb27dfbc640 1 -- start start 2024-03-17T23:47:42.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.863+0000 7fb27dfbc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2780b9660 0x7fb2781544a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:42.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.863+0000 7fb27dfbc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2780d3910 0x7fb278157680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:42.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.863+0000 7fb27dfbc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2781615d0 0x7fb2780cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:42.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.863+0000 7fb27dfbc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb2780d0c60 con 0x7fb2780b9660 2024-03-17T23:47:42.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.863+0000 7fb27dfbc640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb2780d33f0 con 0x7fb2781615d0 2024-03-17T23:47:42.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.863+0000 7fb27dfbc640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb2780d0de0 con 0x7fb2780d3910 2024-03-17T23:47:42.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.864+0000 7fb2777fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2780b9660 0x7fb2781544a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:42.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.864+0000 7fb2777fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2780b9660 0x7fb2781544a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46356/0 (socket says 172.21.15.150:46356) 2024-03-17T23:47:42.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.864+0000 7fb2777fe640 1 -- 172.21.15.150:0/233288272 learned_addr learned my addr 172.21.15.150:0/233288272 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:42.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.864+0000 7fb277fff640 1 --2- 172.21.15.150:0/233288272 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2781615d0 0x7fb2780cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:42.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.864+0000 7fb276ffd640 1 --2- 172.21.15.150:0/233288272 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2780d3910 0x7fb278157680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:42.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.864+0000 7fb2777fe640 1 -- 172.21.15.150:0/233288272 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2780d3910 msgr2=0x7fb278157680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:42.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.864+0000 7fb2777fe640 1 --2- 172.21.15.150:0/233288272 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2780d3910 0x7fb278157680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.864+0000 7fb2777fe640 1 -- 172.21.15.150:0/233288272 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2781615d0 msgr2=0x7fb2780cd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:42.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.864+0000 7fb2777fe640 1 --2- 172.21.15.150:0/233288272 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2781615d0 0x7fb2780cd5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:42.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.864+0000 7fb2777fe640 1 -- 172.21.15.150:0/233288272 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2780cdb30 con 0x7fb2780b9660 2024-03-17T23:47:42.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.864+0000 7fb2777fe640 1 --2- 172.21.15.150:0/233288272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2780b9660 0x7fb2781544a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb26800c440 tx=0x7fb26800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:42.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.864+0000 7fb25ffff640 1 -- 172.21.15.150:0/233288272 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb268017070 con 0x7fb2780b9660 2024-03-17T23:47:42.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.864+0000 7fb25ffff640 1 -- 172.21.15.150:0/233288272 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb268005ce0 con 0x7fb2780b9660 2024-03-17T23:47:42.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.864+0000 7fb27dfbc640 1 -- 172.21.15.150:0/233288272 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb278148940 con 0x7fb2780b9660 2024-03-17T23:47:42.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.864+0000 7fb25ffff640 1 -- 172.21.15.150:0/233288272 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb268007270 con 0x7fb2780b9660 2024-03-17T23:47:42.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.864+0000 7fb27dfbc640 1 -- 172.21.15.150:0/233288272 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb278148f00 con 0x7fb2780b9660 2024-03-17T23:47:42.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.865+0000 7fb27dfbc640 1 -- 172.21.15.150:0/233288272 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2780688b0 con 0x7fb2780b9660 2024-03-17T23:47:42.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.866+0000 7fb25ffff640 1 -- 172.21.15.150:0/233288272 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb26801a070 con 0x7fb2780b9660 2024-03-17T23:47:42.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.866+0000 7fb25ffff640 1 --2- 172.21.15.150:0/233288272 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb26406bbd0 0x7fb26406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:42.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.866+0000 7fb25ffff640 1 -- 172.21.15.150:0/233288272 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fb26808a0e0 con 0x7fb2780b9660 2024-03-17T23:47:42.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.866+0000 7fb276ffd640 1 --2- 172.21.15.150:0/233288272 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb26406bbd0 0x7fb26406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:42.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.866+0000 7fb276ffd640 1 --2- 172.21.15.150:0/233288272 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb26406bbd0 0x7fb26406e090 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb26c036000 tx=0x7fb26c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:42.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:42.870+0000 7fb25ffff640 1 -- 172.21.15.150:0/233288272 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb268058250 con 0x7fb2780b9660 2024-03-17T23:47:43.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.045+0000 7fb27dfbc640 1 -- 172.21.15.150:0/233288272 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb2780ce050 con 0x7fb2780b9660 2024-03-17T23:47:43.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.045+0000 7fb25ffff640 1 -- 172.21.15.150:0/233288272 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7fb2680770c0 con 0x7fb2780b9660 2024-03-17T23:47:43.046 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:47:43.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.047+0000 7fb27dfbc640 1 -- 172.21.15.150:0/233288272 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb26406bbd0 msgr2=0x7fb26406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:43.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.047+0000 7fb27dfbc640 1 --2- 172.21.15.150:0/233288272 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb26406bbd0 0x7fb26406e090 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb26c036000 tx=0x7fb26c03b040 comp rx=0 tx=0).stop 2024-03-17T23:47:43.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.047+0000 7fb27dfbc640 1 -- 172.21.15.150:0/233288272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2780b9660 msgr2=0x7fb2781544a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:43.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.047+0000 7fb27dfbc640 1 --2- 172.21.15.150:0/233288272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2780b9660 0x7fb2781544a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb26800c440 tx=0x7fb26800c900 comp rx=0 tx=0).stop 2024-03-17T23:47:43.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.047+0000 7fb27dfbc640 1 -- 172.21.15.150:0/233288272 shutdown_connections 2024-03-17T23:47:43.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.047+0000 7fb27dfbc640 1 --2- 172.21.15.150:0/233288272 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb26406bbd0 0x7fb26406e090 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.047+0000 7fb27dfbc640 1 --2- 172.21.15.150:0/233288272 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb2781615d0 0x7fb2780cd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.047+0000 7fb27dfbc640 1 --2- 172.21.15.150:0/233288272 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb2780d3910 0x7fb278157680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.047+0000 7fb27dfbc640 1 --2- 172.21.15.150:0/233288272 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb2780b9660 0x7fb2781544a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.047+0000 7fb27dfbc640 1 -- 172.21.15.150:0/233288272 >> 172.21.15.150:0/233288272 conn(0x7fb2780730b0 msgr2=0x7fb2780b6f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:43.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.047+0000 7fb27dfbc640 1 -- 172.21.15.150:0/233288272 shutdown_connections 2024-03-17T23:47:43.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.047+0000 7fb27dfbc640 1 -- 172.21.15.150:0/233288272 wait complete. 2024-03-17T23:47:43.059 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:47:43.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.162+0000 7f99cc397640 1 Processor -- start 2024-03-17T23:47:43.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.162+0000 7f99cc397640 1 -- start start 2024-03-17T23:47:43.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.163+0000 7f99cc397640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f99c4161590 0x7f99c4067690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:43.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.163+0000 7f99cc397640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99c4072a20 0x7f99c40ad180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:43.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.163+0000 7f99cc397640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99c4161970 con 0x7f99c40d0140 2024-03-17T23:47:43.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.163+0000 7f99cc397640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f99c406e580 con 0x7f99c4161590 2024-03-17T23:47:43.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.163+0000 7f99cc397640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f99c406e7d0 con 0x7f99c4072a20 2024-03-17T23:47:43.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.163+0000 7f99ca133640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f99c4161590 0x7f99c4067690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:43.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.163+0000 7f99ca934640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f99c40d0140 0x7f99c406b470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59252/0 (socket says 172.21.15.150:59252) 2024-03-17T23:47:43.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.163+0000 7f99ca934640 1 -- 172.21.15.150:0/523370435 learned_addr learned my addr 172.21.15.150:0/523370435 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:43.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.163+0000 7f99c9932640 1 --2- 172.21.15.150:0/523370435 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99c4072a20 0x7f99c40ad180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:43.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.163+0000 7f99c9131640 1 -- 172.21.15.150:0/523370435 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 403201142 0 0) 0x7f99c4161970 con 0x7f99c40d0140 2024-03-17T23:47:43.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.163+0000 7f99c9131640 1 -- 172.21.15.150:0/523370435 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99b0003520 con 0x7f99c40d0140 2024-03-17T23:47:43.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.163+0000 7f99ca133640 1 -- 172.21.15.150:0/523370435 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99c4072a20 msgr2=0x7f99c40ad180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:43.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.163+0000 7f99ca133640 1 --2- 172.21.15.150:0/523370435 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99c4072a20 0x7f99c40ad180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.163+0000 7f99ca133640 1 -- 172.21.15.150:0/523370435 >> v1:172.21.15.150:6789/0 conn(0x7f99c40d0140 legacy=0x7f99c406b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:43.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.163+0000 7f99ca133640 1 -- 172.21.15.150:0/523370435 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99c406eab0 con 0x7f99c4161590 2024-03-17T23:47:43.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.164+0000 7f99ca133640 1 --2- 172.21.15.150:0/523370435 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99c4161590 0x7f99c4067690 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f99c00097a0 tx=0x7f99c002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f809a8b6a447dfa server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:43.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.164+0000 7f99c9131640 1 -- 172.21.15.150:0/523370435 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c0038070 con 0x7f99c4161590 2024-03-17T23:47:43.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.164+0000 7f99c9131640 1 -- 172.21.15.150:0/523370435 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99c0031700 con 0x7f99c4161590 2024-03-17T23:47:43.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.164+0000 7f99c9131640 1 -- 172.21.15.150:0/523370435 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c0031ad0 con 0x7f99c4161590 2024-03-17T23:47:43.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.165+0000 7f99cc397640 1 -- 172.21.15.150:0/523370435 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99c4161590 msgr2=0x7f99c4067690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:43.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.165+0000 7f99cc397640 1 --2- 172.21.15.150:0/523370435 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99c4161590 0x7f99c4067690 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f99c00097a0 tx=0x7f99c002b7a0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.165+0000 7f99cc397640 1 -- 172.21.15.150:0/523370435 shutdown_connections 2024-03-17T23:47:43.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.165+0000 7f99cc397640 1 --2- 172.21.15.150:0/523370435 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99c4072a20 0x7f99c40ad180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.165+0000 7f99cc397640 1 --2- 172.21.15.150:0/523370435 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99c4161590 0x7f99c4067690 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.165+0000 7f99cc397640 1 -- 172.21.15.150:0/523370435 >> 172.21.15.150:0/523370435 conn(0x7f99c40b3da0 msgr2=0x7f99c40b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:43.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.165+0000 7f99cc397640 1 -- 172.21.15.150:0/523370435 shutdown_connections 2024-03-17T23:47:43.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.165+0000 7f99cc397640 1 -- 172.21.15.150:0/523370435 wait complete. 2024-03-17T23:47:43.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.166+0000 7f99cc397640 1 Processor -- start 2024-03-17T23:47:43.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.166+0000 7f99cc397640 1 -- start start 2024-03-17T23:47:43.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.167+0000 7f99cc397640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99c4072a20 0x7f99c4140df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:43.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.167+0000 7f99cc397640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99c40d0140 0x7f99c4143ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:43.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.167+0000 7f99cc397640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f99c4161590 0x7f99c41530a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:43.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.167+0000 7f99cc397640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f99c4099c20 con 0x7f99c40d0140 2024-03-17T23:47:43.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.167+0000 7f99cc397640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f99c40d05b0 con 0x7f99c4161590 2024-03-17T23:47:43.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.167+0000 7f99cc397640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f99c4151d60 con 0x7f99c4072a20 2024-03-17T23:47:43.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.167+0000 7f99c9932640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99c40d0140 0x7f99c4143ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:43.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.167+0000 7f99c9932640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99c40d0140 0x7f99c4143ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46376/0 (socket says 172.21.15.150:46376) 2024-03-17T23:47:43.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.167+0000 7f99c9932640 1 -- 172.21.15.150:0/1833674339 learned_addr learned my addr 172.21.15.150:0/1833674339 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:43.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.167+0000 7f99ca133640 1 --2- 172.21.15.150:0/1833674339 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99c4072a20 0x7f99c4140df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:43.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.167+0000 7f99ca934640 1 --2- 172.21.15.150:0/1833674339 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f99c4161590 0x7f99c41530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:43.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.167+0000 7f99c9932640 1 -- 172.21.15.150:0/1833674339 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99c4072a20 msgr2=0x7f99c4140df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:43.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.167+0000 7f99c9932640 1 --2- 172.21.15.150:0/1833674339 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99c4072a20 0x7f99c4140df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.167+0000 7f99c9932640 1 -- 172.21.15.150:0/1833674339 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f99c4161590 msgr2=0x7f99c41530a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:43.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.167+0000 7f99c9932640 1 --2- 172.21.15.150:0/1833674339 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f99c4161590 0x7f99c41530a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.167+0000 7f99c9932640 1 -- 172.21.15.150:0/1833674339 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99c4144420 con 0x7f99c40d0140 2024-03-17T23:47:43.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.167+0000 7f99c9932640 1 --2- 172.21.15.150:0/1833674339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99c40d0140 0x7f99c4143ee0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f99b400e9e0 tx=0x7f99b400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:43.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.167+0000 7f99baffd640 1 -- 172.21.15.150:0/1833674339 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b4017070 con 0x7f99c40d0140 2024-03-17T23:47:43.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.167+0000 7f99baffd640 1 -- 172.21.15.150:0/1833674339 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99b400c9d0 con 0x7f99c40d0140 2024-03-17T23:47:43.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.167+0000 7f99cc397640 1 -- 172.21.15.150:0/1833674339 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99c4152000 con 0x7f99c40d0140 2024-03-17T23:47:43.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.167+0000 7f99baffd640 1 -- 172.21.15.150:0/1833674339 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b400cdc0 con 0x7f99c40d0140 2024-03-17T23:47:43.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.167+0000 7f99cc397640 1 -- 172.21.15.150:0/1833674339 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99c41525c0 con 0x7f99c40d0140 2024-03-17T23:47:43.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.168+0000 7f99cc397640 1 -- 172.21.15.150:0/1833674339 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99c40681a0 con 0x7f99c40d0140 2024-03-17T23:47:43.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.169+0000 7f99baffd640 1 -- 172.21.15.150:0/1833674339 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f99b401a070 con 0x7f99c40d0140 2024-03-17T23:47:43.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.169+0000 7f99baffd640 1 --2- 172.21.15.150:0/1833674339 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f99a806bbd0 0x7f99a806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:43.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.169+0000 7f99baffd640 1 -- 172.21.15.150:0/1833674339 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f99b408ae50 con 0x7f99c40d0140 2024-03-17T23:47:43.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.169+0000 7f99ca133640 1 --2- 172.21.15.150:0/1833674339 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f99a806bbd0 0x7f99a806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:43.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.169+0000 7f99ca133640 1 --2- 172.21.15.150:0/1833674339 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f99a806bbd0 0x7f99a806e090 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f99c0035000 tx=0x7f99c0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:43.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.173+0000 7f99baffd640 1 -- 172.21.15.150:0/1833674339 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f99b4058ef0 con 0x7f99c40d0140 2024-03-17T23:47:43.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.346+0000 7f99cc397640 1 -- 172.21.15.150:0/1833674339 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f99c409ab90 con 0x7f99c40d0140 2024-03-17T23:47:43.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.347+0000 7f99baffd640 1 -- 172.21.15.150:0/1833674339 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7f99b405cf90 con 0x7f99c40d0140 2024-03-17T23:47:43.347 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:43.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.348+0000 7f99cc397640 1 -- 172.21.15.150:0/1833674339 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f99a806bbd0 msgr2=0x7f99a806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:43.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.348+0000 7f99cc397640 1 --2- 172.21.15.150:0/1833674339 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f99a806bbd0 0x7f99a806e090 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f99c0035000 tx=0x7f99c0008030 comp rx=0 tx=0).stop 2024-03-17T23:47:43.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.348+0000 7f99cc397640 1 -- 172.21.15.150:0/1833674339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99c40d0140 msgr2=0x7f99c4143ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:43.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.348+0000 7f99cc397640 1 --2- 172.21.15.150:0/1833674339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99c40d0140 0x7f99c4143ee0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f99b400e9e0 tx=0x7f99b400eea0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.348+0000 7f99cc397640 1 -- 172.21.15.150:0/1833674339 shutdown_connections 2024-03-17T23:47:43.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.348+0000 7f99cc397640 1 --2- 172.21.15.150:0/1833674339 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f99a806bbd0 0x7f99a806e090 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.348+0000 7f99cc397640 1 --2- 172.21.15.150:0/1833674339 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f99c4161590 0x7f99c41530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.348+0000 7f99cc397640 1 --2- 172.21.15.150:0/1833674339 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99c40d0140 0x7f99c4143ee0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.348+0000 7f99cc397640 1 --2- 172.21.15.150:0/1833674339 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f99c4072a20 0x7f99c4140df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.348+0000 7f99cc397640 1 -- 172.21.15.150:0/1833674339 >> 172.21.15.150:0/1833674339 conn(0x7f99c40b3da0 msgr2=0x7f99c40b1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:43.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.349+0000 7f99cc397640 1 -- 172.21.15.150:0/1833674339 shutdown_connections 2024-03-17T23:47:43.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.349+0000 7f99cc397640 1 -- 172.21.15.150:0/1833674339 wait complete. 2024-03-17T23:47:43.360 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:47:43.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.463+0000 7fd42fcb2640 1 Processor -- start 2024-03-17T23:47:43.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.464+0000 7fd42fcb2640 1 -- start start 2024-03-17T23:47:43.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.464+0000 7fd42fcb2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd4281615d0 0x7fd4280d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:43.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.464+0000 7fd42fcb2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4280d0e30 0x7fd4280b6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:43.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.464+0000 7fd42fcb2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4280d2870 con 0x7fd4280d3910 2024-03-17T23:47:43.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.464+0000 7fd42fcb2640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fd4281619b0 con 0x7fd4281615d0 2024-03-17T23:47:43.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.464+0000 7fd42fcb2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd4280d2b50 con 0x7fd4280d0e30 2024-03-17T23:47:43.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.465+0000 7fd42da4e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd4281615d0 0x7fd4280d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:43.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.465+0000 7fd42d24d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd4280d3910 0x7fd4280d0720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59254/0 (socket says 172.21.15.150:59254) 2024-03-17T23:47:43.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.465+0000 7fd42d24d640 1 -- 172.21.15.150:0/79278517 learned_addr learned my addr 172.21.15.150:0/79278517 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:43.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.465+0000 7fd42e24f640 1 --2- 172.21.15.150:0/79278517 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4280d0e30 0x7fd4280b6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:43.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.465+0000 7fd42da4e640 1 -- 172.21.15.150:0/79278517 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4280d0e30 msgr2=0x7fd4280b6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:43.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.465+0000 7fd42da4e640 1 --2- 172.21.15.150:0/79278517 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4280d0e30 0x7fd4280b6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.465+0000 7fd42da4e640 1 -- 172.21.15.150:0/79278517 >> v1:172.21.15.150:6789/0 conn(0x7fd4280d3910 legacy=0x7fd4280d0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:43.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.465+0000 7fd42da4e640 1 -- 172.21.15.150:0/79278517 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4280d2e30 con 0x7fd4281615d0 2024-03-17T23:47:43.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.465+0000 7fd42ca4c640 1 -- 172.21.15.150:0/79278517 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3815959210 0 0) 0x7fd4280d2870 con 0x7fd4280d3910 2024-03-17T23:47:43.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.466+0000 7fd42da4e640 1 --2- 172.21.15.150:0/79278517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4281615d0 0x7fd4280d2020 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd4240097a0 tx=0x7fd42402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed1660a27beedbcd server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:43.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.466+0000 7fd42ca4c640 1 -- 172.21.15.150:0/79278517 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd424035020 con 0x7fd4281615d0 2024-03-17T23:47:43.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.466+0000 7fd42ca4c640 1 -- 172.21.15.150:0/79278517 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd424031830 con 0x7fd4281615d0 2024-03-17T23:47:43.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.466+0000 7fd42ca4c640 1 -- 172.21.15.150:0/79278517 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd424031c00 con 0x7fd4281615d0 2024-03-17T23:47:43.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.466+0000 7fd42fcb2640 1 -- 172.21.15.150:0/79278517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4281615d0 msgr2=0x7fd4280d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:43.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.466+0000 7fd42fcb2640 1 --2- 172.21.15.150:0/79278517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4281615d0 0x7fd4280d2020 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fd4240097a0 tx=0x7fd42402b8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.466+0000 7fd42fcb2640 1 -- 172.21.15.150:0/79278517 shutdown_connections 2024-03-17T23:47:43.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.466+0000 7fd42fcb2640 1 --2- 172.21.15.150:0/79278517 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4280d0e30 0x7fd4280b6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.466+0000 7fd42fcb2640 1 --2- 172.21.15.150:0/79278517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4281615d0 0x7fd4280d2020 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.466+0000 7fd42fcb2640 1 -- 172.21.15.150:0/79278517 >> 172.21.15.150:0/79278517 conn(0x7fd4280730b0 msgr2=0x7fd4280b5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:43.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.466+0000 7fd42fcb2640 1 -- 172.21.15.150:0/79278517 shutdown_connections 2024-03-17T23:47:43.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.466+0000 7fd42fcb2640 1 -- 172.21.15.150:0/79278517 wait complete. 2024-03-17T23:47:43.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.468+0000 7fd42fcb2640 1 Processor -- start 2024-03-17T23:47:43.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.468+0000 7fd42fcb2640 1 -- start start 2024-03-17T23:47:43.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.468+0000 7fd42fcb2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4280d0e30 0x7fd4280ac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:43.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.468+0000 7fd42fcb2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4280d3910 0x7fd4280aa790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:43.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.468+0000 7fd42fcb2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4281615d0 0x7fd4280cb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:43.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.468+0000 7fd42fcb2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd4280d2560 con 0x7fd4280d0e30 2024-03-17T23:47:43.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.468+0000 7fd42fcb2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd4280d33f0 con 0x7fd4281615d0 2024-03-17T23:47:43.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.468+0000 7fd42fcb2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd4280b9660 con 0x7fd4280d3910 2024-03-17T23:47:43.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.468+0000 7fd42da4e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4280d0e30 0x7fd4280ac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:43.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.468+0000 7fd42da4e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4280d0e30 0x7fd4280ac1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46400/0 (socket says 172.21.15.150:46400) 2024-03-17T23:47:43.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.468+0000 7fd42da4e640 1 -- 172.21.15.150:0/196907432 learned_addr learned my addr 172.21.15.150:0/196907432 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:43.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.468+0000 7fd42d24d640 1 --2- 172.21.15.150:0/196907432 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4280d3910 0x7fd4280aa790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:43.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.469+0000 7fd42e24f640 1 --2- 172.21.15.150:0/196907432 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4281615d0 0x7fd4280cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:43.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.469+0000 7fd42da4e640 1 -- 172.21.15.150:0/196907432 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4280d3910 msgr2=0x7fd4280aa790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:43.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.469+0000 7fd42da4e640 1 --2- 172.21.15.150:0/196907432 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4280d3910 0x7fd4280aa790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.469+0000 7fd42da4e640 1 -- 172.21.15.150:0/196907432 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4281615d0 msgr2=0x7fd4280cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:43.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.469+0000 7fd42da4e640 1 --2- 172.21.15.150:0/196907432 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4281615d0 0x7fd4280cb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.469+0000 7fd42da4e640 1 -- 172.21.15.150:0/196907432 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4280caf90 con 0x7fd4280d0e30 2024-03-17T23:47:43.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.469+0000 7fd42da4e640 1 --2- 172.21.15.150:0/196907432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4280d0e30 0x7fd4280ac1c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd4240098c0 tx=0x7fd4240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:43.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.469+0000 7fd41e7fc640 1 -- 172.21.15.150:0/196907432 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd424040070 con 0x7fd4280d0e30 2024-03-17T23:47:43.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.469+0000 7fd41e7fc640 1 -- 172.21.15.150:0/196907432 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd42403e070 con 0x7fd4280d0e30 2024-03-17T23:47:43.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.469+0000 7fd41e7fc640 1 -- 172.21.15.150:0/196907432 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd424035040 con 0x7fd4280d0e30 2024-03-17T23:47:43.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.469+0000 7fd42fcb2640 1 -- 172.21.15.150:0/196907432 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4281f40f0 con 0x7fd4280d0e30 2024-03-17T23:47:43.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.469+0000 7fd42fcb2640 1 -- 172.21.15.150:0/196907432 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4281f46b0 con 0x7fd4280d0e30 2024-03-17T23:47:43.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.470+0000 7fd42fcb2640 1 -- 172.21.15.150:0/196907432 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4280688b0 con 0x7fd4280d0e30 2024-03-17T23:47:43.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.470+0000 7fd41e7fc640 1 -- 172.21.15.150:0/196907432 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd424038020 con 0x7fd4280d0e30 2024-03-17T23:47:43.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.471+0000 7fd41e7fc640 1 --2- 172.21.15.150:0/196907432 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd40c06bbd0 0x7fd40c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:43.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.471+0000 7fd41e7fc640 1 -- 172.21.15.150:0/196907432 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fd4240ac2c0 con 0x7fd4280d0e30 2024-03-17T23:47:43.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.471+0000 7fd42d24d640 1 --2- 172.21.15.150:0/196907432 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd40c06bbd0 0x7fd40c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:43.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.471+0000 7fd42d24d640 1 --2- 172.21.15.150:0/196907432 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd40c06bbd0 0x7fd40c06e090 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd4280b8f20 tx=0x7fd418007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:43.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.475+0000 7fd41e7fc640 1 -- 172.21.15.150:0/196907432 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd424031800 con 0x7fd4280d0e30 2024-03-17T23:47:43.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.648+0000 7fd42fcb2640 1 -- 172.21.15.150:0/196907432 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd4281f4880 con 0x7fd4280d0e30 2024-03-17T23:47:43.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.649+0000 7fd41e7fc640 1 -- 172.21.15.150:0/196907432 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7fd42407a360 con 0x7fd4280d0e30 2024-03-17T23:47:43.649 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:47:43.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.651+0000 7fd42fcb2640 1 -- 172.21.15.150:0/196907432 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd40c06bbd0 msgr2=0x7fd40c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:43.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.651+0000 7fd42fcb2640 1 --2- 172.21.15.150:0/196907432 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd40c06bbd0 0x7fd40c06e090 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd4280b8f20 tx=0x7fd418007d50 comp rx=0 tx=0).stop 2024-03-17T23:47:43.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.651+0000 7fd42fcb2640 1 -- 172.21.15.150:0/196907432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4280d0e30 msgr2=0x7fd4280ac1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:43.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.651+0000 7fd42fcb2640 1 --2- 172.21.15.150:0/196907432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4280d0e30 0x7fd4280ac1c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd4240098c0 tx=0x7fd4240046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.651+0000 7fd42fcb2640 1 -- 172.21.15.150:0/196907432 shutdown_connections 2024-03-17T23:47:43.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.651+0000 7fd42fcb2640 1 --2- 172.21.15.150:0/196907432 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd40c06bbd0 0x7fd40c06e090 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.651+0000 7fd42fcb2640 1 --2- 172.21.15.150:0/196907432 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4281615d0 0x7fd4280cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.651+0000 7fd42fcb2640 1 --2- 172.21.15.150:0/196907432 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4280d3910 0x7fd4280aa790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.651+0000 7fd42fcb2640 1 --2- 172.21.15.150:0/196907432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4280d0e30 0x7fd4280ac1c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.651+0000 7fd42fcb2640 1 -- 172.21.15.150:0/196907432 >> 172.21.15.150:0/196907432 conn(0x7fd4280730b0 msgr2=0x7fd42806d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:43.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.651+0000 7fd42fcb2640 1 -- 172.21.15.150:0/196907432 shutdown_connections 2024-03-17T23:47:43.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.651+0000 7fd42fcb2640 1 -- 172.21.15.150:0/196907432 wait complete. 2024-03-17T23:47:43.662 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-17T23:47:43.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.765+0000 7f9b72187640 1 Processor -- start 2024-03-17T23:47:43.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.765+0000 7f9b72187640 1 -- start start 2024-03-17T23:47:43.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.766+0000 7f9b72187640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9b6c06f400 0x7f9b6c0cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:43.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.766+0000 7f9b72187640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9b6c0ce4d0 0x7f9b6c165a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:43.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.766+0000 7f9b72187640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b6c145170 con 0x7f9b6c144bb0 2024-03-17T23:47:43.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.766+0000 7f9b72187640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f9b6c0ce940 con 0x7f9b6c06f400 2024-03-17T23:47:43.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.766+0000 7f9b72187640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f9b6c145450 con 0x7f9b6c0ce4d0 2024-03-17T23:47:43.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.766+0000 7f9b6affd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9b6c06f400 0x7f9b6c0cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:43.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.766+0000 7f9b6affd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9b6c06f400 0x7f9b6c0cdf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46416/0 (socket says 172.21.15.150:46416) 2024-03-17T23:47:43.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.766+0000 7f9b6affd640 1 -- 172.21.15.150:0/2144430018 learned_addr learned my addr 172.21.15.150:0/2144430018 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:43.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.766+0000 7f9b6b7fe640 1 --1- 172.21.15.150:0/2144430018 >> v1:172.21.15.157:6789/0 conn(0x7f9b6c144bb0 0x7f9b6c15ce50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:46826/0 (socket says 172.21.15.150:46826) 2024-03-17T23:47:43.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.766+0000 7f9b6bfff640 1 --2- 172.21.15.150:0/2144430018 >> v2:172.21.15.157:3300/0 conn(0x7f9b6c0ce4d0 0x7f9b6c165a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:43.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.766+0000 7f9b6affd640 1 -- 172.21.15.150:0/2144430018 >> v2:172.21.15.157:3300/0 conn(0x7f9b6c0ce4d0 msgr2=0x7f9b6c165a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:43.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.766+0000 7f9b6affd640 1 --2- 172.21.15.150:0/2144430018 >> v2:172.21.15.157:3300/0 conn(0x7f9b6c0ce4d0 0x7f9b6c165a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.766+0000 7f9b6affd640 1 -- 172.21.15.150:0/2144430018 >> v1:172.21.15.157:6789/0 conn(0x7f9b6c144bb0 legacy=0x7f9b6c15ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:43.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.766+0000 7f9b6affd640 1 -- 172.21.15.150:0/2144430018 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b6c145730 con 0x7f9b6c06f400 2024-03-17T23:47:43.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.766+0000 7f9b6a7fc640 1 -- 172.21.15.150:0/2144430018 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4212813198 0 0) 0x7f9b6c145170 con 0x7f9b6c144bb0 2024-03-17T23:47:43.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.767+0000 7f9b6affd640 1 --2- 172.21.15.150:0/2144430018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b6c06f400 0x7f9b6c0cdf90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c06c330 tx=0x7f9b6002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b863bdb0d6e9327e server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:43.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.767+0000 7f9b6a7fc640 1 -- 172.21.15.150:0/2144430018 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b60035020 con 0x7f9b6c06f400 2024-03-17T23:47:43.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.767+0000 7f9b6a7fc640 1 -- 172.21.15.150:0/2144430018 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b60031870 con 0x7f9b6c06f400 2024-03-17T23:47:43.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.767+0000 7f9b6a7fc640 1 -- 172.21.15.150:0/2144430018 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b60031c40 con 0x7f9b6c06f400 2024-03-17T23:47:43.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.768+0000 7f9b72187640 1 -- 172.21.15.150:0/2144430018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b6c06f400 msgr2=0x7f9b6c0cdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:43.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.768+0000 7f9b72187640 1 --2- 172.21.15.150:0/2144430018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b6c06f400 0x7f9b6c0cdf90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c06c330 tx=0x7f9b6002b910 comp rx=0 tx=0).stop 2024-03-17T23:47:43.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.768+0000 7f9b72187640 1 -- 172.21.15.150:0/2144430018 shutdown_connections 2024-03-17T23:47:43.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.768+0000 7f9b72187640 1 --2- 172.21.15.150:0/2144430018 >> v2:172.21.15.157:3300/0 conn(0x7f9b6c0ce4d0 0x7f9b6c165a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.768+0000 7f9b72187640 1 --2- 172.21.15.150:0/2144430018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b6c06f400 0x7f9b6c0cdf90 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.768+0000 7f9b72187640 1 -- 172.21.15.150:0/2144430018 >> 172.21.15.150:0/2144430018 conn(0x7f9b6c06b190 msgr2=0x7f9b6c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:43.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.768+0000 7f9b72187640 1 -- 172.21.15.150:0/2144430018 shutdown_connections 2024-03-17T23:47:43.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.768+0000 7f9b72187640 1 -- 172.21.15.150:0/2144430018 wait complete. 2024-03-17T23:47:43.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.769+0000 7f9b72187640 1 Processor -- start 2024-03-17T23:47:43.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.769+0000 7f9b72187640 1 -- start start 2024-03-17T23:47:43.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.770+0000 7f9b72187640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b6c06f400 0x7f9b6c0ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:43.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.770+0000 7f9b72187640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9b6c0ce4d0 0x7f9b6c0b4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:43.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.770+0000 7f9b72187640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b6c144bb0 0x7f9b6c1f3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:43.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.770+0000 7f9b72187640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c06bee0 con 0x7f9b6c144bb0 2024-03-17T23:47:43.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.770+0000 7f9b72187640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c06bd60 con 0x7f9b6c06f400 2024-03-17T23:47:43.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.770+0000 7f9b72187640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9b6c165f70 con 0x7f9b6c0ce4d0 2024-03-17T23:47:43.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.770+0000 7f9b6bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b6c144bb0 0x7f9b6c1f3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:43.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.770+0000 7f9b6bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b6c144bb0 0x7f9b6c1f3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46422/0 (socket says 172.21.15.150:46422) 2024-03-17T23:47:43.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.770+0000 7f9b6bfff640 1 -- 172.21.15.150:0/2880873100 learned_addr learned my addr 172.21.15.150:0/2880873100 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:43.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.770+0000 7f9b6b7fe640 1 --2- 172.21.15.150:0/2880873100 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b6c06f400 0x7f9b6c0ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:43.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.770+0000 7f9b6affd640 1 --2- 172.21.15.150:0/2880873100 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9b6c0ce4d0 0x7f9b6c0b4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:43.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.770+0000 7f9b6bfff640 1 -- 172.21.15.150:0/2880873100 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9b6c0ce4d0 msgr2=0x7f9b6c0b4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:43.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.770+0000 7f9b6bfff640 1 --2- 172.21.15.150:0/2880873100 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9b6c0ce4d0 0x7f9b6c0b4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.770+0000 7f9b6bfff640 1 -- 172.21.15.150:0/2880873100 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b6c06f400 msgr2=0x7f9b6c0ae6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:43.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.770+0000 7f9b6bfff640 1 --2- 172.21.15.150:0/2880873100 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b6c06f400 0x7f9b6c0ae6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.770+0000 7f9b6bfff640 1 -- 172.21.15.150:0/2880873100 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b6c0ad630 con 0x7f9b6c144bb0 2024-03-17T23:47:43.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.770+0000 7f9b6bfff640 1 --2- 172.21.15.150:0/2880873100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b6c144bb0 0x7f9b6c1f3f90 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9b6400efc0 tx=0x7f9b6400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:43.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.771+0000 7f9b53fff640 1 -- 172.21.15.150:0/2880873100 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b64018020 con 0x7f9b6c144bb0 2024-03-17T23:47:43.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.771+0000 7f9b53fff640 1 -- 172.21.15.150:0/2880873100 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b64014480 con 0x7f9b6c144bb0 2024-03-17T23:47:43.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.771+0000 7f9b53fff640 1 -- 172.21.15.150:0/2880873100 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b64014870 con 0x7f9b6c144bb0 2024-03-17T23:47:43.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.771+0000 7f9b72187640 1 -- 172.21.15.150:0/2880873100 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b6c0ad910 con 0x7f9b6c144bb0 2024-03-17T23:47:43.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.771+0000 7f9b72187640 1 -- 172.21.15.150:0/2880873100 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b6c0aded0 con 0x7f9b6c144bb0 2024-03-17T23:47:43.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.772+0000 7f9b72187640 1 -- 172.21.15.150:0/2880873100 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b6c0681a0 con 0x7f9b6c144bb0 2024-03-17T23:47:43.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.772+0000 7f9b53fff640 1 -- 172.21.15.150:0/2880873100 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9b64005d10 con 0x7f9b6c144bb0 2024-03-17T23:47:43.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.772+0000 7f9b53fff640 1 --2- 172.21.15.150:0/2880873100 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f9b5806bbd0 0x7f9b5806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:43.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.773+0000 7f9b6b7fe640 1 --2- 172.21.15.150:0/2880873100 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f9b5806bbd0 0x7f9b5806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:43.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.773+0000 7f9b53fff640 1 -- 172.21.15.150:0/2880873100 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f9b6408b6b0 con 0x7f9b6c144bb0 2024-03-17T23:47:43.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.773+0000 7f9b6b7fe640 1 --2- 172.21.15.150:0/2880873100 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f9b5806bbd0 0x7f9b5806e090 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c000f60 tx=0x7f9b5c007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:43.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.777+0000 7f9b53fff640 1 -- 172.21.15.150:0/2880873100 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9b64022070 con 0x7f9b6c144bb0 2024-03-17T23:47:43.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.951+0000 7f9b72187640 1 -- 172.21.15.150:0/2880873100 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f9b6c0cde80 con 0x7f9b6c144bb0 2024-03-17T23:47:43.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.952+0000 7f9b53fff640 1 -- 172.21.15.150:0/2880873100 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7f9b64010040 con 0x7f9b6c144bb0 2024-03-17T23:47:43.952 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:43.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.954+0000 7f9b72187640 1 -- 172.21.15.150:0/2880873100 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f9b5806bbd0 msgr2=0x7f9b5806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:43.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.954+0000 7f9b72187640 1 --2- 172.21.15.150:0/2880873100 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f9b5806bbd0 0x7f9b5806e090 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c000f60 tx=0x7f9b5c007d50 comp rx=0 tx=0).stop 2024-03-17T23:47:43.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.954+0000 7f9b72187640 1 -- 172.21.15.150:0/2880873100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b6c144bb0 msgr2=0x7f9b6c1f3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:43.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.954+0000 7f9b72187640 1 --2- 172.21.15.150:0/2880873100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b6c144bb0 0x7f9b6c1f3f90 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9b6400efc0 tx=0x7f9b6400c490 comp rx=0 tx=0).stop 2024-03-17T23:47:43.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.954+0000 7f9b72187640 1 -- 172.21.15.150:0/2880873100 shutdown_connections 2024-03-17T23:47:43.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.954+0000 7f9b72187640 1 --2- 172.21.15.150:0/2880873100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b6c144bb0 0x7f9b6c1f3f90 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.954+0000 7f9b72187640 1 --2- 172.21.15.150:0/2880873100 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9b6c0ce4d0 0x7f9b6c0b4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.954+0000 7f9b72187640 1 --2- 172.21.15.150:0/2880873100 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f9b5806bbd0 0x7f9b5806e090 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.954+0000 7f9b72187640 1 --2- 172.21.15.150:0/2880873100 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b6c06f400 0x7f9b6c0ae6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:43.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.954+0000 7f9b72187640 1 -- 172.21.15.150:0/2880873100 >> 172.21.15.150:0/2880873100 conn(0x7f9b6c06b190 msgr2=0x7f9b6c13ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:43.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.954+0000 7f9b72187640 1 -- 172.21.15.150:0/2880873100 shutdown_connections 2024-03-17T23:47:43.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:43.954+0000 7f9b72187640 1 -- 172.21.15.150:0/2880873100 wait complete. 2024-03-17T23:47:43.966 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-17T23:47:44.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.068+0000 7fb5244de640 1 Processor -- start 2024-03-17T23:47:44.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.068+0000 7fb5244de640 1 -- start start 2024-03-17T23:47:44.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.069+0000 7fb5244de640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb51c06ccb0 0x7fb51c09a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:44.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.069+0000 7fb5244de640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb51c06c750 con 0x7fb51c06d090 2024-03-17T23:47:44.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.069+0000 7fb5244de640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb51c15edf0 con 0x7fb51c0d04c0 2024-03-17T23:47:44.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.069+0000 7fb5244de640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fb51c0d0030 con 0x7fb51c06ccb0 2024-03-17T23:47:44.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.069+0000 7fb52227a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb51c06ccb0 0x7fb51c09a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:44.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.069+0000 7fb521a79640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb51c06d090 0x7fb51c15e4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59266/0 (socket says 172.21.15.150:59266) 2024-03-17T23:47:44.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.069+0000 7fb521a79640 1 -- 172.21.15.150:0/3897877593 learned_addr learned my addr 172.21.15.150:0/3897877593 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:44.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.069+0000 7fb52227a640 1 -- 172.21.15.150:0/3897877593 >> v1:172.21.15.157:6789/0 conn(0x7fb51c0d04c0 legacy=0x7fb51c0d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:44.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.069+0000 7fb521278640 1 -- 172.21.15.150:0/3897877593 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1965461293 0 0) 0x7fb51c15edf0 con 0x7fb51c0d04c0 2024-03-17T23:47:44.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.069+0000 7fb52227a640 1 -- 172.21.15.150:0/3897877593 >> v1:172.21.15.150:6789/0 conn(0x7fb51c06d090 legacy=0x7fb51c15e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:44.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.069+0000 7fb52227a640 1 -- 172.21.15.150:0/3897877593 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb51c15f0d0 con 0x7fb51c06ccb0 2024-03-17T23:47:44.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.070+0000 7fb52227a640 1 --2- 172.21.15.150:0/3897877593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb51c06ccb0 0x7fb51c09a830 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb518009d90 tx=0x7fb51802bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebcbe2347db404ae server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:44.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.070+0000 7fb521278640 1 -- 172.21.15.150:0/3897877593 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb518039070 con 0x7fb51c06ccb0 2024-03-17T23:47:44.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.070+0000 7fb521278640 1 -- 172.21.15.150:0/3897877593 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb51802cab0 con 0x7fb51c06ccb0 2024-03-17T23:47:44.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.070+0000 7fb521278640 1 -- 172.21.15.150:0/3897877593 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb518034440 con 0x7fb51c06ccb0 2024-03-17T23:47:44.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.070+0000 7fb5244de640 1 -- 172.21.15.150:0/3897877593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb51c06ccb0 msgr2=0x7fb51c09a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:44.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.070+0000 7fb5244de640 1 --2- 172.21.15.150:0/3897877593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb51c06ccb0 0x7fb51c09a830 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb518009d90 tx=0x7fb51802bd80 comp rx=0 tx=0).stop 2024-03-17T23:47:44.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.071+0000 7fb5244de640 1 -- 172.21.15.150:0/3897877593 shutdown_connections 2024-03-17T23:47:44.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.071+0000 7fb5244de640 1 --2- 172.21.15.150:0/3897877593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb51c06ccb0 0x7fb51c09a830 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:44.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.071+0000 7fb5244de640 1 -- 172.21.15.150:0/3897877593 >> 172.21.15.150:0/3897877593 conn(0x7fb51c0b36b0 msgr2=0x7fb51c0b3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:44.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.071+0000 7fb5244de640 1 -- 172.21.15.150:0/3897877593 shutdown_connections 2024-03-17T23:47:44.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.071+0000 7fb5244de640 1 -- 172.21.15.150:0/3897877593 wait complete. 2024-03-17T23:47:44.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.072+0000 7fb5244de640 1 Processor -- start 2024-03-17T23:47:44.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.072+0000 7fb5244de640 1 -- start start 2024-03-17T23:47:44.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.072+0000 7fb5244de640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb51c06ccb0 0x7fb51c15d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:44.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.072+0000 7fb5244de640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb51c06d090 0x7fb51c14e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:44.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.072+0000 7fb5244de640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb51c0d04c0 0x7fb51c0b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:44.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.072+0000 7fb5244de640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb51c06d4f0 con 0x7fb51c0d04c0 2024-03-17T23:47:44.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.072+0000 7fb5244de640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb51c15f690 con 0x7fb51c06d090 2024-03-17T23:47:44.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.072+0000 7fb5244de640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb51c0d1060 con 0x7fb51c06ccb0 2024-03-17T23:47:44.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.072+0000 7fb522a7b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb51c0d04c0 0x7fb51c0b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:44.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.073+0000 7fb522a7b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb51c0d04c0 0x7fb51c0b4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46436/0 (socket says 172.21.15.150:46436) 2024-03-17T23:47:44.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.073+0000 7fb522a7b640 1 -- 172.21.15.150:0/2151636151 learned_addr learned my addr 172.21.15.150:0/2151636151 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:44.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.073+0000 7fb52227a640 1 --2- 172.21.15.150:0/2151636151 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb51c06ccb0 0x7fb51c15d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:44.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.073+0000 7fb521a79640 1 --2- 172.21.15.150:0/2151636151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb51c06d090 0x7fb51c14e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:44.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.073+0000 7fb522a7b640 1 -- 172.21.15.150:0/2151636151 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb51c06ccb0 msgr2=0x7fb51c15d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:44.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.073+0000 7fb522a7b640 1 --2- 172.21.15.150:0/2151636151 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb51c06ccb0 0x7fb51c15d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:44.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.073+0000 7fb522a7b640 1 -- 172.21.15.150:0/2151636151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb51c06d090 msgr2=0x7fb51c14e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:44.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.073+0000 7fb522a7b640 1 --2- 172.21.15.150:0/2151636151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb51c06d090 0x7fb51c14e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:44.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.073+0000 7fb522a7b640 1 -- 172.21.15.150:0/2151636151 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb51c0c8980 con 0x7fb51c0d04c0 2024-03-17T23:47:44.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.073+0000 7fb522a7b640 1 --2- 172.21.15.150:0/2151636151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb51c0d04c0 0x7fb51c0b4d00 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb51400c3e0 tx=0x7fb51400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:44.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.073+0000 7fb512ffd640 1 -- 172.21.15.150:0/2151636151 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb514017070 con 0x7fb51c0d04c0 2024-03-17T23:47:44.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.073+0000 7fb512ffd640 1 -- 172.21.15.150:0/2151636151 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5140035f0 con 0x7fb51c0d04c0 2024-03-17T23:47:44.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.073+0000 7fb5244de640 1 -- 172.21.15.150:0/2151636151 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb51c0c5a20 con 0x7fb51c0d04c0 2024-03-17T23:47:44.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.073+0000 7fb512ffd640 1 -- 172.21.15.150:0/2151636151 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5140039e0 con 0x7fb51c0d04c0 2024-03-17T23:47:44.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.073+0000 7fb5244de640 1 -- 172.21.15.150:0/2151636151 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb51c0c86a0 con 0x7fb51c0d04c0 2024-03-17T23:47:44.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.074+0000 7fb5244de640 1 -- 172.21.15.150:0/2151636151 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb51c15bf00 con 0x7fb51c0d04c0 2024-03-17T23:47:44.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.075+0000 7fb512ffd640 1 -- 172.21.15.150:0/2151636151 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb51401a070 con 0x7fb51c0d04c0 2024-03-17T23:47:44.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.075+0000 7fb512ffd640 1 --2- 172.21.15.150:0/2151636151 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb50006bbd0 0x7fb50006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:44.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.075+0000 7fb512ffd640 1 -- 172.21.15.150:0/2151636151 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fb51408a170 con 0x7fb51c0d04c0 2024-03-17T23:47:44.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.075+0000 7fb52227a640 1 --2- 172.21.15.150:0/2151636151 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb50006bbd0 0x7fb50006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:44.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.075+0000 7fb52227a640 1 --2- 172.21.15.150:0/2151636151 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb50006bbd0 0x7fb50006e090 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb518036000 tx=0x7fb518030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:44.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.079+0000 7fb512ffd640 1 -- 172.21.15.150:0/2151636151 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb51c15bf00 con 0x7fb51c0d04c0 2024-03-17T23:47:44.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.265+0000 7fb5244de640 1 -- 172.21.15.150:0/2151636151 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb51c0681a0 con 0x7fb51c0d04c0 2024-03-17T23:47:44.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.266+0000 7fb512ffd640 1 -- 172.21.15.150:0/2151636151 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) ==== 80+0+0 (secure 0 0 0) 0x7fb514058210 con 0x7fb51c0d04c0 2024-03-17T23:47:44.267 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-03-17T23:47:44.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.268+0000 7fb5244de640 1 -- 172.21.15.150:0/2151636151 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb50006bbd0 msgr2=0x7fb50006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:44.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.268+0000 7fb5244de640 1 --2- 172.21.15.150:0/2151636151 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb50006bbd0 0x7fb50006e090 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb518036000 tx=0x7fb518030000 comp rx=0 tx=0).stop 2024-03-17T23:47:44.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.268+0000 7fb5244de640 1 -- 172.21.15.150:0/2151636151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb51c0d04c0 msgr2=0x7fb51c0b4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:44.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.268+0000 7fb5244de640 1 --2- 172.21.15.150:0/2151636151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb51c0d04c0 0x7fb51c0b4d00 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb51400c3e0 tx=0x7fb51400c8a0 comp rx=0 tx=0).stop 2024-03-17T23:47:44.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.268+0000 7fb5244de640 1 -- 172.21.15.150:0/2151636151 shutdown_connections 2024-03-17T23:47:44.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.268+0000 7fb5244de640 1 --2- 172.21.15.150:0/2151636151 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb51c0d04c0 0x7fb51c0b4d00 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:44.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.268+0000 7fb5244de640 1 --2- 172.21.15.150:0/2151636151 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb50006bbd0 0x7fb50006e090 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:44.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.268+0000 7fb5244de640 1 --2- 172.21.15.150:0/2151636151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb51c06d090 0x7fb51c14e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:44.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.268+0000 7fb5244de640 1 --2- 172.21.15.150:0/2151636151 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb51c06ccb0 0x7fb51c15d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:44.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.268+0000 7fb5244de640 1 -- 172.21.15.150:0/2151636151 >> 172.21.15.150:0/2151636151 conn(0x7fb51c0b36b0 msgr2=0x7fb51c073b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:44.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.268+0000 7fb5244de640 1 -- 172.21.15.150:0/2151636151 shutdown_connections 2024-03-17T23:47:44.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.268+0000 7fb5244de640 1 -- 172.21.15.150:0/2151636151 wait complete. 2024-03-17T23:47:44.279 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi150.front.sepia.ceph.com... 2024-03-17T23:47:44.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:44.280 DEBUG:teuthology.orchestra.run.smithi150:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-17T23:47:44.307 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:47:44.307 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi150.front.sepia.ceph.com 2024-03-17T23:47:44.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:44.307 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-17T23:47:44.362 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:47:44.363 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:47:44.379 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:47:44.379 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:47:44.435 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:44.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.535+0000 7f5599097640 1 Processor -- start 2024-03-17T23:47:44.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.535+0000 7f5599097640 1 -- start start 2024-03-17T23:47:44.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.536+0000 7f5599097640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f559415de30 0x7f55941619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:44.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.536+0000 7f5599097640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f55940cf120 0x7f55940d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:44.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.536+0000 7f5599097640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55940b9cc0 con 0x7f55940b9860 2024-03-17T23:47:44.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.536+0000 7f5599097640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f5594162940 con 0x7f55940cf120 2024-03-17T23:47:44.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.536+0000 7f5599097640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f55941620a0 con 0x7f559415de30 2024-03-17T23:47:44.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.536+0000 7f559359e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f55940cf120 0x7f55940d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:44.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.536+0000 7f559359e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f55940cf120 0x7f55940d2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46448/0 (socket says 172.21.15.150:46448) 2024-03-17T23:47:44.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.536+0000 7f559359e640 1 -- 172.21.15.150:0/3986083395 learned_addr learned my addr 172.21.15.150:0/3986083395 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:44.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.536+0000 7f5592d9d640 1 --2- 172.21.15.150:0/3986083395 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f559415de30 0x7f55941619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:44.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.536+0000 7f559359e640 1 -- 172.21.15.150:0/3986083395 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f559415de30 msgr2=0x7f55941619a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:44.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.536+0000 7f559359e640 1 --2- 172.21.15.150:0/3986083395 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f559415de30 0x7f55941619a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:44.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.536+0000 7f559359e640 1 -- 172.21.15.150:0/3986083395 >> v1:172.21.15.157:6789/0 conn(0x7f55940b9860 legacy=0x7f55940cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:44.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.536+0000 7f559359e640 1 -- 172.21.15.150:0/3986083395 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5594162380 con 0x7f55940cf120 2024-03-17T23:47:44.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.537+0000 7f559359e640 1 --2- 172.21.15.150:0/3986083395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55940cf120 0x7f55940d2690 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f558c009870 tx=0x7f558c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0565f8c0bdef420 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:44.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.537+0000 7f5591d9b640 1 -- 172.21.15.150:0/3986083395 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f558c035020 con 0x7f55940cf120 2024-03-17T23:47:44.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.537+0000 7f5591d9b640 1 -- 172.21.15.150:0/3986083395 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f558c031640 con 0x7f55940cf120 2024-03-17T23:47:44.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.537+0000 7f5591d9b640 1 -- 172.21.15.150:0/3986083395 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f558c02d040 con 0x7f55940cf120 2024-03-17T23:47:44.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.538+0000 7f5599097640 1 -- 172.21.15.150:0/3986083395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55940cf120 msgr2=0x7f55940d2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:44.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.538+0000 7f5599097640 1 --2- 172.21.15.150:0/3986083395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55940cf120 0x7f55940d2690 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f558c009870 tx=0x7f558c02b8f0 comp rx=0 tx=0).stop 2024-03-17T23:47:44.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.538+0000 7f5599097640 1 -- 172.21.15.150:0/3986083395 shutdown_connections 2024-03-17T23:47:44.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.538+0000 7f5599097640 1 --2- 172.21.15.150:0/3986083395 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55940cf120 0x7f55940d2690 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:44.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.538+0000 7f5599097640 1 --2- 172.21.15.150:0/3986083395 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f559415de30 0x7f55941619a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:44.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.538+0000 7f5599097640 1 -- 172.21.15.150:0/3986083395 >> 172.21.15.150:0/3986083395 conn(0x7f55940730b0 msgr2=0x7f55940d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:44.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.538+0000 7f5599097640 1 -- 172.21.15.150:0/3986083395 shutdown_connections 2024-03-17T23:47:44.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.538+0000 7f5599097640 1 -- 172.21.15.150:0/3986083395 wait complete. 2024-03-17T23:47:44.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.539+0000 7f5599097640 1 Processor -- start 2024-03-17T23:47:44.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.539+0000 7f5599097640 1 -- start start 2024-03-17T23:47:44.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.540+0000 7f5599097640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55940b9860 0x7f559415ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:44.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.540+0000 7f5599097640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55940cf120 0x7f55941f3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:44.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.540+0000 7f5599097640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f559415de30 0x7f55940ab140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:44.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.540+0000 7f5599097640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f559415e210 con 0x7f559415de30 2024-03-17T23:47:44.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.540+0000 7f5599097640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f55940d2170 con 0x7f55940cf120 2024-03-17T23:47:44.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.540+0000 7f5599097640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f55940d2bd0 con 0x7f55940b9860 2024-03-17T23:47:44.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.540+0000 7f559359e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f559415de30 0x7f55940ab140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:44.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.540+0000 7f5592d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55940b9860 0x7f559415ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:44.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.540+0000 7f559359e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f559415de30 0x7f55940ab140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46450/0 (socket says 172.21.15.150:46450) 2024-03-17T23:47:44.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.540+0000 7f559359e640 1 -- 172.21.15.150:0/2954029249 learned_addr learned my addr 172.21.15.150:0/2954029249 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:44.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.540+0000 7f559259c640 1 --2- 172.21.15.150:0/2954029249 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55940cf120 0x7f55941f3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:44.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.540+0000 7f5592d9d640 1 -- 172.21.15.150:0/2954029249 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55940cf120 msgr2=0x7f55941f3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:44.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.540+0000 7f5592d9d640 1 --2- 172.21.15.150:0/2954029249 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55940cf120 0x7f55941f3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:44.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.540+0000 7f5592d9d640 1 -- 172.21.15.150:0/2954029249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f559415de30 msgr2=0x7f55940ab140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:44.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.540+0000 7f5592d9d640 1 --2- 172.21.15.150:0/2954029249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f559415de30 0x7f55940ab140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:44.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.541+0000 7f5592d9d640 1 -- 172.21.15.150:0/2954029249 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f559415f5f0 con 0x7f55940b9860 2024-03-17T23:47:44.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.541+0000 7f559359e640 1 --2- 172.21.15.150:0/2954029249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f559415de30 0x7f55940ab140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:44.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.541+0000 7f5592d9d640 1 --2- 172.21.15.150:0/2954029249 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55940b9860 0x7f559415ce00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f558400ebc0 tx=0x7f558400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:44.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.541+0000 7f557b7fe640 1 -- 172.21.15.150:0/2954029249 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5584018020 con 0x7f55940b9860 2024-03-17T23:47:44.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.541+0000 7f557b7fe640 1 -- 172.21.15.150:0/2954029249 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f558400cda0 con 0x7f55940b9860 2024-03-17T23:47:44.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.541+0000 7f557b7fe640 1 -- 172.21.15.150:0/2954029249 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55840146c0 con 0x7f55940b9860 2024-03-17T23:47:44.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.541+0000 7f5599097640 1 -- 172.21.15.150:0/2954029249 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f559415f8d0 con 0x7f55940b9860 2024-03-17T23:47:44.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.541+0000 7f5599097640 1 -- 172.21.15.150:0/2954029249 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f559415fe90 con 0x7f55940b9860 2024-03-17T23:47:44.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.542+0000 7f5599097640 1 -- 172.21.15.150:0/2954029249 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55940681a0 con 0x7f55940b9860 2024-03-17T23:47:44.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.542+0000 7f557b7fe640 1 -- 172.21.15.150:0/2954029249 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5584005d10 con 0x7f55940b9860 2024-03-17T23:47:44.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.543+0000 7f557b7fe640 1 --2- 172.21.15.150:0/2954029249 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f558006bbd0 0x7f558006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:44.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.543+0000 7f557b7fe640 1 -- 172.21.15.150:0/2954029249 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f558408ba00 con 0x7f55940b9860 2024-03-17T23:47:44.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.543+0000 7f559259c640 1 --2- 172.21.15.150:0/2954029249 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f558006bbd0 0x7f558006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:44.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.543+0000 7f559259c640 1 --2- 172.21.15.150:0/2954029249 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f558006bbd0 0x7f558006e090 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f55940b76c0 tx=0x7f5588009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:44.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.547+0000 7f557b7fe640 1 -- 172.21.15.150:0/2954029249 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5584022030 con 0x7f55940b9860 2024-03-17T23:47:44.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.721+0000 7f5599097640 1 -- 172.21.15.150:0/2954029249 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f55940ca6a0 con 0x7f55940b9860 2024-03-17T23:47:44.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.722+0000 7f557b7fe640 1 -- 172.21.15.150:0/2954029249 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18881 (secure 0 0 0) 0x7f5584010040 con 0x7f55940b9860 2024-03-17T23:47:44.722 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:44.723 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":28,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:37.031864+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.e","mappings":[{"from":7,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:44.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.724+0000 7f5599097640 1 -- 172.21.15.150:0/2954029249 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f558006bbd0 msgr2=0x7f558006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:44.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.724+0000 7f5599097640 1 --2- 172.21.15.150:0/2954029249 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f558006bbd0 0x7f558006e090 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f55940b76c0 tx=0x7f5588009040 comp rx=0 tx=0).stop 2024-03-17T23:47:44.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.724+0000 7f5599097640 1 -- 172.21.15.150:0/2954029249 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55940b9860 msgr2=0x7f559415ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:44.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.724+0000 7f5599097640 1 --2- 172.21.15.150:0/2954029249 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55940b9860 0x7f559415ce00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f558400ebc0 tx=0x7f558400c490 comp rx=0 tx=0).stop 2024-03-17T23:47:44.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.724+0000 7f5599097640 1 -- 172.21.15.150:0/2954029249 shutdown_connections 2024-03-17T23:47:44.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.724+0000 7f5599097640 1 --2- 172.21.15.150:0/2954029249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f559415de30 0x7f55940ab140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:44.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.724+0000 7f5599097640 1 --2- 172.21.15.150:0/2954029249 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f558006bbd0 0x7f558006e090 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:44.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.724+0000 7f5599097640 1 --2- 172.21.15.150:0/2954029249 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55940cf120 0x7f55941f3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:44.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.724+0000 7f5599097640 1 --2- 172.21.15.150:0/2954029249 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55940b9860 0x7f559415ce00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:44.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.724+0000 7f5599097640 1 -- 172.21.15.150:0/2954029249 >> 172.21.15.150:0/2954029249 conn(0x7f55940730b0 msgr2=0x7f55940b8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:44.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.724+0000 7f5599097640 1 -- 172.21.15.150:0/2954029249 shutdown_connections 2024-03-17T23:47:44.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.724+0000 7f5599097640 1 -- 172.21.15.150:0/2954029249 wait complete. 2024-03-17T23:47:44.736 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:46:53.610678+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:46:54.227335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-17T23:46:57.949235+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-17T23:47:44.736 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:47:44.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.840+0000 7f9439636640 1 Processor -- start 2024-03-17T23:47:44.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.840+0000 7f9439636640 1 -- start start 2024-03-17T23:47:44.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.840+0000 7f9439636640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9434155320 0x7f943415bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:44.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.840+0000 7f9439636640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f943406dd30 0x7f94340ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:44.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.840+0000 7f9439636640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94340cf3b0 con 0x7f94340cd250 2024-03-17T23:47:44.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.840+0000 7f9439636640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f943415c550 con 0x7f943406dd30 2024-03-17T23:47:44.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.840+0000 7f9439636640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f94340cf690 con 0x7f9434155320 2024-03-17T23:47:44.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.841+0000 7f94327fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f943406dd30 0x7f94340ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:44.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.841+0000 7f9432ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9434155320 0x7f943415bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:44.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.841+0000 7f94327fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f943406dd30 0x7f94340ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46454/0 (socket says 172.21.15.150:46454) 2024-03-17T23:47:44.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.841+0000 7f94337fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f94340cd250 0x7f943415e980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:46840/0 (socket says 172.21.15.150:46840) 2024-03-17T23:47:44.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.841+0000 7f94327fc640 1 -- 172.21.15.150:0/591444111 learned_addr learned my addr 172.21.15.150:0/591444111 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:44.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.841+0000 7f94327fc640 1 -- 172.21.15.150:0/591444111 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9434155320 msgr2=0x7f943415bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:44.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.841+0000 7f94327fc640 1 --2- 172.21.15.150:0/591444111 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9434155320 0x7f943415bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:44.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.841+0000 7f94327fc640 1 -- 172.21.15.150:0/591444111 >> v1:172.21.15.157:6789/0 conn(0x7f94340cd250 legacy=0x7f943415e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:44.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.841+0000 7f9431ffb640 1 -- 172.21.15.150:0/591444111 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2969457220 0 0) 0x7f94340cf3b0 con 0x7f94340cd250 2024-03-17T23:47:44.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.841+0000 7f94327fc640 1 -- 172.21.15.150:0/591444111 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f943415c7a0 con 0x7f943406dd30 2024-03-17T23:47:44.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.842+0000 7f94327fc640 1 --2- 172.21.15.150:0/591444111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f943406dd30 0x7f94340ccd10 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9428009870 tx=0x7f942802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4441eae7f643ebad server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:44.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.842+0000 7f9431ffb640 1 -- 172.21.15.150:0/591444111 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9428034070 con 0x7f943406dd30 2024-03-17T23:47:44.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.842+0000 7f9431ffb640 1 -- 172.21.15.150:0/591444111 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9428030840 con 0x7f943406dd30 2024-03-17T23:47:44.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.842+0000 7f9431ffb640 1 -- 172.21.15.150:0/591444111 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9428030c10 con 0x7f943406dd30 2024-03-17T23:47:44.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.842+0000 7f9439636640 1 -- 172.21.15.150:0/591444111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f943406dd30 msgr2=0x7f94340ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:44.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.842+0000 7f9439636640 1 --2- 172.21.15.150:0/591444111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f943406dd30 0x7f94340ccd10 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9428009870 tx=0x7f942802b980 comp rx=0 tx=0).stop 2024-03-17T23:47:44.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.842+0000 7f9439636640 1 -- 172.21.15.150:0/591444111 shutdown_connections 2024-03-17T23:47:44.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.842+0000 7f9439636640 1 --2- 172.21.15.150:0/591444111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f943406dd30 0x7f94340ccd10 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:44.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.842+0000 7f9439636640 1 --2- 172.21.15.150:0/591444111 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9434155320 0x7f943415bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:44.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.842+0000 7f9439636640 1 -- 172.21.15.150:0/591444111 >> 172.21.15.150:0/591444111 conn(0x7f943406b190 msgr2=0x7f943406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:44.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.842+0000 7f9439636640 1 -- 172.21.15.150:0/591444111 shutdown_connections 2024-03-17T23:47:44.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.842+0000 7f9439636640 1 -- 172.21.15.150:0/591444111 wait complete. 2024-03-17T23:47:44.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.844+0000 7f9439636640 1 Processor -- start 2024-03-17T23:47:44.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.844+0000 7f9439636640 1 -- start start 2024-03-17T23:47:44.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.844+0000 7f9439636640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f943406dd30 0x7f943413fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:44.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.844+0000 7f9439636640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94340cd250 0x7f943414ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:44.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.844+0000 7f9439636640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9434155320 0x7f943414e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:44.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.844+0000 7f9439636640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9434155700 con 0x7f943406dd30 2024-03-17T23:47:44.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.844+0000 7f9439636640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f943406e190 con 0x7f94340cd250 2024-03-17T23:47:44.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.844+0000 7f9439636640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9434155880 con 0x7f9434155320 2024-03-17T23:47:44.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.844+0000 7f9432ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f943406dd30 0x7f943413fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:44.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.845+0000 7f9432ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f943406dd30 0x7f943413fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46460/0 (socket says 172.21.15.150:46460) 2024-03-17T23:47:44.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.845+0000 7f9432ffd640 1 -- 172.21.15.150:0/3309915159 learned_addr learned my addr 172.21.15.150:0/3309915159 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:44.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.845+0000 7f94327fc640 1 --2- 172.21.15.150:0/3309915159 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94340cd250 0x7f943414ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:44.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.845+0000 7f94337fe640 1 --2- 172.21.15.150:0/3309915159 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9434155320 0x7f943414e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:44.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.845+0000 7f9432ffd640 1 -- 172.21.15.150:0/3309915159 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9434155320 msgr2=0x7f943414e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:44.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.845+0000 7f9432ffd640 1 --2- 172.21.15.150:0/3309915159 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9434155320 0x7f943414e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:44.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.845+0000 7f9432ffd640 1 -- 172.21.15.150:0/3309915159 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94340cd250 msgr2=0x7f943414ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:44.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.845+0000 7f9432ffd640 1 --2- 172.21.15.150:0/3309915159 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94340cd250 0x7f943414ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:44.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.845+0000 7f9432ffd640 1 -- 172.21.15.150:0/3309915159 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f943414e960 con 0x7f943406dd30 2024-03-17T23:47:44.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.845+0000 7f9432ffd640 1 --2- 172.21.15.150:0/3309915159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f943406dd30 0x7f943413fcf0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f942400bfd0 tx=0x7f942400ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:44.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.845+0000 7f941b7fe640 1 -- 172.21.15.150:0/3309915159 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942400de00 con 0x7f943406dd30 2024-03-17T23:47:44.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.845+0000 7f941b7fe640 1 -- 172.21.15.150:0/3309915159 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9424010440 con 0x7f943406dd30 2024-03-17T23:47:44.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.845+0000 7f9439636640 1 -- 172.21.15.150:0/3309915159 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f943414ec40 con 0x7f943406dd30 2024-03-17T23:47:44.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.845+0000 7f941b7fe640 1 -- 172.21.15.150:0/3309915159 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9424010830 con 0x7f943406dd30 2024-03-17T23:47:44.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.845+0000 7f9439636640 1 -- 172.21.15.150:0/3309915159 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f943414f200 con 0x7f943406dd30 2024-03-17T23:47:44.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.846+0000 7f9439636640 1 -- 172.21.15.150:0/3309915159 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94340681a0 con 0x7f943406dd30 2024-03-17T23:47:44.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.847+0000 7f941b7fe640 1 -- 172.21.15.150:0/3309915159 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f94240109d0 con 0x7f943406dd30 2024-03-17T23:47:44.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.847+0000 7f941b7fe640 1 --2- 172.21.15.150:0/3309915159 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f942006bbd0 0x7f942006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:44.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.847+0000 7f94327fc640 1 --2- 172.21.15.150:0/3309915159 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f942006bbd0 0x7f942006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:44.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.847+0000 7f941b7fe640 1 -- 172.21.15.150:0/3309915159 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f942408b610 con 0x7f943406dd30 2024-03-17T23:47:44.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.848+0000 7f94327fc640 1 --2- 172.21.15.150:0/3309915159 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f942006bbd0 0x7f942006e090 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9428036000 tx=0x7f9428008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:44.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:44.852+0000 7f941b7fe640 1 -- 172.21.15.150:0/3309915159 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9424014030 con 0x7f943406dd30 2024-03-17T23:47:45.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.022+0000 7f9439636640 1 -- 172.21.15.150:0/3309915159 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9434154610 con 0x7f943406dd30 2024-03-17T23:47:45.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.022+0000 7f941b7fe640 1 -- 172.21.15.150:0/3309915159 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7f94240596b0 con 0x7f943406dd30 2024-03-17T23:47:45.023 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:47:45.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.024+0000 7f9439636640 1 -- 172.21.15.150:0/3309915159 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f942006bbd0 msgr2=0x7f942006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:45.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.024+0000 7f9439636640 1 --2- 172.21.15.150:0/3309915159 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f942006bbd0 0x7f942006e090 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9428036000 tx=0x7f9428008030 comp rx=0 tx=0).stop 2024-03-17T23:47:45.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.024+0000 7f9439636640 1 -- 172.21.15.150:0/3309915159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f943406dd30 msgr2=0x7f943413fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:45.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.024+0000 7f9439636640 1 --2- 172.21.15.150:0/3309915159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f943406dd30 0x7f943413fcf0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f942400bfd0 tx=0x7f942400ee60 comp rx=0 tx=0).stop 2024-03-17T23:47:45.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.024+0000 7f9439636640 1 -- 172.21.15.150:0/3309915159 shutdown_connections 2024-03-17T23:47:45.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.024+0000 7f9439636640 1 --2- 172.21.15.150:0/3309915159 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9434155320 0x7f943414e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.024+0000 7f9439636640 1 --2- 172.21.15.150:0/3309915159 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f942006bbd0 0x7f942006e090 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.024+0000 7f9439636640 1 --2- 172.21.15.150:0/3309915159 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94340cd250 0x7f943414ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.024+0000 7f9439636640 1 --2- 172.21.15.150:0/3309915159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f943406dd30 0x7f943413fcf0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.024+0000 7f9439636640 1 -- 172.21.15.150:0/3309915159 >> 172.21.15.150:0/3309915159 conn(0x7f943406b190 msgr2=0x7f943415a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:45.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.024+0000 7f9439636640 1 -- 172.21.15.150:0/3309915159 shutdown_connections 2024-03-17T23:47:45.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.024+0000 7f9439636640 1 -- 172.21.15.150:0/3309915159 wait complete. 2024-03-17T23:47:45.036 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:47:45.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.139+0000 7f110ab09640 1 Processor -- start 2024-03-17T23:47:45.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.139+0000 7f110ab09640 1 -- start start 2024-03-17T23:47:45.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.140+0000 7f110ab09640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1104155220 0x7f110415c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:45.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.140+0000 7f110ab09640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11040b5170 con 0x7f11040ad7f0 2024-03-17T23:47:45.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.140+0000 7f110ab09640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f110415bdf0 con 0x7f110415c8a0 2024-03-17T23:47:45.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.140+0000 7f110ab09640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f110415c0d0 con 0x7f1104155220 2024-03-17T23:47:45.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.140+0000 7f11090a6640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f11040ad7f0 0x7f11040adc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59274/0 (socket says 172.21.15.150:59274) 2024-03-17T23:47:45.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.140+0000 7f11090a6640 1 -- 172.21.15.150:0/3049361112 learned_addr learned my addr 172.21.15.150:0/3049361112 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:45.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.140+0000 7f11088a5640 1 --2- 172.21.15.150:0/3049361112 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1104155220 0x7f110415c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:45.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.140+0000 7f11037fe640 1 -- 172.21.15.150:0/3049361112 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2362807334 0 0) 0x7f11040b5170 con 0x7f11040ad7f0 2024-03-17T23:47:45.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.140+0000 7f11037fe640 1 -- 172.21.15.150:0/3049361112 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10f4003520 con 0x7f11040ad7f0 2024-03-17T23:47:45.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.140+0000 7f11037fe640 1 -- 172.21.15.150:0/3049361112 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3793261671 0 0) 0x7f110415bdf0 con 0x7f110415c8a0 2024-03-17T23:47:45.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.140+0000 7f11037fe640 1 -- 172.21.15.150:0/3049361112 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11040b5170 con 0x7f110415c8a0 2024-03-17T23:47:45.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.140+0000 7f11088a5640 1 -- 172.21.15.150:0/3049361112 >> v1:172.21.15.157:6789/0 conn(0x7f110415c8a0 legacy=0x7f11040b4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:45.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.140+0000 7f11088a5640 1 -- 172.21.15.150:0/3049361112 >> v1:172.21.15.150:6789/0 conn(0x7f11040ad7f0 legacy=0x7f11040adc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:45.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.140+0000 7f11088a5640 1 -- 172.21.15.150:0/3049361112 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11040b36a0 con 0x7f1104155220 2024-03-17T23:47:45.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.141+0000 7f11088a5640 1 --2- 172.21.15.150:0/3049361112 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1104155220 0x7f110415c360 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f10f80097d0 tx=0x7f10f802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77d8de1f9e5c16c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:45.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.141+0000 7f11037fe640 1 -- 172.21.15.150:0/3049361112 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f8035020 con 0x7f1104155220 2024-03-17T23:47:45.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.141+0000 7f11037fe640 1 -- 172.21.15.150:0/3049361112 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10f8031890 con 0x7f1104155220 2024-03-17T23:47:45.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.141+0000 7f11037fe640 1 -- 172.21.15.150:0/3049361112 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f8031c40 con 0x7f1104155220 2024-03-17T23:47:45.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.142+0000 7f110ab09640 1 -- 172.21.15.150:0/3049361112 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1104155220 msgr2=0x7f110415c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:45.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.142+0000 7f110ab09640 1 --2- 172.21.15.150:0/3049361112 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1104155220 0x7f110415c360 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f10f80097d0 tx=0x7f10f802b930 comp rx=0 tx=0).stop 2024-03-17T23:47:45.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.142+0000 7f110ab09640 1 -- 172.21.15.150:0/3049361112 shutdown_connections 2024-03-17T23:47:45.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.142+0000 7f110ab09640 1 --2- 172.21.15.150:0/3049361112 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1104155220 0x7f110415c360 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.142+0000 7f110ab09640 1 -- 172.21.15.150:0/3049361112 >> 172.21.15.150:0/3049361112 conn(0x7f110406b370 msgr2=0x7f110406b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:45.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.142+0000 7f110ab09640 1 -- 172.21.15.150:0/3049361112 shutdown_connections 2024-03-17T23:47:45.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.142+0000 7f110ab09640 1 -- 172.21.15.150:0/3049361112 wait complete. 2024-03-17T23:47:45.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.143+0000 7f110ab09640 1 Processor -- start 2024-03-17T23:47:45.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.143+0000 7f110ab09640 1 -- start start 2024-03-17T23:47:45.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.144+0000 7f110ab09640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11040ad7f0 0x7f1104140e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:45.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.144+0000 7f110ab09640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1104155220 0x7f1104143950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:45.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.144+0000 7f110ab09640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110415c8a0 0x7f1104143e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:45.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.144+0000 7f110ab09640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1104155600 con 0x7f110415c8a0 2024-03-17T23:47:45.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.144+0000 7f110ab09640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f11040b3c60 con 0x7f1104155220 2024-03-17T23:47:45.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.144+0000 7f110ab09640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f11040b4890 con 0x7f11040ad7f0 2024-03-17T23:47:45.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.144+0000 7f11090a6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110415c8a0 0x7f1104143e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:45.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.144+0000 7f11088a5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11040ad7f0 0x7f1104140e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:45.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.144+0000 7f11090a6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110415c8a0 0x7f1104143e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46466/0 (socket says 172.21.15.150:46466) 2024-03-17T23:47:45.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.144+0000 7f11088a5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11040ad7f0 0x7f1104140e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:38650/0 (socket says 172.21.15.150:38650) 2024-03-17T23:47:45.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.144+0000 7f11090a6640 1 -- 172.21.15.150:0/547498461 learned_addr learned my addr 172.21.15.150:0/547498461 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:45.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.144+0000 7f1103fff640 1 --2- 172.21.15.150:0/547498461 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1104155220 0x7f1104143950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:45.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.144+0000 7f11088a5640 1 -- 172.21.15.150:0/547498461 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1104155220 msgr2=0x7f1104143950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:45.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.144+0000 7f11088a5640 1 --2- 172.21.15.150:0/547498461 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1104155220 0x7f1104143950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.144+0000 7f11088a5640 1 -- 172.21.15.150:0/547498461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110415c8a0 msgr2=0x7f1104143e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:45.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.144+0000 7f11088a5640 1 --2- 172.21.15.150:0/547498461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110415c8a0 0x7f1104143e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.144+0000 7f11088a5640 1 -- 172.21.15.150:0/547498461 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1104141960 con 0x7f11040ad7f0 2024-03-17T23:47:45.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.144+0000 7f11090a6640 1 --2- 172.21.15.150:0/547498461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110415c8a0 0x7f1104143e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:45.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.144+0000 7f11088a5640 1 --2- 172.21.15.150:0/547498461 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11040ad7f0 0x7f1104140e70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f10f8033000 tx=0x7f10f80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:45.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.145+0000 7f11017fa640 1 -- 172.21.15.150:0/547498461 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f8035040 con 0x7f11040ad7f0 2024-03-17T23:47:45.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.145+0000 7f11017fa640 1 -- 172.21.15.150:0/547498461 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10f8030d20 con 0x7f11040ad7f0 2024-03-17T23:47:45.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.145+0000 7f110ab09640 1 -- 172.21.15.150:0/547498461 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11041444f0 con 0x7f11040ad7f0 2024-03-17T23:47:45.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.145+0000 7f11017fa640 1 -- 172.21.15.150:0/547498461 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f8006e10 con 0x7f11040ad7f0 2024-03-17T23:47:45.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.145+0000 7f110ab09640 1 -- 172.21.15.150:0/547498461 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1104141eb0 con 0x7f11040ad7f0 2024-03-17T23:47:45.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.146+0000 7f110ab09640 1 -- 172.21.15.150:0/547498461 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11040681a0 con 0x7f11040ad7f0 2024-03-17T23:47:45.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.146+0000 7f11017fa640 1 -- 172.21.15.150:0/547498461 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f10f8038020 con 0x7f11040ad7f0 2024-03-17T23:47:45.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.146+0000 7f11017fa640 1 --2- 172.21.15.150:0/547498461 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f10e806bbd0 0x7f10e806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:45.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.146+0000 7f11017fa640 1 -- 172.21.15.150:0/547498461 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f10f80acf70 con 0x7f11040ad7f0 2024-03-17T23:47:45.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.146+0000 7f1103fff640 1 --2- 172.21.15.150:0/547498461 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f10e806bbd0 0x7f10e806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:45.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.147+0000 7f1103fff640 1 --2- 172.21.15.150:0/547498461 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f10e806bbd0 0x7f10e806e090 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f10f0007fd0 tx=0x7f10f0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:45.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.151+0000 7f11017fa640 1 -- 172.21.15.150:0/547498461 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f10f802d040 con 0x7f11040ad7f0 2024-03-17T23:47:45.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.323+0000 7f110ab09640 1 -- 172.21.15.150:0/547498461 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f110406f7a0 con 0x7f11040ad7f0 2024-03-17T23:47:45.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.324+0000 7f11017fa640 1 -- 172.21.15.150:0/547498461 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7f10f807afb0 con 0x7f11040ad7f0 2024-03-17T23:47:45.325 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:45.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.326+0000 7f110ab09640 1 -- 172.21.15.150:0/547498461 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f10e806bbd0 msgr2=0x7f10e806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:45.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.326+0000 7f110ab09640 1 --2- 172.21.15.150:0/547498461 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f10e806bbd0 0x7f10e806e090 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f10f0007fd0 tx=0x7f10f0009040 comp rx=0 tx=0).stop 2024-03-17T23:47:45.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.326+0000 7f110ab09640 1 -- 172.21.15.150:0/547498461 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11040ad7f0 msgr2=0x7f1104140e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:45.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.326+0000 7f110ab09640 1 --2- 172.21.15.150:0/547498461 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11040ad7f0 0x7f1104140e70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f10f8033000 tx=0x7f10f80046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.326+0000 7f110ab09640 1 -- 172.21.15.150:0/547498461 shutdown_connections 2024-03-17T23:47:45.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.326+0000 7f110ab09640 1 --2- 172.21.15.150:0/547498461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f110415c8a0 0x7f1104143e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.326+0000 7f110ab09640 1 --2- 172.21.15.150:0/547498461 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f10e806bbd0 0x7f10e806e090 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.326+0000 7f110ab09640 1 --2- 172.21.15.150:0/547498461 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1104155220 0x7f1104143950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.326+0000 7f110ab09640 1 --2- 172.21.15.150:0/547498461 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11040ad7f0 0x7f1104140e70 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.326+0000 7f110ab09640 1 -- 172.21.15.150:0/547498461 >> 172.21.15.150:0/547498461 conn(0x7f110406b370 msgr2=0x7f11040b2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:45.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.326+0000 7f110ab09640 1 -- 172.21.15.150:0/547498461 shutdown_connections 2024-03-17T23:47:45.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.326+0000 7f110ab09640 1 -- 172.21.15.150:0/547498461 wait complete. 2024-03-17T23:47:45.338 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:47:45.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.443+0000 7f67ec3b2640 1 Processor -- start 2024-03-17T23:47:45.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.444+0000 7f67ec3b2640 1 -- start start 2024-03-17T23:47:45.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.444+0000 7f67ec3b2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f67e406f400 0x7f67e40cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:45.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.444+0000 7f67ec3b2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67e4145170 con 0x7f67e4144bb0 2024-03-17T23:47:45.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.445+0000 7f67ec3b2640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67e4145450 con 0x7f67e40ce4d0 2024-03-17T23:47:45.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.445+0000 7f67ec3b2640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f67e40ce940 con 0x7f67e406f400 2024-03-17T23:47:45.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.445+0000 7f67ea14e640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f67e4144bb0 0x7f67e415ce50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59282/0 (socket says 172.21.15.150:59282) 2024-03-17T23:47:45.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.445+0000 7f67ea14e640 1 -- 172.21.15.150:0/410942055 learned_addr learned my addr 172.21.15.150:0/410942055 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:45.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.445+0000 7f67e994d640 1 --2- 172.21.15.150:0/410942055 >> v2:172.21.15.157:3300/0 conn(0x7f67e406f400 0x7f67e40cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:45.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.445+0000 7f67e914c640 1 -- 172.21.15.150:0/410942055 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1747589352 0 0) 0x7f67e4145170 con 0x7f67e4144bb0 2024-03-17T23:47:45.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.445+0000 7f67e914c640 1 -- 172.21.15.150:0/410942055 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67d0003520 con 0x7f67e4144bb0 2024-03-17T23:47:45.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.445+0000 7f67e994d640 1 -- 172.21.15.150:0/410942055 >> v1:172.21.15.157:6789/0 conn(0x7f67e40ce4d0 legacy=0x7f67e4165a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:45.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.445+0000 7f67e914c640 1 -- 172.21.15.150:0/410942055 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 344875237 0 0) 0x7f67e4145450 con 0x7f67e40ce4d0 2024-03-17T23:47:45.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.445+0000 7f67e994d640 1 -- 172.21.15.150:0/410942055 >> v1:172.21.15.150:6789/0 conn(0x7f67e4144bb0 legacy=0x7f67e415ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:45.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.445+0000 7f67e994d640 1 -- 172.21.15.150:0/410942055 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67e4145730 con 0x7f67e406f400 2024-03-17T23:47:45.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.446+0000 7f67e994d640 1 --2- 172.21.15.150:0/410942055 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67e406f400 0x7f67e40cdf90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f67d40097a0 tx=0x7f67d402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b947b58cc0a46d00 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:45.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.446+0000 7f67e914c640 1 -- 172.21.15.150:0/410942055 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d4035020 con 0x7f67e406f400 2024-03-17T23:47:45.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.446+0000 7f67e914c640 1 -- 172.21.15.150:0/410942055 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67d4031830 con 0x7f67e406f400 2024-03-17T23:47:45.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.446+0000 7f67e914c640 1 -- 172.21.15.150:0/410942055 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d4031c00 con 0x7f67e406f400 2024-03-17T23:47:45.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.447+0000 7f67ec3b2640 1 -- 172.21.15.150:0/410942055 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67e406f400 msgr2=0x7f67e40cdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:45.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.447+0000 7f67ec3b2640 1 --2- 172.21.15.150:0/410942055 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67e406f400 0x7f67e40cdf90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f67d40097a0 tx=0x7f67d402b8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.447+0000 7f67ec3b2640 1 -- 172.21.15.150:0/410942055 shutdown_connections 2024-03-17T23:47:45.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.447+0000 7f67ec3b2640 1 --2- 172.21.15.150:0/410942055 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67e406f400 0x7f67e40cdf90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.447+0000 7f67ec3b2640 1 -- 172.21.15.150:0/410942055 >> 172.21.15.150:0/410942055 conn(0x7f67e406b190 msgr2=0x7f67e406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:45.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.447+0000 7f67ec3b2640 1 -- 172.21.15.150:0/410942055 shutdown_connections 2024-03-17T23:47:45.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.447+0000 7f67ec3b2640 1 -- 172.21.15.150:0/410942055 wait complete. 2024-03-17T23:47:45.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.448+0000 7f67ec3b2640 1 Processor -- start 2024-03-17T23:47:45.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.448+0000 7f67ec3b2640 1 -- start start 2024-03-17T23:47:45.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.449+0000 7f67ec3b2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67e406f400 0x7f67e40b4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:45.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.449+0000 7f67ec3b2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67e40ce4d0 0x7f67e41554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:45.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.449+0000 7f67ec3b2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67e4144bb0 0x7f67e4139420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:45.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.449+0000 7f67ec3b2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f67e406bee0 con 0x7f67e406f400 2024-03-17T23:47:45.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.449+0000 7f67ec3b2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f67e406bd60 con 0x7f67e40ce4d0 2024-03-17T23:47:45.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.449+0000 7f67ec3b2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f67e4167150 con 0x7f67e4144bb0 2024-03-17T23:47:45.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.449+0000 7f67ea14e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67e406f400 0x7f67e40b4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:45.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.449+0000 7f67ea14e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67e406f400 0x7f67e40b4890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46476/0 (socket says 172.21.15.150:46476) 2024-03-17T23:47:45.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.449+0000 7f67ea14e640 1 -- 172.21.15.150:0/1975874751 learned_addr learned my addr 172.21.15.150:0/1975874751 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:45.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.449+0000 7f67e994d640 1 --2- 172.21.15.150:0/1975874751 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67e40ce4d0 0x7f67e41554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:45.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.449+0000 7f67ea94f640 1 --2- 172.21.15.150:0/1975874751 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67e4144bb0 0x7f67e4139420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:45.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.449+0000 7f67ea14e640 1 -- 172.21.15.150:0/1975874751 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67e4144bb0 msgr2=0x7f67e4139420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:45.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.449+0000 7f67ea14e640 1 --2- 172.21.15.150:0/1975874751 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67e4144bb0 0x7f67e4139420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.449+0000 7f67ea14e640 1 -- 172.21.15.150:0/1975874751 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67e40ce4d0 msgr2=0x7f67e41554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:45.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.449+0000 7f67ea14e640 1 --2- 172.21.15.150:0/1975874751 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67e40ce4d0 0x7f67e41554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.449+0000 7f67ea14e640 1 -- 172.21.15.150:0/1975874751 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67e4153f00 con 0x7f67e406f400 2024-03-17T23:47:45.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.449+0000 7f67ea14e640 1 --2- 172.21.15.150:0/1975874751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67e406f400 0x7f67e40b4890 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f67e000c430 tx=0x7f67e000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:45.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.449+0000 7f67daffd640 1 -- 172.21.15.150:0/1975874751 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e0017070 con 0x7f67e406f400 2024-03-17T23:47:45.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.449+0000 7f67daffd640 1 -- 172.21.15.150:0/1975874751 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67e0005ce0 con 0x7f67e406f400 2024-03-17T23:47:45.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.449+0000 7f67daffd640 1 -- 172.21.15.150:0/1975874751 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e00072f0 con 0x7f67e406f400 2024-03-17T23:47:45.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.449+0000 7f67ec3b2640 1 -- 172.21.15.150:0/1975874751 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67e41415c0 con 0x7f67e406f400 2024-03-17T23:47:45.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.450+0000 7f67ec3b2640 1 -- 172.21.15.150:0/1975874751 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67e4153c20 con 0x7f67e406f400 2024-03-17T23:47:45.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.450+0000 7f67ec3b2640 1 -- 172.21.15.150:0/1975874751 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67e40681a0 con 0x7f67e406f400 2024-03-17T23:47:45.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.451+0000 7f67daffd640 1 -- 172.21.15.150:0/1975874751 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f67e001a070 con 0x7f67e406f400 2024-03-17T23:47:45.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.452+0000 7f67daffd640 1 --2- 172.21.15.150:0/1975874751 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f67c806bbd0 0x7f67c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:45.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.452+0000 7f67e994d640 1 --2- 172.21.15.150:0/1975874751 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f67c806bbd0 0x7f67c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:45.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.452+0000 7f67daffd640 1 -- 172.21.15.150:0/1975874751 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f67e008a320 con 0x7f67e406f400 2024-03-17T23:47:45.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.452+0000 7f67e994d640 1 --2- 172.21.15.150:0/1975874751 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f67c806bbd0 0x7f67c806e090 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f67d4036000 tx=0x7f67d403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:45.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.456+0000 7f67daffd640 1 -- 172.21.15.150:0/1975874751 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f67e0058320 con 0x7f67e406f400 2024-03-17T23:47:45.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.636+0000 7f67ec3b2640 1 -- 172.21.15.150:0/1975874751 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f67e4144360 con 0x7f67e406f400 2024-03-17T23:47:45.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.637+0000 7f67daffd640 1 -- 172.21.15.150:0/1975874751 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7f67e005c3c0 con 0x7f67e406f400 2024-03-17T23:47:45.637 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:47:45.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.639+0000 7f67ec3b2640 1 -- 172.21.15.150:0/1975874751 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f67c806bbd0 msgr2=0x7f67c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:45.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.639+0000 7f67ec3b2640 1 --2- 172.21.15.150:0/1975874751 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f67c806bbd0 0x7f67c806e090 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f67d4036000 tx=0x7f67d403c040 comp rx=0 tx=0).stop 2024-03-17T23:47:45.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.639+0000 7f67ec3b2640 1 -- 172.21.15.150:0/1975874751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67e406f400 msgr2=0x7f67e40b4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:45.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.639+0000 7f67ec3b2640 1 --2- 172.21.15.150:0/1975874751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67e406f400 0x7f67e40b4890 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f67e000c430 tx=0x7f67e000c8f0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.639+0000 7f67ec3b2640 1 -- 172.21.15.150:0/1975874751 shutdown_connections 2024-03-17T23:47:45.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.639+0000 7f67ec3b2640 1 --2- 172.21.15.150:0/1975874751 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67e4144bb0 0x7f67e4139420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.639+0000 7f67ec3b2640 1 --2- 172.21.15.150:0/1975874751 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f67c806bbd0 0x7f67c806e090 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.639+0000 7f67ec3b2640 1 --2- 172.21.15.150:0/1975874751 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67e40ce4d0 0x7f67e41554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.639+0000 7f67ec3b2640 1 --2- 172.21.15.150:0/1975874751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67e406f400 0x7f67e40b4890 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.639+0000 7f67ec3b2640 1 -- 172.21.15.150:0/1975874751 >> 172.21.15.150:0/1975874751 conn(0x7f67e406b190 msgr2=0x7f67e40b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:45.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.639+0000 7f67ec3b2640 1 -- 172.21.15.150:0/1975874751 shutdown_connections 2024-03-17T23:47:45.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.639+0000 7f67ec3b2640 1 -- 172.21.15.150:0/1975874751 wait complete. 2024-03-17T23:47:45.650 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-17T23:47:45.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.757+0000 7f4c7020b640 1 Processor -- start 2024-03-17T23:47:45.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.758+0000 7f4c7020b640 1 -- start start 2024-03-17T23:47:45.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.758+0000 7f4c7020b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4c681615d0 0x7f4c680d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:45.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.758+0000 7f4c7020b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4c680d0e30 0x7f4c680b6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:45.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.758+0000 7f4c7020b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c680d2870 con 0x7f4c680d3910 2024-03-17T23:47:45.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.758+0000 7f4c7020b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f4c681619b0 con 0x7f4c680d0e30 2024-03-17T23:47:45.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.758+0000 7f4c7020b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f4c680d2b50 con 0x7f4c681615d0 2024-03-17T23:47:45.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.758+0000 7f4c6e7a8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4c680d0e30 0x7f4c680b6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:45.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.759+0000 7f4c6e7a8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4c680d0e30 0x7f4c680b6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50250/0 (socket says 172.21.15.150:50250) 2024-03-17T23:47:45.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.759+0000 7f4c6d7a6640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f4c680d3910 0x7f4c680d0720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:38914/0 (socket says 172.21.15.150:38914) 2024-03-17T23:47:45.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.759+0000 7f4c6e7a8640 1 -- 172.21.15.150:0/52174305 learned_addr learned my addr 172.21.15.150:0/52174305 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:45.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.759+0000 7f4c6dfa7640 1 --2- 172.21.15.150:0/52174305 >> v2:172.21.15.157:3300/0 conn(0x7f4c681615d0 0x7f4c680d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:45.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.759+0000 7f4c6dfa7640 1 -- 172.21.15.150:0/52174305 >> v2:172.21.15.150:3300/0 conn(0x7f4c680d0e30 msgr2=0x7f4c680b6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:45.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.759+0000 7f4c6dfa7640 1 --2- 172.21.15.150:0/52174305 >> v2:172.21.15.150:3300/0 conn(0x7f4c680d0e30 0x7f4c680b6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.759+0000 7f4c6dfa7640 1 -- 172.21.15.150:0/52174305 >> v1:172.21.15.157:6789/0 conn(0x7f4c680d3910 legacy=0x7f4c680d0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:45.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.759+0000 7f4c6cfa5640 1 -- 172.21.15.150:0/52174305 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4075417027 0 0) 0x7f4c680d2870 con 0x7f4c680d3910 2024-03-17T23:47:45.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.759+0000 7f4c6dfa7640 1 -- 172.21.15.150:0/52174305 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c680d2e30 con 0x7f4c681615d0 2024-03-17T23:47:45.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.760+0000 7f4c6dfa7640 1 --2- 172.21.15.150:0/52174305 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c681615d0 0x7f4c680d2020 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4c6806d7c0 tx=0x7f4c6002bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6d3579c7eef3176 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:45.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.760+0000 7f4c6cfa5640 1 -- 172.21.15.150:0/52174305 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c60039070 con 0x7f4c681615d0 2024-03-17T23:47:45.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.760+0000 7f4c6cfa5640 1 -- 172.21.15.150:0/52174305 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c6002cc80 con 0x7f4c681615d0 2024-03-17T23:47:45.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.760+0000 7f4c6cfa5640 1 -- 172.21.15.150:0/52174305 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c60034660 con 0x7f4c681615d0 2024-03-17T23:47:45.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.760+0000 7f4c7020b640 1 -- 172.21.15.150:0/52174305 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c681615d0 msgr2=0x7f4c680d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:45.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.760+0000 7f4c7020b640 1 --2- 172.21.15.150:0/52174305 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c681615d0 0x7f4c680d2020 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4c6806d7c0 tx=0x7f4c6002bed0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.760+0000 7f4c7020b640 1 -- 172.21.15.150:0/52174305 shutdown_connections 2024-03-17T23:47:45.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.760+0000 7f4c7020b640 1 --2- 172.21.15.150:0/52174305 >> v2:172.21.15.150:3300/0 conn(0x7f4c680d0e30 0x7f4c680b6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.760+0000 7f4c7020b640 1 --2- 172.21.15.150:0/52174305 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c681615d0 0x7f4c680d2020 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.760+0000 7f4c7020b640 1 -- 172.21.15.150:0/52174305 >> 172.21.15.150:0/52174305 conn(0x7f4c680730b0 msgr2=0x7f4c680b5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:45.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.761+0000 7f4c7020b640 1 -- 172.21.15.150:0/52174305 shutdown_connections 2024-03-17T23:47:45.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.761+0000 7f4c7020b640 1 -- 172.21.15.150:0/52174305 wait complete. 2024-03-17T23:47:45.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.762+0000 7f4c7020b640 1 Processor -- start 2024-03-17T23:47:45.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.762+0000 7f4c7020b640 1 -- start start 2024-03-17T23:47:45.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.762+0000 7f4c7020b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c680d0e30 0x7f4c680c9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:45.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.762+0000 7f4c7020b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c680d3910 0x7f4c681e6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:45.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.762+0000 7f4c7020b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c681615d0 0x7f4c6815ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:45.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.762+0000 7f4c7020b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4c680d2560 con 0x7f4c681615d0 2024-03-17T23:47:45.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.762+0000 7f4c7020b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4c680d33f0 con 0x7f4c680d3910 2024-03-17T23:47:45.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.762+0000 7f4c7020b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4c680b9660 con 0x7f4c680d0e30 2024-03-17T23:47:45.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.763+0000 7f4c6e7a8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c681615d0 0x7f4c6815ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:45.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.763+0000 7f4c6e7a8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c681615d0 0x7f4c6815ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50266/0 (socket says 172.21.15.150:50266) 2024-03-17T23:47:45.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.763+0000 7f4c6e7a8640 1 -- 172.21.15.150:0/2064488609 learned_addr learned my addr 172.21.15.150:0/2064488609 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:45.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.763+0000 7f4c6d7a6640 1 --2- 172.21.15.150:0/2064488609 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c680d3910 0x7f4c681e6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:45.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.763+0000 7f4c6dfa7640 1 --2- 172.21.15.150:0/2064488609 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c680d0e30 0x7f4c680c9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:45.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.763+0000 7f4c6e7a8640 1 -- 172.21.15.150:0/2064488609 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c680d0e30 msgr2=0x7f4c680c9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:45.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.763+0000 7f4c6e7a8640 1 --2- 172.21.15.150:0/2064488609 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c680d0e30 0x7f4c680c9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.763+0000 7f4c6e7a8640 1 -- 172.21.15.150:0/2064488609 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c680d3910 msgr2=0x7f4c681e6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:45.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.763+0000 7f4c6e7a8640 1 --2- 172.21.15.150:0/2064488609 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c680d3910 0x7f4c681e6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.763+0000 7f4c6e7a8640 1 -- 172.21.15.150:0/2064488609 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c681e72d0 con 0x7f4c681615d0 2024-03-17T23:47:45.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.763+0000 7f4c6dfa7640 1 --2- 172.21.15.150:0/2064488609 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c680d0e30 0x7f4c680c9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:45.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.763+0000 7f4c6e7a8640 1 --2- 172.21.15.150:0/2064488609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c681615d0 0x7f4c6815ce80 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4c6400e870 tx=0x7f4c6400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:45.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.763+0000 7f4c5e7fc640 1 -- 172.21.15.150:0/2064488609 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c64017070 con 0x7f4c681615d0 2024-03-17T23:47:45.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.763+0000 7f4c5e7fc640 1 -- 172.21.15.150:0/2064488609 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c6400c940 con 0x7f4c681615d0 2024-03-17T23:47:45.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.763+0000 7f4c7020b640 1 -- 172.21.15.150:0/2064488609 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c681e75b0 con 0x7f4c681615d0 2024-03-17T23:47:45.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.763+0000 7f4c5e7fc640 1 -- 172.21.15.150:0/2064488609 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6400cd30 con 0x7f4c681615d0 2024-03-17T23:47:45.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.763+0000 7f4c7020b640 1 -- 172.21.15.150:0/2064488609 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c681e7b70 con 0x7f4c681615d0 2024-03-17T23:47:45.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.765+0000 7f4c7020b640 1 -- 172.21.15.150:0/2064488609 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c680688b0 con 0x7f4c681615d0 2024-03-17T23:47:45.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.765+0000 7f4c5e7fc640 1 -- 172.21.15.150:0/2064488609 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4c6401a070 con 0x7f4c681615d0 2024-03-17T23:47:45.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.765+0000 7f4c5e7fc640 1 --2- 172.21.15.150:0/2064488609 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4c4c06bbd0 0x7f4c4c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:45.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.765+0000 7f4c5e7fc640 1 -- 172.21.15.150:0/2064488609 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f4c6408b150 con 0x7f4c681615d0 2024-03-17T23:47:45.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.765+0000 7f4c6dfa7640 1 --2- 172.21.15.150:0/2064488609 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4c4c06bbd0 0x7f4c4c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:45.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.766+0000 7f4c6dfa7640 1 --2- 172.21.15.150:0/2064488609 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4c4c06bbd0 0x7f4c4c06e090 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4c60036000 tx=0x7f4c60030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:45.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.770+0000 7f4c5e7fc640 1 -- 172.21.15.150:0/2064488609 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4c64059200 con 0x7f4c681615d0 2024-03-17T23:47:45.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.946+0000 7f4c7020b640 1 -- 172.21.15.150:0/2064488609 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4c680b90c0 con 0x7f4c681615d0 2024-03-17T23:47:45.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.947+0000 7f4c5e7fc640 1 -- 172.21.15.150:0/2064488609 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7f4c6405d2a0 con 0x7f4c681615d0 2024-03-17T23:47:45.947 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:45.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.949+0000 7f4c7020b640 1 -- 172.21.15.150:0/2064488609 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4c4c06bbd0 msgr2=0x7f4c4c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:45.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.949+0000 7f4c7020b640 1 --2- 172.21.15.150:0/2064488609 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4c4c06bbd0 0x7f4c4c06e090 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4c60036000 tx=0x7f4c60030000 comp rx=0 tx=0).stop 2024-03-17T23:47:45.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.949+0000 7f4c7020b640 1 -- 172.21.15.150:0/2064488609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c681615d0 msgr2=0x7f4c6815ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:45.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.949+0000 7f4c7020b640 1 --2- 172.21.15.150:0/2064488609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c681615d0 0x7f4c6815ce80 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4c6400e870 tx=0x7f4c6400ed30 comp rx=0 tx=0).stop 2024-03-17T23:47:45.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.949+0000 7f4c7020b640 1 -- 172.21.15.150:0/2064488609 shutdown_connections 2024-03-17T23:47:45.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.949+0000 7f4c7020b640 1 --2- 172.21.15.150:0/2064488609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c681615d0 0x7f4c6815ce80 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.949+0000 7f4c7020b640 1 --2- 172.21.15.150:0/2064488609 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4c4c06bbd0 0x7f4c4c06e090 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.949+0000 7f4c7020b640 1 --2- 172.21.15.150:0/2064488609 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c680d3910 0x7f4c681e6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.949+0000 7f4c7020b640 1 --2- 172.21.15.150:0/2064488609 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c680d0e30 0x7f4c680c9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:45.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.949+0000 7f4c7020b640 1 -- 172.21.15.150:0/2064488609 >> 172.21.15.150:0/2064488609 conn(0x7f4c680730b0 msgr2=0x7f4c6815eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:45.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.949+0000 7f4c7020b640 1 -- 172.21.15.150:0/2064488609 shutdown_connections 2024-03-17T23:47:45.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:45.949+0000 7f4c7020b640 1 -- 172.21.15.150:0/2064488609 wait complete. 2024-03-17T23:47:45.960 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-17T23:47:46.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.064+0000 7ff338a7e640 1 Processor -- start 2024-03-17T23:47:46.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.064+0000 7ff338a7e640 1 -- start start 2024-03-17T23:47:46.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.066+0000 7ff338a7e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff3340cf130 0x7ff334168ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:46.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.066+0000 7ff338a7e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff33406c2e0 con 0x7ff3340670b0 2024-03-17T23:47:46.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.066+0000 7ff338a7e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3340b5ab0 con 0x7ff33415e110 2024-03-17T23:47:46.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.066+0000 7ff338a7e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7ff3340b5d90 con 0x7ff3340cf130 2024-03-17T23:47:46.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.066+0000 7ff331d9b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7ff3340670b0 0x7ff33406bbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49770/0 (socket says 172.21.15.150:49770) 2024-03-17T23:47:46.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.066+0000 7ff331d9b640 1 -- 172.21.15.150:0/2602608150 learned_addr learned my addr 172.21.15.150:0/2602608150 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:46.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.066+0000 7ff332d9d640 1 --2- 172.21.15.150:0/2602608150 >> v2:172.21.15.150:3300/0 conn(0x7ff3340cf130 0x7ff334168ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:46.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.066+0000 7ff332d9d640 1 -- 172.21.15.150:0/2602608150 >> v1:172.21.15.157:6789/0 conn(0x7ff33415e110 legacy=0x7ff33406b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:46.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.066+0000 7ff33159a640 1 -- 172.21.15.150:0/2602608150 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 140143824 0 0) 0x7ff3340b5ab0 con 0x7ff33415e110 2024-03-17T23:47:46.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.066+0000 7ff332d9d640 1 -- 172.21.15.150:0/2602608150 >> v1:172.21.15.150:6789/0 conn(0x7ff3340670b0 legacy=0x7ff33406bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:46.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.066+0000 7ff332d9d640 1 -- 172.21.15.150:0/2602608150 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3340b6070 con 0x7ff3340cf130 2024-03-17T23:47:46.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.067+0000 7ff332d9d640 1 --2- 172.21.15.150:0/2602608150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3340cf130 0x7ff334168ba0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff32c009d90 tx=0x7ff32c02bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3cb920c049710ac server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:46.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.067+0000 7ff33159a640 1 -- 172.21.15.150:0/2602608150 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff32c039070 con 0x7ff3340cf130 2024-03-17T23:47:46.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.067+0000 7ff33159a640 1 -- 172.21.15.150:0/2602608150 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff32c02cab0 con 0x7ff3340cf130 2024-03-17T23:47:46.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.067+0000 7ff33159a640 1 -- 172.21.15.150:0/2602608150 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff32c034440 con 0x7ff3340cf130 2024-03-17T23:47:46.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.068+0000 7ff338a7e640 1 -- 172.21.15.150:0/2602608150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3340cf130 msgr2=0x7ff334168ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:46.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.068+0000 7ff338a7e640 1 --2- 172.21.15.150:0/2602608150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3340cf130 0x7ff334168ba0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff32c009d90 tx=0x7ff32c02bd80 comp rx=0 tx=0).stop 2024-03-17T23:47:46.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.068+0000 7ff338a7e640 1 -- 172.21.15.150:0/2602608150 shutdown_connections 2024-03-17T23:47:46.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.068+0000 7ff338a7e640 1 --2- 172.21.15.150:0/2602608150 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3340cf130 0x7ff334168ba0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:46.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.068+0000 7ff338a7e640 1 -- 172.21.15.150:0/2602608150 >> 172.21.15.150:0/2602608150 conn(0x7ff334099a70 msgr2=0x7ff3340b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:46.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.068+0000 7ff338a7e640 1 -- 172.21.15.150:0/2602608150 shutdown_connections 2024-03-17T23:47:46.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.068+0000 7ff338a7e640 1 -- 172.21.15.150:0/2602608150 wait complete. 2024-03-17T23:47:46.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.069+0000 7ff338a7e640 1 Processor -- start 2024-03-17T23:47:46.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.069+0000 7ff338a7e640 1 -- start start 2024-03-17T23:47:46.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.070+0000 7ff338a7e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3340670b0 0x7ff3341505b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:46.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.070+0000 7ff338a7e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3340cf130 0x7ff33406b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:46.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.070+0000 7ff338a7e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff33415e110 0x7ff3341f7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:46.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.070+0000 7ff338a7e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff3341690e0 con 0x7ff3340670b0 2024-03-17T23:47:46.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.070+0000 7ff338a7e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff3340b1bd0 con 0x7ff3340cf130 2024-03-17T23:47:46.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.070+0000 7ff338a7e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff334169260 con 0x7ff33415e110 2024-03-17T23:47:46.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.070+0000 7ff33259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3340670b0 0x7ff3341505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:46.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.070+0000 7ff33259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3340670b0 0x7ff3341505b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50288/0 (socket says 172.21.15.150:50288) 2024-03-17T23:47:46.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.070+0000 7ff33259c640 1 -- 172.21.15.150:0/990748888 learned_addr learned my addr 172.21.15.150:0/990748888 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:46.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.070+0000 7ff331d9b640 1 --2- 172.21.15.150:0/990748888 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3340cf130 0x7ff33406b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:46.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.070+0000 7ff332d9d640 1 --2- 172.21.15.150:0/990748888 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff33415e110 0x7ff3341f7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:46.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.070+0000 7ff33259c640 1 -- 172.21.15.150:0/990748888 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff33415e110 msgr2=0x7ff3341f7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:46.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.070+0000 7ff33259c640 1 --2- 172.21.15.150:0/990748888 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff33415e110 0x7ff3341f7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:46.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.070+0000 7ff33259c640 1 -- 172.21.15.150:0/990748888 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3340cf130 msgr2=0x7ff33406b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:46.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.070+0000 7ff33259c640 1 --2- 172.21.15.150:0/990748888 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3340cf130 0x7ff33406b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:46.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.070+0000 7ff33259c640 1 -- 172.21.15.150:0/990748888 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3341547d0 con 0x7ff3340670b0 2024-03-17T23:47:46.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.071+0000 7ff33259c640 1 --2- 172.21.15.150:0/990748888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3340670b0 0x7ff3341505b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff32400c3c0 tx=0x7ff32400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:46.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.071+0000 7ff322ffd640 1 -- 172.21.15.150:0/990748888 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff324017070 con 0x7ff3340670b0 2024-03-17T23:47:46.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.071+0000 7ff322ffd640 1 -- 172.21.15.150:0/990748888 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3240035f0 con 0x7ff3340670b0 2024-03-17T23:47:46.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.071+0000 7ff338a7e640 1 -- 172.21.15.150:0/990748888 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff334154b60 con 0x7ff3340670b0 2024-03-17T23:47:46.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.071+0000 7ff322ffd640 1 -- 172.21.15.150:0/990748888 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3240039e0 con 0x7ff3340670b0 2024-03-17T23:47:46.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.071+0000 7ff338a7e640 1 -- 172.21.15.150:0/990748888 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3341544f0 con 0x7ff3340670b0 2024-03-17T23:47:46.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.072+0000 7ff338a7e640 1 -- 172.21.15.150:0/990748888 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3340688b0 con 0x7ff3340670b0 2024-03-17T23:47:46.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.072+0000 7ff322ffd640 1 -- 172.21.15.150:0/990748888 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff32401a070 con 0x7ff3340670b0 2024-03-17T23:47:46.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.073+0000 7ff322ffd640 1 --2- 172.21.15.150:0/990748888 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff31406bbd0 0x7ff31406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:46.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.073+0000 7ff322ffd640 1 -- 172.21.15.150:0/990748888 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7ff32408a0e0 con 0x7ff3340670b0 2024-03-17T23:47:46.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.073+0000 7ff331d9b640 1 --2- 172.21.15.150:0/990748888 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff31406bbd0 0x7ff31406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:46.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.073+0000 7ff331d9b640 1 --2- 172.21.15.150:0/990748888 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff31406bbd0 0x7ff31406e090 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff3340b0ac0 tx=0x7ff328007e20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:46.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.081+0000 7ff322ffd640 1 -- 172.21.15.150:0/990748888 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff324058210 con 0x7ff3340670b0 2024-03-17T23:47:46.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.274+0000 7ff338a7e640 1 -- 172.21.15.150:0/990748888 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff334168c40 con 0x7ff3340670b0 2024-03-17T23:47:46.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.274+0000 7ff322ffd640 1 -- 172.21.15.150:0/990748888 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) ==== 80+0+0 (secure 0 0 0) 0x7ff32405c2b0 con 0x7ff3340670b0 2024-03-17T23:47:46.275 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-03-17T23:47:46.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.276+0000 7ff338a7e640 1 -- 172.21.15.150:0/990748888 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff31406bbd0 msgr2=0x7ff31406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:46.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.276+0000 7ff338a7e640 1 --2- 172.21.15.150:0/990748888 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff31406bbd0 0x7ff31406e090 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff3340b0ac0 tx=0x7ff328007e20 comp rx=0 tx=0).stop 2024-03-17T23:47:46.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.276+0000 7ff338a7e640 1 -- 172.21.15.150:0/990748888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3340670b0 msgr2=0x7ff3341505b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:46.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.276+0000 7ff338a7e640 1 --2- 172.21.15.150:0/990748888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3340670b0 0x7ff3341505b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff32400c3c0 tx=0x7ff32400c880 comp rx=0 tx=0).stop 2024-03-17T23:47:46.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.276+0000 7ff338a7e640 1 -- 172.21.15.150:0/990748888 shutdown_connections 2024-03-17T23:47:46.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.276+0000 7ff338a7e640 1 --2- 172.21.15.150:0/990748888 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff33415e110 0x7ff3341f7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:46.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.276+0000 7ff338a7e640 1 --2- 172.21.15.150:0/990748888 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff31406bbd0 0x7ff31406e090 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:46.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.276+0000 7ff338a7e640 1 --2- 172.21.15.150:0/990748888 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3340cf130 0x7ff33406b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:46.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.276+0000 7ff338a7e640 1 --2- 172.21.15.150:0/990748888 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3340670b0 0x7ff3341505b0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:46.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.276+0000 7ff338a7e640 1 -- 172.21.15.150:0/990748888 >> 172.21.15.150:0/990748888 conn(0x7ff334099a70 msgr2=0x7ff3340cfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:46.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.276+0000 7ff338a7e640 1 -- 172.21.15.150:0/990748888 shutdown_connections 2024-03-17T23:47:46.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.276+0000 7ff338a7e640 1 -- 172.21.15.150:0/990748888 wait complete. 2024-03-17T23:47:46.288 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:47:46.288 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:47:46.304 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:46.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.404+0000 7f626c034640 1 Processor -- start 2024-03-17T23:47:46.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.404+0000 7f626c034640 1 -- start start 2024-03-17T23:47:46.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.405+0000 7f626c034640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62640c8210 0x7f62640cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:46.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.405+0000 7f626c034640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f62640ccb80 0x7f62640cd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:46.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.405+0000 7f626c034640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62640cd550 con 0x7f626415be40 2024-03-17T23:47:46.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.405+0000 7f626c034640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f62640c5d70 con 0x7f62640ccb80 2024-03-17T23:47:46.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.405+0000 7f626c034640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f626409b260 con 0x7f62640c8210 2024-03-17T23:47:46.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.405+0000 7f626a5d1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f62640ccb80 0x7f62640cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:46.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.405+0000 7f626a5d1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f62640ccb80 0x7f62640cd010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50300/0 (socket says 172.21.15.150:50300) 2024-03-17T23:47:46.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.405+0000 7f6269dd0640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f626415be40 0x7f626409a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49780/0 (socket says 172.21.15.150:49780) 2024-03-17T23:47:46.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.405+0000 7f626a5d1640 1 -- 172.21.15.150:0/2161199824 learned_addr learned my addr 172.21.15.150:0/2161199824 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:46.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.405+0000 7f62695cf640 1 --2- 172.21.15.150:0/2161199824 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62640c8210 0x7f62640cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:46.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.405+0000 7f626a5d1640 1 -- 172.21.15.150:0/2161199824 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62640c8210 msgr2=0x7f62640cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:46.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.405+0000 7f626a5d1640 1 --2- 172.21.15.150:0/2161199824 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62640c8210 0x7f62640cc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:46.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.405+0000 7f626a5d1640 1 -- 172.21.15.150:0/2161199824 >> v1:172.21.15.150:6789/0 conn(0x7f626415be40 legacy=0x7f626409a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:46.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.405+0000 7f626a5d1640 1 -- 172.21.15.150:0/2161199824 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f626409af80 con 0x7f62640ccb80 2024-03-17T23:47:46.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.406+0000 7f626a5d1640 1 --2- 172.21.15.150:0/2161199824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62640ccb80 0x7f62640cd010 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f625c009e50 tx=0x7f625c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=802a074f8de61a7d server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:46.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.406+0000 7f6268dce640 1 -- 172.21.15.150:0/2161199824 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625c02cb00 con 0x7f62640ccb80 2024-03-17T23:47:46.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.406+0000 7f6268dce640 1 -- 172.21.15.150:0/2161199824 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f625c02cca0 con 0x7f62640ccb80 2024-03-17T23:47:46.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.406+0000 7f6268dce640 1 -- 172.21.15.150:0/2161199824 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625c034610 con 0x7f62640ccb80 2024-03-17T23:47:46.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.407+0000 7f626c034640 1 -- 172.21.15.150:0/2161199824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62640ccb80 msgr2=0x7f62640cd010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:46.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.407+0000 7f626c034640 1 --2- 172.21.15.150:0/2161199824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62640ccb80 0x7f62640cd010 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f625c009e50 tx=0x7f625c02bf60 comp rx=0 tx=0).stop 2024-03-17T23:47:46.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.407+0000 7f626c034640 1 -- 172.21.15.150:0/2161199824 shutdown_connections 2024-03-17T23:47:46.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.407+0000 7f626c034640 1 --2- 172.21.15.150:0/2161199824 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f62640c8210 0x7f62640cc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:46.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.407+0000 7f626c034640 1 --2- 172.21.15.150:0/2161199824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62640ccb80 0x7f62640cd010 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:46.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.407+0000 7f626c034640 1 -- 172.21.15.150:0/2161199824 >> 172.21.15.150:0/2161199824 conn(0x7f62640b3620 msgr2=0x7f62640b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:46.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.407+0000 7f626c034640 1 -- 172.21.15.150:0/2161199824 shutdown_connections 2024-03-17T23:47:46.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.407+0000 7f626c034640 1 -- 172.21.15.150:0/2161199824 wait complete. 2024-03-17T23:47:46.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.408+0000 7f626c034640 1 Processor -- start 2024-03-17T23:47:46.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.408+0000 7f626c034640 1 -- start start 2024-03-17T23:47:46.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.409+0000 7f626c034640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62640c8210 0x7f6264150270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:46.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.409+0000 7f626c034640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62640ccb80 0x7f6264152780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:46.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.409+0000 7f626c034640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f626415be40 0x7f626414e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:46.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.409+0000 7f626c034640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f626415c220 con 0x7f62640c8210 2024-03-17T23:47:46.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.409+0000 7f626c034640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f62640c5a00 con 0x7f62640ccb80 2024-03-17T23:47:46.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.409+0000 7f626c034640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f62641659e0 con 0x7f626415be40 2024-03-17T23:47:46.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.409+0000 7f6269dd0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62640c8210 0x7f6264150270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:46.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.409+0000 7f6269dd0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62640c8210 0x7f6264150270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50308/0 (socket says 172.21.15.150:50308) 2024-03-17T23:47:46.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.409+0000 7f6269dd0640 1 -- 172.21.15.150:0/2307585305 learned_addr learned my addr 172.21.15.150:0/2307585305 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:46.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.409+0000 7f626a5d1640 1 --2- 172.21.15.150:0/2307585305 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f626415be40 0x7f626414e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:46.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.409+0000 7f62695cf640 1 --2- 172.21.15.150:0/2307585305 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62640ccb80 0x7f6264152780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:46.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.409+0000 7f6269dd0640 1 -- 172.21.15.150:0/2307585305 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f626415be40 msgr2=0x7f626414e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:46.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.409+0000 7f6269dd0640 1 --2- 172.21.15.150:0/2307585305 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f626415be40 0x7f626414e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:46.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.409+0000 7f6269dd0640 1 -- 172.21.15.150:0/2307585305 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62640ccb80 msgr2=0x7f6264152780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:46.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.409+0000 7f6269dd0640 1 --2- 172.21.15.150:0/2307585305 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62640ccb80 0x7f6264152780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:46.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.409+0000 7f6269dd0640 1 -- 172.21.15.150:0/2307585305 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f626415a5d0 con 0x7f62640c8210 2024-03-17T23:47:46.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.409+0000 7f6269dd0640 1 --2- 172.21.15.150:0/2307585305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62640c8210 0x7f6264150270 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f626000c400 tx=0x7f626000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:46.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.409+0000 7f625a7fc640 1 -- 172.21.15.150:0/2307585305 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62600074e0 con 0x7f62640c8210 2024-03-17T23:47:46.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.409+0000 7f625a7fc640 1 -- 172.21.15.150:0/2307585305 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6260007680 con 0x7f62640c8210 2024-03-17T23:47:46.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.410+0000 7f626c034640 1 -- 172.21.15.150:0/2307585305 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62641520b0 con 0x7f62640c8210 2024-03-17T23:47:46.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.410+0000 7f625a7fc640 1 -- 172.21.15.150:0/2307585305 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6260004a00 con 0x7f62640c8210 2024-03-17T23:47:46.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.410+0000 7f626c034640 1 -- 172.21.15.150:0/2307585305 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f626414e9b0 con 0x7f62640c8210 2024-03-17T23:47:46.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.411+0000 7f626c034640 1 -- 172.21.15.150:0/2307585305 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62640688b0 con 0x7f62640c8210 2024-03-17T23:47:46.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.411+0000 7f625a7fc640 1 -- 172.21.15.150:0/2307585305 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f626001a020 con 0x7f62640c8210 2024-03-17T23:47:46.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.411+0000 7f625a7fc640 1 --2- 172.21.15.150:0/2307585305 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f624806bc00 0x7f624806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:46.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.411+0000 7f625a7fc640 1 -- 172.21.15.150:0/2307585305 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f626008a4e0 con 0x7f62640c8210 2024-03-17T23:47:46.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.411+0000 7f62695cf640 1 --2- 172.21.15.150:0/2307585305 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f624806bc00 0x7f624806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:46.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.412+0000 7f62695cf640 1 --2- 172.21.15.150:0/2307585305 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f624806bc00 0x7f624806e0c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f62641f5130 tx=0x7f6254009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:46.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.416+0000 7f625a7fc640 1 -- 172.21.15.150:0/2307585305 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6260058580 con 0x7f62640c8210 2024-03-17T23:47:46.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.597+0000 7f626c034640 1 -- 172.21.15.150:0/2307585305 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f62640b2ae0 con 0x7f62640c8210 2024-03-17T23:47:46.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.599+0000 7f625a7fc640 1 -- 172.21.15.150:0/2307585305 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18881 (secure 0 0 0) 0x7f626005c620 con 0x7f62640c8210 2024-03-17T23:47:46.601 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:46.601 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":28,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:37.031864+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.e","mappings":[{"from":7,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:46.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.601+0000 7f626c034640 1 -- 172.21.15.150:0/2307585305 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f624806bc00 msgr2=0x7f624806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:46.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.601+0000 7f626c034640 1 --2- 172.21.15.150:0/2307585305 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f624806bc00 0x7f624806e0c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f62641f5130 tx=0x7f6254009190 comp rx=0 tx=0).stop 2024-03-17T23:47:46.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.601+0000 7f626c034640 1 -- 172.21.15.150:0/2307585305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62640c8210 msgr2=0x7f6264150270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:46.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.601+0000 7f626c034640 1 --2- 172.21.15.150:0/2307585305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62640c8210 0x7f6264150270 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f626000c400 tx=0x7f626000c8c0 comp rx=0 tx=0).stop 2024-03-17T23:47:46.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.601+0000 7f626c034640 1 -- 172.21.15.150:0/2307585305 shutdown_connections 2024-03-17T23:47:46.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.601+0000 7f626c034640 1 --2- 172.21.15.150:0/2307585305 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f626415be40 0x7f626414e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:46.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.601+0000 7f626c034640 1 --2- 172.21.15.150:0/2307585305 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f624806bc00 0x7f624806e0c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:46.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.601+0000 7f626c034640 1 --2- 172.21.15.150:0/2307585305 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62640ccb80 0x7f6264152780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:46.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.601+0000 7f626c034640 1 --2- 172.21.15.150:0/2307585305 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62640c8210 0x7f6264150270 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:46.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.601+0000 7f626c034640 1 -- 172.21.15.150:0/2307585305 >> 172.21.15.150:0/2307585305 conn(0x7f62640b3620 msgr2=0x7f6264070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:46.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.602+0000 7f626c034640 1 -- 172.21.15.150:0/2307585305 shutdown_connections 2024-03-17T23:47:46.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.602+0000 7f626c034640 1 -- 172.21.15.150:0/2307585305 wait complete. 2024-03-17T23:47:46.615 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:46:53.610678+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:46:54.227335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-17T23:46:57.949235+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-17T23:47:46.615 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:47:46.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.717+0000 7fc8dffd4640 1 Processor -- start 2024-03-17T23:47:46.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.717+0000 7fc8dffd4640 1 -- start start 2024-03-17T23:47:46.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.718+0000 7fc8dffd4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc8d806f400 0x7fc8d80cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:46.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.718+0000 7fc8dffd4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8d80ce4d0 0x7fc8d8165a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:46.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.718+0000 7fc8dffd4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8d8145170 con 0x7fc8d8144bb0 2024-03-17T23:47:46.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.718+0000 7fc8dffd4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fc8d80ce940 con 0x7fc8d806f400 2024-03-17T23:47:46.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.718+0000 7fc8dffd4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc8d8145450 con 0x7fc8d80ce4d0 2024-03-17T23:47:46.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.718+0000 7fc8dd56f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc8d806f400 0x7fc8d80cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:46.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.718+0000 7fc8ddd70640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fc8d8144bb0 0x7fc8d815ce50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49790/0 (socket says 172.21.15.150:49790) 2024-03-17T23:47:46.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.718+0000 7fc8ddd70640 1 -- 172.21.15.150:0/2955595563 learned_addr learned my addr 172.21.15.150:0/2955595563 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:46.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.718+0000 7fc8de571640 1 --2- 172.21.15.150:0/2955595563 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8d80ce4d0 0x7fc8d8165a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:46.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.718+0000 7fc8dd56f640 1 -- 172.21.15.150:0/2955595563 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8d80ce4d0 msgr2=0x7fc8d8165a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:46.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.718+0000 7fc8dd56f640 1 --2- 172.21.15.150:0/2955595563 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8d80ce4d0 0x7fc8d8165a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:46.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.719+0000 7fc8dd56f640 1 -- 172.21.15.150:0/2955595563 >> v1:172.21.15.150:6789/0 conn(0x7fc8d8144bb0 legacy=0x7fc8d815ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:46.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.719+0000 7fc8dcd6e640 1 -- 172.21.15.150:0/2955595563 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1469621104 0 0) 0x7fc8d8145170 con 0x7fc8d8144bb0 2024-03-17T23:47:46.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.719+0000 7fc8dd56f640 1 -- 172.21.15.150:0/2955595563 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d8145730 con 0x7fc8d806f400 2024-03-17T23:47:46.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.719+0000 7fc8dd56f640 1 --2- 172.21.15.150:0/2955595563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8d806f400 0x7fc8d80cdf90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc8c80097a0 tx=0x7fc8c802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e50a4a787918ed68 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:46.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.719+0000 7fc8dcd6e640 1 -- 172.21.15.150:0/2955595563 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c80315a0 con 0x7fc8d806f400 2024-03-17T23:47:46.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.719+0000 7fc8dcd6e640 1 -- 172.21.15.150:0/2955595563 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8c8037070 con 0x7fc8d806f400 2024-03-17T23:47:46.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.719+0000 7fc8dcd6e640 1 -- 172.21.15.150:0/2955595563 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c8031960 con 0x7fc8d806f400 2024-03-17T23:47:46.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.720+0000 7fc8dffd4640 1 -- 172.21.15.150:0/2955595563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8d806f400 msgr2=0x7fc8d80cdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:46.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.720+0000 7fc8dffd4640 1 --2- 172.21.15.150:0/2955595563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8d806f400 0x7fc8d80cdf90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc8c80097a0 tx=0x7fc8c802b840 comp rx=0 tx=0).stop 2024-03-17T23:47:46.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.720+0000 7fc8dffd4640 1 -- 172.21.15.150:0/2955595563 shutdown_connections 2024-03-17T23:47:46.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.720+0000 7fc8dffd4640 1 --2- 172.21.15.150:0/2955595563 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8d80ce4d0 0x7fc8d8165a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:46.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.720+0000 7fc8dffd4640 1 --2- 172.21.15.150:0/2955595563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8d806f400 0x7fc8d80cdf90 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:46.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.720+0000 7fc8dffd4640 1 -- 172.21.15.150:0/2955595563 >> 172.21.15.150:0/2955595563 conn(0x7fc8d806b190 msgr2=0x7fc8d806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:46.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.720+0000 7fc8dffd4640 1 -- 172.21.15.150:0/2955595563 shutdown_connections 2024-03-17T23:47:46.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.720+0000 7fc8dffd4640 1 -- 172.21.15.150:0/2955595563 wait complete. 2024-03-17T23:47:46.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.721+0000 7fc8dffd4640 1 Processor -- start 2024-03-17T23:47:46.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.721+0000 7fc8dffd4640 1 -- start start 2024-03-17T23:47:46.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.722+0000 7fc8dffd4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8d806f400 0x7fc8d80ac770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:46.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.722+0000 7fc8dffd4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8d80ce4d0 0x7fc8d80a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:46.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.722+0000 7fc8dffd4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8d8144bb0 0x7fc8d81f4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:46.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.722+0000 7fc8dffd4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d806bee0 con 0x7fc8d806f400 2024-03-17T23:47:46.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.722+0000 7fc8dffd4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d806bd60 con 0x7fc8d80ce4d0 2024-03-17T23:47:46.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.722+0000 7fc8dffd4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc8d8165f70 con 0x7fc8d8144bb0 2024-03-17T23:47:46.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.722+0000 7fc8ddd70640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8d806f400 0x7fc8d80ac770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:46.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.722+0000 7fc8ddd70640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8d806f400 0x7fc8d80ac770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50316/0 (socket says 172.21.15.150:50316) 2024-03-17T23:47:46.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.722+0000 7fc8ddd70640 1 -- 172.21.15.150:0/3184501685 learned_addr learned my addr 172.21.15.150:0/3184501685 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:46.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.722+0000 7fc8dd56f640 1 --2- 172.21.15.150:0/3184501685 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8d80ce4d0 0x7fc8d80a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:46.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.722+0000 7fc8de571640 1 --2- 172.21.15.150:0/3184501685 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8d8144bb0 0x7fc8d81f4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:46.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.722+0000 7fc8ddd70640 1 -- 172.21.15.150:0/3184501685 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8d8144bb0 msgr2=0x7fc8d81f4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:46.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.722+0000 7fc8ddd70640 1 --2- 172.21.15.150:0/3184501685 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8d8144bb0 0x7fc8d81f4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:46.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.722+0000 7fc8ddd70640 1 -- 172.21.15.150:0/3184501685 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8d80ce4d0 msgr2=0x7fc8d80a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:46.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.722+0000 7fc8ddd70640 1 --2- 172.21.15.150:0/3184501685 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8d80ce4d0 0x7fc8d80a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:46.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.722+0000 7fc8ddd70640 1 -- 172.21.15.150:0/3184501685 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d80a9720 con 0x7fc8d806f400 2024-03-17T23:47:46.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.722+0000 7fc8ddd70640 1 --2- 172.21.15.150:0/3184501685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8d806f400 0x7fc8d80ac770 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc8d400c4d0 tx=0x7fc8d400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:46.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.723+0000 7fc8ce7fc640 1 -- 172.21.15.150:0/3184501685 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d4018070 con 0x7fc8d806f400 2024-03-17T23:47:46.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.723+0000 7fc8ce7fc640 1 -- 172.21.15.150:0/3184501685 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8d4007500 con 0x7fc8d806f400 2024-03-17T23:47:46.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.723+0000 7fc8dffd4640 1 -- 172.21.15.150:0/3184501685 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8d80a9a00 con 0x7fc8d806f400 2024-03-17T23:47:46.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.723+0000 7fc8ce7fc640 1 -- 172.21.15.150:0/3184501685 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d4004a20 con 0x7fc8d806f400 2024-03-17T23:47:46.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.723+0000 7fc8dffd4640 1 -- 172.21.15.150:0/3184501685 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8d80ae880 con 0x7fc8d806f400 2024-03-17T23:47:46.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.724+0000 7fc8dffd4640 1 -- 172.21.15.150:0/3184501685 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8d8070020 con 0x7fc8d806f400 2024-03-17T23:47:46.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.724+0000 7fc8ce7fc640 1 -- 172.21.15.150:0/3184501685 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc8d401b070 con 0x7fc8d806f400 2024-03-17T23:47:46.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.724+0000 7fc8ce7fc640 1 --2- 172.21.15.150:0/3184501685 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc8bc06bbd0 0x7fc8bc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:46.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.725+0000 7fc8ce7fc640 1 -- 172.21.15.150:0/3184501685 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fc8d408a060 con 0x7fc8d806f400 2024-03-17T23:47:46.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.725+0000 7fc8dd56f640 1 --2- 172.21.15.150:0/3184501685 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc8bc06bbd0 0x7fc8bc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:46.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.725+0000 7fc8dd56f640 1 --2- 172.21.15.150:0/3184501685 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc8bc06bbd0 0x7fc8bc06e090 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8036000 tx=0x7fc8c803d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:46.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.729+0000 7fc8ce7fc640 1 -- 172.21.15.150:0/3184501685 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc8d4058190 con 0x7fc8d806f400 2024-03-17T23:47:46.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.900+0000 7fc8dffd4640 1 -- 172.21.15.150:0/3184501685 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc8d80681a0 con 0x7fc8d806f400 2024-03-17T23:47:46.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.900+0000 7fc8ce7fc640 1 -- 172.21.15.150:0/3184501685 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7fc8d405c230 con 0x7fc8d806f400 2024-03-17T23:47:46.902 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:47:46.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.902+0000 7fc8dffd4640 1 -- 172.21.15.150:0/3184501685 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc8bc06bbd0 msgr2=0x7fc8bc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:46.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.902+0000 7fc8dffd4640 1 --2- 172.21.15.150:0/3184501685 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc8bc06bbd0 0x7fc8bc06e090 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8036000 tx=0x7fc8c803d040 comp rx=0 tx=0).stop 2024-03-17T23:47:46.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.902+0000 7fc8dffd4640 1 -- 172.21.15.150:0/3184501685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8d806f400 msgr2=0x7fc8d80ac770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:46.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.902+0000 7fc8dffd4640 1 --2- 172.21.15.150:0/3184501685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8d806f400 0x7fc8d80ac770 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc8d400c4d0 tx=0x7fc8d400c990 comp rx=0 tx=0).stop 2024-03-17T23:47:46.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.903+0000 7fc8dffd4640 1 -- 172.21.15.150:0/3184501685 shutdown_connections 2024-03-17T23:47:46.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.903+0000 7fc8dffd4640 1 --2- 172.21.15.150:0/3184501685 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8d8144bb0 0x7fc8d81f4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:46.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.903+0000 7fc8dffd4640 1 --2- 172.21.15.150:0/3184501685 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc8bc06bbd0 0x7fc8bc06e090 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:46.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.903+0000 7fc8dffd4640 1 --2- 172.21.15.150:0/3184501685 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8d80ce4d0 0x7fc8d80a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:46.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.903+0000 7fc8dffd4640 1 --2- 172.21.15.150:0/3184501685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8d806f400 0x7fc8d80ac770 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:46.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.903+0000 7fc8dffd4640 1 -- 172.21.15.150:0/3184501685 >> 172.21.15.150:0/3184501685 conn(0x7fc8d806b190 msgr2=0x7fc8d8142c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:46.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.903+0000 7fc8dffd4640 1 -- 172.21.15.150:0/3184501685 shutdown_connections 2024-03-17T23:47:46.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:46.903+0000 7fc8dffd4640 1 -- 172.21.15.150:0/3184501685 wait complete. 2024-03-17T23:47:46.915 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:47:47.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.015+0000 7f5969bcd640 1 Processor -- start 2024-03-17T23:47:47.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.016+0000 7f5969bcd640 1 -- start start 2024-03-17T23:47:47.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.016+0000 7f5969bcd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f596415dff0 0x7f59640d0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:47.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.016+0000 7f5969bcd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f59641628d0 0x7f59640d0b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:47.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.016+0000 7f5969bcd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f59640b3120 0x7f59640b5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:47.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.016+0000 7f5969bcd640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f59640ce8b0 con 0x7f59640b3120 2024-03-17T23:47:47.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.016+0000 7f5969bcd640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f5964161df0 con 0x7f596415dff0 2024-03-17T23:47:47.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.016+0000 7f5969bcd640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f59641620d0 con 0x7f59641628d0 2024-03-17T23:47:47.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.016+0000 7f5963fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f59640b3120 0x7f59640b5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:47.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.016+0000 7f59637fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f596415dff0 0x7f59640d0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:47.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.017+0000 7f5963fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f59640b3120 0x7f59640b5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50326/0 (socket says 172.21.15.150:50326) 2024-03-17T23:47:47.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.017+0000 7f5963fff640 1 -- 172.21.15.150:0/2729185340 learned_addr learned my addr 172.21.15.150:0/2729185340 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:47.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.017+0000 7f595affd640 1 --2- 172.21.15.150:0/2729185340 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f59641628d0 0x7f59640d0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:47.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.017+0000 7f5963fff640 1 -- 172.21.15.150:0/2729185340 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f59641628d0 msgr2=0x7f59640d0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:47.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.017+0000 7f5963fff640 1 --2- 172.21.15.150:0/2729185340 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f59641628d0 0x7f59640d0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.017+0000 7f5963fff640 1 -- 172.21.15.150:0/2729185340 >> v2:172.21.15.157:3300/0 conn(0x7f596415dff0 msgr2=0x7f59640d0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:47.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.017+0000 7f5963fff640 1 --2- 172.21.15.150:0/2729185340 >> v2:172.21.15.157:3300/0 conn(0x7f596415dff0 0x7f59640d0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.017+0000 7f5963fff640 1 -- 172.21.15.150:0/2729185340 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59641623b0 con 0x7f59640b3120 2024-03-17T23:47:47.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.017+0000 7f59637fe640 1 --2- 172.21.15.150:0/2729185340 >> v2:172.21.15.157:3300/0 conn(0x7f596415dff0 0x7f59640d0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:47.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.017+0000 7f5963fff640 1 --2- 172.21.15.150:0/2729185340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59640b3120 0x7f59640b5780 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f59640b6d20 tx=0x7f595c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dac8463bb9f8c20 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:47.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.018+0000 7f5962ffd640 1 -- 172.21.15.150:0/2729185340 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595c02c9e0 con 0x7f59640b3120 2024-03-17T23:47:47.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.018+0000 7f5962ffd640 1 -- 172.21.15.150:0/2729185340 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f595c02cb80 con 0x7f59640b3120 2024-03-17T23:47:47.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.018+0000 7f5962ffd640 1 -- 172.21.15.150:0/2729185340 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595c0343f0 con 0x7f59640b3120 2024-03-17T23:47:47.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.018+0000 7f5969bcd640 1 -- 172.21.15.150:0/2729185340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59640b3120 msgr2=0x7f59640b5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:47.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.018+0000 7f5969bcd640 1 --2- 172.21.15.150:0/2729185340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59640b3120 0x7f59640b5780 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f59640b6d20 tx=0x7f595c02be40 comp rx=0 tx=0).stop 2024-03-17T23:47:47.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.018+0000 7f5969bcd640 1 -- 172.21.15.150:0/2729185340 shutdown_connections 2024-03-17T23:47:47.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.018+0000 7f5969bcd640 1 --2- 172.21.15.150:0/2729185340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59640b3120 0x7f59640b5780 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.018+0000 7f5969bcd640 1 --2- 172.21.15.150:0/2729185340 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f59641628d0 0x7f59640d0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.018+0000 7f5969bcd640 1 --2- 172.21.15.150:0/2729185340 >> v2:172.21.15.157:3300/0 conn(0x7f596415dff0 0x7f59640d0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.018+0000 7f5969bcd640 1 -- 172.21.15.150:0/2729185340 >> 172.21.15.150:0/2729185340 conn(0x7f596406c9f0 msgr2=0x7f596406ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:47.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.018+0000 7f5969bcd640 1 -- 172.21.15.150:0/2729185340 shutdown_connections 2024-03-17T23:47:47.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.018+0000 7f5969bcd640 1 -- 172.21.15.150:0/2729185340 wait complete. 2024-03-17T23:47:47.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.019+0000 7f5969bcd640 1 Processor -- start 2024-03-17T23:47:47.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.020+0000 7f5969bcd640 1 -- start start 2024-03-17T23:47:47.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.020+0000 7f5969bcd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59640b3120 0x7f59640b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:47.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.020+0000 7f5969bcd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f596415dff0 0x7f59640aa700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:47.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.020+0000 7f5969bcd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59641628d0 0x7f59640aac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:47.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.020+0000 7f5969bcd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f596415e3d0 con 0x7f59640b3120 2024-03-17T23:47:47.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.020+0000 7f5969bcd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f59640b2bd0 con 0x7f59641628d0 2024-03-17T23:47:47.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.020+0000 7f5969bcd640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5964160700 con 0x7f596415dff0 2024-03-17T23:47:47.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.020+0000 7f59637fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59640b3120 0x7f59640b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:47.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.020+0000 7f59637fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59640b3120 0x7f59640b1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50332/0 (socket says 172.21.15.150:50332) 2024-03-17T23:47:47.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.020+0000 7f59637fe640 1 -- 172.21.15.150:0/1823219558 learned_addr learned my addr 172.21.15.150:0/1823219558 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:47.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.020+0000 7f595affd640 1 --2- 172.21.15.150:0/1823219558 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f596415dff0 0x7f59640aa700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:47.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.020+0000 7f5963fff640 1 --2- 172.21.15.150:0/1823219558 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59641628d0 0x7f59640aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:47.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.021+0000 7f59637fe640 1 -- 172.21.15.150:0/1823219558 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f596415dff0 msgr2=0x7f59640aa700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:47.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.021+0000 7f59637fe640 1 --2- 172.21.15.150:0/1823219558 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f596415dff0 0x7f59640aa700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.021+0000 7f59637fe640 1 -- 172.21.15.150:0/1823219558 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59641628d0 msgr2=0x7f59640aac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:47.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.021+0000 7f59637fe640 1 --2- 172.21.15.150:0/1823219558 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59641628d0 0x7f59640aac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.021+0000 7f59637fe640 1 -- 172.21.15.150:0/1823219558 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59641609e0 con 0x7f59640b3120 2024-03-17T23:47:47.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.021+0000 7f59637fe640 1 --2- 172.21.15.150:0/1823219558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59640b3120 0x7f59640b1ca0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f595400e980 tx=0x7f595400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:47.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.021+0000 7f5960ff9640 1 -- 172.21.15.150:0/1823219558 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5954017070 con 0x7f59640b3120 2024-03-17T23:47:47.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.021+0000 7f5960ff9640 1 -- 172.21.15.150:0/1823219558 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f595400ca00 con 0x7f59640b3120 2024-03-17T23:47:47.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.021+0000 7f5969bcd640 1 -- 172.21.15.150:0/1823219558 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5964160cc0 con 0x7f59640b3120 2024-03-17T23:47:47.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.021+0000 7f5960ff9640 1 -- 172.21.15.150:0/1823219558 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f595400cdd0 con 0x7f59640b3120 2024-03-17T23:47:47.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.021+0000 7f5969bcd640 1 -- 172.21.15.150:0/1823219558 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59641f4b60 con 0x7f59640b3120 2024-03-17T23:47:47.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.022+0000 7f5969bcd640 1 -- 172.21.15.150:0/1823219558 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59640681a0 con 0x7f59640b3120 2024-03-17T23:47:47.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.022+0000 7f5960ff9640 1 -- 172.21.15.150:0/1823219558 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f595401a070 con 0x7f59640b3120 2024-03-17T23:47:47.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.023+0000 7f5960ff9640 1 --2- 172.21.15.150:0/1823219558 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f594406bc00 0x7f594406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:47.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.023+0000 7f5960ff9640 1 -- 172.21.15.150:0/1823219558 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f595408afa0 con 0x7f59640b3120 2024-03-17T23:47:47.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.023+0000 7f595affd640 1 --2- 172.21.15.150:0/1823219558 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f594406bc00 0x7f594406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:47.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.023+0000 7f595affd640 1 --2- 172.21.15.150:0/1823219558 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f594406bc00 0x7f594406e0c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f594c009aa0 tx=0x7f594c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:47.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.027+0000 7f5960ff9640 1 -- 172.21.15.150:0/1823219558 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f59540590d0 con 0x7f59640b3120 2024-03-17T23:47:47.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.201+0000 7f5969bcd640 1 -- 172.21.15.150:0/1823219558 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f59640d1a10 con 0x7f59640b3120 2024-03-17T23:47:47.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.202+0000 7f5960ff9640 1 -- 172.21.15.150:0/1823219558 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7f595405d170 con 0x7f59640b3120 2024-03-17T23:47:47.204 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:47.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.204+0000 7f5969bcd640 1 -- 172.21.15.150:0/1823219558 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f594406bc00 msgr2=0x7f594406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:47.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.204+0000 7f5969bcd640 1 --2- 172.21.15.150:0/1823219558 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f594406bc00 0x7f594406e0c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f594c009aa0 tx=0x7f594c009190 comp rx=0 tx=0).stop 2024-03-17T23:47:47.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.204+0000 7f5969bcd640 1 -- 172.21.15.150:0/1823219558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59640b3120 msgr2=0x7f59640b1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:47.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.204+0000 7f5969bcd640 1 --2- 172.21.15.150:0/1823219558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59640b3120 0x7f59640b1ca0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f595400e980 tx=0x7f595400ee40 comp rx=0 tx=0).stop 2024-03-17T23:47:47.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.204+0000 7f5969bcd640 1 -- 172.21.15.150:0/1823219558 shutdown_connections 2024-03-17T23:47:47.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.204+0000 7f5969bcd640 1 --2- 172.21.15.150:0/1823219558 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f594406bc00 0x7f594406e0c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.204+0000 7f5969bcd640 1 --2- 172.21.15.150:0/1823219558 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59641628d0 0x7f59640aac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.204+0000 7f5969bcd640 1 --2- 172.21.15.150:0/1823219558 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f596415dff0 0x7f59640aa700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.204+0000 7f5969bcd640 1 --2- 172.21.15.150:0/1823219558 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59640b3120 0x7f59640b1ca0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.204+0000 7f5969bcd640 1 -- 172.21.15.150:0/1823219558 >> 172.21.15.150:0/1823219558 conn(0x7f596406c9f0 msgr2=0x7f59640b6010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:47.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.204+0000 7f5969bcd640 1 -- 172.21.15.150:0/1823219558 shutdown_connections 2024-03-17T23:47:47.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.204+0000 7f5969bcd640 1 -- 172.21.15.150:0/1823219558 wait complete. 2024-03-17T23:47:47.216 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:47:47.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.317+0000 7f8667a4d640 1 Processor -- start 2024-03-17T23:47:47.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.318+0000 7f8667a4d640 1 -- start start 2024-03-17T23:47:47.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.318+0000 7f8667a4d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8660161520 0x7f86600b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:47.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.318+0000 7f8667a4d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f86600b2bf0 0x7f86600d05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:47.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.318+0000 7f8667a4d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8660161d30 con 0x7f86600d0b10 2024-03-17T23:47:47.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.318+0000 7f8667a4d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f86600b9bf0 con 0x7f8660161520 2024-03-17T23:47:47.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.318+0000 7f8667a4d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8660162940 con 0x7f86600b2bf0 2024-03-17T23:47:47.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.319+0000 7f86657e9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8660161520 0x7f86600b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:47.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.319+0000 7f8664fe8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f86600b2bf0 0x7f86600d05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:47.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.319+0000 7f86657e9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8660161520 0x7f86600b96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50346/0 (socket says 172.21.15.150:50346) 2024-03-17T23:47:47.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.319+0000 7f86657e9640 1 -- 172.21.15.150:0/2281202573 learned_addr learned my addr 172.21.15.150:0/2281202573 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:47.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.319+0000 7f8665fea640 1 --1- 172.21.15.150:0/2281202573 >> v1:172.21.15.157:6789/0 conn(0x7f86600d0b10 0x7f86600b57a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:38946/0 (socket says 172.21.15.150:38946) 2024-03-17T23:47:47.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.319+0000 7f86657e9640 1 -- 172.21.15.150:0/2281202573 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f86600b2bf0 msgr2=0x7f86600d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:47.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.319+0000 7f86657e9640 1 --2- 172.21.15.150:0/2281202573 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f86600b2bf0 0x7f86600d05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.319+0000 7f86657e9640 1 -- 172.21.15.150:0/2281202573 >> v1:172.21.15.157:6789/0 conn(0x7f86600d0b10 legacy=0x7f86600b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:47.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.319+0000 7f86657e9640 1 -- 172.21.15.150:0/2281202573 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8660162010 con 0x7f8660161520 2024-03-17T23:47:47.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.320+0000 7f86657e9640 1 --2- 172.21.15.150:0/2281202573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8660161520 0x7f86600b96b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8654009870 tx=0x7f865402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5338793bf49ba6f server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:47.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.320+0000 7f8653fff640 1 -- 172.21.15.150:0/2281202573 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86540317c0 con 0x7f8660161520 2024-03-17T23:47:47.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.320+0000 7f8653fff640 1 -- 172.21.15.150:0/2281202573 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8654037070 con 0x7f8660161520 2024-03-17T23:47:47.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.320+0000 7f8653fff640 1 -- 172.21.15.150:0/2281202573 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8654031b80 con 0x7f8660161520 2024-03-17T23:47:47.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.320+0000 7f8667a4d640 1 -- 172.21.15.150:0/2281202573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8660161520 msgr2=0x7f86600b96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:47.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.320+0000 7f8667a4d640 1 --2- 172.21.15.150:0/2281202573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8660161520 0x7f86600b96b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8654009870 tx=0x7f865402b860 comp rx=0 tx=0).stop 2024-03-17T23:47:47.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.320+0000 7f8667a4d640 1 -- 172.21.15.150:0/2281202573 shutdown_connections 2024-03-17T23:47:47.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.320+0000 7f8667a4d640 1 --2- 172.21.15.150:0/2281202573 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f86600b2bf0 0x7f86600d05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.320+0000 7f8667a4d640 1 --2- 172.21.15.150:0/2281202573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8660161520 0x7f86600b96b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.320+0000 7f8667a4d640 1 -- 172.21.15.150:0/2281202573 >> 172.21.15.150:0/2281202573 conn(0x7f866006ca00 msgr2=0x7f866006ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:47.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.321+0000 7f8667a4d640 1 -- 172.21.15.150:0/2281202573 shutdown_connections 2024-03-17T23:47:47.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.321+0000 7f8667a4d640 1 -- 172.21.15.150:0/2281202573 wait complete. 2024-03-17T23:47:47.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.322+0000 7f8667a4d640 1 Processor -- start 2024-03-17T23:47:47.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.322+0000 7f8667a4d640 1 -- start start 2024-03-17T23:47:47.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.322+0000 7f8667a4d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f86600b2bf0 0x7f866015b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:47.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.322+0000 7f8667a4d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86600d0b10 0x7f86600b5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:47.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.322+0000 7f8667a4d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8660161520 0x7f86600b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:47.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.322+0000 7f8667a4d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f86601625d0 con 0x7f86600d0b10 2024-03-17T23:47:47.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.322+0000 7f8667a4d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f86600d0f60 con 0x7f86600b2bf0 2024-03-17T23:47:47.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.322+0000 7f8667a4d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8660161900 con 0x7f8660161520 2024-03-17T23:47:47.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.323+0000 7f8664fe8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86600d0b10 0x7f86600b5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:47.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.323+0000 7f8664fe8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86600d0b10 0x7f86600b5f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50358/0 (socket says 172.21.15.150:50358) 2024-03-17T23:47:47.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.323+0000 7f8664fe8640 1 -- 172.21.15.150:0/1926266731 learned_addr learned my addr 172.21.15.150:0/1926266731 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:47.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.323+0000 7f86657e9640 1 --2- 172.21.15.150:0/1926266731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f86600b2bf0 0x7f866015b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:47.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.323+0000 7f8665fea640 1 --2- 172.21.15.150:0/1926266731 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8660161520 0x7f86600b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:47.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.323+0000 7f8664fe8640 1 -- 172.21.15.150:0/1926266731 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8660161520 msgr2=0x7f86600b7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:47.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.323+0000 7f8664fe8640 1 --2- 172.21.15.150:0/1926266731 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8660161520 0x7f86600b7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.323+0000 7f8664fe8640 1 -- 172.21.15.150:0/1926266731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f86600b2bf0 msgr2=0x7f866015b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:47.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.323+0000 7f8664fe8640 1 --2- 172.21.15.150:0/1926266731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f86600b2bf0 0x7f866015b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.323+0000 7f8664fe8640 1 -- 172.21.15.150:0/1926266731 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86600b4e90 con 0x7f86600d0b10 2024-03-17T23:47:47.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.323+0000 7f8664fe8640 1 --2- 172.21.15.150:0/1926266731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86600d0b10 0x7f86600b5f00 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f865c00e870 tx=0x7f865c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:47.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.323+0000 7f8651ffb640 1 -- 172.21.15.150:0/1926266731 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c017070 con 0x7f86600d0b10 2024-03-17T23:47:47.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.323+0000 7f8651ffb640 1 -- 172.21.15.150:0/1926266731 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f865c00c940 con 0x7f86600d0b10 2024-03-17T23:47:47.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.323+0000 7f8667a4d640 1 -- 172.21.15.150:0/1926266731 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86600b5420 con 0x7f86600d0b10 2024-03-17T23:47:47.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.323+0000 7f8651ffb640 1 -- 172.21.15.150:0/1926266731 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865c00cd30 con 0x7f86600d0b10 2024-03-17T23:47:47.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.323+0000 7f8667a4d640 1 -- 172.21.15.150:0/1926266731 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86600b59e0 con 0x7f86600d0b10 2024-03-17T23:47:47.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.324+0000 7f8667a4d640 1 -- 172.21.15.150:0/1926266731 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86600b6560 con 0x7f86600d0b10 2024-03-17T23:47:47.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.325+0000 7f8651ffb640 1 -- 172.21.15.150:0/1926266731 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f865c01a070 con 0x7f86600d0b10 2024-03-17T23:47:47.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.325+0000 7f8651ffb640 1 --2- 172.21.15.150:0/1926266731 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f864406bbd0 0x7f864406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:47.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.325+0000 7f8651ffb640 1 -- 172.21.15.150:0/1926266731 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f865c08ad20 con 0x7f86600d0b10 2024-03-17T23:47:47.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.325+0000 7f86657e9640 1 --2- 172.21.15.150:0/1926266731 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f864406bbd0 0x7f864406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:47.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.325+0000 7f86657e9640 1 --2- 172.21.15.150:0/1926266731 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f864406bbd0 0x7f864406e090 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8654036000 tx=0x7f865403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:47.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.329+0000 7f8651ffb640 1 -- 172.21.15.150:0/1926266731 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f865c058d30 con 0x7f86600d0b10 2024-03-17T23:47:47.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.502+0000 7f8667a4d640 1 -- 172.21.15.150:0/1926266731 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f86600681a0 con 0x7f86600d0b10 2024-03-17T23:47:47.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.502+0000 7f8651ffb640 1 -- 172.21.15.150:0/1926266731 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7f865c05cdd0 con 0x7f86600d0b10 2024-03-17T23:47:47.504 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:47:47.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.504+0000 7f8667a4d640 1 -- 172.21.15.150:0/1926266731 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f864406bbd0 msgr2=0x7f864406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:47.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.504+0000 7f8667a4d640 1 --2- 172.21.15.150:0/1926266731 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f864406bbd0 0x7f864406e090 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8654036000 tx=0x7f865403d040 comp rx=0 tx=0).stop 2024-03-17T23:47:47.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.504+0000 7f8667a4d640 1 -- 172.21.15.150:0/1926266731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86600d0b10 msgr2=0x7f86600b5f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:47.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.504+0000 7f8667a4d640 1 --2- 172.21.15.150:0/1926266731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86600d0b10 0x7f86600b5f00 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f865c00e870 tx=0x7f865c00ed30 comp rx=0 tx=0).stop 2024-03-17T23:47:47.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.504+0000 7f8667a4d640 1 -- 172.21.15.150:0/1926266731 shutdown_connections 2024-03-17T23:47:47.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.504+0000 7f8667a4d640 1 --2- 172.21.15.150:0/1926266731 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8660161520 0x7f86600b7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.504+0000 7f8667a4d640 1 --2- 172.21.15.150:0/1926266731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86600d0b10 0x7f86600b5f00 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.504+0000 7f8667a4d640 1 --2- 172.21.15.150:0/1926266731 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f864406bbd0 0x7f864406e090 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.504+0000 7f8667a4d640 1 --2- 172.21.15.150:0/1926266731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f86600b2bf0 0x7f866015b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.504+0000 7f8667a4d640 1 -- 172.21.15.150:0/1926266731 >> 172.21.15.150:0/1926266731 conn(0x7f866006ca00 msgr2=0x7f86600d1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:47.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.504+0000 7f8667a4d640 1 -- 172.21.15.150:0/1926266731 shutdown_connections 2024-03-17T23:47:47.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.504+0000 7f8667a4d640 1 -- 172.21.15.150:0/1926266731 wait complete. 2024-03-17T23:47:47.517 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-17T23:47:47.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.619+0000 7f0f6d589640 1 Processor -- start 2024-03-17T23:47:47.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.619+0000 7f0f6d589640 1 -- start start 2024-03-17T23:47:47.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.620+0000 7f0f6d589640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f68161520 0x7f0f680b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:47.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.620+0000 7f0f6d589640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0f680d0ce0 0x7f0f680d1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:47.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.620+0000 7f0f6d589640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f68161d30 con 0x7f0f680b2bf0 2024-03-17T23:47:47.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.620+0000 7f0f6d589640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f0f680b9bf0 con 0x7f0f680d0ce0 2024-03-17T23:47:47.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.620+0000 7f0f6d589640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0f68162940 con 0x7f0f68161520 2024-03-17T23:47:47.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.620+0000 7f0f677fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0f680d0ce0 0x7f0f680d1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:47.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.620+0000 7f0f66ffd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f68161520 0x7f0f680b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:47.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.620+0000 7f0f667fc640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f0f680b2bf0 0x7f0f680d05d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:38960/0 (socket says 172.21.15.150:38960) 2024-03-17T23:47:47.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.620+0000 7f0f667fc640 1 -- 172.21.15.150:0/1238835372 learned_addr learned my addr 172.21.15.150:0/1238835372 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:47.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.620+0000 7f0f65ffb640 1 -- 172.21.15.150:0/1238835372 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2357883907 0 0) 0x7f0f68161d30 con 0x7f0f680b2bf0 2024-03-17T23:47:47.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.621+0000 7f0f66ffd640 1 -- 172.21.15.150:0/1238835372 >> v2:172.21.15.157:3300/0 conn(0x7f0f680d0ce0 msgr2=0x7f0f680d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:47.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.621+0000 7f0f66ffd640 1 --2- 172.21.15.150:0/1238835372 >> v2:172.21.15.157:3300/0 conn(0x7f0f680d0ce0 0x7f0f680d1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.621+0000 7f0f66ffd640 1 -- 172.21.15.150:0/1238835372 >> v1:172.21.15.157:6789/0 conn(0x7f0f680b2bf0 legacy=0x7f0f680d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:47.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.621+0000 7f0f66ffd640 1 -- 172.21.15.150:0/1238835372 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f68162010 con 0x7f0f68161520 2024-03-17T23:47:47.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.621+0000 7f0f677fe640 1 --2- 172.21.15.150:0/1238835372 >> v2:172.21.15.157:3300/0 conn(0x7f0f680d0ce0 0x7f0f680d1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:47.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.621+0000 7f0f66ffd640 1 --2- 172.21.15.150:0/1238835372 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f68161520 0x7f0f680b96b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0f580097a0 tx=0x7f0f5802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=612ce74a11576550 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:47.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.621+0000 7f0f65ffb640 1 -- 172.21.15.150:0/1238835372 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f58038070 con 0x7f0f68161520 2024-03-17T23:47:47.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.621+0000 7f0f65ffb640 1 -- 172.21.15.150:0/1238835372 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f58031700 con 0x7f0f68161520 2024-03-17T23:47:47.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.621+0000 7f0f65ffb640 1 -- 172.21.15.150:0/1238835372 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f58031ad0 con 0x7f0f68161520 2024-03-17T23:47:47.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.622+0000 7f0f6d589640 1 -- 172.21.15.150:0/1238835372 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f68161520 msgr2=0x7f0f680b96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:47.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.622+0000 7f0f6d589640 1 --2- 172.21.15.150:0/1238835372 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f68161520 0x7f0f680b96b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0f580097a0 tx=0x7f0f5802b7a0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.622+0000 7f0f6d589640 1 -- 172.21.15.150:0/1238835372 shutdown_connections 2024-03-17T23:47:47.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.622+0000 7f0f6d589640 1 --2- 172.21.15.150:0/1238835372 >> v2:172.21.15.157:3300/0 conn(0x7f0f680d0ce0 0x7f0f680d1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.622+0000 7f0f6d589640 1 --2- 172.21.15.150:0/1238835372 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f68161520 0x7f0f680b96b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.622+0000 7f0f6d589640 1 -- 172.21.15.150:0/1238835372 >> 172.21.15.150:0/1238835372 conn(0x7f0f6806ca00 msgr2=0x7f0f6806ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:47.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.622+0000 7f0f6d589640 1 -- 172.21.15.150:0/1238835372 shutdown_connections 2024-03-17T23:47:47.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.622+0000 7f0f6d589640 1 -- 172.21.15.150:0/1238835372 wait complete. 2024-03-17T23:47:47.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.623+0000 7f0f6d589640 1 Processor -- start 2024-03-17T23:47:47.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.623+0000 7f0f6d589640 1 -- start start 2024-03-17T23:47:47.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.624+0000 7f0f6d589640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f680b2bf0 0x7f0f6815fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:47.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.624+0000 7f0f6d589640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f680d0ce0 0x7f0f680b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:47.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.624+0000 7f0f6d589640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f68161520 0x7f0f681f3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:47.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.624+0000 7f0f6d589640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0f68161900 con 0x7f0f68161520 2024-03-17T23:47:47.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.624+0000 7f0f6d589640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0f681625d0 con 0x7f0f680d0ce0 2024-03-17T23:47:47.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.624+0000 7f0f6d589640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0f680d24e0 con 0x7f0f680b2bf0 2024-03-17T23:47:47.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.624+0000 7f0f677fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f68161520 0x7f0f681f3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:47.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.624+0000 7f0f667fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f680d0ce0 0x7f0f680b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:47.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.624+0000 7f0f677fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f68161520 0x7f0f681f3f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50366/0 (socket says 172.21.15.150:50366) 2024-03-17T23:47:47.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.624+0000 7f0f677fe640 1 -- 172.21.15.150:0/155872991 learned_addr learned my addr 172.21.15.150:0/155872991 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:47.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.624+0000 7f0f66ffd640 1 --2- 172.21.15.150:0/155872991 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f680b2bf0 0x7f0f6815fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:47.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.624+0000 7f0f667fc640 1 --2- 172.21.15.150:0/155872991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f680d0ce0 0x7f0f680b7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:58578/0 (socket says 172.21.15.150:58578) 2024-03-17T23:47:47.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.624+0000 7f0f66ffd640 1 -- 172.21.15.150:0/155872991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f680d0ce0 msgr2=0x7f0f680b7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:47.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.624+0000 7f0f66ffd640 1 --2- 172.21.15.150:0/155872991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f680d0ce0 0x7f0f680b7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.624+0000 7f0f66ffd640 1 -- 172.21.15.150:0/155872991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f68161520 msgr2=0x7f0f681f3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:47.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.624+0000 7f0f66ffd640 1 --2- 172.21.15.150:0/155872991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f68161520 0x7f0f681f3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.624+0000 7f0f66ffd640 1 -- 172.21.15.150:0/155872991 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f680cc410 con 0x7f0f680b2bf0 2024-03-17T23:47:47.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.625+0000 7f0f677fe640 1 --2- 172.21.15.150:0/155872991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f68161520 0x7f0f681f3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:47.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.625+0000 7f0f667fc640 1 --2- 172.21.15.150:0/155872991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f680d0ce0 0x7f0f680b7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:47.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.625+0000 7f0f66ffd640 1 --2- 172.21.15.150:0/155872991 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f680b2bf0 0x7f0f6815fda0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0f58033000 tx=0x7f0f580046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:47.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.625+0000 7f0f4f7fe640 1 -- 172.21.15.150:0/155872991 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f58043070 con 0x7f0f680b2bf0 2024-03-17T23:47:47.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.625+0000 7f0f4f7fe640 1 -- 172.21.15.150:0/155872991 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f58004020 con 0x7f0f680b2bf0 2024-03-17T23:47:47.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.625+0000 7f0f6d589640 1 -- 172.21.15.150:0/155872991 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f680cc6f0 con 0x7f0f680b2bf0 2024-03-17T23:47:47.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.625+0000 7f0f4f7fe640 1 -- 172.21.15.150:0/155872991 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f58038040 con 0x7f0f680b2bf0 2024-03-17T23:47:47.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.625+0000 7f0f6d589640 1 -- 172.21.15.150:0/155872991 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f680cccb0 con 0x7f0f680b2bf0 2024-03-17T23:47:47.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.626+0000 7f0f6d589640 1 -- 172.21.15.150:0/155872991 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f680681a0 con 0x7f0f680b2bf0 2024-03-17T23:47:47.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.626+0000 7f0f4f7fe640 1 -- 172.21.15.150:0/155872991 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0f58033820 con 0x7f0f680b2bf0 2024-03-17T23:47:47.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.627+0000 7f0f4f7fe640 1 --2- 172.21.15.150:0/155872991 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0f5406bbd0 0x7f0f5406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:47.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.627+0000 7f0f667fc640 1 --2- 172.21.15.150:0/155872991 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0f5406bbd0 0x7f0f5406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:47.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.627+0000 7f0f4f7fe640 1 -- 172.21.15.150:0/155872991 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f0f580ac9e0 con 0x7f0f680b2bf0 2024-03-17T23:47:47.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.627+0000 7f0f667fc640 1 --2- 172.21.15.150:0/155872991 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0f5406bbd0 0x7f0f5406e090 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c000f60 tx=0x7f0f5c007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:47.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.631+0000 7f0f4f7fe640 1 -- 172.21.15.150:0/155872991 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0f58030070 con 0x7f0f680b2bf0 2024-03-17T23:47:47.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.803+0000 7f0f6d589640 1 -- 172.21.15.150:0/155872991 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f0f680d19c0 con 0x7f0f680b2bf0 2024-03-17T23:47:47.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.803+0000 7f0f4f7fe640 1 -- 172.21.15.150:0/155872991 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7f0f5807ab90 con 0x7f0f680b2bf0 2024-03-17T23:47:47.805 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:47.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.805+0000 7f0f6d589640 1 -- 172.21.15.150:0/155872991 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0f5406bbd0 msgr2=0x7f0f5406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:47.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.805+0000 7f0f6d589640 1 --2- 172.21.15.150:0/155872991 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0f5406bbd0 0x7f0f5406e090 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c000f60 tx=0x7f0f5c007d50 comp rx=0 tx=0).stop 2024-03-17T23:47:47.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.805+0000 7f0f6d589640 1 -- 172.21.15.150:0/155872991 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f680b2bf0 msgr2=0x7f0f6815fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:47.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.806+0000 7f0f6d589640 1 --2- 172.21.15.150:0/155872991 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f680b2bf0 0x7f0f6815fda0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0f58033000 tx=0x7f0f580046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.806+0000 7f0f6d589640 1 -- 172.21.15.150:0/155872991 shutdown_connections 2024-03-17T23:47:47.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.806+0000 7f0f6d589640 1 --2- 172.21.15.150:0/155872991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f68161520 0x7f0f681f3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.806+0000 7f0f6d589640 1 --2- 172.21.15.150:0/155872991 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0f5406bbd0 0x7f0f5406e090 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.806+0000 7f0f6d589640 1 --2- 172.21.15.150:0/155872991 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f680d0ce0 0x7f0f680b7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.806+0000 7f0f6d589640 1 --2- 172.21.15.150:0/155872991 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f680b2bf0 0x7f0f6815fda0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.806+0000 7f0f6d589640 1 -- 172.21.15.150:0/155872991 >> 172.21.15.150:0/155872991 conn(0x7f0f6806ca00 msgr2=0x7f0f680b9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:47.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.806+0000 7f0f6d589640 1 -- 172.21.15.150:0/155872991 shutdown_connections 2024-03-17T23:47:47.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.806+0000 7f0f6d589640 1 -- 172.21.15.150:0/155872991 wait complete. 2024-03-17T23:47:47.818 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-17T23:47:47.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.918+0000 7fc640b28640 1 Processor -- start 2024-03-17T23:47:47.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.918+0000 7fc640b28640 1 -- start start 2024-03-17T23:47:47.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.919+0000 7fc640b28640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc63c15e030 0x7fc63c06b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:47.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.919+0000 7fc640b28640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc63c0cdc10 0x7fc63c0b0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:47.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.919+0000 7fc640b28640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc63c0ce160 con 0x7fc63c06b6d0 2024-03-17T23:47:47.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.919+0000 7fc640b28640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc63c0cfed0 con 0x7fc63c0cdc10 2024-03-17T23:47:47.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.919+0000 7fc640b28640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc63c0d01b0 con 0x7fc63c15e030 2024-03-17T23:47:47.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.919+0000 7fc63a59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc63c15e030 0x7fc63c06b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:47.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.919+0000 7fc63ad9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc63c0cdc10 0x7fc63c0b0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:47.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.919+0000 7fc639d9b640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fc63c06b6d0 0x7fc63c0cd500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:38972/0 (socket says 172.21.15.150:38972) 2024-03-17T23:47:47.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.919+0000 7fc639d9b640 1 -- 172.21.15.150:0/569307711 learned_addr learned my addr 172.21.15.150:0/569307711 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:47.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.920+0000 7fc63959a640 1 -- 172.21.15.150:0/569307711 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1645943286 0 0) 0x7fc63c0ce160 con 0x7fc63c06b6d0 2024-03-17T23:47:47.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.920+0000 7fc63959a640 1 -- 172.21.15.150:0/569307711 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc624003520 con 0x7fc63c06b6d0 2024-03-17T23:47:47.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.920+0000 7fc63ad9d640 1 -- 172.21.15.150:0/569307711 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc63c15e030 msgr2=0x7fc63c06b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:47.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.920+0000 7fc63ad9d640 1 --2- 172.21.15.150:0/569307711 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc63c15e030 0x7fc63c06b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.920+0000 7fc63ad9d640 1 -- 172.21.15.150:0/569307711 >> v1:172.21.15.157:6789/0 conn(0x7fc63c06b6d0 legacy=0x7fc63c0cd500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:47.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.920+0000 7fc63ad9d640 1 -- 172.21.15.150:0/569307711 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63c0d0490 con 0x7fc63c0cdc10 2024-03-17T23:47:47.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.920+0000 7fc63a59c640 1 --2- 172.21.15.150:0/569307711 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc63c15e030 0x7fc63c06b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:47.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.921+0000 7fc63ad9d640 1 --2- 172.21.15.150:0/569307711 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc63c0cdc10 0x7fc63c0b0260 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc6340097a0 tx=0x7fc63402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f3fa4f279d02414 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:47.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.921+0000 7fc63959a640 1 -- 172.21.15.150:0/569307711 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc634035020 con 0x7fc63c0cdc10 2024-03-17T23:47:47.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.921+0000 7fc63959a640 1 -- 172.21.15.150:0/569307711 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc634031830 con 0x7fc63c0cdc10 2024-03-17T23:47:47.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.921+0000 7fc63959a640 1 -- 172.21.15.150:0/569307711 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc634031c00 con 0x7fc63c0cdc10 2024-03-17T23:47:47.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.921+0000 7fc640b28640 1 -- 172.21.15.150:0/569307711 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc63c0cdc10 msgr2=0x7fc63c0b0260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:47.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.921+0000 7fc640b28640 1 --2- 172.21.15.150:0/569307711 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc63c0cdc10 0x7fc63c0b0260 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc6340097a0 tx=0x7fc63402b8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.921+0000 7fc640b28640 1 -- 172.21.15.150:0/569307711 shutdown_connections 2024-03-17T23:47:47.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.921+0000 7fc640b28640 1 --2- 172.21.15.150:0/569307711 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc63c0cdc10 0x7fc63c0b0260 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.921+0000 7fc640b28640 1 --2- 172.21.15.150:0/569307711 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc63c15e030 0x7fc63c06b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.921+0000 7fc640b28640 1 -- 172.21.15.150:0/569307711 >> 172.21.15.150:0/569307711 conn(0x7fc63c099a70 msgr2=0x7fc63c099e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:47.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.921+0000 7fc640b28640 1 -- 172.21.15.150:0/569307711 shutdown_connections 2024-03-17T23:47:47.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.921+0000 7fc640b28640 1 -- 172.21.15.150:0/569307711 wait complete. 2024-03-17T23:47:47.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.923+0000 7fc640b28640 1 Processor -- start 2024-03-17T23:47:47.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.923+0000 7fc640b28640 1 -- start start 2024-03-17T23:47:47.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.923+0000 7fc640b28640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc63c06b6d0 0x7fc63c09bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:47.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.923+0000 7fc640b28640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc63c0cdc10 0x7fc63c14ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:47.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.923+0000 7fc640b28640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc63c15e030 0x7fc63c15fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:47.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.923+0000 7fc640b28640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc63c0b07a0 con 0x7fc63c15e030 2024-03-17T23:47:47.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.923+0000 7fc640b28640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc63c15e410 con 0x7fc63c06b6d0 2024-03-17T23:47:47.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.923+0000 7fc640b28640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc63c0b0920 con 0x7fc63c0cdc10 2024-03-17T23:47:47.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.924+0000 7fc639d9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc63c0cdc10 0x7fc63c14ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:47.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.924+0000 7fc63ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc63c15e030 0x7fc63c15fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:47.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.924+0000 7fc639d9b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc63c0cdc10 0x7fc63c14ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:60670/0 (socket says 172.21.15.150:60670) 2024-03-17T23:47:47.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.924+0000 7fc63ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc63c15e030 0x7fc63c15fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50372/0 (socket says 172.21.15.150:50372) 2024-03-17T23:47:47.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.924+0000 7fc63a59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc63c06b6d0 0x7fc63c09bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:47.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.924+0000 7fc63ad9d640 1 -- 172.21.15.150:0/1856800111 learned_addr learned my addr 172.21.15.150:0/1856800111 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:47.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.924+0000 7fc63ad9d640 1 -- 172.21.15.150:0/1856800111 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc63c0cdc10 msgr2=0x7fc63c14ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:47.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.924+0000 7fc63ad9d640 1 --2- 172.21.15.150:0/1856800111 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc63c0cdc10 0x7fc63c14ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.924+0000 7fc63ad9d640 1 -- 172.21.15.150:0/1856800111 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc63c06b6d0 msgr2=0x7fc63c09bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:47.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.924+0000 7fc63ad9d640 1 --2- 172.21.15.150:0/1856800111 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc63c06b6d0 0x7fc63c09bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:47.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.924+0000 7fc63ad9d640 1 -- 172.21.15.150:0/1856800111 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63c0c8050 con 0x7fc63c15e030 2024-03-17T23:47:47.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.924+0000 7fc639d9b640 1 --2- 172.21.15.150:0/1856800111 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc63c0cdc10 0x7fc63c14ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:47.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.924+0000 7fc63a59c640 1 --2- 172.21.15.150:0/1856800111 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc63c06b6d0 0x7fc63c09bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:47.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.924+0000 7fc63ad9d640 1 --2- 172.21.15.150:0/1856800111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc63c15e030 0x7fc63c15fb30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc6340097a0 tx=0x7fc6340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:47.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.924+0000 7fc62affd640 1 -- 172.21.15.150:0/1856800111 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc634035040 con 0x7fc63c15e030 2024-03-17T23:47:47.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.924+0000 7fc62affd640 1 -- 172.21.15.150:0/1856800111 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc63403b840 con 0x7fc63c15e030 2024-03-17T23:47:47.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.924+0000 7fc640b28640 1 -- 172.21.15.150:0/1856800111 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc63c0c8330 con 0x7fc63c15e030 2024-03-17T23:47:47.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.924+0000 7fc62affd640 1 -- 172.21.15.150:0/1856800111 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63403bc30 con 0x7fc63c15e030 2024-03-17T23:47:47.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.924+0000 7fc640b28640 1 -- 172.21.15.150:0/1856800111 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc63c0c88f0 con 0x7fc63c15e030 2024-03-17T23:47:47.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.925+0000 7fc640b28640 1 -- 172.21.15.150:0/1856800111 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc63c0688b0 con 0x7fc63c15e030 2024-03-17T23:47:47.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.926+0000 7fc62affd640 1 -- 172.21.15.150:0/1856800111 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc634038020 con 0x7fc63c15e030 2024-03-17T23:47:47.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.926+0000 7fc62affd640 1 --2- 172.21.15.150:0/1856800111 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc61c06bbd0 0x7fc61c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:47.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.926+0000 7fc62affd640 1 -- 172.21.15.150:0/1856800111 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fc6340acdc0 con 0x7fc63c15e030 2024-03-17T23:47:47.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.926+0000 7fc63a59c640 1 --2- 172.21.15.150:0/1856800111 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc61c06bbd0 0x7fc61c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:47.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.926+0000 7fc63a59c640 1 --2- 172.21.15.150:0/1856800111 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc61c06bbd0 0x7fc61c06e090 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc62c009fd0 tx=0x7fc62c00b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:47.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:47.930+0000 7fc62affd640 1 -- 172.21.15.150:0/1856800111 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc63407ae60 con 0x7fc63c15e030 2024-03-17T23:47:48.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.119+0000 7fc640b28640 1 -- 172.21.15.150:0/1856800111 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc63c150470 con 0x7fc63c15e030 2024-03-17T23:47:48.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.120+0000 7fc62affd640 1 -- 172.21.15.150:0/1856800111 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) ==== 80+0+0 (secure 0 0 0) 0x7fc63407fe20 con 0x7fc63c15e030 2024-03-17T23:47:48.121 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-03-17T23:47:48.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.122+0000 7fc640b28640 1 -- 172.21.15.150:0/1856800111 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc61c06bbd0 msgr2=0x7fc61c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:48.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.122+0000 7fc640b28640 1 --2- 172.21.15.150:0/1856800111 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc61c06bbd0 0x7fc61c06e090 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc62c009fd0 tx=0x7fc62c00b040 comp rx=0 tx=0).stop 2024-03-17T23:47:48.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.122+0000 7fc640b28640 1 -- 172.21.15.150:0/1856800111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc63c15e030 msgr2=0x7fc63c15fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:48.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.122+0000 7fc640b28640 1 --2- 172.21.15.150:0/1856800111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc63c15e030 0x7fc63c15fb30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc6340097a0 tx=0x7fc6340046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:48.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.122+0000 7fc640b28640 1 -- 172.21.15.150:0/1856800111 shutdown_connections 2024-03-17T23:47:48.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.122+0000 7fc640b28640 1 --2- 172.21.15.150:0/1856800111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc63c15e030 0x7fc63c15fb30 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:48.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.122+0000 7fc640b28640 1 --2- 172.21.15.150:0/1856800111 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc63c0cdc10 0x7fc63c14ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:48.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.122+0000 7fc640b28640 1 --2- 172.21.15.150:0/1856800111 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc61c06bbd0 0x7fc61c06e090 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:48.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.122+0000 7fc640b28640 1 --2- 172.21.15.150:0/1856800111 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc63c06b6d0 0x7fc63c09bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:48.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.122+0000 7fc640b28640 1 -- 172.21.15.150:0/1856800111 >> 172.21.15.150:0/1856800111 conn(0x7fc63c099a70 msgr2=0x7fc63c0cbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:48.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.122+0000 7fc640b28640 1 -- 172.21.15.150:0/1856800111 shutdown_connections 2024-03-17T23:47:48.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.122+0000 7fc640b28640 1 -- 172.21.15.150:0/1856800111 wait complete. 2024-03-17T23:47:48.134 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi157.front.sepia.ceph.com... 2024-03-17T23:47:48.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:48.135 DEBUG:teuthology.orchestra.run.smithi157:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-03-17T23:47:48.163 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:47:48.163 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi157.front.sepia.ceph.com 2024-03-17T23:47:48.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:48.163 DEBUG:teuthology.orchestra.run.smithi157:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-17T23:47:48.462 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:47:48.462 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:47:48.478 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:47:48.478 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:47:48.534 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:48.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.673+0000 7ff7e8a44640 1 Processor -- start 2024-03-17T23:47:48.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.673+0000 7ff7e8a44640 1 -- start start 2024-03-17T23:47:48.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.673+0000 7ff7e8a44640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff7e406b8a0 0x7ff7e41626d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:48.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.673+0000 7ff7e8a44640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7e40affc0 0x7ff7e40cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:48.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.673+0000 7ff7e8a44640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7e40c7410 con 0x7ff7e415e0d0 2024-03-17T23:47:48.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.673+0000 7ff7e8a44640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff7e4162c10 con 0x7ff7e406b8a0 2024-03-17T23:47:48.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.673+0000 7ff7e8a44640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff7e40cb630 con 0x7ff7e40affc0 2024-03-17T23:47:48.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.674+0000 7ff7dbfff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff7e406b8a0 0x7ff7e41626d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:48.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.674+0000 7ff7e259c640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7ff7e415e0d0 0x7ff7e406b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:38986/0 (socket says 172.21.15.150:38986) 2024-03-17T23:47:48.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.674+0000 7ff7e259c640 1 -- 172.21.15.150:0/83711651 learned_addr learned my addr 172.21.15.150:0/83711651 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:48.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.674+0000 7ff7e2d9d640 1 --2- 172.21.15.150:0/83711651 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7e40affc0 0x7ff7e40cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:48.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.674+0000 7ff7e2d9d640 1 -- 172.21.15.150:0/83711651 >> v2:172.21.15.157:3300/0 conn(0x7ff7e406b8a0 msgr2=0x7ff7e41626d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:48.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.674+0000 7ff7e2d9d640 1 --2- 172.21.15.150:0/83711651 >> v2:172.21.15.157:3300/0 conn(0x7ff7e406b8a0 0x7ff7e41626d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:48.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.674+0000 7ff7e2d9d640 1 -- 172.21.15.150:0/83711651 >> v1:172.21.15.157:6789/0 conn(0x7ff7e415e0d0 legacy=0x7ff7e406b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:48.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.674+0000 7ff7e2d9d640 1 -- 172.21.15.150:0/83711651 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7e40c76f0 con 0x7ff7e40affc0 2024-03-17T23:47:48.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.675+0000 7ff7e2d9d640 1 --2- 172.21.15.150:0/83711651 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7e40affc0 0x7ff7e40cbac0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc009d90 tx=0x7ff7dc02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4a52c6b1d7c4e01 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:48.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.675+0000 7ff7e1d9b640 1 -- 172.21.15.150:0/83711651 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc039070 con 0x7ff7e40affc0 2024-03-17T23:47:48.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.675+0000 7ff7e1d9b640 1 -- 172.21.15.150:0/83711651 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7dc02cc50 con 0x7ff7e40affc0 2024-03-17T23:47:48.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.675+0000 7ff7e1d9b640 1 -- 172.21.15.150:0/83711651 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc034440 con 0x7ff7e40affc0 2024-03-17T23:47:48.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.675+0000 7ff7e8a44640 1 -- 172.21.15.150:0/83711651 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7e40affc0 msgr2=0x7ff7e40cbac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:48.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.675+0000 7ff7e8a44640 1 --2- 172.21.15.150:0/83711651 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7e40affc0 0x7ff7e40cbac0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc009d90 tx=0x7ff7dc02bea0 comp rx=0 tx=0).stop 2024-03-17T23:47:48.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.675+0000 7ff7e8a44640 1 -- 172.21.15.150:0/83711651 shutdown_connections 2024-03-17T23:47:48.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.675+0000 7ff7e8a44640 1 --2- 172.21.15.150:0/83711651 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7e40affc0 0x7ff7e40cbac0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:48.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.675+0000 7ff7e8a44640 1 --2- 172.21.15.150:0/83711651 >> v2:172.21.15.157:3300/0 conn(0x7ff7e406b8a0 0x7ff7e41626d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:48.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.675+0000 7ff7e8a44640 1 -- 172.21.15.150:0/83711651 >> 172.21.15.150:0/83711651 conn(0x7ff7e4099a70 msgr2=0x7ff7e4099e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:48.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.675+0000 7ff7e8a44640 1 -- 172.21.15.150:0/83711651 shutdown_connections 2024-03-17T23:47:48.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.675+0000 7ff7e8a44640 1 -- 172.21.15.150:0/83711651 wait complete. 2024-03-17T23:47:48.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.677+0000 7ff7e8a44640 1 Processor -- start 2024-03-17T23:47:48.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.677+0000 7ff7e8a44640 1 -- start start 2024-03-17T23:47:48.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.677+0000 7ff7e8a44640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7e406b8a0 0x7ff7e415bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:48.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.677+0000 7ff7e8a44640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7e40affc0 0x7ff7e4156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:48.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.677+0000 7ff7e8a44640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7e415e0d0 0x7ff7e4158c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:48.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.677+0000 7ff7e8a44640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff7e415e4b0 con 0x7ff7e406b8a0 2024-03-17T23:47:48.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.677+0000 7ff7e8a44640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff7e40c7030 con 0x7ff7e415e0d0 2024-03-17T23:47:48.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.677+0000 7ff7e8a44640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff7e415c500 con 0x7ff7e40affc0 2024-03-17T23:47:48.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.678+0000 7ff7e259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7e406b8a0 0x7ff7e415bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:48.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.678+0000 7ff7e259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7e406b8a0 0x7ff7e415bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50384/0 (socket says 172.21.15.150:50384) 2024-03-17T23:47:48.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.678+0000 7ff7e259c640 1 -- 172.21.15.150:0/2582271751 learned_addr learned my addr 172.21.15.150:0/2582271751 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:48.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.678+0000 7ff7e2d9d640 1 --2- 172.21.15.150:0/2582271751 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7e415e0d0 0x7ff7e4158c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:48.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.678+0000 7ff7dbfff640 1 --2- 172.21.15.150:0/2582271751 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7e40affc0 0x7ff7e4156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:48.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.678+0000 7ff7e259c640 1 -- 172.21.15.150:0/2582271751 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7e40affc0 msgr2=0x7ff7e4156f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:48.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.678+0000 7ff7e259c640 1 --2- 172.21.15.150:0/2582271751 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7e40affc0 0x7ff7e4156f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:48.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.678+0000 7ff7e259c640 1 -- 172.21.15.150:0/2582271751 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7e415e0d0 msgr2=0x7ff7e4158c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:48.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.678+0000 7ff7e259c640 1 --2- 172.21.15.150:0/2582271751 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7e415e0d0 0x7ff7e4158c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:48.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.678+0000 7ff7e259c640 1 -- 172.21.15.150:0/2582271751 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7e40ca490 con 0x7ff7e406b8a0 2024-03-17T23:47:48.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.678+0000 7ff7e2d9d640 1 --2- 172.21.15.150:0/2582271751 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7e415e0d0 0x7ff7e4158c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:48.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.678+0000 7ff7e259c640 1 --2- 172.21.15.150:0/2582271751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7e406b8a0 0x7ff7e415bfc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff7d400c430 tx=0x7ff7d400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:48.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.678+0000 7ff7daffd640 1 -- 172.21.15.150:0/2582271751 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d4017070 con 0x7ff7e406b8a0 2024-03-17T23:47:48.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.678+0000 7ff7daffd640 1 -- 172.21.15.150:0/2582271751 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7d4005ce0 con 0x7ff7e406b8a0 2024-03-17T23:47:48.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.678+0000 7ff7e8a44640 1 -- 172.21.15.150:0/2582271751 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7e415b4e0 con 0x7ff7e406b8a0 2024-03-17T23:47:48.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.678+0000 7ff7daffd640 1 -- 172.21.15.150:0/2582271751 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d40072f0 con 0x7ff7e406b8a0 2024-03-17T23:47:48.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.678+0000 7ff7e8a44640 1 -- 172.21.15.150:0/2582271751 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7e415baa0 con 0x7ff7e406b8a0 2024-03-17T23:47:48.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.679+0000 7ff7e8a44640 1 -- 172.21.15.150:0/2582271751 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7e40688b0 con 0x7ff7e406b8a0 2024-03-17T23:47:48.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.680+0000 7ff7daffd640 1 -- 172.21.15.150:0/2582271751 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff7d401a070 con 0x7ff7e406b8a0 2024-03-17T23:47:48.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.680+0000 7ff7daffd640 1 --2- 172.21.15.150:0/2582271751 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff7c406bc00 0x7ff7c406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:48.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.680+0000 7ff7daffd640 1 -- 172.21.15.150:0/2582271751 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7ff7d408a160 con 0x7ff7e406b8a0 2024-03-17T23:47:48.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.680+0000 7ff7dbfff640 1 --2- 172.21.15.150:0/2582271751 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff7c406bc00 0x7ff7c406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:48.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.681+0000 7ff7dbfff640 1 --2- 172.21.15.150:0/2582271751 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff7c406bc00 0x7ff7c406e0c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff7e41f7210 tx=0x7ff7cc009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:48.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.685+0000 7ff7daffd640 1 -- 172.21.15.150:0/2582271751 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff7d4058290 con 0x7ff7e406b8a0 2024-03-17T23:47:48.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.860+0000 7ff7e8a44640 1 -- 172.21.15.150:0/2582271751 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff7e4073000 con 0x7ff7e406b8a0 2024-03-17T23:47:48.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.861+0000 7ff7daffd640 1 -- 172.21.15.150:0/2582271751 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18881 (secure 0 0 0) 0x7ff7d405c330 con 0x7ff7e406b8a0 2024-03-17T23:47:48.865 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:48.865 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":28,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:37.031864+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.e","mappings":[{"from":7,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:48.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.863+0000 7ff7e8a44640 1 -- 172.21.15.150:0/2582271751 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff7c406bc00 msgr2=0x7ff7c406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:48.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.863+0000 7ff7e8a44640 1 --2- 172.21.15.150:0/2582271751 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff7c406bc00 0x7ff7c406e0c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff7e41f7210 tx=0x7ff7cc009190 comp rx=0 tx=0).stop 2024-03-17T23:47:48.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.863+0000 7ff7e8a44640 1 -- 172.21.15.150:0/2582271751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7e406b8a0 msgr2=0x7ff7e415bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:48.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.863+0000 7ff7e8a44640 1 --2- 172.21.15.150:0/2582271751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7e406b8a0 0x7ff7e415bfc0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff7d400c430 tx=0x7ff7d400c8f0 comp rx=0 tx=0).stop 2024-03-17T23:47:48.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.863+0000 7ff7e8a44640 1 -- 172.21.15.150:0/2582271751 shutdown_connections 2024-03-17T23:47:48.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.863+0000 7ff7e8a44640 1 --2- 172.21.15.150:0/2582271751 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff7c406bc00 0x7ff7c406e0c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:48.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.863+0000 7ff7e8a44640 1 --2- 172.21.15.150:0/2582271751 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7e415e0d0 0x7ff7e4158c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:48.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.863+0000 7ff7e8a44640 1 --2- 172.21.15.150:0/2582271751 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7e40affc0 0x7ff7e4156f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:48.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.863+0000 7ff7e8a44640 1 --2- 172.21.15.150:0/2582271751 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7e406b8a0 0x7ff7e415bfc0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:48.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.863+0000 7ff7e8a44640 1 -- 172.21.15.150:0/2582271751 >> 172.21.15.150:0/2582271751 conn(0x7ff7e4099a70 msgr2=0x7ff7e4161c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:48.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.863+0000 7ff7e8a44640 1 -- 172.21.15.150:0/2582271751 shutdown_connections 2024-03-17T23:47:48.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.863+0000 7ff7e8a44640 1 -- 172.21.15.150:0/2582271751 wait complete. 2024-03-17T23:47:48.876 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:46:53.610678+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:46:54.227335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-17T23:46:57.949235+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-17T23:47:48.876 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:47:48.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.978+0000 7fd15ee63640 1 Processor -- start 2024-03-17T23:47:48.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.979+0000 7fd15ee63640 1 -- start start 2024-03-17T23:47:48.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.979+0000 7fd15ee63640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd158099e20 0x7fd15809a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:48.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.979+0000 7fd15ee63640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1580d1210 0x7fd158167ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:48.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.979+0000 7fd15ee63640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd15809a740 con 0x7fd15815e460 2024-03-17T23:47:48.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.980+0000 7fd15ee63640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fd15806ca90 con 0x7fd158099e20 2024-03-17T23:47:48.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.980+0000 7fd15ee63640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd15815e990 con 0x7fd1580d1210 2024-03-17T23:47:48.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.980+0000 7fd15cbff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd158099e20 0x7fd15809a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:48.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.980+0000 7fd157fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd15815e460 0x7fd15806c380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49802/0 (socket says 172.21.15.150:49802) 2024-03-17T23:47:48.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.980+0000 7fd15cbff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd158099e20 0x7fd15809a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50386/0 (socket says 172.21.15.150:50386) 2024-03-17T23:47:48.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.980+0000 7fd157fff640 1 -- 172.21.15.150:0/3170928818 learned_addr learned my addr 172.21.15.150:0/3170928818 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:48.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.980+0000 7fd15d400640 1 --2- 172.21.15.150:0/3170928818 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1580d1210 0x7fd158167ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:48.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.980+0000 7fd15cbff640 1 -- 172.21.15.150:0/3170928818 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1580d1210 msgr2=0x7fd158167ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:48.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.980+0000 7fd15cbff640 1 --2- 172.21.15.150:0/3170928818 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1580d1210 0x7fd158167ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:48.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.980+0000 7fd15cbff640 1 -- 172.21.15.150:0/3170928818 >> v1:172.21.15.150:6789/0 conn(0x7fd15815e460 legacy=0x7fd15806c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:48.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.980+0000 7fd15cbff640 1 -- 172.21.15.150:0/3170928818 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd15815ec70 con 0x7fd158099e20 2024-03-17T23:47:48.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.981+0000 7fd15cbff640 1 --2- 172.21.15.150:0/3170928818 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd158099e20 0x7fd15809a200 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd1480097a0 tx=0x7fd14802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eccaa96e2765ef5 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:48.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.981+0000 7fd1577fe640 1 -- 172.21.15.150:0/3170928818 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd148035020 con 0x7fd158099e20 2024-03-17T23:47:48.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.981+0000 7fd1577fe640 1 -- 172.21.15.150:0/3170928818 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd148031830 con 0x7fd158099e20 2024-03-17T23:47:48.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.981+0000 7fd1577fe640 1 -- 172.21.15.150:0/3170928818 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd148031c00 con 0x7fd158099e20 2024-03-17T23:47:48.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.981+0000 7fd15ee63640 1 -- 172.21.15.150:0/3170928818 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd158099e20 msgr2=0x7fd15809a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:48.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.981+0000 7fd15ee63640 1 --2- 172.21.15.150:0/3170928818 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd158099e20 0x7fd15809a200 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd1480097a0 tx=0x7fd14802b8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:48.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.981+0000 7fd15ee63640 1 -- 172.21.15.150:0/3170928818 shutdown_connections 2024-03-17T23:47:48.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.981+0000 7fd15ee63640 1 --2- 172.21.15.150:0/3170928818 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd1580d1210 0x7fd158167ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:48.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.981+0000 7fd15ee63640 1 --2- 172.21.15.150:0/3170928818 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd158099e20 0x7fd15809a200 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:48.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.981+0000 7fd15ee63640 1 -- 172.21.15.150:0/3170928818 >> 172.21.15.150:0/3170928818 conn(0x7fd1580b3630 msgr2=0x7fd1580b3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:48.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.982+0000 7fd15ee63640 1 -- 172.21.15.150:0/3170928818 shutdown_connections 2024-03-17T23:47:48.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.982+0000 7fd15ee63640 1 -- 172.21.15.150:0/3170928818 wait complete. 2024-03-17T23:47:48.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.983+0000 7fd15ee63640 1 Processor -- start 2024-03-17T23:47:48.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.983+0000 7fd15ee63640 1 -- start start 2024-03-17T23:47:48.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.983+0000 7fd15ee63640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd158099e20 0x7fd1580c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:48.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.983+0000 7fd15ee63640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1580d1210 0x7fd158150f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:48.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.983+0000 7fd15ee63640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd15815e460 0x7fd1580b4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:48.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.984+0000 7fd15ee63640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd1580cfcd0 con 0x7fd15815e460 2024-03-17T23:47:48.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.984+0000 7fd15ee63640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd15815f230 con 0x7fd1580d1210 2024-03-17T23:47:48.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.984+0000 7fd15ee63640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd15806d570 con 0x7fd158099e20 2024-03-17T23:47:48.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.984+0000 7fd15d400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd15815e460 0x7fd1580b4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:48.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.984+0000 7fd15d400640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd15815e460 0x7fd1580b4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50396/0 (socket says 172.21.15.150:50396) 2024-03-17T23:47:48.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.984+0000 7fd15d400640 1 -- 172.21.15.150:0/3298654113 learned_addr learned my addr 172.21.15.150:0/3298654113 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:48.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.984+0000 7fd15cbff640 1 --2- 172.21.15.150:0/3298654113 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd158099e20 0x7fd1580c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:48.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.984+0000 7fd157fff640 1 --2- 172.21.15.150:0/3298654113 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1580d1210 0x7fd158150f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:48.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.984+0000 7fd15d400640 1 -- 172.21.15.150:0/3298654113 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd158099e20 msgr2=0x7fd1580c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:48.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.984+0000 7fd15d400640 1 --2- 172.21.15.150:0/3298654113 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd158099e20 0x7fd1580c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:48.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.984+0000 7fd15d400640 1 -- 172.21.15.150:0/3298654113 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1580d1210 msgr2=0x7fd158150f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:48.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.984+0000 7fd15d400640 1 --2- 172.21.15.150:0/3298654113 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1580d1210 0x7fd158150f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:48.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.984+0000 7fd15d400640 1 -- 172.21.15.150:0/3298654113 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd158151720 con 0x7fd15815e460 2024-03-17T23:47:48.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.984+0000 7fd15d400640 1 --2- 172.21.15.150:0/3298654113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd15815e460 0x7fd1580b4990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd150009cb0 tx=0x7fd15000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:48.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.984+0000 7fd1557fa640 1 -- 172.21.15.150:0/3298654113 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd150018020 con 0x7fd15815e460 2024-03-17T23:47:48.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.984+0000 7fd1557fa640 1 -- 172.21.15.150:0/3298654113 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd150014480 con 0x7fd15815e460 2024-03-17T23:47:48.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.984+0000 7fd15ee63640 1 -- 172.21.15.150:0/3298654113 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd158155db0 con 0x7fd15815e460 2024-03-17T23:47:48.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.984+0000 7fd1557fa640 1 -- 172.21.15.150:0/3298654113 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd150014870 con 0x7fd15815e460 2024-03-17T23:47:48.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.984+0000 7fd15ee63640 1 -- 172.21.15.150:0/3298654113 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1580c92a0 con 0x7fd15815e460 2024-03-17T23:47:48.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.985+0000 7fd15ee63640 1 -- 172.21.15.150:0/3298654113 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1580681a0 con 0x7fd15815e460 2024-03-17T23:47:48.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.986+0000 7fd1557fa640 1 -- 172.21.15.150:0/3298654113 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd150005d10 con 0x7fd15815e460 2024-03-17T23:47:48.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.986+0000 7fd1557fa640 1 --2- 172.21.15.150:0/3298654113 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd14406bbd0 0x7fd14406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:48.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.986+0000 7fd1557fa640 1 -- 172.21.15.150:0/3298654113 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fd15008ba40 con 0x7fd15815e460 2024-03-17T23:47:48.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.986+0000 7fd15cbff640 1 --2- 172.21.15.150:0/3298654113 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd14406bbd0 0x7fd14406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:48.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.987+0000 7fd15cbff640 1 --2- 172.21.15.150:0/3298654113 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd14406bbd0 0x7fd14406e090 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd148036000 tx=0x7fd14803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:48.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:48.990+0000 7fd1557fa640 1 -- 172.21.15.150:0/3298654113 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd150022030 con 0x7fd15815e460 2024-03-17T23:47:49.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.161+0000 7fd15ee63640 1 -- 172.21.15.150:0/3298654113 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd1580aef60 con 0x7fd15815e460 2024-03-17T23:47:49.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.161+0000 7fd1557fa640 1 -- 172.21.15.150:0/3298654113 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7fd15008be20 con 0x7fd15815e460 2024-03-17T23:47:49.163 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:47:49.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.163+0000 7fd15ee63640 1 -- 172.21.15.150:0/3298654113 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd14406bbd0 msgr2=0x7fd14406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:49.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.163+0000 7fd15ee63640 1 --2- 172.21.15.150:0/3298654113 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd14406bbd0 0x7fd14406e090 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd148036000 tx=0x7fd14803c040 comp rx=0 tx=0).stop 2024-03-17T23:47:49.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.163+0000 7fd15ee63640 1 -- 172.21.15.150:0/3298654113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd15815e460 msgr2=0x7fd1580b4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:49.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.163+0000 7fd15ee63640 1 --2- 172.21.15.150:0/3298654113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd15815e460 0x7fd1580b4990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd150009cb0 tx=0x7fd15000c490 comp rx=0 tx=0).stop 2024-03-17T23:47:49.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.163+0000 7fd15ee63640 1 -- 172.21.15.150:0/3298654113 shutdown_connections 2024-03-17T23:47:49.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.163+0000 7fd15ee63640 1 --2- 172.21.15.150:0/3298654113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd15815e460 0x7fd1580b4990 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:49.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.163+0000 7fd15ee63640 1 --2- 172.21.15.150:0/3298654113 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd14406bbd0 0x7fd14406e090 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:49.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.163+0000 7fd15ee63640 1 --2- 172.21.15.150:0/3298654113 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1580d1210 0x7fd158150f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:49.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.163+0000 7fd15ee63640 1 --2- 172.21.15.150:0/3298654113 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd158099e20 0x7fd1580c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:49.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.163+0000 7fd15ee63640 1 -- 172.21.15.150:0/3298654113 >> 172.21.15.150:0/3298654113 conn(0x7fd1580b3630 msgr2=0x7fd15815f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:49.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.163+0000 7fd15ee63640 1 -- 172.21.15.150:0/3298654113 shutdown_connections 2024-03-17T23:47:49.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.163+0000 7fd15ee63640 1 -- 172.21.15.150:0/3298654113 wait complete. 2024-03-17T23:47:49.176 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:47:49.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.278+0000 7f6798910640 1 Processor -- start 2024-03-17T23:47:49.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.278+0000 7f6798910640 1 -- start start 2024-03-17T23:47:49.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.279+0000 7f6798910640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6794155320 0x7f679415bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:49.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.279+0000 7f6798910640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f67940cd420 0x7f679415ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:49.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.279+0000 7f6798910640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67940cf3b0 con 0x7f679406dd30 2024-03-17T23:47:49.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.279+0000 7f6798910640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f679415c550 con 0x7f67940cd420 2024-03-17T23:47:49.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.279+0000 7f6798910640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f67940cf690 con 0x7f6794155320 2024-03-17T23:47:49.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.279+0000 7f6792d9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f67940cd420 0x7f679415ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:49.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.279+0000 7f6791d9b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f679406dd30 0x7f67940ccd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49816/0 (socket says 172.21.15.150:49816) 2024-03-17T23:47:49.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.279+0000 7f6792d9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f67940cd420 0x7f679415ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50410/0 (socket says 172.21.15.150:50410) 2024-03-17T23:47:49.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.279+0000 7f6791d9b640 1 -- 172.21.15.150:0/3649131409 learned_addr learned my addr 172.21.15.150:0/3649131409 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:49.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.279+0000 7f679259c640 1 --2- 172.21.15.150:0/3649131409 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6794155320 0x7f679415bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:49.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.279+0000 7f6792d9d640 1 -- 172.21.15.150:0/3649131409 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6794155320 msgr2=0x7f679415bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:49.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.279+0000 7f6792d9d640 1 --2- 172.21.15.150:0/3649131409 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6794155320 0x7f679415bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:49.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.279+0000 7f6792d9d640 1 -- 172.21.15.150:0/3649131409 >> v1:172.21.15.150:6789/0 conn(0x7f679406dd30 legacy=0x7f67940ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:49.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.279+0000 7f6792d9d640 1 -- 172.21.15.150:0/3649131409 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f679415c7a0 con 0x7f67940cd420 2024-03-17T23:47:49.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.279+0000 7f679159a640 1 -- 172.21.15.150:0/3649131409 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2906473839 0 0) 0x7f67940cf3b0 con 0x7f679406dd30 2024-03-17T23:47:49.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.280+0000 7f6792d9d640 1 --2- 172.21.15.150:0/3649131409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67940cd420 0x7f679415ccc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f678400bfa0 tx=0x7f678402e0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87a491266bdaff4f server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:49.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.280+0000 7f679159a640 1 -- 172.21.15.150:0/3649131409 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f678402ea40 con 0x7f67940cd420 2024-03-17T23:47:49.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.280+0000 7f679159a640 1 -- 172.21.15.150:0/3649131409 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f678402ebe0 con 0x7f67940cd420 2024-03-17T23:47:49.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.280+0000 7f679159a640 1 -- 172.21.15.150:0/3649131409 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6784009540 con 0x7f67940cd420 2024-03-17T23:47:49.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.281+0000 7f6798910640 1 -- 172.21.15.150:0/3649131409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67940cd420 msgr2=0x7f679415ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:49.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.281+0000 7f6798910640 1 --2- 172.21.15.150:0/3649131409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67940cd420 0x7f679415ccc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f678400bfa0 tx=0x7f678402e0a0 comp rx=0 tx=0).stop 2024-03-17T23:47:49.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.281+0000 7f6798910640 1 -- 172.21.15.150:0/3649131409 shutdown_connections 2024-03-17T23:47:49.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.281+0000 7f6798910640 1 --2- 172.21.15.150:0/3649131409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67940cd420 0x7f679415ccc0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:49.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.281+0000 7f6798910640 1 --2- 172.21.15.150:0/3649131409 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6794155320 0x7f679415bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:49.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.281+0000 7f6798910640 1 -- 172.21.15.150:0/3649131409 >> 172.21.15.150:0/3649131409 conn(0x7f679406b190 msgr2=0x7f679406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:49.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.281+0000 7f6798910640 1 -- 172.21.15.150:0/3649131409 shutdown_connections 2024-03-17T23:47:49.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.281+0000 7f6798910640 1 -- 172.21.15.150:0/3649131409 wait complete. 2024-03-17T23:47:49.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.282+0000 7f6798910640 1 Processor -- start 2024-03-17T23:47:49.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.282+0000 7f6798910640 1 -- start start 2024-03-17T23:47:49.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.283+0000 7f6798910640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f679406dd30 0x7f67940b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:49.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.283+0000 7f6798910640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67940cd420 0x7f67940b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:49.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.283+0000 7f6798910640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6794155320 0x7f679415dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:49.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.283+0000 7f6798910640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6794155700 con 0x7f679406dd30 2024-03-17T23:47:49.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.283+0000 7f6798910640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f679406e190 con 0x7f67940cd420 2024-03-17T23:47:49.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.283+0000 7f6798910640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6794155880 con 0x7f6794155320 2024-03-17T23:47:49.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.283+0000 7f679259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f679406dd30 0x7f67940b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:49.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.283+0000 7f679259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f679406dd30 0x7f67940b3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50420/0 (socket says 172.21.15.150:50420) 2024-03-17T23:47:49.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.283+0000 7f679259c640 1 -- 172.21.15.150:0/1589617484 learned_addr learned my addr 172.21.15.150:0/1589617484 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:49.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.283+0000 7f6791d9b640 1 --2- 172.21.15.150:0/1589617484 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67940cd420 0x7f67940b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:49.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.283+0000 7f6792d9d640 1 --2- 172.21.15.150:0/1589617484 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6794155320 0x7f679415dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:49.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.283+0000 7f679259c640 1 -- 172.21.15.150:0/1589617484 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6794155320 msgr2=0x7f679415dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:49.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.283+0000 7f679259c640 1 --2- 172.21.15.150:0/1589617484 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6794155320 0x7f679415dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:49.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.283+0000 7f679259c640 1 -- 172.21.15.150:0/1589617484 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67940cd420 msgr2=0x7f67940b41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:49.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.283+0000 7f679259c640 1 --2- 172.21.15.150:0/1589617484 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67940cd420 0x7f67940b41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:49.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.283+0000 7f679259c640 1 -- 172.21.15.150:0/1589617484 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67941459a0 con 0x7f679406dd30 2024-03-17T23:47:49.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.284+0000 7f679259c640 1 --2- 172.21.15.150:0/1589617484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f679406dd30 0x7f67940b3ca0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f678c00e870 tx=0x7f678c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:49.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.284+0000 7f6782ffd640 1 -- 172.21.15.150:0/1589617484 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f678c017070 con 0x7f679406dd30 2024-03-17T23:47:49.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.284+0000 7f6782ffd640 1 -- 172.21.15.150:0/1589617484 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f678c00c940 con 0x7f679406dd30 2024-03-17T23:47:49.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.284+0000 7f6798910640 1 -- 172.21.15.150:0/1589617484 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67940c8fe0 con 0x7f679406dd30 2024-03-17T23:47:49.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.284+0000 7f6782ffd640 1 -- 172.21.15.150:0/1589617484 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f678c00cd30 con 0x7f679406dd30 2024-03-17T23:47:49.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.284+0000 7f6798910640 1 -- 172.21.15.150:0/1589617484 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67940c95a0 con 0x7f679406dd30 2024-03-17T23:47:49.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.285+0000 7f6798910640 1 -- 172.21.15.150:0/1589617484 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67940681a0 con 0x7f679406dd30 2024-03-17T23:47:49.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.285+0000 7f6782ffd640 1 -- 172.21.15.150:0/1589617484 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f678c01a070 con 0x7f679406dd30 2024-03-17T23:47:49.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.285+0000 7f6782ffd640 1 --2- 172.21.15.150:0/1589617484 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f677406bc00 0x7f677406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:49.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.286+0000 7f6791d9b640 1 --2- 172.21.15.150:0/1589617484 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f677406bc00 0x7f677406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:49.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.286+0000 7f6782ffd640 1 -- 172.21.15.150:0/1589617484 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f678c08abc0 con 0x7f679406dd30 2024-03-17T23:47:49.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.286+0000 7f6791d9b640 1 --2- 172.21.15.150:0/1589617484 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f677406bc00 0x7f677406e0c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6788007f70 tx=0x7f6788007f00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:49.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.290+0000 7f6782ffd640 1 -- 172.21.15.150:0/1589617484 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f678c058cf0 con 0x7f679406dd30 2024-03-17T23:47:49.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.460+0000 7f6798910640 1 -- 172.21.15.150:0/1589617484 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6794153f60 con 0x7f679406dd30 2024-03-17T23:47:49.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.461+0000 7f6782ffd640 1 -- 172.21.15.150:0/1589617484 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7f678c05cd90 con 0x7f679406dd30 2024-03-17T23:47:49.463 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:49.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.463+0000 7f6798910640 1 -- 172.21.15.150:0/1589617484 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f677406bc00 msgr2=0x7f677406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:49.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.463+0000 7f6798910640 1 --2- 172.21.15.150:0/1589617484 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f677406bc00 0x7f677406e0c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6788007f70 tx=0x7f6788007f00 comp rx=0 tx=0).stop 2024-03-17T23:47:49.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.463+0000 7f6798910640 1 -- 172.21.15.150:0/1589617484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f679406dd30 msgr2=0x7f67940b3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:49.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.463+0000 7f6798910640 1 --2- 172.21.15.150:0/1589617484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f679406dd30 0x7f67940b3ca0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f678c00e870 tx=0x7f678c00ed30 comp rx=0 tx=0).stop 2024-03-17T23:47:49.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.463+0000 7f6798910640 1 -- 172.21.15.150:0/1589617484 shutdown_connections 2024-03-17T23:47:49.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.463+0000 7f6798910640 1 --2- 172.21.15.150:0/1589617484 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6794155320 0x7f679415dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:49.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.463+0000 7f6798910640 1 --2- 172.21.15.150:0/1589617484 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f677406bc00 0x7f677406e0c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:49.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.463+0000 7f6798910640 1 --2- 172.21.15.150:0/1589617484 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67940cd420 0x7f67940b41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:49.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.463+0000 7f6798910640 1 --2- 172.21.15.150:0/1589617484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f679406dd30 0x7f67940b3ca0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:49.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.463+0000 7f6798910640 1 -- 172.21.15.150:0/1589617484 >> 172.21.15.150:0/1589617484 conn(0x7f679406b190 msgr2=0x7f679406f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:49.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.463+0000 7f6798910640 1 -- 172.21.15.150:0/1589617484 shutdown_connections 2024-03-17T23:47:49.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.463+0000 7f6798910640 1 -- 172.21.15.150:0/1589617484 wait complete. 2024-03-17T23:47:49.476 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:47:49.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.577+0000 7f5e4e48c640 1 Processor -- start 2024-03-17T23:47:49.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.577+0000 7f5e4e48c640 1 -- start start 2024-03-17T23:47:49.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.578+0000 7f5e4e48c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5e480cd420 0x7f5e48165d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:49.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.578+0000 7f5e4e48c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e480cf3b0 con 0x7f5e48155320 2024-03-17T23:47:49.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.578+0000 7f5e4e48c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e480cf690 con 0x7f5e4815bfd0 2024-03-17T23:47:49.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.578+0000 7f5e4e48c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f5e4815c550 con 0x7f5e480cd420 2024-03-17T23:47:49.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.578+0000 7f5e47fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f5e48155320 0x7f5e4806dd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49818/0 (socket says 172.21.15.150:49818) 2024-03-17T23:47:49.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.578+0000 7f5e47fff640 1 -- 172.21.15.150:0/3185424775 learned_addr learned my addr 172.21.15.150:0/3185424775 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:49.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.578+0000 7f5e4ca29640 1 --2- 172.21.15.150:0/3185424775 >> v2:172.21.15.157:3300/0 conn(0x7f5e480cd420 0x7f5e48165d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:49.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.578+0000 7f5e46ffd640 1 -- 172.21.15.150:0/3185424775 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2605450839 0 0) 0x7f5e480cf3b0 con 0x7f5e48155320 2024-03-17T23:47:49.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.578+0000 7f5e46ffd640 1 -- 172.21.15.150:0/3185424775 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e30003520 con 0x7f5e48155320 2024-03-17T23:47:49.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.578+0000 7f5e46ffd640 1 -- 172.21.15.150:0/3185424775 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 412298387 0 0) 0x7f5e480cf690 con 0x7f5e4815bfd0 2024-03-17T23:47:49.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.578+0000 7f5e46ffd640 1 -- 172.21.15.150:0/3185424775 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e480cf3b0 con 0x7f5e4815bfd0 2024-03-17T23:47:49.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.578+0000 7f5e4ca29640 1 -- 172.21.15.150:0/3185424775 >> v1:172.21.15.157:6789/0 conn(0x7f5e4815bfd0 legacy=0x7f5e480ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:49.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.578+0000 7f5e4ca29640 1 -- 172.21.15.150:0/3185424775 >> v1:172.21.15.150:6789/0 conn(0x7f5e48155320 legacy=0x7f5e4806dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:49.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.578+0000 7f5e4ca29640 1 -- 172.21.15.150:0/3185424775 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e4815c7a0 con 0x7f5e480cd420 2024-03-17T23:47:49.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.579+0000 7f5e4ca29640 1 --2- 172.21.15.150:0/3185424775 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e480cd420 0x7f5e48165d10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5e40009d90 tx=0x7f5e4002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f76065a1d5e97ff server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:49.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.579+0000 7f5e46ffd640 1 -- 172.21.15.150:0/3185424775 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e40039070 con 0x7f5e480cd420 2024-03-17T23:47:49.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.579+0000 7f5e46ffd640 1 -- 172.21.15.150:0/3185424775 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e4002cd00 con 0x7f5e480cd420 2024-03-17T23:47:49.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.579+0000 7f5e46ffd640 1 -- 172.21.15.150:0/3185424775 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e40034660 con 0x7f5e480cd420 2024-03-17T23:47:49.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.580+0000 7f5e4e48c640 1 -- 172.21.15.150:0/3185424775 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e480cd420 msgr2=0x7f5e48165d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:49.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.580+0000 7f5e4e48c640 1 --2- 172.21.15.150:0/3185424775 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e480cd420 0x7f5e48165d10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5e40009d90 tx=0x7f5e4002bf50 comp rx=0 tx=0).stop 2024-03-17T23:47:49.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.580+0000 7f5e4e48c640 1 -- 172.21.15.150:0/3185424775 shutdown_connections 2024-03-17T23:47:49.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.580+0000 7f5e4e48c640 1 --2- 172.21.15.150:0/3185424775 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e480cd420 0x7f5e48165d10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:49.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.580+0000 7f5e4e48c640 1 -- 172.21.15.150:0/3185424775 >> 172.21.15.150:0/3185424775 conn(0x7f5e4806b190 msgr2=0x7f5e4806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:49.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.580+0000 7f5e4e48c640 1 -- 172.21.15.150:0/3185424775 shutdown_connections 2024-03-17T23:47:49.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.580+0000 7f5e4e48c640 1 -- 172.21.15.150:0/3185424775 wait complete. 2024-03-17T23:47:49.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.581+0000 7f5e4e48c640 1 Processor -- start 2024-03-17T23:47:49.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.581+0000 7f5e4e48c640 1 -- start start 2024-03-17T23:47:49.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.582+0000 7f5e4e48c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e480cd420 0x7f5e48142420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:49.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.582+0000 7f5e4e48c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e48155320 0x7f5e48071cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:49.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.582+0000 7f5e4e48c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e4815bfd0 0x7f5e48072210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:49.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.582+0000 7f5e4e48c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5e48155880 con 0x7f5e4815bfd0 2024-03-17T23:47:49.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.582+0000 7f5e4e48c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5e48155700 con 0x7f5e480cd420 2024-03-17T23:47:49.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.582+0000 7f5e4e48c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5e4815dcd0 con 0x7f5e48155320 2024-03-17T23:47:49.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.582+0000 7f5e4ca29640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e4815bfd0 0x7f5e48072210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:49.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.582+0000 7f5e47fff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e480cd420 0x7f5e48142420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:49.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.582+0000 7f5e4ca29640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e4815bfd0 0x7f5e48072210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50428/0 (socket says 172.21.15.150:50428) 2024-03-17T23:47:49.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.582+0000 7f5e4ca29640 1 -- 172.21.15.150:0/279726243 learned_addr learned my addr 172.21.15.150:0/279726243 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:49.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.582+0000 7f5e477fe640 1 --2- 172.21.15.150:0/279726243 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e48155320 0x7f5e48071cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:49.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.582+0000 7f5e4ca29640 1 -- 172.21.15.150:0/279726243 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e48155320 msgr2=0x7f5e48071cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:49.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.582+0000 7f5e4ca29640 1 --2- 172.21.15.150:0/279726243 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e48155320 0x7f5e48071cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:49.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.582+0000 7f5e4ca29640 1 -- 172.21.15.150:0/279726243 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e480cd420 msgr2=0x7f5e48142420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:49.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.582+0000 7f5e4ca29640 1 --2- 172.21.15.150:0/279726243 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e480cd420 0x7f5e48142420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:49.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.582+0000 7f5e4ca29640 1 -- 172.21.15.150:0/279726243 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e481587b0 con 0x7f5e4815bfd0 2024-03-17T23:47:49.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.582+0000 7f5e47fff640 1 --2- 172.21.15.150:0/279726243 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e480cd420 0x7f5e48142420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:49.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.582+0000 7f5e477fe640 1 --2- 172.21.15.150:0/279726243 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e48155320 0x7f5e48071cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:49.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.582+0000 7f5e4ca29640 1 --2- 172.21.15.150:0/279726243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e4815bfd0 0x7f5e48072210 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5e40009eb0 tx=0x7f5e40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:49.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.583+0000 7f5e44ff9640 1 -- 172.21.15.150:0/279726243 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e40039050 con 0x7f5e4815bfd0 2024-03-17T23:47:49.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.583+0000 7f5e44ff9640 1 -- 172.21.15.150:0/279726243 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e40045070 con 0x7f5e4815bfd0 2024-03-17T23:47:49.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.583+0000 7f5e4e48c640 1 -- 172.21.15.150:0/279726243 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e480c9880 con 0x7f5e4815bfd0 2024-03-17T23:47:49.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.583+0000 7f5e44ff9640 1 -- 172.21.15.150:0/279726243 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e40009450 con 0x7f5e4815bfd0 2024-03-17T23:47:49.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.583+0000 7f5e4e48c640 1 -- 172.21.15.150:0/279726243 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e481579f0 con 0x7f5e4815bfd0 2024-03-17T23:47:49.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.584+0000 7f5e4e48c640 1 -- 172.21.15.150:0/279726243 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e480681a0 con 0x7f5e4815bfd0 2024-03-17T23:47:49.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.584+0000 7f5e44ff9640 1 -- 172.21.15.150:0/279726243 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5e4003e850 con 0x7f5e4815bfd0 2024-03-17T23:47:49.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.584+0000 7f5e44ff9640 1 --2- 172.21.15.150:0/279726243 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5e3406bbd0 0x7f5e3406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:49.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.584+0000 7f5e44ff9640 1 -- 172.21.15.150:0/279726243 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f5e400adbf0 con 0x7f5e4815bfd0 2024-03-17T23:47:49.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.584+0000 7f5e47fff640 1 --2- 172.21.15.150:0/279726243 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5e3406bbd0 0x7f5e3406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:49.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.585+0000 7f5e47fff640 1 --2- 172.21.15.150:0/279726243 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5e3406bbd0 0x7f5e3406e090 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c000f60 tx=0x7f5e3c007d70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:49.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.589+0000 7f5e44ff9640 1 -- 172.21.15.150:0/279726243 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5e4007bdb0 con 0x7f5e4815bfd0 2024-03-17T23:47:49.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.762+0000 7f5e4e48c640 1 -- 172.21.15.150:0/279726243 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5e480ce3c0 con 0x7f5e4815bfd0 2024-03-17T23:47:49.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.762+0000 7f5e44ff9640 1 -- 172.21.15.150:0/279726243 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7f5e4007fe50 con 0x7f5e4815bfd0 2024-03-17T23:47:49.764 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:47:49.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.764+0000 7f5e4e48c640 1 -- 172.21.15.150:0/279726243 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5e3406bbd0 msgr2=0x7f5e3406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:49.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.764+0000 7f5e4e48c640 1 --2- 172.21.15.150:0/279726243 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5e3406bbd0 0x7f5e3406e090 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c000f60 tx=0x7f5e3c007d70 comp rx=0 tx=0).stop 2024-03-17T23:47:49.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.764+0000 7f5e4e48c640 1 -- 172.21.15.150:0/279726243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e4815bfd0 msgr2=0x7f5e48072210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:49.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.764+0000 7f5e4e48c640 1 --2- 172.21.15.150:0/279726243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e4815bfd0 0x7f5e48072210 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5e40009eb0 tx=0x7f5e40004a40 comp rx=0 tx=0).stop 2024-03-17T23:47:49.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.764+0000 7f5e4e48c640 1 -- 172.21.15.150:0/279726243 shutdown_connections 2024-03-17T23:47:49.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.764+0000 7f5e4e48c640 1 --2- 172.21.15.150:0/279726243 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e4815bfd0 0x7f5e48072210 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:49.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.764+0000 7f5e4e48c640 1 --2- 172.21.15.150:0/279726243 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e48155320 0x7f5e48071cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:49.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.764+0000 7f5e4e48c640 1 --2- 172.21.15.150:0/279726243 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5e3406bbd0 0x7f5e3406e090 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:49.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.764+0000 7f5e4e48c640 1 --2- 172.21.15.150:0/279726243 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e480cd420 0x7f5e48142420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:49.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.764+0000 7f5e4e48c640 1 -- 172.21.15.150:0/279726243 >> 172.21.15.150:0/279726243 conn(0x7f5e4806b190 msgr2=0x7f5e48153310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:49.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.764+0000 7f5e4e48c640 1 -- 172.21.15.150:0/279726243 shutdown_connections 2024-03-17T23:47:49.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.764+0000 7f5e4e48c640 1 -- 172.21.15.150:0/279726243 wait complete. 2024-03-17T23:47:49.777 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-17T23:47:49.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.878+0000 7f7e63990640 1 Processor -- start 2024-03-17T23:47:49.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.878+0000 7f7e63990640 1 -- start start 2024-03-17T23:47:49.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.879+0000 7f7e63990640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7e5c0d0ce0 0x7f7e5c0b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:49.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.879+0000 7f7e63990640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e5c161d30 con 0x7f7e5c0b2bf0 2024-03-17T23:47:49.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.879+0000 7f7e63990640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e5c162010 con 0x7f7e5c161520 2024-03-17T23:47:49.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.879+0000 7f7e63990640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f7e5c0b9bf0 con 0x7f7e5c0d0ce0 2024-03-17T23:47:49.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.879+0000 7f7e61f2d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f7e5c0b2bf0 0x7f7e5c0d1fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49824/0 (socket says 172.21.15.150:49824) 2024-03-17T23:47:49.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.879+0000 7f7e61f2d640 1 -- 172.21.15.150:0/3914869350 learned_addr learned my addr 172.21.15.150:0/3914869350 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:49.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.879+0000 7f7e60f2b640 1 --2- 172.21.15.150:0/3914869350 >> v2:172.21.15.157:3300/0 conn(0x7f7e5c0d0ce0 0x7f7e5c0b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:49.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.880+0000 7f7e53fff640 1 -- 172.21.15.150:0/3914869350 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 184997555 0 0) 0x7f7e5c161d30 con 0x7f7e5c0b2bf0 2024-03-17T23:47:49.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.880+0000 7f7e53fff640 1 -- 172.21.15.150:0/3914869350 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e48003520 con 0x7f7e5c0b2bf0 2024-03-17T23:47:49.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.880+0000 7f7e53fff640 1 -- 172.21.15.150:0/3914869350 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 712425170 0 0) 0x7f7e5c162010 con 0x7f7e5c161520 2024-03-17T23:47:49.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.880+0000 7f7e53fff640 1 -- 172.21.15.150:0/3914869350 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e5c161d30 con 0x7f7e5c161520 2024-03-17T23:47:49.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.880+0000 7f7e60f2b640 1 -- 172.21.15.150:0/3914869350 >> v1:172.21.15.157:6789/0 conn(0x7f7e5c161520 legacy=0x7f7e5c0d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:49.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.880+0000 7f7e60f2b640 1 -- 172.21.15.150:0/3914869350 >> v1:172.21.15.150:6789/0 conn(0x7f7e5c0b2bf0 legacy=0x7f7e5c0d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:49.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.880+0000 7f7e60f2b640 1 -- 172.21.15.150:0/3914869350 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e5c1622f0 con 0x7f7e5c0d0ce0 2024-03-17T23:47:49.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.880+0000 7f7e60f2b640 1 --2- 172.21.15.150:0/3914869350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7e5c0d0ce0 0x7f7e5c0b96b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c0097a0 tx=0x7f7e4c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5651e6bedcf7a99 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:49.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.881+0000 7f7e53fff640 1 -- 172.21.15.150:0/3914869350 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c0315a0 con 0x7f7e5c0d0ce0 2024-03-17T23:47:49.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.881+0000 7f7e53fff640 1 -- 172.21.15.150:0/3914869350 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e4c037070 con 0x7f7e5c0d0ce0 2024-03-17T23:47:49.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.881+0000 7f7e53fff640 1 -- 172.21.15.150:0/3914869350 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c031940 con 0x7f7e5c0d0ce0 2024-03-17T23:47:49.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.881+0000 7f7e63990640 1 -- 172.21.15.150:0/3914869350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7e5c0d0ce0 msgr2=0x7f7e5c0b96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:49.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.881+0000 7f7e63990640 1 --2- 172.21.15.150:0/3914869350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7e5c0d0ce0 0x7f7e5c0b96b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c0097a0 tx=0x7f7e4c02b840 comp rx=0 tx=0).stop 2024-03-17T23:47:49.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.881+0000 7f7e63990640 1 -- 172.21.15.150:0/3914869350 shutdown_connections 2024-03-17T23:47:49.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.881+0000 7f7e63990640 1 --2- 172.21.15.150:0/3914869350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7e5c0d0ce0 0x7f7e5c0b96b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:49.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.881+0000 7f7e63990640 1 -- 172.21.15.150:0/3914869350 >> 172.21.15.150:0/3914869350 conn(0x7f7e5c06ca00 msgr2=0x7f7e5c06ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:49.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.881+0000 7f7e63990640 1 -- 172.21.15.150:0/3914869350 shutdown_connections 2024-03-17T23:47:49.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.881+0000 7f7e63990640 1 -- 172.21.15.150:0/3914869350 wait complete. 2024-03-17T23:47:49.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.882+0000 7f7e63990640 1 Processor -- start 2024-03-17T23:47:49.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.882+0000 7f7e63990640 1 -- start start 2024-03-17T23:47:49.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.883+0000 7f7e63990640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7e5c0b2bf0 0x7f7e5c0b4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:49.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.883+0000 7f7e63990640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e5c0d0ce0 0x7f7e5c0cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:49.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.883+0000 7f7e63990640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7e5c161520 0x7f7e5c15ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:49.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.883+0000 7f7e63990640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7e5c161900 con 0x7f7e5c0d0ce0 2024-03-17T23:47:49.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.883+0000 7f7e63990640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7e5c162940 con 0x7f7e5c161520 2024-03-17T23:47:49.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.883+0000 7f7e63990640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7e5c0d26b0 con 0x7f7e5c0b2bf0 2024-03-17T23:47:49.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.883+0000 7f7e60f2b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e5c0d0ce0 0x7f7e5c0cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:49.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.883+0000 7f7e60f2b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e5c0d0ce0 0x7f7e5c0cc4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50440/0 (socket says 172.21.15.150:50440) 2024-03-17T23:47:49.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.883+0000 7f7e60f2b640 1 -- 172.21.15.150:0/2676949532 learned_addr learned my addr 172.21.15.150:0/2676949532 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:49.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.883+0000 7f7e6172c640 1 --2- 172.21.15.150:0/2676949532 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7e5c0b2bf0 0x7f7e5c0b4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:49.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.883+0000 7f7e61f2d640 1 --2- 172.21.15.150:0/2676949532 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7e5c161520 0x7f7e5c15ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:49.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.884+0000 7f7e60f2b640 1 -- 172.21.15.150:0/2676949532 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7e5c0b2bf0 msgr2=0x7f7e5c0b4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:49.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.884+0000 7f7e60f2b640 1 --2- 172.21.15.150:0/2676949532 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7e5c0b2bf0 0x7f7e5c0b4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:49.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.884+0000 7f7e60f2b640 1 -- 172.21.15.150:0/2676949532 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7e5c161520 msgr2=0x7f7e5c15ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:49.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.884+0000 7f7e60f2b640 1 --2- 172.21.15.150:0/2676949532 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7e5c161520 0x7f7e5c15ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:49.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.884+0000 7f7e60f2b640 1 -- 172.21.15.150:0/2676949532 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e5c0b38d0 con 0x7f7e5c0d0ce0 2024-03-17T23:47:49.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.884+0000 7f7e6172c640 1 --2- 172.21.15.150:0/2676949532 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7e5c0b2bf0 0x7f7e5c0b4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:49.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.884+0000 7f7e60f2b640 1 --2- 172.21.15.150:0/2676949532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e5c0d0ce0 0x7f7e5c0cc4f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c000c00 tx=0x7f7e4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:49.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.884+0000 7f7e51ffb640 1 -- 172.21.15.150:0/2676949532 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c032630 con 0x7f7e5c0d0ce0 2024-03-17T23:47:49.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.884+0000 7f7e51ffb640 1 -- 172.21.15.150:0/2676949532 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e4c0327d0 con 0x7f7e5c0d0ce0 2024-03-17T23:47:49.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.884+0000 7f7e51ffb640 1 -- 172.21.15.150:0/2676949532 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c032bc0 con 0x7f7e5c0d0ce0 2024-03-17T23:47:49.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.884+0000 7f7e63990640 1 -- 172.21.15.150:0/2676949532 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e5c0b3bb0 con 0x7f7e5c0d0ce0 2024-03-17T23:47:49.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.884+0000 7f7e63990640 1 -- 172.21.15.150:0/2676949532 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e5c0b4170 con 0x7f7e5c0d0ce0 2024-03-17T23:47:49.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.885+0000 7f7e63990640 1 -- 172.21.15.150:0/2676949532 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e5c0681a0 con 0x7f7e5c0d0ce0 2024-03-17T23:47:49.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.886+0000 7f7e51ffb640 1 -- 172.21.15.150:0/2676949532 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7e4c03b3f0 con 0x7f7e5c0d0ce0 2024-03-17T23:47:49.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.886+0000 7f7e51ffb640 1 --2- 172.21.15.150:0/2676949532 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7e4006bbd0 0x7f7e4006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:49.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.886+0000 7f7e51ffb640 1 -- 172.21.15.150:0/2676949532 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f7e4c02f030 con 0x7f7e5c0d0ce0 2024-03-17T23:47:49.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.886+0000 7f7e6172c640 1 --2- 172.21.15.150:0/2676949532 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7e4006bbd0 0x7f7e4006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:49.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.886+0000 7f7e6172c640 1 --2- 172.21.15.150:0/2676949532 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7e4006bbd0 0x7f7e4006e090 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7e58007fd0 tx=0x7f7e58009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:49.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:49.890+0000 7f7e51ffb640 1 -- 172.21.15.150:0/2676949532 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7e4c040070 con 0x7f7e5c0d0ce0 2024-03-17T23:47:50.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.060+0000 7f7e63990640 1 -- 172.21.15.150:0/2676949532 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f7e5c0d19c0 con 0x7f7e5c0d0ce0 2024-03-17T23:47:50.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.061+0000 7f7e51ffb640 1 -- 172.21.15.150:0/2676949532 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7f7e4c07ab30 con 0x7f7e5c0d0ce0 2024-03-17T23:47:50.062 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:50.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.063+0000 7f7e63990640 1 -- 172.21.15.150:0/2676949532 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7e4006bbd0 msgr2=0x7f7e4006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:50.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.063+0000 7f7e63990640 1 --2- 172.21.15.150:0/2676949532 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7e4006bbd0 0x7f7e4006e090 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7e58007fd0 tx=0x7f7e58009040 comp rx=0 tx=0).stop 2024-03-17T23:47:50.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.063+0000 7f7e63990640 1 -- 172.21.15.150:0/2676949532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e5c0d0ce0 msgr2=0x7f7e5c0cc4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:50.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.063+0000 7f7e63990640 1 --2- 172.21.15.150:0/2676949532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e5c0d0ce0 0x7f7e5c0cc4f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c000c00 tx=0x7f7e4c0046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.063+0000 7f7e63990640 1 -- 172.21.15.150:0/2676949532 shutdown_connections 2024-03-17T23:47:50.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.063+0000 7f7e63990640 1 --2- 172.21.15.150:0/2676949532 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7e4006bbd0 0x7f7e4006e090 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.063+0000 7f7e63990640 1 --2- 172.21.15.150:0/2676949532 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7e5c161520 0x7f7e5c15ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.063+0000 7f7e63990640 1 --2- 172.21.15.150:0/2676949532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7e5c0d0ce0 0x7f7e5c0cc4f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.063+0000 7f7e63990640 1 --2- 172.21.15.150:0/2676949532 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7e5c0b2bf0 0x7f7e5c0b4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.063+0000 7f7e63990640 1 -- 172.21.15.150:0/2676949532 >> 172.21.15.150:0/2676949532 conn(0x7f7e5c06ca00 msgr2=0x7f7e5c0c9fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:50.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.063+0000 7f7e63990640 1 -- 172.21.15.150:0/2676949532 shutdown_connections 2024-03-17T23:47:50.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.063+0000 7f7e63990640 1 -- 172.21.15.150:0/2676949532 wait complete. 2024-03-17T23:47:50.076 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-17T23:47:50.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.176+0000 7f2e7696f640 1 Processor -- start 2024-03-17T23:47:50.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.176+0000 7f2e7696f640 1 -- start start 2024-03-17T23:47:50.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.177+0000 7f2e7696f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2e70142520 0x7f2e700cc510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:50.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.177+0000 7f2e7696f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e700ce610 0x7f2e70165b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:50.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.177+0000 7f2e7696f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e7006bf60 con 0x7f2e700cca50 2024-03-17T23:47:50.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.177+0000 7f2e7696f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f2e70142900 con 0x7f2e70142520 2024-03-17T23:47:50.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.177+0000 7f2e7696f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2e700cd900 con 0x7f2e700ce610 2024-03-17T23:47:50.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.177+0000 7f2e6ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2e70142520 0x7f2e700cc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:50.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.177+0000 7f2e6ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2e70142520 0x7f2e700cc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50450/0 (socket says 172.21.15.150:50450) 2024-03-17T23:47:50.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.177+0000 7f2e6ffff640 1 -- 172.21.15.150:0/3727785612 learned_addr learned my addr 172.21.15.150:0/3727785612 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:50.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.177+0000 7f2e74f0c640 1 --2- 172.21.15.150:0/3727785612 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e700ce610 0x7f2e70165b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:50.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.178+0000 7f2e6ffff640 1 -- 172.21.15.150:0/3727785612 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e700ce610 msgr2=0x7f2e70165b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:50.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.178+0000 7f2e6ffff640 1 --2- 172.21.15.150:0/3727785612 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e700ce610 0x7f2e70165b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.178+0000 7f2e6ffff640 1 -- 172.21.15.150:0/3727785612 >> v1:172.21.15.157:6789/0 conn(0x7f2e700cca50 legacy=0x7f2e700cdf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:50.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.178+0000 7f2e6effd640 1 -- 172.21.15.150:0/3727785612 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2184577873 0 0) 0x7f2e7006bf60 con 0x7f2e700cca50 2024-03-17T23:47:50.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.178+0000 7f2e6ffff640 1 -- 172.21.15.150:0/3727785612 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e700cd060 con 0x7f2e70142520 2024-03-17T23:47:50.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.178+0000 7f2e6ffff640 1 --2- 172.21.15.150:0/3727785612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e70142520 0x7f2e700cc510 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2e600098a0 tx=0x7f2e6002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d8c69b484afa44a server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:50.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.178+0000 7f2e6effd640 1 -- 172.21.15.150:0/3727785612 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e60035020 con 0x7f2e70142520 2024-03-17T23:47:50.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.179+0000 7f2e6effd640 1 -- 172.21.15.150:0/3727785612 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e60031640 con 0x7f2e70142520 2024-03-17T23:47:50.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.179+0000 7f2e6effd640 1 -- 172.21.15.150:0/3727785612 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e6002d040 con 0x7f2e70142520 2024-03-17T23:47:50.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.179+0000 7f2e7696f640 1 -- 172.21.15.150:0/3727785612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e70142520 msgr2=0x7f2e700cc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:50.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.179+0000 7f2e7696f640 1 --2- 172.21.15.150:0/3727785612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e70142520 0x7f2e700cc510 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2e600098a0 tx=0x7f2e6002b8f0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.179+0000 7f2e7696f640 1 -- 172.21.15.150:0/3727785612 shutdown_connections 2024-03-17T23:47:50.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.179+0000 7f2e7696f640 1 --2- 172.21.15.150:0/3727785612 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e700ce610 0x7f2e70165b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.179+0000 7f2e7696f640 1 --2- 172.21.15.150:0/3727785612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e70142520 0x7f2e700cc510 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.179+0000 7f2e7696f640 1 -- 172.21.15.150:0/3727785612 >> 172.21.15.150:0/3727785612 conn(0x7f2e7006b190 msgr2=0x7f2e7006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:50.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.179+0000 7f2e7696f640 1 -- 172.21.15.150:0/3727785612 shutdown_connections 2024-03-17T23:47:50.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.179+0000 7f2e7696f640 1 -- 172.21.15.150:0/3727785612 wait complete. 2024-03-17T23:47:50.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.180+0000 7f2e7696f640 1 Processor -- start 2024-03-17T23:47:50.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.180+0000 7f2e7696f640 1 -- start start 2024-03-17T23:47:50.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.181+0000 7f2e7696f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e700cca50 0x7f2e70138680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:50.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.181+0000 7f2e7696f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e700ce610 0x7f2e7013bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:50.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.181+0000 7f2e7696f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e70142520 0x7f2e700aa070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:50.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.181+0000 7f2e7696f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7006de60 con 0x7f2e70142520 2024-03-17T23:47:50.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.181+0000 7f2e7696f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2e700cd620 con 0x7f2e700ce610 2024-03-17T23:47:50.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.181+0000 7f2e7696f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2e7006f390 con 0x7f2e700cca50 2024-03-17T23:47:50.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.181+0000 7f2e74f0c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e70142520 0x7f2e700aa070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:50.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.181+0000 7f2e74f0c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e70142520 0x7f2e700aa070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50460/0 (socket says 172.21.15.150:50460) 2024-03-17T23:47:50.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.181+0000 7f2e6ffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e700cca50 0x7f2e70138680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:50.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.181+0000 7f2e74f0c640 1 -- 172.21.15.150:0/1875216877 learned_addr learned my addr 172.21.15.150:0/1875216877 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:50.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.181+0000 7f2e6f7fe640 1 --2- 172.21.15.150:0/1875216877 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e700ce610 0x7f2e7013bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:50.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.182+0000 7f2e74f0c640 1 -- 172.21.15.150:0/1875216877 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e700cca50 msgr2=0x7f2e70138680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:50.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.182+0000 7f2e74f0c640 1 --2- 172.21.15.150:0/1875216877 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e700cca50 0x7f2e70138680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.182+0000 7f2e74f0c640 1 -- 172.21.15.150:0/1875216877 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e700ce610 msgr2=0x7f2e7013bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:50.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.182+0000 7f2e74f0c640 1 --2- 172.21.15.150:0/1875216877 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e700ce610 0x7f2e7013bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.182+0000 7f2e74f0c640 1 -- 172.21.15.150:0/1875216877 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e7013acc0 con 0x7f2e70142520 2024-03-17T23:47:50.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.182+0000 7f2e6ffff640 1 --2- 172.21.15.150:0/1875216877 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e700cca50 0x7f2e70138680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:50.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.182+0000 7f2e74f0c640 1 --2- 172.21.15.150:0/1875216877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e70142520 0x7f2e700aa070 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2e6800ef90 tx=0x7f2e6800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:50.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.182+0000 7f2e6cff9640 1 -- 172.21.15.150:0/1875216877 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e68018020 con 0x7f2e70142520 2024-03-17T23:47:50.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.182+0000 7f2e6cff9640 1 -- 172.21.15.150:0/1875216877 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e68014480 con 0x7f2e70142520 2024-03-17T23:47:50.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.182+0000 7f2e7696f640 1 -- 172.21.15.150:0/1875216877 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e7013afa0 con 0x7f2e70142520 2024-03-17T23:47:50.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.182+0000 7f2e6cff9640 1 -- 172.21.15.150:0/1875216877 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e68014870 con 0x7f2e70142520 2024-03-17T23:47:50.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.182+0000 7f2e7696f640 1 -- 172.21.15.150:0/1875216877 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e7013b560 con 0x7f2e70142520 2024-03-17T23:47:50.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.183+0000 7f2e7696f640 1 -- 172.21.15.150:0/1875216877 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e700681a0 con 0x7f2e70142520 2024-03-17T23:47:50.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.183+0000 7f2e6cff9640 1 -- 172.21.15.150:0/1875216877 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2e68005d10 con 0x7f2e70142520 2024-03-17T23:47:50.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.184+0000 7f2e6cff9640 1 --2- 172.21.15.150:0/1875216877 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2e5c06bc00 0x7f2e5c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:50.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.184+0000 7f2e6cff9640 1 -- 172.21.15.150:0/1875216877 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f2e6808bbe0 con 0x7f2e70142520 2024-03-17T23:47:50.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.184+0000 7f2e6ffff640 1 --2- 172.21.15.150:0/1875216877 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2e5c06bc00 0x7f2e5c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:50.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.184+0000 7f2e6ffff640 1 --2- 172.21.15.150:0/1875216877 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2e5c06bc00 0x7f2e5c06e0c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2e60036000 tx=0x7f2e6003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:50.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.188+0000 7f2e6cff9640 1 -- 172.21.15.150:0/1875216877 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2e68022030 con 0x7f2e70142520 2024-03-17T23:47:50.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.370+0000 7f2e7696f640 1 -- 172.21.15.150:0/1875216877 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2e70074210 con 0x7f2e70142520 2024-03-17T23:47:50.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.370+0000 7f2e6cff9640 1 -- 172.21.15.150:0/1875216877 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) ==== 80+0+0 (secure 0 0 0) 0x7f2e68010040 con 0x7f2e70142520 2024-03-17T23:47:50.372 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-03-17T23:47:50.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.372+0000 7f2e7696f640 1 -- 172.21.15.150:0/1875216877 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2e5c06bc00 msgr2=0x7f2e5c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:50.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.372+0000 7f2e7696f640 1 --2- 172.21.15.150:0/1875216877 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2e5c06bc00 0x7f2e5c06e0c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2e60036000 tx=0x7f2e6003c040 comp rx=0 tx=0).stop 2024-03-17T23:47:50.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.372+0000 7f2e7696f640 1 -- 172.21.15.150:0/1875216877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e70142520 msgr2=0x7f2e700aa070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:50.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.372+0000 7f2e7696f640 1 --2- 172.21.15.150:0/1875216877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e70142520 0x7f2e700aa070 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2e6800ef90 tx=0x7f2e6800c490 comp rx=0 tx=0).stop 2024-03-17T23:47:50.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.372+0000 7f2e7696f640 1 -- 172.21.15.150:0/1875216877 shutdown_connections 2024-03-17T23:47:50.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.372+0000 7f2e7696f640 1 --2- 172.21.15.150:0/1875216877 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2e70142520 0x7f2e700aa070 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.372+0000 7f2e7696f640 1 --2- 172.21.15.150:0/1875216877 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2e5c06bc00 0x7f2e5c06e0c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.372+0000 7f2e7696f640 1 --2- 172.21.15.150:0/1875216877 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2e700ce610 0x7f2e7013bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.372+0000 7f2e7696f640 1 --2- 172.21.15.150:0/1875216877 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2e700cca50 0x7f2e70138680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.372+0000 7f2e7696f640 1 -- 172.21.15.150:0/1875216877 >> 172.21.15.150:0/1875216877 conn(0x7f2e7006b190 msgr2=0x7f2e70144ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:50.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.373+0000 7f2e7696f640 1 -- 172.21.15.150:0/1875216877 shutdown_connections 2024-03-17T23:47:50.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.373+0000 7f2e7696f640 1 -- 172.21.15.150:0/1875216877 wait complete. 2024-03-17T23:47:50.385 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:47:50.385 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:47:50.401 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:50.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.539+0000 7f97b8d90640 1 Processor -- start 2024-03-17T23:47:50.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.539+0000 7f97b8d90640 1 -- start start 2024-03-17T23:47:50.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.540+0000 7f97b8d90640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f97b415e1c0 0x7f97b4067550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:50.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.540+0000 7f97b8d90640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97b4099c80 con 0x7f97b415f720 2024-03-17T23:47:50.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.540+0000 7f97b8d90640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97b406b330 con 0x7f97b406ec90 2024-03-17T23:47:50.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.540+0000 7f97b8d90640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f97b4099f60 con 0x7f97b415e1c0 2024-03-17T23:47:50.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.540+0000 7f97b259c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f97b415e1c0 0x7f97b4067550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:50.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.540+0000 7f97b259c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f97b415e1c0 0x7f97b4067550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50476/0 (socket says 172.21.15.150:50476) 2024-03-17T23:47:50.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.540+0000 7f97b259c640 1 -- 172.21.15.150:0/1828199870 learned_addr learned my addr 172.21.15.150:0/1828199870 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:50.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.540+0000 7f97b1d9b640 1 --1- 172.21.15.150:0/1828199870 >> v1:172.21.15.150:6789/0 conn(0x7f97b415f720 0x7f97b406e580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49834/0 (socket says 172.21.15.150:49834) 2024-03-17T23:47:50.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.541+0000 7f97b259c640 1 -- 172.21.15.150:0/1828199870 >> v1:172.21.15.157:6789/0 conn(0x7f97b406ec90 legacy=0x7f97b415e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:50.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.541+0000 7f97b259c640 1 -- 172.21.15.150:0/1828199870 >> v1:172.21.15.150:6789/0 conn(0x7f97b415f720 legacy=0x7f97b406e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:50.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.541+0000 7f97b259c640 1 -- 172.21.15.150:0/1828199870 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b406b610 con 0x7f97b415e1c0 2024-03-17T23:47:50.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.541+0000 7f97b159a640 1 -- 172.21.15.150:0/1828199870 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1874420190 0 0) 0x7f97b406b330 con 0x7f97b406ec90 2024-03-17T23:47:50.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.541+0000 7f97b259c640 1 --2- 172.21.15.150:0/1828199870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97b415e1c0 0x7f97b4067550 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f97a4009e50 tx=0x7f97a402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81d9044515ecdbae server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:50.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.541+0000 7f97b159a640 1 -- 172.21.15.150:0/1828199870 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a402cbb0 con 0x7f97b415e1c0 2024-03-17T23:47:50.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.541+0000 7f97b159a640 1 -- 172.21.15.150:0/1828199870 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97a402cd50 con 0x7f97b415e1c0 2024-03-17T23:47:50.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.541+0000 7f97b159a640 1 -- 172.21.15.150:0/1828199870 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a4034610 con 0x7f97b415e1c0 2024-03-17T23:47:50.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.542+0000 7f97b8d90640 1 -- 172.21.15.150:0/1828199870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97b415e1c0 msgr2=0x7f97b4067550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:50.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.542+0000 7f97b8d90640 1 --2- 172.21.15.150:0/1828199870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97b415e1c0 0x7f97b4067550 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f97a4009e50 tx=0x7f97a402c010 comp rx=0 tx=0).stop 2024-03-17T23:47:50.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.542+0000 7f97b8d90640 1 -- 172.21.15.150:0/1828199870 shutdown_connections 2024-03-17T23:47:50.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.542+0000 7f97b8d90640 1 --2- 172.21.15.150:0/1828199870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97b415e1c0 0x7f97b4067550 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.542+0000 7f97b8d90640 1 -- 172.21.15.150:0/1828199870 >> 172.21.15.150:0/1828199870 conn(0x7f97b40b3da0 msgr2=0x7f97b40b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:50.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.542+0000 7f97b8d90640 1 -- 172.21.15.150:0/1828199870 shutdown_connections 2024-03-17T23:47:50.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.542+0000 7f97b8d90640 1 -- 172.21.15.150:0/1828199870 wait complete. 2024-03-17T23:47:50.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.543+0000 7f97b8d90640 1 Processor -- start 2024-03-17T23:47:50.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.543+0000 7f97b8d90640 1 -- start start 2024-03-17T23:47:50.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.544+0000 7f97b8d90640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97b406ec90 0x7f97b414ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:50.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.544+0000 7f97b8d90640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97b415e1c0 0x7f97b41f3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:50.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.544+0000 7f97b8d90640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97b415f720 0x7f97b415c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:50.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.544+0000 7f97b8d90640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f97b4072ba0 con 0x7f97b415f720 2024-03-17T23:47:50.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.544+0000 7f97b8d90640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f97b4072a20 con 0x7f97b406ec90 2024-03-17T23:47:50.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.544+0000 7f97b8d90640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f97b4072d20 con 0x7f97b415e1c0 2024-03-17T23:47:50.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.544+0000 7f97b2d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97b415f720 0x7f97b415c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:50.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.544+0000 7f97b2d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97b415f720 0x7f97b415c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50492/0 (socket says 172.21.15.150:50492) 2024-03-17T23:47:50.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.544+0000 7f97b2d9d640 1 -- 172.21.15.150:0/589776169 learned_addr learned my addr 172.21.15.150:0/589776169 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:50.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.544+0000 7f97b259c640 1 --2- 172.21.15.150:0/589776169 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97b406ec90 0x7f97b414ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:50.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.544+0000 7f97b1d9b640 1 --2- 172.21.15.150:0/589776169 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97b415e1c0 0x7f97b41f3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:50.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.545+0000 7f97b2d9d640 1 -- 172.21.15.150:0/589776169 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97b415e1c0 msgr2=0x7f97b41f3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:50.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.545+0000 7f97b2d9d640 1 --2- 172.21.15.150:0/589776169 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97b415e1c0 0x7f97b41f3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.545+0000 7f97b2d9d640 1 -- 172.21.15.150:0/589776169 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97b406ec90 msgr2=0x7f97b414ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:50.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.545+0000 7f97b2d9d640 1 --2- 172.21.15.150:0/589776169 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97b406ec90 0x7f97b414ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.545+0000 7f97b2d9d640 1 -- 172.21.15.150:0/589776169 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b41f4750 con 0x7f97b415f720 2024-03-17T23:47:50.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.545+0000 7f97b259c640 1 --2- 172.21.15.150:0/589776169 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97b406ec90 0x7f97b414ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:50.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.545+0000 7f97b1d9b640 1 --2- 172.21.15.150:0/589776169 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97b415e1c0 0x7f97b41f3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:50.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.545+0000 7f97b2d9d640 1 --2- 172.21.15.150:0/589776169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97b415f720 0x7f97b415c630 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00c410 tx=0x7f97ac00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:50.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.545+0000 7f97a2ffd640 1 -- 172.21.15.150:0/589776169 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac017070 con 0x7f97b415f720 2024-03-17T23:47:50.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.545+0000 7f97a2ffd640 1 -- 172.21.15.150:0/589776169 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97ac0035f0 con 0x7f97b415f720 2024-03-17T23:47:50.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.545+0000 7f97b8d90640 1 -- 172.21.15.150:0/589776169 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97b41f4a30 con 0x7f97b415f720 2024-03-17T23:47:50.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.545+0000 7f97a2ffd640 1 -- 172.21.15.150:0/589776169 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac003a60 con 0x7f97b415f720 2024-03-17T23:47:50.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.545+0000 7f97b8d90640 1 -- 172.21.15.150:0/589776169 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97b41f52b0 con 0x7f97b415f720 2024-03-17T23:47:50.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.546+0000 7f97b8d90640 1 -- 172.21.15.150:0/589776169 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97b40681a0 con 0x7f97b415f720 2024-03-17T23:47:50.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.546+0000 7f97a2ffd640 1 -- 172.21.15.150:0/589776169 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f97ac01a070 con 0x7f97b415f720 2024-03-17T23:47:50.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.547+0000 7f97a2ffd640 1 --2- 172.21.15.150:0/589776169 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f979406bbd0 0x7f979406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:50.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.547+0000 7f97a2ffd640 1 -- 172.21.15.150:0/589776169 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f97ac08a1f0 con 0x7f97b415f720 2024-03-17T23:47:50.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.547+0000 7f97b259c640 1 --2- 172.21.15.150:0/589776169 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f979406bbd0 0x7f979406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:50.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.547+0000 7f97b259c640 1 --2- 172.21.15.150:0/589776169 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f979406bbd0 0x7f979406e090 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f97a4035000 tx=0x7f97a4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:50.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.551+0000 7f97a2ffd640 1 -- 172.21.15.150:0/589776169 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f97ac058290 con 0x7f97b415f720 2024-03-17T23:47:50.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.729+0000 7f97b8d90640 1 -- 172.21.15.150:0/589776169 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f97b41f4c00 con 0x7f97b415f720 2024-03-17T23:47:50.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.730+0000 7f97a2ffd640 1 -- 172.21.15.150:0/589776169 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18881 (secure 0 0 0) 0x7f97ac05c330 con 0x7f97b415f720 2024-03-17T23:47:50.732 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:50.732 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":28,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:37.031864+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.e","mappings":[{"from":7,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:50.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.733+0000 7f97b8d90640 1 -- 172.21.15.150:0/589776169 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f979406bbd0 msgr2=0x7f979406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:50.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.733+0000 7f97b8d90640 1 --2- 172.21.15.150:0/589776169 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f979406bbd0 0x7f979406e090 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f97a4035000 tx=0x7f97a4030000 comp rx=0 tx=0).stop 2024-03-17T23:47:50.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.733+0000 7f97b8d90640 1 -- 172.21.15.150:0/589776169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97b415f720 msgr2=0x7f97b415c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:50.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.733+0000 7f97b8d90640 1 --2- 172.21.15.150:0/589776169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97b415f720 0x7f97b415c630 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00c410 tx=0x7f97ac00c810 comp rx=0 tx=0).stop 2024-03-17T23:47:50.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.733+0000 7f97b8d90640 1 -- 172.21.15.150:0/589776169 shutdown_connections 2024-03-17T23:47:50.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.733+0000 7f97b8d90640 1 --2- 172.21.15.150:0/589776169 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97b415f720 0x7f97b415c630 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.733+0000 7f97b8d90640 1 --2- 172.21.15.150:0/589776169 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f97b415e1c0 0x7f97b41f3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.733+0000 7f97b8d90640 1 --2- 172.21.15.150:0/589776169 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f979406bbd0 0x7f979406e090 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.733+0000 7f97b8d90640 1 --2- 172.21.15.150:0/589776169 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97b406ec90 0x7f97b414ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.733+0000 7f97b8d90640 1 -- 172.21.15.150:0/589776169 >> 172.21.15.150:0/589776169 conn(0x7f97b40b3da0 msgr2=0x7f97b406e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:50.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.733+0000 7f97b8d90640 1 -- 172.21.15.150:0/589776169 shutdown_connections 2024-03-17T23:47:50.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.733+0000 7f97b8d90640 1 -- 172.21.15.150:0/589776169 wait complete. 2024-03-17T23:47:50.745 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:46:53.610678+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:46:54.227335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-17T23:46:57.949235+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-17T23:47:50.746 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:47:50.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.848+0000 7f5934e1f640 1 Processor -- start 2024-03-17T23:47:50.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.848+0000 7f5934e1f640 1 -- start start 2024-03-17T23:47:50.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.849+0000 7f5934e1f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5930155320 0x7f593015bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:50.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.849+0000 7f5934e1f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f593006dd30 0x7f59300ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:50.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.849+0000 7f5934e1f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59300cf3b0 con 0x7f59300cd250 2024-03-17T23:47:50.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.849+0000 7f5934e1f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f593015c550 con 0x7f593006dd30 2024-03-17T23:47:50.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.849+0000 7f5934e1f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f59300cf690 con 0x7f5930155320 2024-03-17T23:47:50.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.850+0000 7f592dd9b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f593006dd30 0x7f59300ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:50.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.850+0000 7f592e59c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5930155320 0x7f593015bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:50.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.850+0000 7f592ed9d640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f59300cd250 0x7f593015e980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:39030/0 (socket says 172.21.15.150:39030) 2024-03-17T23:47:50.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.850+0000 7f592ed9d640 1 -- 172.21.15.150:0/2552055905 learned_addr learned my addr 172.21.15.150:0/2552055905 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:50.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.850+0000 7f592dd9b640 1 -- 172.21.15.150:0/2552055905 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5930155320 msgr2=0x7f593015bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:50.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.850+0000 7f592dd9b640 1 --2- 172.21.15.150:0/2552055905 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5930155320 0x7f593015bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.850+0000 7f592dd9b640 1 -- 172.21.15.150:0/2552055905 >> v1:172.21.15.157:6789/0 conn(0x7f59300cd250 legacy=0x7f593015e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:50.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.850+0000 7f592d59a640 1 -- 172.21.15.150:0/2552055905 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 459156211 0 0) 0x7f59300cf3b0 con 0x7f59300cd250 2024-03-17T23:47:50.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.850+0000 7f592dd9b640 1 -- 172.21.15.150:0/2552055905 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f593015c7a0 con 0x7f593006dd30 2024-03-17T23:47:50.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.850+0000 7f592e59c640 1 --2- 172.21.15.150:0/2552055905 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5930155320 0x7f593015bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:50.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.851+0000 7f592dd9b640 1 --2- 172.21.15.150:0/2552055905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f593006dd30 0x7f59300ccd10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f59240097a0 tx=0x7f592402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d429310167ff590c server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:50.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.851+0000 7f592d59a640 1 -- 172.21.15.150:0/2552055905 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5924035020 con 0x7f593006dd30 2024-03-17T23:47:50.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.851+0000 7f592d59a640 1 -- 172.21.15.150:0/2552055905 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5924031830 con 0x7f593006dd30 2024-03-17T23:47:50.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.851+0000 7f592d59a640 1 -- 172.21.15.150:0/2552055905 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5924031c00 con 0x7f593006dd30 2024-03-17T23:47:50.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.851+0000 7f5934e1f640 1 -- 172.21.15.150:0/2552055905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f593006dd30 msgr2=0x7f59300ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:50.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.851+0000 7f5934e1f640 1 --2- 172.21.15.150:0/2552055905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f593006dd30 0x7f59300ccd10 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f59240097a0 tx=0x7f592402b8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.851+0000 7f5934e1f640 1 -- 172.21.15.150:0/2552055905 shutdown_connections 2024-03-17T23:47:50.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.851+0000 7f5934e1f640 1 --2- 172.21.15.150:0/2552055905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f593006dd30 0x7f59300ccd10 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.851+0000 7f5934e1f640 1 --2- 172.21.15.150:0/2552055905 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5930155320 0x7f593015bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.851+0000 7f5934e1f640 1 -- 172.21.15.150:0/2552055905 >> 172.21.15.150:0/2552055905 conn(0x7f593006b190 msgr2=0x7f593006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:50.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.851+0000 7f5934e1f640 1 -- 172.21.15.150:0/2552055905 shutdown_connections 2024-03-17T23:47:50.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.851+0000 7f5934e1f640 1 -- 172.21.15.150:0/2552055905 wait complete. 2024-03-17T23:47:50.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.853+0000 7f5934e1f640 1 Processor -- start 2024-03-17T23:47:50.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.853+0000 7f5934e1f640 1 -- start start 2024-03-17T23:47:50.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.853+0000 7f5934e1f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f593006dd30 0x7f593013b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:50.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.853+0000 7f5934e1f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59300cd250 0x7f593013aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:50.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.853+0000 7f5934e1f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5930155320 0x7f593013af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:50.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.853+0000 7f5934e1f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5930155700 con 0x7f5930155320 2024-03-17T23:47:50.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.853+0000 7f5934e1f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f593006e190 con 0x7f59300cd250 2024-03-17T23:47:50.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.853+0000 7f5934e1f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5930155880 con 0x7f593006dd30 2024-03-17T23:47:50.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.854+0000 7f592ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5930155320 0x7f593013af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:50.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.854+0000 7f592dd9b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59300cd250 0x7f593013aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:50.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.854+0000 7f592ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5930155320 0x7f593013af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50496/0 (socket says 172.21.15.150:50496) 2024-03-17T23:47:50.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.854+0000 7f592ed9d640 1 -- 172.21.15.150:0/622766493 learned_addr learned my addr 172.21.15.150:0/622766493 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:50.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.854+0000 7f592e59c640 1 --2- 172.21.15.150:0/622766493 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f593006dd30 0x7f593013b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:50.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.854+0000 7f592e59c640 1 -- 172.21.15.150:0/622766493 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59300cd250 msgr2=0x7f593013aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:50.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.854+0000 7f592e59c640 1 --2- 172.21.15.150:0/622766493 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59300cd250 0x7f593013aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.854+0000 7f592e59c640 1 -- 172.21.15.150:0/622766493 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5930155320 msgr2=0x7f593013af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:50.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.854+0000 7f592e59c640 1 --2- 172.21.15.150:0/622766493 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5930155320 0x7f593013af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:50.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.854+0000 7f592e59c640 1 -- 172.21.15.150:0/622766493 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f593014eba0 con 0x7f593006dd30 2024-03-17T23:47:50.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.854+0000 7f592ed9d640 1 --2- 172.21.15.150:0/622766493 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5930155320 0x7f593013af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:50.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.854+0000 7f592e59c640 1 --2- 172.21.15.150:0/622766493 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f593006dd30 0x7f593013b5f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5920009e60 tx=0x7f592000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:50.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.854+0000 7f591effd640 1 -- 172.21.15.150:0/622766493 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5920018020 con 0x7f593006dd30 2024-03-17T23:47:50.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.854+0000 7f591effd640 1 -- 172.21.15.150:0/622766493 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5920014680 con 0x7f593006dd30 2024-03-17T23:47:50.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.854+0000 7f591effd640 1 -- 172.21.15.150:0/622766493 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5920014a70 con 0x7f593006dd30 2024-03-17T23:47:50.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.854+0000 7f5934e1f640 1 -- 172.21.15.150:0/622766493 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f593013bb30 con 0x7f593006dd30 2024-03-17T23:47:50.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.854+0000 7f5934e1f640 1 -- 172.21.15.150:0/622766493 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f59301515a0 con 0x7f593006dd30 2024-03-17T23:47:50.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.855+0000 7f5934e1f640 1 -- 172.21.15.150:0/622766493 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59300681a0 con 0x7f593006dd30 2024-03-17T23:47:50.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.856+0000 7f591effd640 1 -- 172.21.15.150:0/622766493 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5920005d10 con 0x7f593006dd30 2024-03-17T23:47:50.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.856+0000 7f591effd640 1 --2- 172.21.15.150:0/622766493 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f591006bbd0 0x7f591006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:50.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.856+0000 7f591effd640 1 -- 172.21.15.150:0/622766493 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f592008ba60 con 0x7f593006dd30 2024-03-17T23:47:50.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.856+0000 7f592dd9b640 1 --2- 172.21.15.150:0/622766493 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f591006bbd0 0x7f591006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:50.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.857+0000 7f592dd9b640 1 --2- 172.21.15.150:0/622766493 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f591006bbd0 0x7f591006e090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5924036000 tx=0x7f592403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:50.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:50.861+0000 7f591effd640 1 -- 172.21.15.150:0/622766493 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5920022030 con 0x7f593006dd30 2024-03-17T23:47:51.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.035+0000 7f5934e1f640 1 -- 172.21.15.150:0/622766493 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5930151910 con 0x7f593006dd30 2024-03-17T23:47:51.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.036+0000 7f591effd640 1 -- 172.21.15.150:0/622766493 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7f5930151910 con 0x7f593006dd30 2024-03-17T23:47:51.037 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:47:51.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.038+0000 7f5934e1f640 1 -- 172.21.15.150:0/622766493 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f591006bbd0 msgr2=0x7f591006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:51.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.038+0000 7f5934e1f640 1 --2- 172.21.15.150:0/622766493 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f591006bbd0 0x7f591006e090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5924036000 tx=0x7f592403d040 comp rx=0 tx=0).stop 2024-03-17T23:47:51.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.038+0000 7f5934e1f640 1 -- 172.21.15.150:0/622766493 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f593006dd30 msgr2=0x7f593013b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:51.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.038+0000 7f5934e1f640 1 --2- 172.21.15.150:0/622766493 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f593006dd30 0x7f593013b5f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5920009e60 tx=0x7f592000c490 comp rx=0 tx=0).stop 2024-03-17T23:47:51.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.038+0000 7f5934e1f640 1 -- 172.21.15.150:0/622766493 shutdown_connections 2024-03-17T23:47:51.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.038+0000 7f5934e1f640 1 --2- 172.21.15.150:0/622766493 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5930155320 0x7f593013af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.038+0000 7f5934e1f640 1 --2- 172.21.15.150:0/622766493 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f591006bbd0 0x7f591006e090 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.038+0000 7f5934e1f640 1 --2- 172.21.15.150:0/622766493 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59300cd250 0x7f593013aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.038+0000 7f5934e1f640 1 --2- 172.21.15.150:0/622766493 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f593006dd30 0x7f593013b5f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.038+0000 7f5934e1f640 1 -- 172.21.15.150:0/622766493 >> 172.21.15.150:0/622766493 conn(0x7f593006b190 msgr2=0x7f593006e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:51.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.038+0000 7f5934e1f640 1 -- 172.21.15.150:0/622766493 shutdown_connections 2024-03-17T23:47:51.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.038+0000 7f5934e1f640 1 -- 172.21.15.150:0/622766493 wait complete. 2024-03-17T23:47:51.051 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:47:51.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.153+0000 7f15e5b22640 1 Processor -- start 2024-03-17T23:47:51.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.153+0000 7f15e5b22640 1 -- start start 2024-03-17T23:47:51.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.154+0000 7f15e5b22640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f15e00ce6a0 0x7f15e0166a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:51.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.154+0000 7f15e5b22640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15e0145170 con 0x7f15e006f400 2024-03-17T23:47:51.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.154+0000 7f15e5b22640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15e0145450 con 0x7f15e0144bb0 2024-03-17T23:47:51.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.154+0000 7f15e5b22640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f15e0145730 con 0x7f15e00ce6a0 2024-03-17T23:47:51.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.154+0000 7f15deffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f15e006f400 0x7f15e00cdf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49844/0 (socket says 172.21.15.150:49844) 2024-03-17T23:47:51.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.154+0000 7f15deffd640 1 -- 172.21.15.150:0/319945659 learned_addr learned my addr 172.21.15.150:0/319945659 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:51.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.154+0000 7f15dffff640 1 --2- 172.21.15.150:0/319945659 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f15e00ce6a0 0x7f15e0166a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:51.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.154+0000 7f15de7fc640 1 -- 172.21.15.150:0/319945659 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 978347220 0 0) 0x7f15e0145170 con 0x7f15e006f400 2024-03-17T23:47:51.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.154+0000 7f15de7fc640 1 -- 172.21.15.150:0/319945659 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15c8003520 con 0x7f15e006f400 2024-03-17T23:47:51.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.154+0000 7f15de7fc640 1 -- 172.21.15.150:0/319945659 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2504348410 0 0) 0x7f15e0145450 con 0x7f15e0144bb0 2024-03-17T23:47:51.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.154+0000 7f15de7fc640 1 -- 172.21.15.150:0/319945659 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15e0145170 con 0x7f15e0144bb0 2024-03-17T23:47:51.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.154+0000 7f15dffff640 1 -- 172.21.15.150:0/319945659 >> v1:172.21.15.157:6789/0 conn(0x7f15e0144bb0 legacy=0x7f15e015ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:51.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.154+0000 7f15dffff640 1 -- 172.21.15.150:0/319945659 >> v1:172.21.15.150:6789/0 conn(0x7f15e006f400 legacy=0x7f15e00cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:51.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.154+0000 7f15dffff640 1 -- 172.21.15.150:0/319945659 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e015d560 con 0x7f15e00ce6a0 2024-03-17T23:47:51.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.155+0000 7f15dffff640 1 --2- 172.21.15.150:0/319945659 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f15e00ce6a0 0x7f15e0166a40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f15d8009dc0 tx=0x7f15d802be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9745d7ca77cc88e server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:51.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.155+0000 7f15de7fc640 1 -- 172.21.15.150:0/319945659 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d8039070 con 0x7f15e00ce6a0 2024-03-17T23:47:51.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.155+0000 7f15de7fc640 1 -- 172.21.15.150:0/319945659 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15d802cc40 con 0x7f15e00ce6a0 2024-03-17T23:47:51.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.155+0000 7f15de7fc640 1 -- 172.21.15.150:0/319945659 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d8034440 con 0x7f15e00ce6a0 2024-03-17T23:47:51.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.156+0000 7f15e5b22640 1 -- 172.21.15.150:0/319945659 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f15e00ce6a0 msgr2=0x7f15e0166a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:51.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.156+0000 7f15e5b22640 1 --2- 172.21.15.150:0/319945659 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f15e00ce6a0 0x7f15e0166a40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f15d8009dc0 tx=0x7f15d802be90 comp rx=0 tx=0).stop 2024-03-17T23:47:51.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.156+0000 7f15e5b22640 1 -- 172.21.15.150:0/319945659 shutdown_connections 2024-03-17T23:47:51.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.156+0000 7f15e5b22640 1 --2- 172.21.15.150:0/319945659 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f15e00ce6a0 0x7f15e0166a40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.156+0000 7f15e5b22640 1 -- 172.21.15.150:0/319945659 >> 172.21.15.150:0/319945659 conn(0x7f15e006b190 msgr2=0x7f15e006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:51.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.156+0000 7f15e5b22640 1 -- 172.21.15.150:0/319945659 shutdown_connections 2024-03-17T23:47:51.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.156+0000 7f15e5b22640 1 -- 172.21.15.150:0/319945659 wait complete. 2024-03-17T23:47:51.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.157+0000 7f15e5b22640 1 Processor -- start 2024-03-17T23:47:51.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.157+0000 7f15e5b22640 1 -- start start 2024-03-17T23:47:51.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.158+0000 7f15e5b22640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15e006f400 0x7f15e015b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:51.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.158+0000 7f15e5b22640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f15e00ce6a0 0x7f15e013b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:51.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.158+0000 7f15e5b22640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15e0144bb0 0x7f15e013a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:51.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.158+0000 7f15e5b22640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f15e0167100 con 0x7f15e0144bb0 2024-03-17T23:47:51.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.158+0000 7f15e5b22640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f15e0166f80 con 0x7f15e006f400 2024-03-17T23:47:51.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.158+0000 7f15e5b22640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f15e0167280 con 0x7f15e00ce6a0 2024-03-17T23:47:51.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.158+0000 7f15dffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15e0144bb0 0x7f15e013a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:51.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.158+0000 7f15dffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15e0144bb0 0x7f15e013a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50512/0 (socket says 172.21.15.150:50512) 2024-03-17T23:47:51.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.158+0000 7f15dffff640 1 -- 172.21.15.150:0/1159154899 learned_addr learned my addr 172.21.15.150:0/1159154899 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:51.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.158+0000 7f15df7fe640 1 --2- 172.21.15.150:0/1159154899 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15e006f400 0x7f15e015b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:51.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.158+0000 7f15deffd640 1 --2- 172.21.15.150:0/1159154899 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f15e00ce6a0 0x7f15e013b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:51.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.158+0000 7f15dffff640 1 -- 172.21.15.150:0/1159154899 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f15e00ce6a0 msgr2=0x7f15e013b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:51.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.158+0000 7f15dffff640 1 --2- 172.21.15.150:0/1159154899 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f15e00ce6a0 0x7f15e013b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.158+0000 7f15dffff640 1 -- 172.21.15.150:0/1159154899 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15e006f400 msgr2=0x7f15e015b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:51.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.158+0000 7f15dffff640 1 --2- 172.21.15.150:0/1159154899 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15e006f400 0x7f15e015b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.158+0000 7f15dffff640 1 -- 172.21.15.150:0/1159154899 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e015be30 con 0x7f15e0144bb0 2024-03-17T23:47:51.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.159+0000 7f15dffff640 1 --2- 172.21.15.150:0/1159154899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15e0144bb0 0x7f15e013a410 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f15d8037040 tx=0x7f15d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:51.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.159+0000 7f15c7fff640 1 -- 172.21.15.150:0/1159154899 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d8039050 con 0x7f15e0144bb0 2024-03-17T23:47:51.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.159+0000 7f15c7fff640 1 -- 172.21.15.150:0/1159154899 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15d8045070 con 0x7f15e0144bb0 2024-03-17T23:47:51.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.159+0000 7f15e5b22640 1 -- 172.21.15.150:0/1159154899 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15e013bc80 con 0x7f15e0144bb0 2024-03-17T23:47:51.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.159+0000 7f15c7fff640 1 -- 172.21.15.150:0/1159154899 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d8040690 con 0x7f15e0144bb0 2024-03-17T23:47:51.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.159+0000 7f15e5b22640 1 -- 172.21.15.150:0/1159154899 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15e013c220 con 0x7f15e0144bb0 2024-03-17T23:47:51.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.160+0000 7f15e5b22640 1 -- 172.21.15.150:0/1159154899 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15e00681a0 con 0x7f15e0144bb0 2024-03-17T23:47:51.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.160+0000 7f15c7fff640 1 -- 172.21.15.150:0/1159154899 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f15d8040830 con 0x7f15e0144bb0 2024-03-17T23:47:51.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.161+0000 7f15c7fff640 1 --2- 172.21.15.150:0/1159154899 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f15cc06bbd0 0x7f15cc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:51.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.161+0000 7f15df7fe640 1 --2- 172.21.15.150:0/1159154899 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f15cc06bbd0 0x7f15cc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:51.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.161+0000 7f15c7fff640 1 -- 172.21.15.150:0/1159154899 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f15d80ad730 con 0x7f15e0144bb0 2024-03-17T23:47:51.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.161+0000 7f15df7fe640 1 --2- 172.21.15.150:0/1159154899 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f15cc06bbd0 0x7f15cc06e090 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f15d0007fd0 tx=0x7f15d0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:51.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.165+0000 7f15c7fff640 1 -- 172.21.15.150:0/1159154899 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f15e015be30 con 0x7f15e0144bb0 2024-03-17T23:47:51.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.339+0000 7f15e5b22640 1 -- 172.21.15.150:0/1159154899 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f15e00cd180 con 0x7f15e0144bb0 2024-03-17T23:47:51.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.340+0000 7f15c7fff640 1 -- 172.21.15.150:0/1159154899 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7f15d807b7d0 con 0x7f15e0144bb0 2024-03-17T23:47:51.341 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:51.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.342+0000 7f15e5b22640 1 -- 172.21.15.150:0/1159154899 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f15cc06bbd0 msgr2=0x7f15cc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:51.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.342+0000 7f15e5b22640 1 --2- 172.21.15.150:0/1159154899 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f15cc06bbd0 0x7f15cc06e090 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f15d0007fd0 tx=0x7f15d0009040 comp rx=0 tx=0).stop 2024-03-17T23:47:51.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.342+0000 7f15e5b22640 1 -- 172.21.15.150:0/1159154899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15e0144bb0 msgr2=0x7f15e013a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:51.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.342+0000 7f15e5b22640 1 --2- 172.21.15.150:0/1159154899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15e0144bb0 0x7f15e013a410 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f15d8037040 tx=0x7f15d8004a40 comp rx=0 tx=0).stop 2024-03-17T23:47:51.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.342+0000 7f15e5b22640 1 -- 172.21.15.150:0/1159154899 shutdown_connections 2024-03-17T23:47:51.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.342+0000 7f15e5b22640 1 --2- 172.21.15.150:0/1159154899 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15e0144bb0 0x7f15e013a410 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.342+0000 7f15e5b22640 1 --2- 172.21.15.150:0/1159154899 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f15e00ce6a0 0x7f15e013b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.342+0000 7f15e5b22640 1 --2- 172.21.15.150:0/1159154899 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f15cc06bbd0 0x7f15cc06e090 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.342+0000 7f15e5b22640 1 --2- 172.21.15.150:0/1159154899 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f15e006f400 0x7f15e015b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.342+0000 7f15e5b22640 1 -- 172.21.15.150:0/1159154899 >> 172.21.15.150:0/1159154899 conn(0x7f15e006b190 msgr2=0x7f15e00b07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:51.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.342+0000 7f15e5b22640 1 -- 172.21.15.150:0/1159154899 shutdown_connections 2024-03-17T23:47:51.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.342+0000 7f15e5b22640 1 -- 172.21.15.150:0/1159154899 wait complete. 2024-03-17T23:47:51.355 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:47:51.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.456+0000 7f0e89668640 1 Processor -- start 2024-03-17T23:47:51.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.456+0000 7f0e89668640 1 -- start start 2024-03-17T23:47:51.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.457+0000 7f0e89668640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0e84099e20 0x7f0e8409a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:51.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.457+0000 7f0e89668640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0e8415e460 0x7f0e8406c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:51.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.457+0000 7f0e89668640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e8409a740 con 0x7f0e8406c8c0 2024-03-17T23:47:51.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.457+0000 7f0e89668640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f0e8415e990 con 0x7f0e84099e20 2024-03-17T23:47:51.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.457+0000 7f0e89668640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0e8415ec70 con 0x7f0e8415e460 2024-03-17T23:47:51.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.457+0000 7f0e82ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0e84099e20 0x7f0e8409a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:51.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.457+0000 7f0e7bfff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0e8415e460 0x7f0e8406c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:51.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.457+0000 7f0e837fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f0e8406c8c0 0x7f0e840d0200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:39046/0 (socket says 172.21.15.150:39046) 2024-03-17T23:47:51.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.457+0000 7f0e82ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0e84099e20 0x7f0e8409a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50522/0 (socket says 172.21.15.150:50522) 2024-03-17T23:47:51.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.457+0000 7f0e837fe640 1 -- 172.21.15.150:0/1770298282 learned_addr learned my addr 172.21.15.150:0/1770298282 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:51.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.457+0000 7f0e82ffd640 1 -- 172.21.15.150:0/1770298282 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0e8415e460 msgr2=0x7f0e8406c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:51.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.457+0000 7f0e82ffd640 1 --2- 172.21.15.150:0/1770298282 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0e8415e460 0x7f0e8406c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.457+0000 7f0e827fc640 1 -- 172.21.15.150:0/1770298282 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4015041299 0 0) 0x7f0e8409a740 con 0x7f0e8406c8c0 2024-03-17T23:47:51.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.457+0000 7f0e82ffd640 1 -- 172.21.15.150:0/1770298282 >> v1:172.21.15.157:6789/0 conn(0x7f0e8406c8c0 legacy=0x7f0e840d0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:51.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.457+0000 7f0e82ffd640 1 -- 172.21.15.150:0/1770298282 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e8415ef50 con 0x7f0e84099e20 2024-03-17T23:47:51.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.458+0000 7f0e82ffd640 1 --2- 172.21.15.150:0/1770298282 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e84099e20 0x7f0e8409a200 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f0e740097a0 tx=0x7f0e7402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87557490d95bf9a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:51.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.458+0000 7f0e827fc640 1 -- 172.21.15.150:0/1770298282 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e74035020 con 0x7f0e84099e20 2024-03-17T23:47:51.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.458+0000 7f0e827fc640 1 -- 172.21.15.150:0/1770298282 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e74031830 con 0x7f0e84099e20 2024-03-17T23:47:51.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.458+0000 7f0e827fc640 1 -- 172.21.15.150:0/1770298282 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e74031c00 con 0x7f0e84099e20 2024-03-17T23:47:51.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.459+0000 7f0e89668640 1 -- 172.21.15.150:0/1770298282 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e84099e20 msgr2=0x7f0e8409a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:51.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.459+0000 7f0e89668640 1 --2- 172.21.15.150:0/1770298282 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e84099e20 0x7f0e8409a200 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f0e740097a0 tx=0x7f0e7402b8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.459+0000 7f0e89668640 1 -- 172.21.15.150:0/1770298282 shutdown_connections 2024-03-17T23:47:51.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.459+0000 7f0e89668640 1 --2- 172.21.15.150:0/1770298282 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0e8415e460 0x7f0e8406c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.459+0000 7f0e89668640 1 --2- 172.21.15.150:0/1770298282 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e84099e20 0x7f0e8409a200 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.459+0000 7f0e89668640 1 -- 172.21.15.150:0/1770298282 >> 172.21.15.150:0/1770298282 conn(0x7f0e840b3630 msgr2=0x7f0e840b3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:51.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.459+0000 7f0e89668640 1 -- 172.21.15.150:0/1770298282 shutdown_connections 2024-03-17T23:47:51.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.459+0000 7f0e89668640 1 -- 172.21.15.150:0/1770298282 wait complete. 2024-03-17T23:47:51.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.460+0000 7f0e89668640 1 Processor -- start 2024-03-17T23:47:51.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.460+0000 7f0e89668640 1 -- start start 2024-03-17T23:47:51.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.460+0000 7f0e89668640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e84099e20 0x7f0e841f4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:51.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.460+0000 7f0e89668640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0e8415e460 0x7f0e841f7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:51.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.460+0000 7f0e89668640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e84071220 0x7f0e841529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:51.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.460+0000 7f0e89668640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8406d570 con 0x7f0e84099e20 2024-03-17T23:47:51.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.460+0000 7f0e89668640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0e840cfcd0 con 0x7f0e84071220 2024-03-17T23:47:51.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.461+0000 7f0e89668640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0e840d0910 con 0x7f0e8415e460 2024-03-17T23:47:51.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.461+0000 7f0e82ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e84099e20 0x7f0e841f4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:51.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.461+0000 7f0e82ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e84099e20 0x7f0e841f4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50530/0 (socket says 172.21.15.150:50530) 2024-03-17T23:47:51.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.461+0000 7f0e82ffd640 1 -- 172.21.15.150:0/516927555 learned_addr learned my addr 172.21.15.150:0/516927555 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:51.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.461+0000 7f0e7bfff640 1 --2- 172.21.15.150:0/516927555 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0e8415e460 0x7f0e841f7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:51.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.461+0000 7f0e837fe640 1 --2- 172.21.15.150:0/516927555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e84071220 0x7f0e841529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:51.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.461+0000 7f0e82ffd640 1 -- 172.21.15.150:0/516927555 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0e8415e460 msgr2=0x7f0e841f7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:51.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.461+0000 7f0e82ffd640 1 --2- 172.21.15.150:0/516927555 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0e8415e460 0x7f0e841f7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.461+0000 7f0e82ffd640 1 -- 172.21.15.150:0/516927555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e84071220 msgr2=0x7f0e841529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:51.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.461+0000 7f0e82ffd640 1 --2- 172.21.15.150:0/516927555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e84071220 0x7f0e841529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.461+0000 7f0e82ffd640 1 -- 172.21.15.150:0/516927555 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e841f7bc0 con 0x7f0e84099e20 2024-03-17T23:47:51.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.461+0000 7f0e837fe640 1 --2- 172.21.15.150:0/516927555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e84071220 0x7f0e841529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:47:51.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.461+0000 7f0e82ffd640 1 --2- 172.21.15.150:0/516927555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e84099e20 0x7f0e841f4ba0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0e740098c0 tx=0x7f0e740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:51.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.461+0000 7f0e7b7fe640 1 -- 172.21.15.150:0/516927555 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e74041070 con 0x7f0e84099e20 2024-03-17T23:47:51.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.461+0000 7f0e7b7fe640 1 -- 172.21.15.150:0/516927555 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e7403b920 con 0x7f0e84099e20 2024-03-17T23:47:51.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.461+0000 7f0e7b7fe640 1 -- 172.21.15.150:0/516927555 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e74035040 con 0x7f0e84099e20 2024-03-17T23:47:51.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.461+0000 7f0e89668640 1 -- 172.21.15.150:0/516927555 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e841f7e40 con 0x7f0e84099e20 2024-03-17T23:47:51.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.461+0000 7f0e89668640 1 -- 172.21.15.150:0/516927555 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e841f8400 con 0x7f0e84099e20 2024-03-17T23:47:51.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.462+0000 7f0e89668640 1 -- 172.21.15.150:0/516927555 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e840681a0 con 0x7f0e84099e20 2024-03-17T23:47:51.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.463+0000 7f0e7b7fe640 1 -- 172.21.15.150:0/516927555 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0e7404c060 con 0x7f0e84099e20 2024-03-17T23:47:51.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.463+0000 7f0e7b7fe640 1 --2- 172.21.15.150:0/516927555 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0e6406bc00 0x7f0e6406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:51.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.463+0000 7f0e7b7fe640 1 -- 172.21.15.150:0/516927555 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f0e740ac6b0 con 0x7f0e84099e20 2024-03-17T23:47:51.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.463+0000 7f0e7bfff640 1 --2- 172.21.15.150:0/516927555 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0e6406bc00 0x7f0e6406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:51.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.464+0000 7f0e7bfff640 1 --2- 172.21.15.150:0/516927555 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0e6406bc00 0x7f0e6406e0c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0e840b3580 tx=0x7f0e6c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:51.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.468+0000 7f0e7b7fe640 1 -- 172.21.15.150:0/516927555 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0e841f7bc0 con 0x7f0e84099e20 2024-03-17T23:47:51.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.639+0000 7f0e89668640 1 -- 172.21.15.150:0/516927555 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0e8406de80 con 0x7f0e84099e20 2024-03-17T23:47:51.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.640+0000 7f0e7b7fe640 1 -- 172.21.15.150:0/516927555 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7f0e7407ada0 con 0x7f0e84099e20 2024-03-17T23:47:51.641 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:47:51.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.642+0000 7f0e89668640 1 -- 172.21.15.150:0/516927555 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0e6406bc00 msgr2=0x7f0e6406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:51.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.642+0000 7f0e89668640 1 --2- 172.21.15.150:0/516927555 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0e6406bc00 0x7f0e6406e0c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0e840b3580 tx=0x7f0e6c009190 comp rx=0 tx=0).stop 2024-03-17T23:47:51.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.642+0000 7f0e89668640 1 -- 172.21.15.150:0/516927555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e84099e20 msgr2=0x7f0e841f4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:51.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.642+0000 7f0e89668640 1 --2- 172.21.15.150:0/516927555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e84099e20 0x7f0e841f4ba0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0e740098c0 tx=0x7f0e740046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.642+0000 7f0e89668640 1 -- 172.21.15.150:0/516927555 shutdown_connections 2024-03-17T23:47:51.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.642+0000 7f0e89668640 1 --2- 172.21.15.150:0/516927555 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0e6406bc00 0x7f0e6406e0c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.642+0000 7f0e89668640 1 --2- 172.21.15.150:0/516927555 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e84071220 0x7f0e841529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.642+0000 7f0e89668640 1 --2- 172.21.15.150:0/516927555 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0e8415e460 0x7f0e841f7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.642+0000 7f0e89668640 1 --2- 172.21.15.150:0/516927555 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e84099e20 0x7f0e841f4ba0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.642+0000 7f0e89668640 1 -- 172.21.15.150:0/516927555 >> 172.21.15.150:0/516927555 conn(0x7f0e840b3630 msgr2=0x7f0e840ae870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:51.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.642+0000 7f0e89668640 1 -- 172.21.15.150:0/516927555 shutdown_connections 2024-03-17T23:47:51.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.642+0000 7f0e89668640 1 -- 172.21.15.150:0/516927555 wait complete. 2024-03-17T23:47:51.654 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-17T23:47:51.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.756+0000 7f7a3fa26640 1 Processor -- start 2024-03-17T23:47:51.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.756+0000 7f7a3fa26640 1 -- start start 2024-03-17T23:47:51.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.757+0000 7f7a3fa26640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a38155320 0x7f7a3815bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:51.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.757+0000 7f7a3fa26640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7a3806dd30 0x7f7a380ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:51.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.757+0000 7f7a3fa26640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a380cf3b0 con 0x7f7a380cd250 2024-03-17T23:47:51.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.757+0000 7f7a3fa26640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f7a3815c550 con 0x7f7a3806dd30 2024-03-17T23:47:51.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.757+0000 7f7a3fa26640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7a380cf690 con 0x7f7a38155320 2024-03-17T23:47:51.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.757+0000 7f7a3cfc1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7a3806dd30 0x7f7a380ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:51.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.757+0000 7f7a3d7c2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a38155320 0x7f7a3815bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:51.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.757+0000 7f7a3cfc1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7a3806dd30 0x7f7a380ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50538/0 (socket says 172.21.15.150:50538) 2024-03-17T23:47:51.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.757+0000 7f7a3cfc1640 1 -- 172.21.15.150:0/4278968597 learned_addr learned my addr 172.21.15.150:0/4278968597 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:51.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.757+0000 7f7a3d7c2640 1 -- 172.21.15.150:0/4278968597 >> v2:172.21.15.150:3300/0 conn(0x7f7a3806dd30 msgr2=0x7f7a380ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:51.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.757+0000 7f7a3d7c2640 1 --2- 172.21.15.150:0/4278968597 >> v2:172.21.15.150:3300/0 conn(0x7f7a3806dd30 0x7f7a380ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.757+0000 7f7a2ffff640 1 -- 172.21.15.150:0/4278968597 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1958226893 0 0) 0x7f7a380cf3b0 con 0x7f7a380cd250 2024-03-17T23:47:51.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.757+0000 7f7a3d7c2640 1 -- 172.21.15.150:0/4278968597 >> v1:172.21.15.157:6789/0 conn(0x7f7a380cd250 legacy=0x7f7a3815e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:51.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.757+0000 7f7a3d7c2640 1 -- 172.21.15.150:0/4278968597 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a3815c7a0 con 0x7f7a38155320 2024-03-17T23:47:51.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.757+0000 7f7a3cfc1640 1 --2- 172.21.15.150:0/4278968597 >> v2:172.21.15.150:3300/0 conn(0x7f7a3806dd30 0x7f7a380ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:51.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.758+0000 7f7a3d7c2640 1 --2- 172.21.15.150:0/4278968597 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a38155320 0x7f7a3815bfd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7a340097a0 tx=0x7f7a3402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=883bcade7a1ad71a server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:51.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.758+0000 7f7a2ffff640 1 -- 172.21.15.150:0/4278968597 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a34035020 con 0x7f7a38155320 2024-03-17T23:47:51.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.758+0000 7f7a2ffff640 1 -- 172.21.15.150:0/4278968597 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a34031830 con 0x7f7a38155320 2024-03-17T23:47:51.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.758+0000 7f7a2ffff640 1 -- 172.21.15.150:0/4278968597 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a34031c00 con 0x7f7a38155320 2024-03-17T23:47:51.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.759+0000 7f7a3fa26640 1 -- 172.21.15.150:0/4278968597 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a38155320 msgr2=0x7f7a3815bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:51.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.759+0000 7f7a3fa26640 1 --2- 172.21.15.150:0/4278968597 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a38155320 0x7f7a3815bfd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7a340097a0 tx=0x7f7a3402b8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.759+0000 7f7a3fa26640 1 -- 172.21.15.150:0/4278968597 shutdown_connections 2024-03-17T23:47:51.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.759+0000 7f7a3fa26640 1 --2- 172.21.15.150:0/4278968597 >> v2:172.21.15.150:3300/0 conn(0x7f7a3806dd30 0x7f7a380ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.759+0000 7f7a3fa26640 1 --2- 172.21.15.150:0/4278968597 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a38155320 0x7f7a3815bfd0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.759+0000 7f7a3fa26640 1 -- 172.21.15.150:0/4278968597 >> 172.21.15.150:0/4278968597 conn(0x7f7a3806b190 msgr2=0x7f7a3806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:51.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.759+0000 7f7a3fa26640 1 -- 172.21.15.150:0/4278968597 shutdown_connections 2024-03-17T23:47:51.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.759+0000 7f7a3fa26640 1 -- 172.21.15.150:0/4278968597 wait complete. 2024-03-17T23:47:51.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.760+0000 7f7a3fa26640 1 Processor -- start 2024-03-17T23:47:51.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.760+0000 7f7a3fa26640 1 -- start start 2024-03-17T23:47:51.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.761+0000 7f7a3fa26640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a3806dd30 0x7f7a38158620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:51.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.761+0000 7f7a3fa26640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a380cd250 0x7f7a38151570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:51.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.761+0000 7f7a3fa26640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a38155320 0x7f7a381f7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:51.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.761+0000 7f7a3fa26640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7a38155700 con 0x7f7a3806dd30 2024-03-17T23:47:51.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.761+0000 7f7a3fa26640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7a3806e190 con 0x7f7a380cd250 2024-03-17T23:47:51.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.761+0000 7f7a3fa26640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7a38155880 con 0x7f7a38155320 2024-03-17T23:47:51.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.761+0000 7f7a3d7c2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a3806dd30 0x7f7a38158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:51.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.761+0000 7f7a3d7c2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a3806dd30 0x7f7a38158620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50542/0 (socket says 172.21.15.150:50542) 2024-03-17T23:47:51.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.761+0000 7f7a3d7c2640 1 -- 172.21.15.150:0/3862923503 learned_addr learned my addr 172.21.15.150:0/3862923503 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:51.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.761+0000 7f7a3cfc1640 1 --2- 172.21.15.150:0/3862923503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a380cd250 0x7f7a38151570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:51.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.761+0000 7f7a3dfc3640 1 --2- 172.21.15.150:0/3862923503 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a38155320 0x7f7a381f7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:51.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.761+0000 7f7a3d7c2640 1 -- 172.21.15.150:0/3862923503 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a38155320 msgr2=0x7f7a381f7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:51.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.761+0000 7f7a3d7c2640 1 --2- 172.21.15.150:0/3862923503 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a38155320 0x7f7a381f7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.761+0000 7f7a3d7c2640 1 -- 172.21.15.150:0/3862923503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a380cd250 msgr2=0x7f7a38151570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:51.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.761+0000 7f7a3d7c2640 1 --2- 172.21.15.150:0/3862923503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a380cd250 0x7f7a38151570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.761+0000 7f7a3d7c2640 1 -- 172.21.15.150:0/3862923503 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a38151ce0 con 0x7f7a3806dd30 2024-03-17T23:47:51.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.761+0000 7f7a3d7c2640 1 --2- 172.21.15.150:0/3862923503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a3806dd30 0x7f7a38158620 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7a340097a0 tx=0x7f7a340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:51.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.761+0000 7f7a2dffb640 1 -- 172.21.15.150:0/3862923503 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a34004230 con 0x7f7a3806dd30 2024-03-17T23:47:51.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.761+0000 7f7a2dffb640 1 -- 172.21.15.150:0/3862923503 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a340043d0 con 0x7f7a3806dd30 2024-03-17T23:47:51.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.762+0000 7f7a3fa26640 1 -- 172.21.15.150:0/3862923503 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a38151fc0 con 0x7f7a3806dd30 2024-03-17T23:47:51.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.762+0000 7f7a2dffb640 1 -- 172.21.15.150:0/3862923503 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3403a8a0 con 0x7f7a3806dd30 2024-03-17T23:47:51.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.762+0000 7f7a3fa26640 1 -- 172.21.15.150:0/3862923503 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a3815a090 con 0x7f7a3806dd30 2024-03-17T23:47:51.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.763+0000 7f7a3fa26640 1 -- 172.21.15.150:0/3862923503 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a380681a0 con 0x7f7a3806dd30 2024-03-17T23:47:51.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.763+0000 7f7a2dffb640 1 -- 172.21.15.150:0/3862923503 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7a34038020 con 0x7f7a3806dd30 2024-03-17T23:47:51.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.763+0000 7f7a2dffb640 1 --2- 172.21.15.150:0/3862923503 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7a1c06bbd0 0x7f7a1c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:51.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.764+0000 7f7a2dffb640 1 -- 172.21.15.150:0/3862923503 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f7a340ace10 con 0x7f7a3806dd30 2024-03-17T23:47:51.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.764+0000 7f7a3cfc1640 1 --2- 172.21.15.150:0/3862923503 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7a1c06bbd0 0x7f7a1c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:51.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.764+0000 7f7a3cfc1640 1 --2- 172.21.15.150:0/3862923503 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7a1c06bbd0 0x7f7a1c06e090 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7a3815f8f0 tx=0x7f7a28009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:51.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.768+0000 7f7a2dffb640 1 -- 172.21.15.150:0/3862923503 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7a34041070 con 0x7f7a3806dd30 2024-03-17T23:47:51.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.942+0000 7f7a3fa26640 1 -- 172.21.15.150:0/3862923503 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f7a380cf270 con 0x7f7a3806dd30 2024-03-17T23:47:51.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.943+0000 7f7a2dffb640 1 -- 172.21.15.150:0/3862923503 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7f7a3407afe0 con 0x7f7a3806dd30 2024-03-17T23:47:51.944 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:51.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.945+0000 7f7a3fa26640 1 -- 172.21.15.150:0/3862923503 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7a1c06bbd0 msgr2=0x7f7a1c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:51.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.945+0000 7f7a3fa26640 1 --2- 172.21.15.150:0/3862923503 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7a1c06bbd0 0x7f7a1c06e090 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7a3815f8f0 tx=0x7f7a28009240 comp rx=0 tx=0).stop 2024-03-17T23:47:51.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.945+0000 7f7a3fa26640 1 -- 172.21.15.150:0/3862923503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a3806dd30 msgr2=0x7f7a38158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:51.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.945+0000 7f7a3fa26640 1 --2- 172.21.15.150:0/3862923503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a3806dd30 0x7f7a38158620 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7a340097a0 tx=0x7f7a340046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.945+0000 7f7a3fa26640 1 -- 172.21.15.150:0/3862923503 shutdown_connections 2024-03-17T23:47:51.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.945+0000 7f7a3fa26640 1 --2- 172.21.15.150:0/3862923503 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a38155320 0x7f7a381f7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.945+0000 7f7a3fa26640 1 --2- 172.21.15.150:0/3862923503 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7a1c06bbd0 0x7f7a1c06e090 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.945+0000 7f7a3fa26640 1 --2- 172.21.15.150:0/3862923503 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a380cd250 0x7f7a38151570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.945+0000 7f7a3fa26640 1 --2- 172.21.15.150:0/3862923503 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a3806dd30 0x7f7a38158620 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:51.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.945+0000 7f7a3fa26640 1 -- 172.21.15.150:0/3862923503 >> 172.21.15.150:0/3862923503 conn(0x7f7a3806b190 msgr2=0x7f7a38158d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:51.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.945+0000 7f7a3fa26640 1 -- 172.21.15.150:0/3862923503 shutdown_connections 2024-03-17T23:47:51.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:51.945+0000 7f7a3fa26640 1 -- 172.21.15.150:0/3862923503 wait complete. 2024-03-17T23:47:51.958 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-17T23:47:52.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.058+0000 7f4e98a19640 1 Processor -- start 2024-03-17T23:47:52.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.058+0000 7f4e98a19640 1 -- start start 2024-03-17T23:47:52.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.059+0000 7f4e98a19640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4e940cfb20 0x7f4e94067360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:52.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.059+0000 7f4e98a19640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4e940ccc90 0x7f4e9406c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:52.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.059+0000 7f4e98a19640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e940af490 con 0x7f4e940b1410 2024-03-17T23:47:52.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.059+0000 7f4e98a19640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f4e9406bdb0 con 0x7f4e940cfb20 2024-03-17T23:47:52.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.059+0000 7f4e98a19640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4e940af730 con 0x7f4e940ccc90 2024-03-17T23:47:52.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.059+0000 7f4e9259c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f4e940b1410 0x7f4e940b17f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49856/0 (socket says 172.21.15.150:49856) 2024-03-17T23:47:52.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.059+0000 7f4e9259c640 1 -- 172.21.15.150:0/2960304272 learned_addr learned my addr 172.21.15.150:0/2960304272 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:52.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.059+0000 7f4e91d9b640 1 --2- 172.21.15.150:0/2960304272 >> v2:172.21.15.157:3300/0 conn(0x7f4e940cfb20 0x7f4e94067360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:52.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.059+0000 7f4e92d9d640 1 --2- 172.21.15.150:0/2960304272 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4e940ccc90 0x7f4e9406c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:52.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.060+0000 7f4e9159a640 1 -- 172.21.15.150:0/2960304272 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2722662302 0 0) 0x7f4e940af490 con 0x7f4e940b1410 2024-03-17T23:47:52.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.060+0000 7f4e9159a640 1 -- 172.21.15.150:0/2960304272 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e7c003520 con 0x7f4e940b1410 2024-03-17T23:47:52.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.060+0000 7f4e91d9b640 1 -- 172.21.15.150:0/2960304272 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4e940ccc90 msgr2=0x7f4e9406c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:52.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.060+0000 7f4e91d9b640 1 --2- 172.21.15.150:0/2960304272 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4e940ccc90 0x7f4e9406c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:52.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.060+0000 7f4e91d9b640 1 -- 172.21.15.150:0/2960304272 >> v1:172.21.15.150:6789/0 conn(0x7f4e940b1410 legacy=0x7f4e940b17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:52.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.060+0000 7f4e91d9b640 1 -- 172.21.15.150:0/2960304272 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e940afa10 con 0x7f4e940cfb20 2024-03-17T23:47:52.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.060+0000 7f4e92d9d640 1 --2- 172.21.15.150:0/2960304272 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4e940ccc90 0x7f4e9406c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:52.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.060+0000 7f4e91d9b640 1 --2- 172.21.15.150:0/2960304272 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4e940cfb20 0x7f4e94067360 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4e880097a0 tx=0x7f4e8802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=527691ba82d9ea98 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:52.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.061+0000 7f4e9159a640 1 -- 172.21.15.150:0/2960304272 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e88035020 con 0x7f4e940cfb20 2024-03-17T23:47:52.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.061+0000 7f4e9159a640 1 -- 172.21.15.150:0/2960304272 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e88031830 con 0x7f4e940cfb20 2024-03-17T23:47:52.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.061+0000 7f4e9159a640 1 -- 172.21.15.150:0/2960304272 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e88031be0 con 0x7f4e940cfb20 2024-03-17T23:47:52.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.061+0000 7f4e98a19640 1 -- 172.21.15.150:0/2960304272 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4e940cfb20 msgr2=0x7f4e94067360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:52.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.061+0000 7f4e98a19640 1 --2- 172.21.15.150:0/2960304272 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4e940cfb20 0x7f4e94067360 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4e880097a0 tx=0x7f4e8802b8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:52.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.061+0000 7f4e98a19640 1 -- 172.21.15.150:0/2960304272 shutdown_connections 2024-03-17T23:47:52.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.061+0000 7f4e98a19640 1 --2- 172.21.15.150:0/2960304272 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4e940ccc90 0x7f4e9406c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:52.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.061+0000 7f4e98a19640 1 --2- 172.21.15.150:0/2960304272 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4e940cfb20 0x7f4e94067360 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:52.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.061+0000 7f4e98a19640 1 -- 172.21.15.150:0/2960304272 >> 172.21.15.150:0/2960304272 conn(0x7f4e94072a60 msgr2=0x7f4e940b5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:52.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.061+0000 7f4e98a19640 1 -- 172.21.15.150:0/2960304272 shutdown_connections 2024-03-17T23:47:52.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.061+0000 7f4e98a19640 1 -- 172.21.15.150:0/2960304272 wait complete. 2024-03-17T23:47:52.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.063+0000 7f4e98a19640 1 Processor -- start 2024-03-17T23:47:52.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.063+0000 7f4e98a19640 1 -- start start 2024-03-17T23:47:52.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.063+0000 7f4e98a19640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e940b1410 0x7f4e94153240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:52.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.063+0000 7f4e98a19640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4e940ccc90 0x7f4e940b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:52.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.063+0000 7f4e98a19640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4e940cfb20 0x7f4e940b2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:52.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.063+0000 7f4e98a19640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4e9415c300 con 0x7f4e940b1410 2024-03-17T23:47:52.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.063+0000 7f4e98a19640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4e9415c180 con 0x7f4e940cfb20 2024-03-17T23:47:52.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.063+0000 7f4e98a19640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4e940cd120 con 0x7f4e940ccc90 2024-03-17T23:47:52.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.063+0000 7f4e9259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e940b1410 0x7f4e94153240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:52.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.063+0000 7f4e9259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e940b1410 0x7f4e94153240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50548/0 (socket says 172.21.15.150:50548) 2024-03-17T23:47:52.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.063+0000 7f4e9259c640 1 -- 172.21.15.150:0/3567869645 learned_addr learned my addr 172.21.15.150:0/3567869645 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:52.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.064+0000 7f4e92d9d640 1 --2- 172.21.15.150:0/3567869645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4e940cfb20 0x7f4e940b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:52.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.064+0000 7f4e91d9b640 1 --2- 172.21.15.150:0/3567869645 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4e940ccc90 0x7f4e940b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:52.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.064+0000 7f4e9259c640 1 -- 172.21.15.150:0/3567869645 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4e940ccc90 msgr2=0x7f4e940b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:52.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.064+0000 7f4e9259c640 1 --2- 172.21.15.150:0/3567869645 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4e940ccc90 0x7f4e940b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:52.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.064+0000 7f4e9259c640 1 -- 172.21.15.150:0/3567869645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4e940cfb20 msgr2=0x7f4e940b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:52.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.064+0000 7f4e9259c640 1 --2- 172.21.15.150:0/3567869645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4e940cfb20 0x7f4e940b2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:52.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.064+0000 7f4e9259c640 1 -- 172.21.15.150:0/3567869645 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e94153780 con 0x7f4e940b1410 2024-03-17T23:47:52.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.064+0000 7f4e9259c640 1 --2- 172.21.15.150:0/3567869645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e940b1410 0x7f4e94153240 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f4e8400caa0 tx=0x7f4e8400cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:52.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.064+0000 7f4e82ffd640 1 -- 172.21.15.150:0/3567869645 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e84017070 con 0x7f4e940b1410 2024-03-17T23:47:52.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.064+0000 7f4e82ffd640 1 -- 172.21.15.150:0/3567869645 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e8400ab40 con 0x7f4e940b1410 2024-03-17T23:47:52.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.064+0000 7f4e98a19640 1 -- 172.21.15.150:0/3567869645 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e94153be0 con 0x7f4e940b1410 2024-03-17T23:47:52.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.064+0000 7f4e82ffd640 1 -- 172.21.15.150:0/3567869645 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e84012400 con 0x7f4e940b1410 2024-03-17T23:47:52.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.064+0000 7f4e98a19640 1 -- 172.21.15.150:0/3567869645 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e941541a0 con 0x7f4e940b1410 2024-03-17T23:47:52.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.065+0000 7f4e98a19640 1 -- 172.21.15.150:0/3567869645 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e9415e840 con 0x7f4e940b1410 2024-03-17T23:47:52.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.066+0000 7f4e82ffd640 1 -- 172.21.15.150:0/3567869645 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4e8401a070 con 0x7f4e940b1410 2024-03-17T23:47:52.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.066+0000 7f4e82ffd640 1 --2- 172.21.15.150:0/3567869645 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4e7406bbd0 0x7f4e7406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:52.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.066+0000 7f4e82ffd640 1 -- 172.21.15.150:0/3567869645 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f4e8408aae0 con 0x7f4e940b1410 2024-03-17T23:47:52.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.066+0000 7f4e91d9b640 1 --2- 172.21.15.150:0/3567869645 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4e7406bbd0 0x7f4e7406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:52.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.066+0000 7f4e91d9b640 1 --2- 172.21.15.150:0/3567869645 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4e7406bbd0 0x7f4e7406e090 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4e88036000 tx=0x7f4e88008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:52.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.070+0000 7f4e82ffd640 1 -- 172.21.15.150:0/3567869645 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4e9415e840 con 0x7f4e940b1410 2024-03-17T23:47:52.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.254+0000 7f4e98a19640 1 -- 172.21.15.150:0/3567869645 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4e940681a0 con 0x7f4e940b1410 2024-03-17T23:47:52.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.254+0000 7f4e82ffd640 1 -- 172.21.15.150:0/3567869645 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) ==== 80+0+0 (secure 0 0 0) 0x7f4e84058b80 con 0x7f4e940b1410 2024-03-17T23:47:52.256 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-03-17T23:47:52.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.256+0000 7f4e98a19640 1 -- 172.21.15.150:0/3567869645 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4e7406bbd0 msgr2=0x7f4e7406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:52.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.256+0000 7f4e98a19640 1 --2- 172.21.15.150:0/3567869645 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4e7406bbd0 0x7f4e7406e090 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4e88036000 tx=0x7f4e88008030 comp rx=0 tx=0).stop 2024-03-17T23:47:52.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.256+0000 7f4e98a19640 1 -- 172.21.15.150:0/3567869645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e940b1410 msgr2=0x7f4e94153240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:52.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.256+0000 7f4e98a19640 1 --2- 172.21.15.150:0/3567869645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e940b1410 0x7f4e94153240 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f4e8400caa0 tx=0x7f4e8400cea0 comp rx=0 tx=0).stop 2024-03-17T23:47:52.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.256+0000 7f4e98a19640 1 -- 172.21.15.150:0/3567869645 shutdown_connections 2024-03-17T23:47:52.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.256+0000 7f4e98a19640 1 --2- 172.21.15.150:0/3567869645 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4e7406bbd0 0x7f4e7406e090 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:52.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.256+0000 7f4e98a19640 1 --2- 172.21.15.150:0/3567869645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4e940cfb20 0x7f4e940b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:52.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.256+0000 7f4e98a19640 1 --2- 172.21.15.150:0/3567869645 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4e940ccc90 0x7f4e940b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:52.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.256+0000 7f4e98a19640 1 --2- 172.21.15.150:0/3567869645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4e940b1410 0x7f4e94153240 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:52.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.256+0000 7f4e98a19640 1 -- 172.21.15.150:0/3567869645 >> 172.21.15.150:0/3567869645 conn(0x7f4e94072a60 msgr2=0x7f4e9406f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:52.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.256+0000 7f4e98a19640 1 -- 172.21.15.150:0/3567869645 shutdown_connections 2024-03-17T23:47:52.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.256+0000 7f4e98a19640 1 -- 172.21.15.150:0/3567869645 wait complete. 2024-03-17T23:47:52.269 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi150.front.sepia.ceph.com... 2024-03-17T23:47:52.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:52.269 DEBUG:teuthology.orchestra.run.smithi150:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-03-17T23:47:52.296 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:47:52.296 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi150.front.sepia.ceph.com 2024-03-17T23:47:52.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:52.297 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-17T23:47:52.351 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:47:52.351 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:47:52.367 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:47:52.368 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:47:52.423 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:52.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.522+0000 7f796be6c640 1 Processor -- start 2024-03-17T23:47:52.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.522+0000 7f796be6c640 1 -- start start 2024-03-17T23:47:52.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.523+0000 7f796be6c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7964155320 0x7f796415bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:52.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.523+0000 7f796be6c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79640cf3b0 con 0x7f796406dd30 2024-03-17T23:47:52.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.523+0000 7f796be6c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79640cf690 con 0x7f79640cd420 2024-03-17T23:47:52.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.523+0000 7f796be6c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f796415c550 con 0x7f7964155320 2024-03-17T23:47:52.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.523+0000 7f7969407640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f796406dd30 0x7f79640ccd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49866/0 (socket says 172.21.15.150:49866) 2024-03-17T23:47:52.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.523+0000 7f7969407640 1 -- 172.21.15.150:0/2309777196 learned_addr learned my addr 172.21.15.150:0/2309777196 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:52.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.523+0000 7f7969c08640 1 --2- 172.21.15.150:0/2309777196 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7964155320 0x7f796415bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:52.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.523+0000 7f7968c06640 1 -- 172.21.15.150:0/2309777196 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2492365055 0 0) 0x7f79640cf3b0 con 0x7f796406dd30 2024-03-17T23:47:52.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.523+0000 7f7968c06640 1 -- 172.21.15.150:0/2309777196 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7950003520 con 0x7f796406dd30 2024-03-17T23:47:52.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.523+0000 7f7968c06640 1 -- 172.21.15.150:0/2309777196 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4236797000 0 0) 0x7f79640cf690 con 0x7f79640cd420 2024-03-17T23:47:52.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.523+0000 7f7968c06640 1 -- 172.21.15.150:0/2309777196 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79640cf3b0 con 0x7f79640cd420 2024-03-17T23:47:52.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.523+0000 7f7969c08640 1 -- 172.21.15.150:0/2309777196 >> v1:172.21.15.157:6789/0 conn(0x7f79640cd420 legacy=0x7f796415ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:52.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.523+0000 7f7969c08640 1 -- 172.21.15.150:0/2309777196 >> v1:172.21.15.150:6789/0 conn(0x7f796406dd30 legacy=0x7f79640ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:52.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.523+0000 7f7969c08640 1 -- 172.21.15.150:0/2309777196 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f796415c7a0 con 0x7f7964155320 2024-03-17T23:47:52.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.524+0000 7f7969c08640 1 --2- 172.21.15.150:0/2309777196 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7964155320 0x7f796415bfd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f79600097d0 tx=0x7f796002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a02fa7863082b03c server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:52.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.524+0000 7f7968c06640 1 -- 172.21.15.150:0/2309777196 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7960035020 con 0x7f7964155320 2024-03-17T23:47:52.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.524+0000 7f7968c06640 1 -- 172.21.15.150:0/2309777196 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7960031890 con 0x7f7964155320 2024-03-17T23:47:52.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.524+0000 7f7968c06640 1 -- 172.21.15.150:0/2309777196 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7960031c40 con 0x7f7964155320 2024-03-17T23:47:52.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.525+0000 7f796be6c640 1 -- 172.21.15.150:0/2309777196 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7964155320 msgr2=0x7f796415bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:52.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.525+0000 7f796be6c640 1 --2- 172.21.15.150:0/2309777196 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7964155320 0x7f796415bfd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f79600097d0 tx=0x7f796002b930 comp rx=0 tx=0).stop 2024-03-17T23:47:52.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.525+0000 7f796be6c640 1 -- 172.21.15.150:0/2309777196 shutdown_connections 2024-03-17T23:47:52.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.525+0000 7f796be6c640 1 --2- 172.21.15.150:0/2309777196 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7964155320 0x7f796415bfd0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:52.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.525+0000 7f796be6c640 1 -- 172.21.15.150:0/2309777196 >> 172.21.15.150:0/2309777196 conn(0x7f796406b190 msgr2=0x7f796406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:52.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.525+0000 7f796be6c640 1 -- 172.21.15.150:0/2309777196 shutdown_connections 2024-03-17T23:47:52.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.525+0000 7f796be6c640 1 -- 172.21.15.150:0/2309777196 wait complete. 2024-03-17T23:47:52.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.526+0000 7f796be6c640 1 Processor -- start 2024-03-17T23:47:52.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.526+0000 7f796be6c640 1 -- start start 2024-03-17T23:47:52.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.527+0000 7f796be6c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f796406dd30 0x7f796413f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:52.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.527+0000 7f796be6c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79640cd420 0x7f7964159da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:52.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.527+0000 7f796be6c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7964155320 0x7f796415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:52.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.527+0000 7f796be6c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7964155700 con 0x7f79640cd420 2024-03-17T23:47:52.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.527+0000 7f796be6c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f796406e190 con 0x7f796406dd30 2024-03-17T23:47:52.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.527+0000 7f796be6c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7964155880 con 0x7f7964155320 2024-03-17T23:47:52.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.527+0000 7f7969407640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79640cd420 0x7f7964159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:52.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.527+0000 7f7969407640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79640cd420 0x7f7964159da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50550/0 (socket says 172.21.15.150:50550) 2024-03-17T23:47:52.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.527+0000 7f7969407640 1 -- 172.21.15.150:0/2251079514 learned_addr learned my addr 172.21.15.150:0/2251079514 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:52.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.527+0000 7f7969c08640 1 --2- 172.21.15.150:0/2251079514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f796406dd30 0x7f796413f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:52.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.527+0000 7f796a409640 1 --2- 172.21.15.150:0/2251079514 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7964155320 0x7f796415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:52.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.527+0000 7f7969407640 1 -- 172.21.15.150:0/2251079514 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7964155320 msgr2=0x7f796415a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:52.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.527+0000 7f7969407640 1 --2- 172.21.15.150:0/2251079514 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7964155320 0x7f796415a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:52.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.527+0000 7f7969407640 1 -- 172.21.15.150:0/2251079514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f796406dd30 msgr2=0x7f796413f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:52.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.527+0000 7f7969407640 1 --2- 172.21.15.150:0/2251079514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f796406dd30 0x7f796413f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:52.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.527+0000 7f7969407640 1 -- 172.21.15.150:0/2251079514 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79641568b0 con 0x7f79640cd420 2024-03-17T23:47:52.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.527+0000 7f7969c08640 1 --2- 172.21.15.150:0/2251079514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f796406dd30 0x7f796413f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:52.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.527+0000 7f7969407640 1 --2- 172.21.15.150:0/2251079514 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79640cd420 0x7f7964159da0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f795400c500 tx=0x7f795400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:52.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.527+0000 7f795a7fc640 1 -- 172.21.15.150:0/2251079514 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7954017070 con 0x7f79640cd420 2024-03-17T23:47:52.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.527+0000 7f795a7fc640 1 -- 172.21.15.150:0/2251079514 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7954005ce0 con 0x7f79640cd420 2024-03-17T23:47:52.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.527+0000 7f796be6c640 1 -- 172.21.15.150:0/2251079514 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f796414d780 con 0x7f79640cd420 2024-03-17T23:47:52.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.527+0000 7f795a7fc640 1 -- 172.21.15.150:0/2251079514 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79540072f0 con 0x7f79640cd420 2024-03-17T23:47:52.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.527+0000 7f796be6c640 1 -- 172.21.15.150:0/2251079514 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f796414dd40 con 0x7f79640cd420 2024-03-17T23:47:52.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.528+0000 7f796be6c640 1 -- 172.21.15.150:0/2251079514 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79640681a0 con 0x7f79640cd420 2024-03-17T23:47:52.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.529+0000 7f795a7fc640 1 -- 172.21.15.150:0/2251079514 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f795401a070 con 0x7f79640cd420 2024-03-17T23:47:52.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.529+0000 7f795a7fc640 1 --2- 172.21.15.150:0/2251079514 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f794806bbd0 0x7f794806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:52.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.529+0000 7f795a7fc640 1 -- 172.21.15.150:0/2251079514 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f795408a140 con 0x7f79640cd420 2024-03-17T23:47:52.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.529+0000 7f7969c08640 1 --2- 172.21.15.150:0/2251079514 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f794806bbd0 0x7f794806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:52.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.530+0000 7f7969c08640 1 --2- 172.21.15.150:0/2251079514 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f794806bbd0 0x7f794806e090 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7960036000 tx=0x7f796003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:52.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.534+0000 7f795a7fc640 1 -- 172.21.15.150:0/2251079514 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7954058270 con 0x7f79640cd420 2024-03-17T23:47:52.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.707+0000 7f796be6c640 1 -- 172.21.15.150:0/2251079514 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7964071f20 con 0x7f79640cd420 2024-03-17T23:47:52.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.709+0000 7f795a7fc640 1 -- 172.21.15.150:0/2251079514 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18881 (secure 0 0 0) 0x7f795405c310 con 0x7f79640cd420 2024-03-17T23:47:52.710 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:52.710 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":28,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:37.031864+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.e","mappings":[{"from":7,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:52.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.711+0000 7f796be6c640 1 -- 172.21.15.150:0/2251079514 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f794806bbd0 msgr2=0x7f794806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:52.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.711+0000 7f796be6c640 1 --2- 172.21.15.150:0/2251079514 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f794806bbd0 0x7f794806e090 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7960036000 tx=0x7f796003c040 comp rx=0 tx=0).stop 2024-03-17T23:47:52.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.711+0000 7f796be6c640 1 -- 172.21.15.150:0/2251079514 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79640cd420 msgr2=0x7f7964159da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:52.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.711+0000 7f796be6c640 1 --2- 172.21.15.150:0/2251079514 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79640cd420 0x7f7964159da0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f795400c500 tx=0x7f795400c9c0 comp rx=0 tx=0).stop 2024-03-17T23:47:52.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.711+0000 7f796be6c640 1 -- 172.21.15.150:0/2251079514 shutdown_connections 2024-03-17T23:47:52.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.711+0000 7f796be6c640 1 --2- 172.21.15.150:0/2251079514 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7964155320 0x7f796415a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:52.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.711+0000 7f796be6c640 1 --2- 172.21.15.150:0/2251079514 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79640cd420 0x7f7964159da0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:52.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.711+0000 7f796be6c640 1 --2- 172.21.15.150:0/2251079514 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f794806bbd0 0x7f794806e090 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:52.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.711+0000 7f796be6c640 1 --2- 172.21.15.150:0/2251079514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f796406dd30 0x7f796413f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:52.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.711+0000 7f796be6c640 1 -- 172.21.15.150:0/2251079514 >> 172.21.15.150:0/2251079514 conn(0x7f796406b190 msgr2=0x7f7964154230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:52.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.711+0000 7f796be6c640 1 -- 172.21.15.150:0/2251079514 shutdown_connections 2024-03-17T23:47:52.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.711+0000 7f796be6c640 1 -- 172.21.15.150:0/2251079514 wait complete. 2024-03-17T23:47:52.723 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:46:53.610678+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:46:54.227335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-17T23:46:57.949235+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-17T23:47:52.724 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:47:52.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.826+0000 7fd24f097640 1 Processor -- start 2024-03-17T23:47:52.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.826+0000 7fd24f097640 1 -- start start 2024-03-17T23:47:52.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.826+0000 7fd24f097640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd248161520 0x7fd2480b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:52.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.826+0000 7fd24f097640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd2480b2bf0 0x7fd2480d05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:52.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.826+0000 7fd24f097640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd248161d30 con 0x7fd2480d0b10 2024-03-17T23:47:52.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.826+0000 7fd24f097640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fd2480b9bf0 con 0x7fd2480b2bf0 2024-03-17T23:47:52.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.826+0000 7fd24f097640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd248162940 con 0x7fd248161520 2024-03-17T23:47:52.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.827+0000 7fd247fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd2480b2bf0 0x7fd2480d05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:52.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.827+0000 7fd247fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd2480b2bf0 0x7fd2480d05d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50554/0 (socket says 172.21.15.150:50554) 2024-03-17T23:47:52.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.827+0000 7fd247fff640 1 -- 172.21.15.150:0/246967618 learned_addr learned my addr 172.21.15.150:0/246967618 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:52.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.827+0000 7fd24ce33640 1 --2- 172.21.15.150:0/246967618 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd248161520 0x7fd2480b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:52.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.827+0000 7fd247fff640 1 -- 172.21.15.150:0/246967618 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd248161520 msgr2=0x7fd2480b96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:52.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.827+0000 7fd247fff640 1 --2- 172.21.15.150:0/246967618 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd248161520 0x7fd2480b96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:52.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.827+0000 7fd247fff640 1 -- 172.21.15.150:0/246967618 >> v1:172.21.15.157:6789/0 conn(0x7fd2480d0b10 legacy=0x7fd2480b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:52.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.827+0000 7fd247fff640 1 -- 172.21.15.150:0/246967618 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd248162010 con 0x7fd2480b2bf0 2024-03-17T23:47:52.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.828+0000 7fd247fff640 1 --2- 172.21.15.150:0/246967618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2480b2bf0 0x7fd2480d05d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd234009870 tx=0x7fd23402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=179cdcd9985b7d3e server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:52.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.828+0000 7fd2477fe640 1 -- 172.21.15.150:0/246967618 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2340317c0 con 0x7fd2480b2bf0 2024-03-17T23:47:52.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.828+0000 7fd2477fe640 1 -- 172.21.15.150:0/246967618 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd234037070 con 0x7fd2480b2bf0 2024-03-17T23:47:52.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.828+0000 7fd2477fe640 1 -- 172.21.15.150:0/246967618 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd234031b80 con 0x7fd2480b2bf0 2024-03-17T23:47:52.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.828+0000 7fd24f097640 1 -- 172.21.15.150:0/246967618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2480b2bf0 msgr2=0x7fd2480d05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:52.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.828+0000 7fd24f097640 1 --2- 172.21.15.150:0/246967618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2480b2bf0 0x7fd2480d05d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd234009870 tx=0x7fd23402b860 comp rx=0 tx=0).stop 2024-03-17T23:47:52.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.828+0000 7fd24f097640 1 -- 172.21.15.150:0/246967618 shutdown_connections 2024-03-17T23:47:52.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.828+0000 7fd24f097640 1 --2- 172.21.15.150:0/246967618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2480b2bf0 0x7fd2480d05d0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:52.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.828+0000 7fd24f097640 1 --2- 172.21.15.150:0/246967618 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd248161520 0x7fd2480b96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:52.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.828+0000 7fd24f097640 1 -- 172.21.15.150:0/246967618 >> 172.21.15.150:0/246967618 conn(0x7fd24806ca00 msgr2=0x7fd24806ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:52.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.828+0000 7fd24f097640 1 -- 172.21.15.150:0/246967618 shutdown_connections 2024-03-17T23:47:52.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.828+0000 7fd24f097640 1 -- 172.21.15.150:0/246967618 wait complete. 2024-03-17T23:47:52.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.829+0000 7fd24f097640 1 Processor -- start 2024-03-17T23:47:52.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.829+0000 7fd24f097640 1 -- start start 2024-03-17T23:47:52.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.830+0000 7fd24f097640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2480b2bf0 0x7fd2481523c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:52.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.830+0000 7fd24f097640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2480d0b10 0x7fd2480b7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:52.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.830+0000 7fd24f097640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd248161520 0x7fd2480b7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:52.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.830+0000 7fd24f097640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd2481625d0 con 0x7fd248161520 2024-03-17T23:47:52.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.830+0000 7fd24f097640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd2480d0f60 con 0x7fd2480b2bf0 2024-03-17T23:47:52.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.830+0000 7fd24f097640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd248161900 con 0x7fd2480d0b10 2024-03-17T23:47:52.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.830+0000 7fd24d634640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd248161520 0x7fd2480b7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:52.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.830+0000 7fd24d634640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd248161520 0x7fd2480b7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50566/0 (socket says 172.21.15.150:50566) 2024-03-17T23:47:52.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.830+0000 7fd24d634640 1 -- 172.21.15.150:0/1605803876 learned_addr learned my addr 172.21.15.150:0/1605803876 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:52.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.830+0000 7fd247fff640 1 --2- 172.21.15.150:0/1605803876 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2480d0b10 0x7fd2480b7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:52.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.830+0000 7fd24ce33640 1 --2- 172.21.15.150:0/1605803876 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2480b2bf0 0x7fd2481523c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:52.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.831+0000 7fd24d634640 1 -- 172.21.15.150:0/1605803876 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2480d0b10 msgr2=0x7fd2480b7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:52.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.831+0000 7fd24d634640 1 --2- 172.21.15.150:0/1605803876 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2480d0b10 0x7fd2480b7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:52.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.831+0000 7fd24d634640 1 -- 172.21.15.150:0/1605803876 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2480b2bf0 msgr2=0x7fd2481523c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:52.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.831+0000 7fd24d634640 1 --2- 172.21.15.150:0/1605803876 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2480b2bf0 0x7fd2481523c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:52.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.831+0000 7fd24d634640 1 -- 172.21.15.150:0/1605803876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd248155c50 con 0x7fd248161520 2024-03-17T23:47:52.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.831+0000 7fd24d634640 1 --2- 172.21.15.150:0/1605803876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd248161520 0x7fd2480b7cc0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd24000c6e0 tx=0x7fd24000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:52.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.831+0000 7fd2457fa640 1 -- 172.21.15.150:0/1605803876 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd240003b70 con 0x7fd248161520 2024-03-17T23:47:52.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.831+0000 7fd2457fa640 1 -- 172.21.15.150:0/1605803876 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd240003d30 con 0x7fd248161520 2024-03-17T23:47:52.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.831+0000 7fd24f097640 1 -- 172.21.15.150:0/1605803876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd248155040 con 0x7fd248161520 2024-03-17T23:47:52.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.831+0000 7fd2457fa640 1 -- 172.21.15.150:0/1605803876 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd240005310 con 0x7fd248161520 2024-03-17T23:47:52.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.831+0000 7fd24f097640 1 -- 172.21.15.150:0/1605803876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd248155600 con 0x7fd248161520 2024-03-17T23:47:52.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.832+0000 7fd24f097640 1 -- 172.21.15.150:0/1605803876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2480681a0 con 0x7fd248161520 2024-03-17T23:47:52.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.832+0000 7fd2457fa640 1 -- 172.21.15.150:0/1605803876 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd2400054b0 con 0x7fd248161520 2024-03-17T23:47:52.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.833+0000 7fd2457fa640 1 --2- 172.21.15.150:0/1605803876 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd22c06bbd0 0x7fd22c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:52.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.833+0000 7fd2457fa640 1 -- 172.21.15.150:0/1605803876 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fd24008ab60 con 0x7fd248161520 2024-03-17T23:47:52.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.833+0000 7fd24ce33640 1 --2- 172.21.15.150:0/1605803876 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd22c06bbd0 0x7fd22c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:52.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.833+0000 7fd24ce33640 1 --2- 172.21.15.150:0/1605803876 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd22c06bbd0 0x7fd22c06e090 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd23c009b40 tx=0x7fd23c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:52.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:52.837+0000 7fd2457fa640 1 -- 172.21.15.150:0/1605803876 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd240058c00 con 0x7fd248161520 2024-03-17T23:47:53.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.010+0000 7fd24f097640 1 -- 172.21.15.150:0/1605803876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd2480d19c0 con 0x7fd248161520 2024-03-17T23:47:53.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.010+0000 7fd2457fa640 1 -- 172.21.15.150:0/1605803876 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7fd24005cca0 con 0x7fd248161520 2024-03-17T23:47:53.012 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:47:53.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.012+0000 7fd24f097640 1 -- 172.21.15.150:0/1605803876 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd22c06bbd0 msgr2=0x7fd22c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:53.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.012+0000 7fd24f097640 1 --2- 172.21.15.150:0/1605803876 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd22c06bbd0 0x7fd22c06e090 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd23c009b40 tx=0x7fd23c009190 comp rx=0 tx=0).stop 2024-03-17T23:47:53.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.012+0000 7fd24f097640 1 -- 172.21.15.150:0/1605803876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd248161520 msgr2=0x7fd2480b7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:53.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.012+0000 7fd24f097640 1 --2- 172.21.15.150:0/1605803876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd248161520 0x7fd2480b7cc0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd24000c6e0 tx=0x7fd24000cba0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.012+0000 7fd24f097640 1 -- 172.21.15.150:0/1605803876 shutdown_connections 2024-03-17T23:47:53.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.012+0000 7fd24f097640 1 --2- 172.21.15.150:0/1605803876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd248161520 0x7fd2480b7cc0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.012+0000 7fd24f097640 1 --2- 172.21.15.150:0/1605803876 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2480d0b10 0x7fd2480b7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.012+0000 7fd24f097640 1 --2- 172.21.15.150:0/1605803876 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd22c06bbd0 0x7fd22c06e090 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.012+0000 7fd24f097640 1 --2- 172.21.15.150:0/1605803876 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2480b2bf0 0x7fd2481523c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.012+0000 7fd24f097640 1 -- 172.21.15.150:0/1605803876 >> 172.21.15.150:0/1605803876 conn(0x7fd24806ca00 msgr2=0x7fd2480d2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:53.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.012+0000 7fd24f097640 1 -- 172.21.15.150:0/1605803876 shutdown_connections 2024-03-17T23:47:53.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.012+0000 7fd24f097640 1 -- 172.21.15.150:0/1605803876 wait complete. 2024-03-17T23:47:53.026 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:47:53.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.128+0000 7ff7b331d640 1 Processor -- start 2024-03-17T23:47:53.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.128+0000 7ff7b331d640 1 -- start start 2024-03-17T23:47:53.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.129+0000 7ff7b331d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff7ac0d0ce0 0x7ff7ac0b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:53.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.129+0000 7ff7b331d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff7ac0b2bf0 0x7ff7ac0d1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:53.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.129+0000 7ff7b331d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7ac161d30 con 0x7ff7ac161520 2024-03-17T23:47:53.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.129+0000 7ff7b331d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7ff7ac0b9bf0 con 0x7ff7ac0b2bf0 2024-03-17T23:47:53.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.129+0000 7ff7b331d640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff7ac162940 con 0x7ff7ac0d0ce0 2024-03-17T23:47:53.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.129+0000 7ff7b18ba640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff7ac0b2bf0 0x7ff7ac0d1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:53.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.129+0000 7ff7b10b9640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7ff7ac161520 0x7ff7ac0d05d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:39086/0 (socket says 172.21.15.150:39086) 2024-03-17T23:47:53.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.129+0000 7ff7b10b9640 1 -- 172.21.15.150:0/4041479754 learned_addr learned my addr 172.21.15.150:0/4041479754 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:53.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.129+0000 7ff7b08b8640 1 --2- 172.21.15.150:0/4041479754 >> v2:172.21.15.157:3300/0 conn(0x7ff7ac0d0ce0 0x7ff7ac0b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:53.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.130+0000 7ff7a3fff640 1 -- 172.21.15.150:0/4041479754 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 376800672 0 0) 0x7ff7ac161d30 con 0x7ff7ac161520 2024-03-17T23:47:53.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.130+0000 7ff7a3fff640 1 -- 172.21.15.150:0/4041479754 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff798003520 con 0x7ff7ac161520 2024-03-17T23:47:53.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.130+0000 7ff7b18ba640 1 -- 172.21.15.150:0/4041479754 >> v2:172.21.15.157:3300/0 conn(0x7ff7ac0d0ce0 msgr2=0x7ff7ac0b96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:53.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.130+0000 7ff7b18ba640 1 --2- 172.21.15.150:0/4041479754 >> v2:172.21.15.157:3300/0 conn(0x7ff7ac0d0ce0 0x7ff7ac0b96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.130+0000 7ff7b18ba640 1 -- 172.21.15.150:0/4041479754 >> v1:172.21.15.157:6789/0 conn(0x7ff7ac161520 legacy=0x7ff7ac0d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:53.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.130+0000 7ff7b18ba640 1 -- 172.21.15.150:0/4041479754 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7ac162010 con 0x7ff7ac0b2bf0 2024-03-17T23:47:53.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.130+0000 7ff7b18ba640 1 --2- 172.21.15.150:0/4041479754 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7ac0b2bf0 0x7ff7ac0d1fa0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac0b6c90 tx=0x7ff7a402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5191422b68dabd57 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:53.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.131+0000 7ff7a3fff640 1 -- 172.21.15.150:0/4041479754 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a4038070 con 0x7ff7ac0b2bf0 2024-03-17T23:47:53.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.131+0000 7ff7a3fff640 1 -- 172.21.15.150:0/4041479754 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7a4031710 con 0x7ff7ac0b2bf0 2024-03-17T23:47:53.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.131+0000 7ff7a3fff640 1 -- 172.21.15.150:0/4041479754 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a4031ae0 con 0x7ff7ac0b2bf0 2024-03-17T23:47:53.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.131+0000 7ff7b331d640 1 -- 172.21.15.150:0/4041479754 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7ac0b2bf0 msgr2=0x7ff7ac0d1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:53.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.131+0000 7ff7b331d640 1 --2- 172.21.15.150:0/4041479754 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7ac0b2bf0 0x7ff7ac0d1fa0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac0b6c90 tx=0x7ff7a402b7b0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.131+0000 7ff7b331d640 1 -- 172.21.15.150:0/4041479754 shutdown_connections 2024-03-17T23:47:53.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.131+0000 7ff7b331d640 1 --2- 172.21.15.150:0/4041479754 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7ac0b2bf0 0x7ff7ac0d1fa0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.131+0000 7ff7b331d640 1 --2- 172.21.15.150:0/4041479754 >> v2:172.21.15.157:3300/0 conn(0x7ff7ac0d0ce0 0x7ff7ac0b96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.131+0000 7ff7b331d640 1 -- 172.21.15.150:0/4041479754 >> 172.21.15.150:0/4041479754 conn(0x7ff7ac06ca00 msgr2=0x7ff7ac06ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:53.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.131+0000 7ff7b331d640 1 -- 172.21.15.150:0/4041479754 shutdown_connections 2024-03-17T23:47:53.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.131+0000 7ff7b331d640 1 -- 172.21.15.150:0/4041479754 wait complete. 2024-03-17T23:47:53.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.132+0000 7ff7b331d640 1 Processor -- start 2024-03-17T23:47:53.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.132+0000 7ff7b331d640 1 -- start start 2024-03-17T23:47:53.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.133+0000 7ff7b331d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7ac0b2bf0 0x7ff7ac0cc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:53.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.133+0000 7ff7b331d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7ac0d0ce0 0x7ff7ac0cc720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:53.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.133+0000 7ff7b331d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7ac161520 0x7ff7ac15a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:53.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.133+0000 7ff7b331d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ac161900 con 0x7ff7ac161520 2024-03-17T23:47:53.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.133+0000 7ff7b331d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ac1625d0 con 0x7ff7ac0b2bf0 2024-03-17T23:47:53.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.133+0000 7ff7b331d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff7ac0d24e0 con 0x7ff7ac0d0ce0 2024-03-17T23:47:53.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.133+0000 7ff7b18ba640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7ac161520 0x7ff7ac15a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:53.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.133+0000 7ff7b18ba640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7ac161520 0x7ff7ac15a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50588/0 (socket says 172.21.15.150:50588) 2024-03-17T23:47:53.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.133+0000 7ff7b18ba640 1 -- 172.21.15.150:0/3384838984 learned_addr learned my addr 172.21.15.150:0/3384838984 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:53.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.133+0000 7ff7b08b8640 1 --2- 172.21.15.150:0/3384838984 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7ac0d0ce0 0x7ff7ac0cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:53.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.133+0000 7ff7b10b9640 1 --2- 172.21.15.150:0/3384838984 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7ac0b2bf0 0x7ff7ac0cc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:53.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.134+0000 7ff7b18ba640 1 -- 172.21.15.150:0/3384838984 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7ac0d0ce0 msgr2=0x7ff7ac0cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:53.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.134+0000 7ff7b18ba640 1 --2- 172.21.15.150:0/3384838984 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7ac0d0ce0 0x7ff7ac0cc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.134+0000 7ff7b18ba640 1 -- 172.21.15.150:0/3384838984 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7ac0b2bf0 msgr2=0x7ff7ac0cc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:53.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.134+0000 7ff7b18ba640 1 --2- 172.21.15.150:0/3384838984 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7ac0b2bf0 0x7ff7ac0cc1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.134+0000 7ff7b18ba640 1 -- 172.21.15.150:0/3384838984 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7ac0cb140 con 0x7ff7ac161520 2024-03-17T23:47:53.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.134+0000 7ff7b18ba640 1 --2- 172.21.15.150:0/3384838984 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7ac161520 0x7ff7ac15a730 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4009890 tx=0x7ff7a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:53.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.134+0000 7ff7a1ffb640 1 -- 172.21.15.150:0/3384838984 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a4042030 con 0x7ff7ac161520 2024-03-17T23:47:53.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.134+0000 7ff7a1ffb640 1 -- 172.21.15.150:0/3384838984 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7a403d670 con 0x7ff7ac161520 2024-03-17T23:47:53.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.134+0000 7ff7b331d640 1 -- 172.21.15.150:0/3384838984 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7ac0cb420 con 0x7ff7ac161520 2024-03-17T23:47:53.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.134+0000 7ff7a1ffb640 1 -- 172.21.15.150:0/3384838984 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a4038040 con 0x7ff7ac161520 2024-03-17T23:47:53.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.134+0000 7ff7b331d640 1 -- 172.21.15.150:0/3384838984 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7ac0cb9e0 con 0x7ff7ac161520 2024-03-17T23:47:53.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.135+0000 7ff7b331d640 1 -- 172.21.15.150:0/3384838984 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7ac0681a0 con 0x7ff7ac161520 2024-03-17T23:47:53.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.136+0000 7ff7a1ffb640 1 -- 172.21.15.150:0/3384838984 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff7a4045070 con 0x7ff7ac161520 2024-03-17T23:47:53.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.136+0000 7ff7a1ffb640 1 --2- 172.21.15.150:0/3384838984 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff79006bc00 0x7ff79006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:53.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.136+0000 7ff7a1ffb640 1 -- 172.21.15.150:0/3384838984 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7ff7a40ac630 con 0x7ff7ac161520 2024-03-17T23:47:53.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.136+0000 7ff7b10b9640 1 --2- 172.21.15.150:0/3384838984 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff79006bc00 0x7ff79006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:53.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.136+0000 7ff7b10b9640 1 --2- 172.21.15.150:0/3384838984 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff79006bc00 0x7ff79006e0c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8005c00 tx=0x7ff7a8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:53.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.140+0000 7ff7a1ffb640 1 -- 172.21.15.150:0/3384838984 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff7a407a760 con 0x7ff7ac161520 2024-03-17T23:47:53.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.312+0000 7ff7b331d640 1 -- 172.21.15.150:0/3384838984 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff7ac0d19c0 con 0x7ff7ac161520 2024-03-17T23:47:53.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.312+0000 7ff7a1ffb640 1 -- 172.21.15.150:0/3384838984 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7ff7a407e800 con 0x7ff7ac161520 2024-03-17T23:47:53.314 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:53.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.314+0000 7ff7b331d640 1 -- 172.21.15.150:0/3384838984 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff79006bc00 msgr2=0x7ff79006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:53.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.314+0000 7ff7b331d640 1 --2- 172.21.15.150:0/3384838984 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff79006bc00 0x7ff79006e0c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8005c00 tx=0x7ff7a8009040 comp rx=0 tx=0).stop 2024-03-17T23:47:53.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.314+0000 7ff7b331d640 1 -- 172.21.15.150:0/3384838984 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7ac161520 msgr2=0x7ff7ac15a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:53.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.314+0000 7ff7b331d640 1 --2- 172.21.15.150:0/3384838984 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7ac161520 0x7ff7ac15a730 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4009890 tx=0x7ff7a40046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.314+0000 7ff7b331d640 1 -- 172.21.15.150:0/3384838984 shutdown_connections 2024-03-17T23:47:53.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.314+0000 7ff7b331d640 1 --2- 172.21.15.150:0/3384838984 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7ac161520 0x7ff7ac15a730 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.314+0000 7ff7b331d640 1 --2- 172.21.15.150:0/3384838984 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7ac0d0ce0 0x7ff7ac0cc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.314+0000 7ff7b331d640 1 --2- 172.21.15.150:0/3384838984 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff79006bc00 0x7ff79006e0c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.314+0000 7ff7b331d640 1 --2- 172.21.15.150:0/3384838984 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff7ac0b2bf0 0x7ff7ac0cc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.314+0000 7ff7b331d640 1 -- 172.21.15.150:0/3384838984 >> 172.21.15.150:0/3384838984 conn(0x7ff7ac06ca00 msgr2=0x7ff7ac0ce7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:53.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.314+0000 7ff7b331d640 1 -- 172.21.15.150:0/3384838984 shutdown_connections 2024-03-17T23:47:53.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.314+0000 7ff7b331d640 1 -- 172.21.15.150:0/3384838984 wait complete. 2024-03-17T23:47:53.327 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:47:53.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.428+0000 7f3dd7c97640 1 Processor -- start 2024-03-17T23:47:53.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.429+0000 7f3dd7c97640 1 -- start start 2024-03-17T23:47:53.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.429+0000 7f3dd7c97640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3dd009a4c0 0x7f3dd0168e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:53.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.429+0000 7f3dd7c97640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dd00d0040 con 0x7f3dd015e0a0 2024-03-17T23:47:53.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.429+0000 7f3dd7c97640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dd015edb0 con 0x7f3dd006c2a0 2024-03-17T23:47:53.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.429+0000 7f3dd7c97640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3dd015eb90 con 0x7f3dd009a4c0 2024-03-17T23:47:53.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.429+0000 7f3dd5a33640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f3dd015e0a0 0x7f3dd015e480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49870/0 (socket says 172.21.15.150:49870) 2024-03-17T23:47:53.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.430+0000 7f3dd5a33640 1 -- 172.21.15.150:0/950484624 learned_addr learned my addr 172.21.15.150:0/950484624 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:53.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.430+0000 7f3dd6234640 1 --2- 172.21.15.150:0/950484624 >> v2:172.21.15.157:3300/0 conn(0x7f3dd009a4c0 0x7f3dd0168e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:53.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.430+0000 7f3dd4a31640 1 -- 172.21.15.150:0/950484624 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2187522873 0 0) 0x7f3dd00d0040 con 0x7f3dd015e0a0 2024-03-17T23:47:53.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.430+0000 7f3dd4a31640 1 -- 172.21.15.150:0/950484624 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3dbc003520 con 0x7f3dd015e0a0 2024-03-17T23:47:53.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.430+0000 7f3dd6234640 1 -- 172.21.15.150:0/950484624 >> v1:172.21.15.157:6789/0 conn(0x7f3dd006c2a0 legacy=0x7f3dd0099db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:53.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.430+0000 7f3dd4a31640 1 -- 172.21.15.150:0/950484624 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4211616864 0 0) 0x7f3dd015edb0 con 0x7f3dd006c2a0 2024-03-17T23:47:53.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.430+0000 7f3dd6234640 1 -- 172.21.15.150:0/950484624 >> v1:172.21.15.150:6789/0 conn(0x7f3dd015e0a0 legacy=0x7f3dd015e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:53.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.430+0000 7f3dd6234640 1 -- 172.21.15.150:0/950484624 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dd015f090 con 0x7f3dd009a4c0 2024-03-17T23:47:53.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.431+0000 7f3dd6234640 1 --2- 172.21.15.150:0/950484624 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3dd009a4c0 0x7f3dd0168e00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8009d90 tx=0x7f3dc802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ecff7de93f5874 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:53.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.431+0000 7f3dd4a31640 1 -- 172.21.15.150:0/950484624 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc8039070 con 0x7f3dd009a4c0 2024-03-17T23:47:53.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.431+0000 7f3dd4a31640 1 -- 172.21.15.150:0/950484624 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dc802cd00 con 0x7f3dd009a4c0 2024-03-17T23:47:53.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.431+0000 7f3dd4a31640 1 -- 172.21.15.150:0/950484624 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc8034660 con 0x7f3dd009a4c0 2024-03-17T23:47:53.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.431+0000 7f3dd7c97640 1 -- 172.21.15.150:0/950484624 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3dd009a4c0 msgr2=0x7f3dd0168e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:53.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.431+0000 7f3dd7c97640 1 --2- 172.21.15.150:0/950484624 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3dd009a4c0 0x7f3dd0168e00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8009d90 tx=0x7f3dc802bf50 comp rx=0 tx=0).stop 2024-03-17T23:47:53.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.431+0000 7f3dd7c97640 1 -- 172.21.15.150:0/950484624 shutdown_connections 2024-03-17T23:47:53.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.431+0000 7f3dd7c97640 1 --2- 172.21.15.150:0/950484624 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3dd009a4c0 0x7f3dd0168e00 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.431+0000 7f3dd7c97640 1 -- 172.21.15.150:0/950484624 >> 172.21.15.150:0/950484624 conn(0x7f3dd00b36c0 msgr2=0x7f3dd00b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:53.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.431+0000 7f3dd7c97640 1 -- 172.21.15.150:0/950484624 shutdown_connections 2024-03-17T23:47:53.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.431+0000 7f3dd7c97640 1 -- 172.21.15.150:0/950484624 wait complete. 2024-03-17T23:47:53.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.433+0000 7f3dd7c97640 1 Processor -- start 2024-03-17T23:47:53.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.433+0000 7f3dd7c97640 1 -- start start 2024-03-17T23:47:53.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.433+0000 7f3dd7c97640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3dd006c2a0 0x7f3dd00b7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:53.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.433+0000 7f3dd7c97640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3dd009a4c0 0x7f3dd009c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:53.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.433+0000 7f3dd7c97640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3dd015e0a0 0x7f3dd015a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:53.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.433+0000 7f3dd7c97640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd00d1530 con 0x7f3dd015e0a0 2024-03-17T23:47:53.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.433+0000 7f3dd7c97640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd015f650 con 0x7f3dd006c2a0 2024-03-17T23:47:53.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.433+0000 7f3dd7c97640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3dd00d16b0 con 0x7f3dd009a4c0 2024-03-17T23:47:53.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.433+0000 7f3dd6234640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3dd015e0a0 0x7f3dd015a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:53.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.433+0000 7f3dd6234640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3dd015e0a0 0x7f3dd015a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50602/0 (socket says 172.21.15.150:50602) 2024-03-17T23:47:53.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.433+0000 7f3dd6234640 1 -- 172.21.15.150:0/102508774 learned_addr learned my addr 172.21.15.150:0/102508774 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:53.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.433+0000 7f3dd5a33640 1 --2- 172.21.15.150:0/102508774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3dd006c2a0 0x7f3dd00b7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:53.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.433+0000 7f3dd5232640 1 --2- 172.21.15.150:0/102508774 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3dd009a4c0 0x7f3dd009c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:53.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.434+0000 7f3dd6234640 1 -- 172.21.15.150:0/102508774 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3dd009a4c0 msgr2=0x7f3dd009c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:53.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.434+0000 7f3dd6234640 1 --2- 172.21.15.150:0/102508774 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3dd009a4c0 0x7f3dd009c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.434+0000 7f3dd6234640 1 -- 172.21.15.150:0/102508774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3dd006c2a0 msgr2=0x7f3dd00b7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:53.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.434+0000 7f3dd6234640 1 --2- 172.21.15.150:0/102508774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3dd006c2a0 0x7f3dd00b7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.434+0000 7f3dd6234640 1 -- 172.21.15.150:0/102508774 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dd00b4740 con 0x7f3dd015e0a0 2024-03-17T23:47:53.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.434+0000 7f3dd6234640 1 --2- 172.21.15.150:0/102508774 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3dd015e0a0 0x7f3dd015a380 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8009d90 tx=0x7f3dc8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:53.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.434+0000 7f3dc67fc640 1 -- 172.21.15.150:0/102508774 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc8039050 con 0x7f3dd015e0a0 2024-03-17T23:47:53.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.434+0000 7f3dc67fc640 1 -- 172.21.15.150:0/102508774 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dc803a040 con 0x7f3dd015e0a0 2024-03-17T23:47:53.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.434+0000 7f3dd7c97640 1 -- 172.21.15.150:0/102508774 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dd00b4a20 con 0x7f3dd015e0a0 2024-03-17T23:47:53.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.434+0000 7f3dc67fc640 1 -- 172.21.15.150:0/102508774 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc8009450 con 0x7f3dd015e0a0 2024-03-17T23:47:53.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.434+0000 7f3dd7c97640 1 -- 172.21.15.150:0/102508774 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dd00b4fe0 con 0x7f3dd015e0a0 2024-03-17T23:47:53.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.435+0000 7f3dd7c97640 1 -- 172.21.15.150:0/102508774 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dd00688b0 con 0x7f3dd015e0a0 2024-03-17T23:47:53.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.436+0000 7f3dc67fc640 1 -- 172.21.15.150:0/102508774 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3dc8047020 con 0x7f3dd015e0a0 2024-03-17T23:47:53.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.436+0000 7f3dc67fc640 1 --2- 172.21.15.150:0/102508774 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3db406bbd0 0x7f3db406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:53.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.436+0000 7f3dd5a33640 1 --2- 172.21.15.150:0/102508774 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3db406bbd0 0x7f3db406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:53.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.436+0000 7f3dc67fc640 1 -- 172.21.15.150:0/102508774 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f3dc80ad010 con 0x7f3dd015e0a0 2024-03-17T23:47:53.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.436+0000 7f3dd5a33640 1 --2- 172.21.15.150:0/102508774 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3db406bbd0 0x7f3db406e090 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc005880 tx=0x7f3dcc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:53.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.440+0000 7f3dc67fc640 1 -- 172.21.15.150:0/102508774 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3dc804e020 con 0x7f3dd015e0a0 2024-03-17T23:47:53.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.613+0000 7f3dd7c97640 1 -- 172.21.15.150:0/102508774 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3dd00b51b0 con 0x7f3dd015e0a0 2024-03-17T23:47:53.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.614+0000 7f3dc67fc640 1 -- 172.21.15.150:0/102508774 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7f3dc807b030 con 0x7f3dd015e0a0 2024-03-17T23:47:53.615 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:47:53.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.616+0000 7f3dd7c97640 1 -- 172.21.15.150:0/102508774 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3db406bbd0 msgr2=0x7f3db406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:53.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.616+0000 7f3dd7c97640 1 --2- 172.21.15.150:0/102508774 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3db406bbd0 0x7f3db406e090 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc005880 tx=0x7f3dcc009040 comp rx=0 tx=0).stop 2024-03-17T23:47:53.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.616+0000 7f3dd7c97640 1 -- 172.21.15.150:0/102508774 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3dd015e0a0 msgr2=0x7f3dd015a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:53.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.616+0000 7f3dd7c97640 1 --2- 172.21.15.150:0/102508774 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3dd015e0a0 0x7f3dd015a380 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8009d90 tx=0x7f3dc8004a40 comp rx=0 tx=0).stop 2024-03-17T23:47:53.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.616+0000 7f3dd7c97640 1 -- 172.21.15.150:0/102508774 shutdown_connections 2024-03-17T23:47:53.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.616+0000 7f3dd7c97640 1 --2- 172.21.15.150:0/102508774 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3dd015e0a0 0x7f3dd015a380 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.616+0000 7f3dd7c97640 1 --2- 172.21.15.150:0/102508774 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3dd009a4c0 0x7f3dd009c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.616+0000 7f3dd7c97640 1 --2- 172.21.15.150:0/102508774 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3db406bbd0 0x7f3db406e090 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.616+0000 7f3dd7c97640 1 --2- 172.21.15.150:0/102508774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3dd006c2a0 0x7f3dd00b7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.616+0000 7f3dd7c97640 1 -- 172.21.15.150:0/102508774 >> 172.21.15.150:0/102508774 conn(0x7f3dd00b36c0 msgr2=0x7f3dd00c8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:53.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.616+0000 7f3dd7c97640 1 -- 172.21.15.150:0/102508774 shutdown_connections 2024-03-17T23:47:53.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.616+0000 7f3dd7c97640 1 -- 172.21.15.150:0/102508774 wait complete. 2024-03-17T23:47:53.628 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-17T23:47:53.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.730+0000 7fd8f0b2f640 1 Processor -- start 2024-03-17T23:47:53.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.730+0000 7fd8f0b2f640 1 -- start start 2024-03-17T23:47:53.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.731+0000 7fd8f0b2f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd8ec1514f0 0x7fd8ec15bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:53.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.731+0000 7fd8f0b2f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd8ec15ab90 0x7fd8ec06f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:53.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.731+0000 7fd8f0b2f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8ec0ce790 con 0x7fd8ec06e5c0 2024-03-17T23:47:53.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.731+0000 7fd8f0b2f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd8ec06fba0 con 0x7fd8ec1514f0 2024-03-17T23:47:53.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.731+0000 7fd8f0b2f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd8ec15b050 con 0x7fd8ec15ab90 2024-03-17T23:47:53.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.731+0000 7fd8ead9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd8ec06e5c0 0x7fd8ec06ea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49876/0 (socket says 172.21.15.150:49876) 2024-03-17T23:47:53.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.731+0000 7fd8ead9d640 1 -- 172.21.15.150:0/1000531588 learned_addr learned my addr 172.21.15.150:0/1000531588 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:53.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.731+0000 7fd8e9d9b640 1 --2- 172.21.15.150:0/1000531588 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd8ec15ab90 0x7fd8ec06f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:53.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.731+0000 7fd8ea59c640 1 --2- 172.21.15.150:0/1000531588 >> v2:172.21.15.157:3300/0 conn(0x7fd8ec1514f0 0x7fd8ec15bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:53.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.732+0000 7fd8e959a640 1 -- 172.21.15.150:0/1000531588 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 629168 0 0) 0x7fd8e4001110 con 0x7fd8ec06e5c0 2024-03-17T23:47:53.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.732+0000 7fd8e9d9b640 1 -- 172.21.15.150:0/1000531588 >> v2:172.21.15.157:3300/0 conn(0x7fd8ec1514f0 msgr2=0x7fd8ec15bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:53.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.732+0000 7fd8e9d9b640 1 --2- 172.21.15.150:0/1000531588 >> v2:172.21.15.157:3300/0 conn(0x7fd8ec1514f0 0x7fd8ec15bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.732+0000 7fd8e9d9b640 1 -- 172.21.15.150:0/1000531588 >> v1:172.21.15.150:6789/0 conn(0x7fd8ec06e5c0 legacy=0x7fd8ec06ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:53.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.732+0000 7fd8e9d9b640 1 -- 172.21.15.150:0/1000531588 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8ec15b330 con 0x7fd8ec15ab90 2024-03-17T23:47:53.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.732+0000 7fd8e9d9b640 1 --2- 172.21.15.150:0/1000531588 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd8ec15ab90 0x7fd8ec06f660 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00097a0 tx=0x7fd8e002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b31631169f8b78e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:53.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.732+0000 7fd8e959a640 1 -- 172.21.15.150:0/1000531588 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0035020 con 0x7fd8ec15ab90 2024-03-17T23:47:53.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.733+0000 7fd8e959a640 1 -- 172.21.15.150:0/1000531588 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8e0031830 con 0x7fd8ec15ab90 2024-03-17T23:47:53.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.733+0000 7fd8e959a640 1 -- 172.21.15.150:0/1000531588 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0031c00 con 0x7fd8ec15ab90 2024-03-17T23:47:53.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.733+0000 7fd8f0b2f640 1 -- 172.21.15.150:0/1000531588 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd8ec15ab90 msgr2=0x7fd8ec06f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:53.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.733+0000 7fd8f0b2f640 1 --2- 172.21.15.150:0/1000531588 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd8ec15ab90 0x7fd8ec06f660 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00097a0 tx=0x7fd8e002b8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.733+0000 7fd8f0b2f640 1 -- 172.21.15.150:0/1000531588 shutdown_connections 2024-03-17T23:47:53.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.733+0000 7fd8f0b2f640 1 --2- 172.21.15.150:0/1000531588 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd8ec15ab90 0x7fd8ec06f660 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.733+0000 7fd8f0b2f640 1 --2- 172.21.15.150:0/1000531588 >> v2:172.21.15.157:3300/0 conn(0x7fd8ec1514f0 0x7fd8ec15bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.733+0000 7fd8f0b2f640 1 -- 172.21.15.150:0/1000531588 >> 172.21.15.150:0/1000531588 conn(0x7fd8ec06b190 msgr2=0x7fd8ec06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:53.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.733+0000 7fd8f0b2f640 1 -- 172.21.15.150:0/1000531588 shutdown_connections 2024-03-17T23:47:53.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.733+0000 7fd8f0b2f640 1 -- 172.21.15.150:0/1000531588 wait complete. 2024-03-17T23:47:53.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.734+0000 7fd8f0b2f640 1 Processor -- start 2024-03-17T23:47:53.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.735+0000 7fd8f0b2f640 1 -- start start 2024-03-17T23:47:53.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.735+0000 7fd8f0b2f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8ec06e5c0 0x7fd8ec14b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:53.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.735+0000 7fd8f0b2f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd8ec1514f0 0x7fd8ec140930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:53.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.735+0000 7fd8f0b2f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8ec15ab90 0x7fd8ec149720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:53.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.735+0000 7fd8f0b2f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ec0af510 con 0x7fd8ec15ab90 2024-03-17T23:47:53.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.735+0000 7fd8f0b2f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ec15b8f0 con 0x7fd8ec06e5c0 2024-03-17T23:47:53.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.735+0000 7fd8f0b2f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd8ec1518d0 con 0x7fd8ec1514f0 2024-03-17T23:47:53.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.735+0000 7fd8ead9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8ec15ab90 0x7fd8ec149720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:53.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.735+0000 7fd8ead9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8ec15ab90 0x7fd8ec149720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50618/0 (socket says 172.21.15.150:50618) 2024-03-17T23:47:53.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.735+0000 7fd8ead9d640 1 -- 172.21.15.150:0/2349731631 learned_addr learned my addr 172.21.15.150:0/2349731631 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:53.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.735+0000 7fd8ea59c640 1 --2- 172.21.15.150:0/2349731631 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8ec06e5c0 0x7fd8ec14b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:53.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.735+0000 7fd8e9d9b640 1 --2- 172.21.15.150:0/2349731631 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd8ec1514f0 0x7fd8ec140930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:53.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.736+0000 7fd8ead9d640 1 -- 172.21.15.150:0/2349731631 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd8ec1514f0 msgr2=0x7fd8ec140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:53.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.736+0000 7fd8ead9d640 1 --2- 172.21.15.150:0/2349731631 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd8ec1514f0 0x7fd8ec140930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.736+0000 7fd8ead9d640 1 -- 172.21.15.150:0/2349731631 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8ec06e5c0 msgr2=0x7fd8ec14b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:53.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.736+0000 7fd8ead9d640 1 --2- 172.21.15.150:0/2349731631 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8ec06e5c0 0x7fd8ec14b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.736+0000 7fd8ead9d640 1 -- 172.21.15.150:0/2349731631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8ec140e70 con 0x7fd8ec15ab90 2024-03-17T23:47:53.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.736+0000 7fd8ea59c640 1 --2- 172.21.15.150:0/2349731631 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8ec06e5c0 0x7fd8ec14b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:53.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.736+0000 7fd8ead9d640 1 --2- 172.21.15.150:0/2349731631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8ec15ab90 0x7fd8ec149720 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd8e400c7b0 tx=0x7fd8e400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:53.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.736+0000 7fd8daffd640 1 -- 172.21.15.150:0/2349731631 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e4018070 con 0x7fd8ec15ab90 2024-03-17T23:47:53.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.736+0000 7fd8daffd640 1 -- 172.21.15.150:0/2349731631 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8e40070d0 con 0x7fd8ec15ab90 2024-03-17T23:47:53.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.736+0000 7fd8f0b2f640 1 -- 172.21.15.150:0/2349731631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8ec14aaa0 con 0x7fd8ec15ab90 2024-03-17T23:47:53.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.736+0000 7fd8daffd640 1 -- 172.21.15.150:0/2349731631 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e40074c0 con 0x7fd8ec15ab90 2024-03-17T23:47:53.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.736+0000 7fd8f0b2f640 1 -- 172.21.15.150:0/2349731631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8ec149f20 con 0x7fd8ec15ab90 2024-03-17T23:47:53.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.737+0000 7fd8f0b2f640 1 -- 172.21.15.150:0/2349731631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8ec0681a0 con 0x7fd8ec15ab90 2024-03-17T23:47:53.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.738+0000 7fd8daffd640 1 -- 172.21.15.150:0/2349731631 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd8e401b070 con 0x7fd8ec15ab90 2024-03-17T23:47:53.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.738+0000 7fd8daffd640 1 --2- 172.21.15.150:0/2349731631 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd8cc06bbd0 0x7fd8cc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:53.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.738+0000 7fd8daffd640 1 -- 172.21.15.150:0/2349731631 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fd8e4089fd0 con 0x7fd8ec15ab90 2024-03-17T23:47:53.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.738+0000 7fd8ea59c640 1 --2- 172.21.15.150:0/2349731631 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd8cc06bbd0 0x7fd8cc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:53.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.738+0000 7fd8ea59c640 1 --2- 172.21.15.150:0/2349731631 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd8cc06bbd0 0x7fd8cc06e090 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0b5100 tx=0x7fd8dc00b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:53.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.742+0000 7fd8daffd640 1 -- 172.21.15.150:0/2349731631 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd8ec0ce790 con 0x7fd8ec15ab90 2024-03-17T23:47:53.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.914+0000 7fd8f0b2f640 1 -- 172.21.15.150:0/2349731631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd8ec0ccec0 con 0x7fd8ec15ab90 2024-03-17T23:47:53.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.914+0000 7fd8daffd640 1 -- 172.21.15.150:0/2349731631 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7fd8e4058070 con 0x7fd8ec15ab90 2024-03-17T23:47:53.916 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:53.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.916+0000 7fd8f0b2f640 1 -- 172.21.15.150:0/2349731631 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd8cc06bbd0 msgr2=0x7fd8cc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:53.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.916+0000 7fd8f0b2f640 1 --2- 172.21.15.150:0/2349731631 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd8cc06bbd0 0x7fd8cc06e090 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0b5100 tx=0x7fd8dc00b040 comp rx=0 tx=0).stop 2024-03-17T23:47:53.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.916+0000 7fd8f0b2f640 1 -- 172.21.15.150:0/2349731631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8ec15ab90 msgr2=0x7fd8ec149720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:53.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.916+0000 7fd8f0b2f640 1 --2- 172.21.15.150:0/2349731631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8ec15ab90 0x7fd8ec149720 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd8e400c7b0 tx=0x7fd8e400cc70 comp rx=0 tx=0).stop 2024-03-17T23:47:53.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.916+0000 7fd8f0b2f640 1 -- 172.21.15.150:0/2349731631 shutdown_connections 2024-03-17T23:47:53.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.916+0000 7fd8f0b2f640 1 --2- 172.21.15.150:0/2349731631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8ec15ab90 0x7fd8ec149720 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.917+0000 7fd8f0b2f640 1 --2- 172.21.15.150:0/2349731631 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd8ec1514f0 0x7fd8ec140930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.917+0000 7fd8f0b2f640 1 --2- 172.21.15.150:0/2349731631 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd8cc06bbd0 0x7fd8cc06e090 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.917+0000 7fd8f0b2f640 1 --2- 172.21.15.150:0/2349731631 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8ec06e5c0 0x7fd8ec14b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:53.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.917+0000 7fd8f0b2f640 1 -- 172.21.15.150:0/2349731631 >> 172.21.15.150:0/2349731631 conn(0x7fd8ec06b190 msgr2=0x7fd8ec147620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:53.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.917+0000 7fd8f0b2f640 1 -- 172.21.15.150:0/2349731631 shutdown_connections 2024-03-17T23:47:53.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:53.917+0000 7fd8f0b2f640 1 -- 172.21.15.150:0/2349731631 wait complete. 2024-03-17T23:47:53.929 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-17T23:47:54.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.029+0000 7fe821427640 1 Processor -- start 2024-03-17T23:47:54.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.029+0000 7fe821427640 1 -- start start 2024-03-17T23:47:54.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.030+0000 7fe821427640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe81c15df40 0x7fe81c0b97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:54.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.030+0000 7fe821427640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe81c15e7d0 0x7fe81c0d2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:54.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.030+0000 7fe821427640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe81c15ec90 con 0x7fe81c15e320 2024-03-17T23:47:54.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.030+0000 7fe821427640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fe81c0b6d40 con 0x7fe81c15df40 2024-03-17T23:47:54.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.030+0000 7fe821427640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe81c15ef70 con 0x7fe81c15e7d0 2024-03-17T23:47:54.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.030+0000 7fe813fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fe81c15e320 0x7fe81c162160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49882/0 (socket says 172.21.15.150:49882) 2024-03-17T23:47:54.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.030+0000 7fe81affd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe81c15df40 0x7fe81c0b97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:54.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.030+0000 7fe813fff640 1 -- 172.21.15.150:0/4209930299 learned_addr learned my addr 172.21.15.150:0/4209930299 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:54.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.030+0000 7fe81b7fe640 1 --2- 172.21.15.150:0/4209930299 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe81c15e7d0 0x7fe81c0d2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:54.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.030+0000 7fe81a7fc640 1 -- 172.21.15.150:0/4209930299 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 543470225 0 0) 0x7fe81c15ec90 con 0x7fe81c15e320 2024-03-17T23:47:54.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.030+0000 7fe81a7fc640 1 -- 172.21.15.150:0/4209930299 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe808003520 con 0x7fe81c15e320 2024-03-17T23:47:54.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.030+0000 7fe81affd640 1 -- 172.21.15.150:0/4209930299 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe81c15e7d0 msgr2=0x7fe81c0d2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:54.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.030+0000 7fe81affd640 1 --2- 172.21.15.150:0/4209930299 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe81c15e7d0 0x7fe81c0d2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.030+0000 7fe81affd640 1 -- 172.21.15.150:0/4209930299 >> v1:172.21.15.150:6789/0 conn(0x7fe81c15e320 legacy=0x7fe81c162160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:54.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.030+0000 7fe81affd640 1 -- 172.21.15.150:0/4209930299 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe81c15f250 con 0x7fe81c15df40 2024-03-17T23:47:54.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.030+0000 7fe81b7fe640 1 --2- 172.21.15.150:0/4209930299 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe81c15e7d0 0x7fe81c0d2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:54.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.031+0000 7fe81affd640 1 --2- 172.21.15.150:0/4209930299 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe81c15df40 0x7fe81c0b97f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe80c0097a0 tx=0x7fe80c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abcab8956510991f server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:54.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.031+0000 7fe81a7fc640 1 -- 172.21.15.150:0/4209930299 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80c034070 con 0x7fe81c15df40 2024-03-17T23:47:54.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.031+0000 7fe81a7fc640 1 -- 172.21.15.150:0/4209930299 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe80c030610 con 0x7fe81c15df40 2024-03-17T23:47:54.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.031+0000 7fe81a7fc640 1 -- 172.21.15.150:0/4209930299 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80c0309e0 con 0x7fe81c15df40 2024-03-17T23:47:54.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.032+0000 7fe821427640 1 -- 172.21.15.150:0/4209930299 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe81c15df40 msgr2=0x7fe81c0b97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:54.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.032+0000 7fe821427640 1 --2- 172.21.15.150:0/4209930299 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe81c15df40 0x7fe81c0b97f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe80c0097a0 tx=0x7fe80c02b960 comp rx=0 tx=0).stop 2024-03-17T23:47:54.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.032+0000 7fe821427640 1 -- 172.21.15.150:0/4209930299 shutdown_connections 2024-03-17T23:47:54.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.032+0000 7fe821427640 1 --2- 172.21.15.150:0/4209930299 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe81c15e7d0 0x7fe81c0d2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.032+0000 7fe821427640 1 --2- 172.21.15.150:0/4209930299 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe81c15df40 0x7fe81c0b97f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.032+0000 7fe821427640 1 -- 172.21.15.150:0/4209930299 >> 172.21.15.150:0/4209930299 conn(0x7fe81c072ef0 msgr2=0x7fe81c0d3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:54.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.032+0000 7fe821427640 1 -- 172.21.15.150:0/4209930299 shutdown_connections 2024-03-17T23:47:54.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.032+0000 7fe821427640 1 -- 172.21.15.150:0/4209930299 wait complete. 2024-03-17T23:47:54.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.033+0000 7fe821427640 1 Processor -- start 2024-03-17T23:47:54.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.033+0000 7fe821427640 1 -- start start 2024-03-17T23:47:54.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.034+0000 7fe821427640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe81c15df40 0x7fe81c15a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:54.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.034+0000 7fe821427640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe81c15e320 0x7fe81c0cd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:54.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.034+0000 7fe821427640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe81c15e7d0 0x7fe81c1f43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:54.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.034+0000 7fe821427640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c162870 con 0x7fe81c15e7d0 2024-03-17T23:47:54.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.034+0000 7fe821427640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c0d2640 con 0x7fe81c15e320 2024-03-17T23:47:54.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.034+0000 7fe821427640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe81c15a570 con 0x7fe81c15df40 2024-03-17T23:47:54.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.034+0000 7fe81b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe81c15e7d0 0x7fe81c1f43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:54.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.034+0000 7fe81b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe81c15e7d0 0x7fe81c1f43a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50628/0 (socket says 172.21.15.150:50628) 2024-03-17T23:47:54.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.034+0000 7fe81b7fe640 1 -- 172.21.15.150:0/4235065998 learned_addr learned my addr 172.21.15.150:0/4235065998 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:54.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.034+0000 7fe813fff640 1 --2- 172.21.15.150:0/4235065998 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe81c15e320 0x7fe81c0cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:54.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.034+0000 7fe81affd640 1 --2- 172.21.15.150:0/4235065998 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe81c15df40 0x7fe81c15a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:54.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.034+0000 7fe81b7fe640 1 -- 172.21.15.150:0/4235065998 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe81c15df40 msgr2=0x7fe81c15a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:54.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.034+0000 7fe81b7fe640 1 --2- 172.21.15.150:0/4235065998 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe81c15df40 0x7fe81c15a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.034+0000 7fe81b7fe640 1 -- 172.21.15.150:0/4235065998 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe81c15e320 msgr2=0x7fe81c0cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:54.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.034+0000 7fe81b7fe640 1 --2- 172.21.15.150:0/4235065998 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe81c15e320 0x7fe81c0cd240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.034+0000 7fe81b7fe640 1 -- 172.21.15.150:0/4235065998 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe81c15a7a0 con 0x7fe81c15e7d0 2024-03-17T23:47:54.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.034+0000 7fe81b7fe640 1 --2- 172.21.15.150:0/4235065998 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe81c15e7d0 0x7fe81c1f43a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe81400efc0 tx=0x7fe81400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:54.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.035+0000 7fe8137fe640 1 -- 172.21.15.150:0/4235065998 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe814018030 con 0x7fe81c15e7d0 2024-03-17T23:47:54.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.035+0000 7fe8137fe640 1 -- 172.21.15.150:0/4235065998 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe81400cd30 con 0x7fe81c15e7d0 2024-03-17T23:47:54.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.035+0000 7fe821427640 1 -- 172.21.15.150:0/4235065998 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe81c15aa80 con 0x7fe81c15e7d0 2024-03-17T23:47:54.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.035+0000 7fe8137fe640 1 -- 172.21.15.150:0/4235065998 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe814003bb0 con 0x7fe81c15e7d0 2024-03-17T23:47:54.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.035+0000 7fe821427640 1 -- 172.21.15.150:0/4235065998 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe81c15b320 con 0x7fe81c15e7d0 2024-03-17T23:47:54.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.036+0000 7fe821427640 1 -- 172.21.15.150:0/4235065998 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe81c15b520 con 0x7fe81c15e7d0 2024-03-17T23:47:54.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.036+0000 7fe8137fe640 1 -- 172.21.15.150:0/4235065998 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe81401b070 con 0x7fe81c15e7d0 2024-03-17T23:47:54.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.036+0000 7fe8137fe640 1 --2- 172.21.15.150:0/4235065998 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe7fc06bbd0 0x7fe7fc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:54.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.037+0000 7fe8137fe640 1 -- 172.21.15.150:0/4235065998 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fe81408b730 con 0x7fe81c15e7d0 2024-03-17T23:47:54.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.037+0000 7fe81affd640 1 --2- 172.21.15.150:0/4235065998 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe7fc06bbd0 0x7fe7fc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:54.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.037+0000 7fe81affd640 1 --2- 172.21.15.150:0/4235065998 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe7fc06bbd0 0x7fe7fc06e090 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe80c036000 tx=0x7fe80c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:54.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.041+0000 7fe8137fe640 1 -- 172.21.15.150:0/4235065998 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe81c15b520 con 0x7fe81c15e7d0 2024-03-17T23:47:54.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.227+0000 7fe821427640 1 -- 172.21.15.150:0/4235065998 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe81c0681a0 con 0x7fe81c15e7d0 2024-03-17T23:47:54.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.227+0000 7fe8137fe640 1 -- 172.21.15.150:0/4235065998 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) ==== 80+0+0 (secure 0 0 0) 0x7fe814010040 con 0x7fe81c15e7d0 2024-03-17T23:47:54.229 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-03-17T23:47:54.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.229+0000 7fe821427640 1 -- 172.21.15.150:0/4235065998 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe7fc06bbd0 msgr2=0x7fe7fc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:54.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.229+0000 7fe821427640 1 --2- 172.21.15.150:0/4235065998 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe7fc06bbd0 0x7fe7fc06e090 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe80c036000 tx=0x7fe80c03b040 comp rx=0 tx=0).stop 2024-03-17T23:47:54.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.229+0000 7fe821427640 1 -- 172.21.15.150:0/4235065998 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe81c15e7d0 msgr2=0x7fe81c1f43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:54.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.229+0000 7fe821427640 1 --2- 172.21.15.150:0/4235065998 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe81c15e7d0 0x7fe81c1f43a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe81400efc0 tx=0x7fe81400c490 comp rx=0 tx=0).stop 2024-03-17T23:47:54.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.229+0000 7fe821427640 1 -- 172.21.15.150:0/4235065998 shutdown_connections 2024-03-17T23:47:54.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.230+0000 7fe821427640 1 --2- 172.21.15.150:0/4235065998 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe81c15e7d0 0x7fe81c1f43a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.230+0000 7fe821427640 1 --2- 172.21.15.150:0/4235065998 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe7fc06bbd0 0x7fe7fc06e090 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.230+0000 7fe821427640 1 --2- 172.21.15.150:0/4235065998 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe81c15e320 0x7fe81c0cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.230+0000 7fe821427640 1 --2- 172.21.15.150:0/4235065998 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe81c15df40 0x7fe81c15a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.230+0000 7fe821427640 1 -- 172.21.15.150:0/4235065998 >> 172.21.15.150:0/4235065998 conn(0x7fe81c072ef0 msgr2=0x7fe81c1542a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:54.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.230+0000 7fe821427640 1 -- 172.21.15.150:0/4235065998 shutdown_connections 2024-03-17T23:47:54.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.230+0000 7fe821427640 1 -- 172.21.15.150:0/4235065998 wait complete. 2024-03-17T23:47:54.242 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:47:54.242 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:47:54.258 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:54.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.358+0000 7f4af4419640 1 Processor -- start 2024-03-17T23:47:54.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.358+0000 7f4af4419640 1 -- start start 2024-03-17T23:47:54.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.359+0000 7f4af4419640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4aec155320 0x7f4aec15bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:54.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.359+0000 7f4af4419640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4aec0cf3b0 con 0x7f4aec06dd30 2024-03-17T23:47:54.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.359+0000 7f4af4419640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4aec0cf690 con 0x7f4aec0cd420 2024-03-17T23:47:54.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.359+0000 7f4af4419640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f4aec15c550 con 0x7f4aec155320 2024-03-17T23:47:54.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.359+0000 7f4af19b4640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f4aec06dd30 0x7f4aec0ccd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49886/0 (socket says 172.21.15.150:49886) 2024-03-17T23:47:54.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.359+0000 7f4af19b4640 1 -- 172.21.15.150:0/2240533936 learned_addr learned my addr 172.21.15.150:0/2240533936 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:54.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.359+0000 7f4af21b5640 1 --2- 172.21.15.150:0/2240533936 >> v2:172.21.15.157:3300/0 conn(0x7f4aec155320 0x7f4aec15bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:54.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.360+0000 7f4af11b3640 1 -- 172.21.15.150:0/2240533936 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1129112557 0 0) 0x7f4aec0cf3b0 con 0x7f4aec06dd30 2024-03-17T23:47:54.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.360+0000 7f4af11b3640 1 -- 172.21.15.150:0/2240533936 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ad8003520 con 0x7f4aec06dd30 2024-03-17T23:47:54.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.360+0000 7f4af11b3640 1 -- 172.21.15.150:0/2240533936 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4110688570 0 0) 0x7f4aec0cf690 con 0x7f4aec0cd420 2024-03-17T23:47:54.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.360+0000 7f4af11b3640 1 -- 172.21.15.150:0/2240533936 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4aec0cf3b0 con 0x7f4aec0cd420 2024-03-17T23:47:54.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.360+0000 7f4af21b5640 1 -- 172.21.15.150:0/2240533936 >> v1:172.21.15.157:6789/0 conn(0x7f4aec0cd420 legacy=0x7f4aec15ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:54.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.360+0000 7f4af21b5640 1 -- 172.21.15.150:0/2240533936 >> v1:172.21.15.150:6789/0 conn(0x7f4aec06dd30 legacy=0x7f4aec0ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:54.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.360+0000 7f4af21b5640 1 -- 172.21.15.150:0/2240533936 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aec15c7a0 con 0x7f4aec155320 2024-03-17T23:47:54.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.360+0000 7f4af21b5640 1 --2- 172.21.15.150:0/2240533936 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4aec155320 0x7f4aec15bfd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4ae8009d90 tx=0x7f4ae802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc76095380e78d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:54.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.361+0000 7f4af11b3640 1 -- 172.21.15.150:0/2240533936 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae8039070 con 0x7f4aec155320 2024-03-17T23:47:54.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.361+0000 7f4af11b3640 1 -- 172.21.15.150:0/2240533936 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ae802cd00 con 0x7f4aec155320 2024-03-17T23:47:54.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.361+0000 7f4af11b3640 1 -- 172.21.15.150:0/2240533936 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae8034660 con 0x7f4aec155320 2024-03-17T23:47:54.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.361+0000 7f4af4419640 1 -- 172.21.15.150:0/2240533936 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4aec155320 msgr2=0x7f4aec15bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:54.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.361+0000 7f4af4419640 1 --2- 172.21.15.150:0/2240533936 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4aec155320 0x7f4aec15bfd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4ae8009d90 tx=0x7f4ae802bf50 comp rx=0 tx=0).stop 2024-03-17T23:47:54.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.361+0000 7f4af4419640 1 -- 172.21.15.150:0/2240533936 shutdown_connections 2024-03-17T23:47:54.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.361+0000 7f4af4419640 1 --2- 172.21.15.150:0/2240533936 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4aec155320 0x7f4aec15bfd0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.361+0000 7f4af4419640 1 -- 172.21.15.150:0/2240533936 >> 172.21.15.150:0/2240533936 conn(0x7f4aec06b190 msgr2=0x7f4aec06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:54.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.361+0000 7f4af4419640 1 -- 172.21.15.150:0/2240533936 shutdown_connections 2024-03-17T23:47:54.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.361+0000 7f4af4419640 1 -- 172.21.15.150:0/2240533936 wait complete. 2024-03-17T23:47:54.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.362+0000 7f4af4419640 1 Processor -- start 2024-03-17T23:47:54.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.362+0000 7f4af4419640 1 -- start start 2024-03-17T23:47:54.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.363+0000 7f4af4419640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4aec06dd30 0x7f4aec156ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:54.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.363+0000 7f4af4419640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4aec0cd420 0x7f4aec1401b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:54.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.363+0000 7f4af4419640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4aec155320 0x7f4aec15a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:54.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.363+0000 7f4af4419640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4aec155700 con 0x7f4aec06dd30 2024-03-17T23:47:54.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.363+0000 7f4af4419640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4aec06e190 con 0x7f4aec155320 2024-03-17T23:47:54.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.363+0000 7f4af4419640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4aec155880 con 0x7f4aec0cd420 2024-03-17T23:47:54.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.363+0000 7f4af21b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4aec06dd30 0x7f4aec156ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:54.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.363+0000 7f4af21b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4aec06dd30 0x7f4aec156ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50640/0 (socket says 172.21.15.150:50640) 2024-03-17T23:47:54.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.363+0000 7f4af21b5640 1 -- 172.21.15.150:0/2718942205 learned_addr learned my addr 172.21.15.150:0/2718942205 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:54.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.363+0000 7f4af29b6640 1 --2- 172.21.15.150:0/2718942205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4aec155320 0x7f4aec15a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:54.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.363+0000 7f4af19b4640 1 --2- 172.21.15.150:0/2718942205 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4aec0cd420 0x7f4aec1401b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:54.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.363+0000 7f4af21b5640 1 -- 172.21.15.150:0/2718942205 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4aec0cd420 msgr2=0x7f4aec1401b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:54.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.363+0000 7f4af21b5640 1 --2- 172.21.15.150:0/2718942205 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4aec0cd420 0x7f4aec1401b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.363+0000 7f4af21b5640 1 -- 172.21.15.150:0/2718942205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4aec155320 msgr2=0x7f4aec15a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:54.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.363+0000 7f4af21b5640 1 --2- 172.21.15.150:0/2718942205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4aec155320 0x7f4aec15a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.364+0000 7f4af21b5640 1 -- 172.21.15.150:0/2718942205 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aec155a30 con 0x7f4aec06dd30 2024-03-17T23:47:54.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.364+0000 7f4af21b5640 1 --2- 172.21.15.150:0/2718942205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4aec06dd30 0x7f4aec156ad0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f4ae8009d90 tx=0x7f4ae8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:54.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.364+0000 7f4ae2ffd640 1 -- 172.21.15.150:0/2718942205 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae8039050 con 0x7f4aec06dd30 2024-03-17T23:47:54.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.364+0000 7f4ae2ffd640 1 -- 172.21.15.150:0/2718942205 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ae8045070 con 0x7f4aec06dd30 2024-03-17T23:47:54.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.364+0000 7f4af4419640 1 -- 172.21.15.150:0/2718942205 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aec155d10 con 0x7f4aec06dd30 2024-03-17T23:47:54.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.364+0000 7f4ae2ffd640 1 -- 172.21.15.150:0/2718942205 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae8009450 con 0x7f4aec06dd30 2024-03-17T23:47:54.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.364+0000 7f4af4419640 1 -- 172.21.15.150:0/2718942205 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aec1562d0 con 0x7f4aec06dd30 2024-03-17T23:47:54.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.365+0000 7f4af4419640 1 -- 172.21.15.150:0/2718942205 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4aec0681a0 con 0x7f4aec06dd30 2024-03-17T23:47:54.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.365+0000 7f4ae2ffd640 1 -- 172.21.15.150:0/2718942205 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4ae803e830 con 0x7f4aec06dd30 2024-03-17T23:47:54.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.366+0000 7f4ae2ffd640 1 --2- 172.21.15.150:0/2718942205 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4ad006bbd0 0x7f4ad006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:54.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.366+0000 7f4ae2ffd640 1 -- 172.21.15.150:0/2718942205 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f4ae80adb00 con 0x7f4aec06dd30 2024-03-17T23:47:54.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.366+0000 7f4af19b4640 1 --2- 172.21.15.150:0/2718942205 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4ad006bbd0 0x7f4ad006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:54.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.366+0000 7f4af19b4640 1 --2- 172.21.15.150:0/2718942205 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4ad006bbd0 0x7f4ad006e090 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4aec0b4370 tx=0x7f4adc007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:54.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.370+0000 7f4ae2ffd640 1 -- 172.21.15.150:0/2718942205 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4ae807bcc0 con 0x7f4aec06dd30 2024-03-17T23:47:54.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.544+0000 7f4af4419640 1 -- 172.21.15.150:0/2718942205 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4aec073da0 con 0x7f4aec06dd30 2024-03-17T23:47:54.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.546+0000 7f4ae2ffd640 1 -- 172.21.15.150:0/2718942205 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18881 (secure 0 0 0) 0x7f4ae807fd60 con 0x7f4aec06dd30 2024-03-17T23:47:54.547 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:54.547 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":28,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:37.031864+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.e","mappings":[{"from":7,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:54.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.548+0000 7f4af4419640 1 -- 172.21.15.150:0/2718942205 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4ad006bbd0 msgr2=0x7f4ad006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.548+0000 7f4af4419640 1 --2- 172.21.15.150:0/2718942205 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4ad006bbd0 0x7f4ad006e090 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4aec0b4370 tx=0x7f4adc007d30 comp rx=0 tx=0).stop 2024-03-17T23:47:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.548+0000 7f4af4419640 1 -- 172.21.15.150:0/2718942205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4aec06dd30 msgr2=0x7f4aec156ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.548+0000 7f4af4419640 1 --2- 172.21.15.150:0/2718942205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4aec06dd30 0x7f4aec156ad0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f4ae8009d90 tx=0x7f4ae8004a40 comp rx=0 tx=0).stop 2024-03-17T23:47:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.548+0000 7f4af4419640 1 -- 172.21.15.150:0/2718942205 shutdown_connections 2024-03-17T23:47:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.548+0000 7f4af4419640 1 --2- 172.21.15.150:0/2718942205 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4ad006bbd0 0x7f4ad006e090 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.548+0000 7f4af4419640 1 --2- 172.21.15.150:0/2718942205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4aec155320 0x7f4aec15a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.548+0000 7f4af4419640 1 --2- 172.21.15.150:0/2718942205 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4aec0cd420 0x7f4aec1401b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.548+0000 7f4af4419640 1 --2- 172.21.15.150:0/2718942205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4aec06dd30 0x7f4aec156ad0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.548+0000 7f4af4419640 1 -- 172.21.15.150:0/2718942205 >> 172.21.15.150:0/2718942205 conn(0x7f4aec06b190 msgr2=0x7f4aec152e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.548+0000 7f4af4419640 1 -- 172.21.15.150:0/2718942205 shutdown_connections 2024-03-17T23:47:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.548+0000 7f4af4419640 1 -- 172.21.15.150:0/2718942205 wait complete. 2024-03-17T23:47:54.560 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:46:53.610678+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:46:54.227335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-17T23:46:57.949235+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-17T23:47:54.561 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:47:54.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.663+0000 7fb9e2a27640 1 Processor -- start 2024-03-17T23:47:54.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.663+0000 7fb9e2a27640 1 -- start start 2024-03-17T23:47:54.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.664+0000 7fb9e2a27640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9dc0ce6a0 0x7fb9dc166a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:54.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.664+0000 7fb9e2a27640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9dc145170 con 0x7fb9dc06f400 2024-03-17T23:47:54.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.664+0000 7fb9e2a27640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9dc145450 con 0x7fb9dc144bb0 2024-03-17T23:47:54.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.664+0000 7fb9e2a27640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb9dc145730 con 0x7fb9dc0ce6a0 2024-03-17T23:47:54.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.664+0000 7fb9db7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb9dc06f400 0x7fb9dc0cdf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49890/0 (socket says 172.21.15.150:49890) 2024-03-17T23:47:54.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.664+0000 7fb9db7fe640 1 -- 172.21.15.150:0/3166925969 learned_addr learned my addr 172.21.15.150:0/3166925969 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:54.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.664+0000 7fb9e0fc4640 1 --2- 172.21.15.150:0/3166925969 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9dc0ce6a0 0x7fb9dc166a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:54.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.664+0000 7fb9daffd640 1 -- 172.21.15.150:0/3166925969 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2796589813 0 0) 0x7fb9dc145170 con 0x7fb9dc06f400 2024-03-17T23:47:54.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.664+0000 7fb9daffd640 1 -- 172.21.15.150:0/3166925969 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9c4003520 con 0x7fb9dc06f400 2024-03-17T23:47:54.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.664+0000 7fb9daffd640 1 -- 172.21.15.150:0/3166925969 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 729758934 0 0) 0x7fb9dc145450 con 0x7fb9dc144bb0 2024-03-17T23:47:54.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.664+0000 7fb9daffd640 1 -- 172.21.15.150:0/3166925969 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9dc145170 con 0x7fb9dc144bb0 2024-03-17T23:47:54.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.664+0000 7fb9e0fc4640 1 -- 172.21.15.150:0/3166925969 >> v1:172.21.15.157:6789/0 conn(0x7fb9dc144bb0 legacy=0x7fb9dc15ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:54.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.664+0000 7fb9e0fc4640 1 -- 172.21.15.150:0/3166925969 >> v1:172.21.15.150:6789/0 conn(0x7fb9dc06f400 legacy=0x7fb9dc0cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:54.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.664+0000 7fb9e0fc4640 1 -- 172.21.15.150:0/3166925969 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9dc15d560 con 0x7fb9dc0ce6a0 2024-03-17T23:47:54.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.665+0000 7fb9e0fc4640 1 --2- 172.21.15.150:0/3166925969 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9dc0ce6a0 0x7fb9dc166a40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4009dc0 tx=0x7fb9d402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cff8bed96793dd30 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:54.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.665+0000 7fb9daffd640 1 -- 172.21.15.150:0/3166925969 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d4039070 con 0x7fb9dc0ce6a0 2024-03-17T23:47:54.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.665+0000 7fb9daffd640 1 -- 172.21.15.150:0/3166925969 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9d402cc40 con 0x7fb9dc0ce6a0 2024-03-17T23:47:54.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.665+0000 7fb9daffd640 1 -- 172.21.15.150:0/3166925969 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d4034440 con 0x7fb9dc0ce6a0 2024-03-17T23:47:54.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.665+0000 7fb9e2a27640 1 -- 172.21.15.150:0/3166925969 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9dc0ce6a0 msgr2=0x7fb9dc166a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:54.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.665+0000 7fb9e2a27640 1 --2- 172.21.15.150:0/3166925969 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9dc0ce6a0 0x7fb9dc166a40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb9d4009dc0 tx=0x7fb9d402be90 comp rx=0 tx=0).stop 2024-03-17T23:47:54.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.666+0000 7fb9e2a27640 1 -- 172.21.15.150:0/3166925969 shutdown_connections 2024-03-17T23:47:54.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.666+0000 7fb9e2a27640 1 --2- 172.21.15.150:0/3166925969 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9dc0ce6a0 0x7fb9dc166a40 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.666+0000 7fb9e2a27640 1 -- 172.21.15.150:0/3166925969 >> 172.21.15.150:0/3166925969 conn(0x7fb9dc06b190 msgr2=0x7fb9dc06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:54.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.666+0000 7fb9e2a27640 1 -- 172.21.15.150:0/3166925969 shutdown_connections 2024-03-17T23:47:54.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.666+0000 7fb9e2a27640 1 -- 172.21.15.150:0/3166925969 wait complete. 2024-03-17T23:47:54.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.667+0000 7fb9e2a27640 1 Processor -- start 2024-03-17T23:47:54.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.667+0000 7fb9e2a27640 1 -- start start 2024-03-17T23:47:54.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.668+0000 7fb9e2a27640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9dc06f400 0x7fb9dc1410e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:54.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.668+0000 7fb9e2a27640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9dc0ce6a0 0x7fb9dc0b0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:54.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.668+0000 7fb9e2a27640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9dc144bb0 0x7fb9dc0b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:54.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.668+0000 7fb9e2a27640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb9dc167100 con 0x7fb9dc06f400 2024-03-17T23:47:54.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.668+0000 7fb9e2a27640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb9dc166f80 con 0x7fb9dc144bb0 2024-03-17T23:47:54.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.668+0000 7fb9e2a27640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb9dc167280 con 0x7fb9dc0ce6a0 2024-03-17T23:47:54.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.668+0000 7fb9dbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9dc06f400 0x7fb9dc1410e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:54.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.668+0000 7fb9dbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9dc06f400 0x7fb9dc1410e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50654/0 (socket says 172.21.15.150:50654) 2024-03-17T23:47:54.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.668+0000 7fb9dbfff640 1 -- 172.21.15.150:0/2376832643 learned_addr learned my addr 172.21.15.150:0/2376832643 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:54.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.668+0000 7fb9e0fc4640 1 --2- 172.21.15.150:0/2376832643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9dc144bb0 0x7fb9dc0b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:54.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.668+0000 7fb9db7fe640 1 --2- 172.21.15.150:0/2376832643 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9dc0ce6a0 0x7fb9dc0b0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:54.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.668+0000 7fb9dbfff640 1 -- 172.21.15.150:0/2376832643 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9dc0ce6a0 msgr2=0x7fb9dc0b0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:54.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.668+0000 7fb9dbfff640 1 --2- 172.21.15.150:0/2376832643 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9dc0ce6a0 0x7fb9dc0b0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.668+0000 7fb9dbfff640 1 -- 172.21.15.150:0/2376832643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9dc144bb0 msgr2=0x7fb9dc0b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:54.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.668+0000 7fb9dbfff640 1 --2- 172.21.15.150:0/2376832643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9dc144bb0 0x7fb9dc0b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.668+0000 7fb9dbfff640 1 -- 172.21.15.150:0/2376832643 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9dc1f7200 con 0x7fb9dc06f400 2024-03-17T23:47:54.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.668+0000 7fb9dbfff640 1 --2- 172.21.15.150:0/2376832643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9dc06f400 0x7fb9dc1410e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc00c530 tx=0x7fb9cc00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:54.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.668+0000 7fb9d8ff9640 1 -- 172.21.15.150:0/2376832643 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc0052d0 con 0x7fb9dc06f400 2024-03-17T23:47:54.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.668+0000 7fb9d8ff9640 1 -- 172.21.15.150:0/2376832643 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9cc005490 con 0x7fb9dc06f400 2024-03-17T23:47:54.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.668+0000 7fb9e2a27640 1 -- 172.21.15.150:0/2376832643 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9dc1f74e0 con 0x7fb9dc06f400 2024-03-17T23:47:54.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.668+0000 7fb9d8ff9640 1 -- 172.21.15.150:0/2376832643 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc012870 con 0x7fb9dc06f400 2024-03-17T23:47:54.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.668+0000 7fb9e2a27640 1 -- 172.21.15.150:0/2376832643 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9dc141620 con 0x7fb9dc06f400 2024-03-17T23:47:54.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.669+0000 7fb9e2a27640 1 -- 172.21.15.150:0/2376832643 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9dc0681a0 con 0x7fb9dc06f400 2024-03-17T23:47:54.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.670+0000 7fb9d8ff9640 1 -- 172.21.15.150:0/2376832643 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb9cc012a10 con 0x7fb9dc06f400 2024-03-17T23:47:54.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.670+0000 7fb9d8ff9640 1 --2- 172.21.15.150:0/2376832643 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb9c806bbd0 0x7fb9c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:54.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.670+0000 7fb9db7fe640 1 --2- 172.21.15.150:0/2376832643 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb9c806bbd0 0x7fb9c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:54.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.670+0000 7fb9d8ff9640 1 -- 172.21.15.150:0/2376832643 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fb9cc08a990 con 0x7fb9dc06f400 2024-03-17T23:47:54.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.671+0000 7fb9db7fe640 1 --2- 172.21.15.150:0/2376832643 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb9c806bbd0 0x7fb9c806e090 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc15bf80 tx=0x7fb9d0007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:54.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.675+0000 7fb9d8ff9640 1 -- 172.21.15.150:0/2376832643 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb9cc058a40 con 0x7fb9dc06f400 2024-03-17T23:47:54.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.845+0000 7fb9e2a27640 1 -- 172.21.15.150:0/2376832643 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb9dc0cd390 con 0x7fb9dc06f400 2024-03-17T23:47:54.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.845+0000 7fb9d8ff9640 1 -- 172.21.15.150:0/2376832643 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7fb9cc05cae0 con 0x7fb9dc06f400 2024-03-17T23:47:54.847 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:47:54.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.847+0000 7fb9e2a27640 1 -- 172.21.15.150:0/2376832643 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb9c806bbd0 msgr2=0x7fb9c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:54.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.847+0000 7fb9e2a27640 1 --2- 172.21.15.150:0/2376832643 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb9c806bbd0 0x7fb9c806e090 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc15bf80 tx=0x7fb9d0007d30 comp rx=0 tx=0).stop 2024-03-17T23:47:54.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.847+0000 7fb9e2a27640 1 -- 172.21.15.150:0/2376832643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9dc06f400 msgr2=0x7fb9dc1410e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:54.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.847+0000 7fb9e2a27640 1 --2- 172.21.15.150:0/2376832643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9dc06f400 0x7fb9dc1410e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc00c530 tx=0x7fb9cc00c9f0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.847+0000 7fb9e2a27640 1 -- 172.21.15.150:0/2376832643 shutdown_connections 2024-03-17T23:47:54.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.847+0000 7fb9e2a27640 1 --2- 172.21.15.150:0/2376832643 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb9c806bbd0 0x7fb9c806e090 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.847+0000 7fb9e2a27640 1 --2- 172.21.15.150:0/2376832643 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb9dc144bb0 0x7fb9dc0b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.847+0000 7fb9e2a27640 1 --2- 172.21.15.150:0/2376832643 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9dc0ce6a0 0x7fb9dc0b0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.847+0000 7fb9e2a27640 1 --2- 172.21.15.150:0/2376832643 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9dc06f400 0x7fb9dc1410e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.847+0000 7fb9e2a27640 1 -- 172.21.15.150:0/2376832643 >> 172.21.15.150:0/2376832643 conn(0x7fb9dc06b190 msgr2=0x7fb9dc071cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:54.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.847+0000 7fb9e2a27640 1 -- 172.21.15.150:0/2376832643 shutdown_connections 2024-03-17T23:47:54.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.847+0000 7fb9e2a27640 1 -- 172.21.15.150:0/2376832643 wait complete. 2024-03-17T23:47:54.860 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:47:54.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.962+0000 7f22b4019640 1 Processor -- start 2024-03-17T23:47:54.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.963+0000 7f22b4019640 1 -- start start 2024-03-17T23:47:54.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.963+0000 7f22b4019640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f22ac1615d0 0x7f22ac0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:54.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.963+0000 7f22b4019640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f22ac0d3910 0x7f22ac0d0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:54.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.963+0000 7f22b4019640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22ac0d2870 con 0x7f22ac0b9660 2024-03-17T23:47:54.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.963+0000 7f22b4019640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f22ac1619b0 con 0x7f22ac1615d0 2024-03-17T23:47:54.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.963+0000 7f22b4019640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f22ac0b7e80 con 0x7f22ac0d3910 2024-03-17T23:47:54.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.964+0000 7f22b1db5640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f22ac1615d0 0x7f22ac0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:54.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.964+0000 7f22b25b6640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f22ac0b9660 0x7f22ac06c790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:39132/0 (socket says 172.21.15.150:39132) 2024-03-17T23:47:54.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.964+0000 7f22b25b6640 1 -- 172.21.15.150:0/2435776364 learned_addr learned my addr 172.21.15.150:0/2435776364 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:54.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.964+0000 7f22b15b4640 1 --2- 172.21.15.150:0/2435776364 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f22ac0d3910 0x7f22ac0d0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:54.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.964+0000 7f22b15b4640 1 -- 172.21.15.150:0/2435776364 >> v2:172.21.15.157:3300/0 conn(0x7f22ac1615d0 msgr2=0x7f22ac0d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:54.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.964+0000 7f22b15b4640 1 --2- 172.21.15.150:0/2435776364 >> v2:172.21.15.157:3300/0 conn(0x7f22ac1615d0 0x7f22ac0d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.964+0000 7f22b15b4640 1 -- 172.21.15.150:0/2435776364 >> v1:172.21.15.157:6789/0 conn(0x7f22ac0b9660 legacy=0x7f22ac06c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:54.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.964+0000 7f22b0db3640 1 -- 172.21.15.150:0/2435776364 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 218787094 0 0) 0x7f22ac0d2870 con 0x7f22ac0b9660 2024-03-17T23:47:54.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.964+0000 7f22b15b4640 1 -- 172.21.15.150:0/2435776364 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22ac0d2b50 con 0x7f22ac0d3910 2024-03-17T23:47:54.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.964+0000 7f22b1db5640 1 --2- 172.21.15.150:0/2435776364 >> v2:172.21.15.157:3300/0 conn(0x7f22ac1615d0 0x7f22ac0d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:54.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.965+0000 7f22b15b4640 1 --2- 172.21.15.150:0/2435776364 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f22ac0d3910 0x7f22ac0d0720 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f229c0097a0 tx=0x7f229c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f78deb04ae4d4e83 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:54.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.965+0000 7f22b0db3640 1 -- 172.21.15.150:0/2435776364 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229c035020 con 0x7f22ac0d3910 2024-03-17T23:47:54.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.965+0000 7f22b0db3640 1 -- 172.21.15.150:0/2435776364 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f229c031830 con 0x7f22ac0d3910 2024-03-17T23:47:54.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.965+0000 7f22b0db3640 1 -- 172.21.15.150:0/2435776364 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229c031c00 con 0x7f22ac0d3910 2024-03-17T23:47:54.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.965+0000 7f22b4019640 1 -- 172.21.15.150:0/2435776364 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f22ac0d3910 msgr2=0x7f22ac0d0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:54.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.965+0000 7f22b4019640 1 --2- 172.21.15.150:0/2435776364 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f22ac0d3910 0x7f22ac0d0720 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f229c0097a0 tx=0x7f229c02b8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.965+0000 7f22b4019640 1 -- 172.21.15.150:0/2435776364 shutdown_connections 2024-03-17T23:47:54.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.965+0000 7f22b4019640 1 --2- 172.21.15.150:0/2435776364 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f22ac0d3910 0x7f22ac0d0720 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.965+0000 7f22b4019640 1 --2- 172.21.15.150:0/2435776364 >> v2:172.21.15.157:3300/0 conn(0x7f22ac1615d0 0x7f22ac0d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.965+0000 7f22b4019640 1 -- 172.21.15.150:0/2435776364 >> 172.21.15.150:0/2435776364 conn(0x7f22ac0730b0 msgr2=0x7f22ac0b5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:54.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.965+0000 7f22b4019640 1 -- 172.21.15.150:0/2435776364 shutdown_connections 2024-03-17T23:47:54.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.965+0000 7f22b4019640 1 -- 172.21.15.150:0/2435776364 wait complete. 2024-03-17T23:47:54.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.966+0000 7f22b4019640 1 Processor -- start 2024-03-17T23:47:54.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.967+0000 7f22b4019640 1 -- start start 2024-03-17T23:47:54.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.967+0000 7f22b4019640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22ac0b9660 0x7f22ac1540b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:54.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.967+0000 7f22b4019640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f22ac0d3910 0x7f22ac0cc380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:54.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.967+0000 7f22b4019640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22ac1615d0 0x7f22ac0b4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:54.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.967+0000 7f22b4019640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f22ac0d33f0 con 0x7f22ac0b9660 2024-03-17T23:47:54.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.967+0000 7f22b4019640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f22ac0d3110 con 0x7f22ac1615d0 2024-03-17T23:47:54.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.967+0000 7f22b4019640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f22ac0d2560 con 0x7f22ac0d3910 2024-03-17T23:47:54.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.967+0000 7f22b1db5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22ac0b9660 0x7f22ac1540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:54.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.967+0000 7f22b1db5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22ac0b9660 0x7f22ac1540b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50662/0 (socket says 172.21.15.150:50662) 2024-03-17T23:47:54.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.967+0000 7f22b1db5640 1 -- 172.21.15.150:0/2251530452 learned_addr learned my addr 172.21.15.150:0/2251530452 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:54.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.967+0000 7f22b15b4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f22ac0d3910 0x7f22ac0cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:54.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.967+0000 7f22b25b6640 1 --2- 172.21.15.150:0/2251530452 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22ac1615d0 0x7f22ac0b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:54.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.968+0000 7f22b1db5640 1 -- 172.21.15.150:0/2251530452 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f22ac0d3910 msgr2=0x7f22ac0cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:54.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.968+0000 7f22b1db5640 1 --2- 172.21.15.150:0/2251530452 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f22ac0d3910 0x7f22ac0cc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.968+0000 7f22b1db5640 1 -- 172.21.15.150:0/2251530452 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22ac1615d0 msgr2=0x7f22ac0b4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:54.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.968+0000 7f22b1db5640 1 --2- 172.21.15.150:0/2251530452 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22ac1615d0 0x7f22ac0b4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:54.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.968+0000 7f22b1db5640 1 -- 172.21.15.150:0/2251530452 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22ac0ccd40 con 0x7f22ac0b9660 2024-03-17T23:47:54.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.968+0000 7f22b1db5640 1 --2- 172.21.15.150:0/2251530452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22ac0b9660 0x7f22ac1540b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f22a800e960 tx=0x7f22a800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:54.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.968+0000 7f22a27fc640 1 -- 172.21.15.150:0/2251530452 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a8017070 con 0x7f22ac0b9660 2024-03-17T23:47:54.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.968+0000 7f22a27fc640 1 -- 172.21.15.150:0/2251530452 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22a800c9a0 con 0x7f22ac0b9660 2024-03-17T23:47:54.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.968+0000 7f22b4019640 1 -- 172.21.15.150:0/2251530452 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22ac0cd020 con 0x7f22ac0b9660 2024-03-17T23:47:54.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.968+0000 7f22a27fc640 1 -- 172.21.15.150:0/2251530452 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a800cd70 con 0x7f22ac0b9660 2024-03-17T23:47:54.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.968+0000 7f22b4019640 1 -- 172.21.15.150:0/2251530452 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22ac0cd5e0 con 0x7f22ac0b9660 2024-03-17T23:47:54.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.969+0000 7f22b4019640 1 -- 172.21.15.150:0/2251530452 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22ac0ca6e0 con 0x7f22ac0b9660 2024-03-17T23:47:54.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.970+0000 7f22a27fc640 1 -- 172.21.15.150:0/2251530452 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f22a801a070 con 0x7f22ac0b9660 2024-03-17T23:47:54.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.970+0000 7f22a27fc640 1 --2- 172.21.15.150:0/2251530452 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f229006bc00 0x7f229006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:54.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.970+0000 7f22a27fc640 1 -- 172.21.15.150:0/2251530452 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f22a808ac10 con 0x7f22ac0b9660 2024-03-17T23:47:54.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.970+0000 7f22b15b4640 1 --2- 172.21.15.150:0/2251530452 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f229006bc00 0x7f229006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:54.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.970+0000 7f22b15b4640 1 --2- 172.21.15.150:0/2251530452 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f229006bc00 0x7f229006e0c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f229c036000 tx=0x7f229c03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:54.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:54.974+0000 7f22a27fc640 1 -- 172.21.15.150:0/2251530452 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f22ac0ca6e0 con 0x7f22ac0b9660 2024-03-17T23:47:55.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.146+0000 7f22b4019640 1 -- 172.21.15.150:0/2251530452 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f22ac067c80 con 0x7f22ac0b9660 2024-03-17T23:47:55.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.146+0000 7f22a27fc640 1 -- 172.21.15.150:0/2251530452 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7f22ac067c80 con 0x7f22ac0b9660 2024-03-17T23:47:55.150 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:55.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.148+0000 7f22b4019640 1 -- 172.21.15.150:0/2251530452 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f229006bc00 msgr2=0x7f229006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:55.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.148+0000 7f22b4019640 1 --2- 172.21.15.150:0/2251530452 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f229006bc00 0x7f229006e0c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f229c036000 tx=0x7f229c03d040 comp rx=0 tx=0).stop 2024-03-17T23:47:55.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.148+0000 7f22b4019640 1 -- 172.21.15.150:0/2251530452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22ac0b9660 msgr2=0x7f22ac1540b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:55.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.148+0000 7f22b4019640 1 --2- 172.21.15.150:0/2251530452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22ac0b9660 0x7f22ac1540b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f22a800e960 tx=0x7f22a800ee20 comp rx=0 tx=0).stop 2024-03-17T23:47:55.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.148+0000 7f22b4019640 1 -- 172.21.15.150:0/2251530452 shutdown_connections 2024-03-17T23:47:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.148+0000 7f22b4019640 1 --2- 172.21.15.150:0/2251530452 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f229006bc00 0x7f229006e0c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.148+0000 7f22b4019640 1 --2- 172.21.15.150:0/2251530452 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22ac1615d0 0x7f22ac0b4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.148+0000 7f22b4019640 1 --2- 172.21.15.150:0/2251530452 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f22ac0d3910 0x7f22ac0cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.148+0000 7f22b4019640 1 --2- 172.21.15.150:0/2251530452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22ac0b9660 0x7f22ac1540b0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.148+0000 7f22b4019640 1 -- 172.21.15.150:0/2251530452 >> 172.21.15.150:0/2251530452 conn(0x7f22ac0730b0 msgr2=0x7f22ac15f050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.149+0000 7f22b4019640 1 -- 172.21.15.150:0/2251530452 shutdown_connections 2024-03-17T23:47:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.149+0000 7f22b4019640 1 -- 172.21.15.150:0/2251530452 wait complete. 2024-03-17T23:47:55.161 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:47:55.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.264+0000 7f67f284b640 1 Processor -- start 2024-03-17T23:47:55.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.264+0000 7f67f284b640 1 -- start start 2024-03-17T23:47:55.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.264+0000 7f67f284b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f67ec1615d0 0x7f67ec0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:55.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.264+0000 7f67f284b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f67ec0d0e30 0x7f67ec0b6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:55.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.264+0000 7f67f284b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67ec0d2870 con 0x7f67ec0d3910 2024-03-17T23:47:55.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.264+0000 7f67f284b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f67ec1619b0 con 0x7f67ec0d0e30 2024-03-17T23:47:55.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.264+0000 7f67f284b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f67ec0d2b50 con 0x7f67ec1615d0 2024-03-17T23:47:55.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.265+0000 7f67f0de8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f67ec0d0e30 0x7f67ec0b6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:55.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.265+0000 7f67f0de8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f67ec0d0e30 0x7f67ec0b6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50668/0 (socket says 172.21.15.150:50668) 2024-03-17T23:47:55.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.265+0000 7f67f0de8640 1 -- 172.21.15.150:0/1319452246 learned_addr learned my addr 172.21.15.150:0/1319452246 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:55.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.265+0000 7f67ebfff640 1 --2- 172.21.15.150:0/1319452246 >> v2:172.21.15.157:3300/0 conn(0x7f67ec1615d0 0x7f67ec0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:55.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.265+0000 7f67f0de8640 1 -- 172.21.15.150:0/1319452246 >> v2:172.21.15.157:3300/0 conn(0x7f67ec1615d0 msgr2=0x7f67ec0d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:55.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.265+0000 7f67f0de8640 1 --2- 172.21.15.150:0/1319452246 >> v2:172.21.15.157:3300/0 conn(0x7f67ec1615d0 0x7f67ec0d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.265+0000 7f67f0de8640 1 -- 172.21.15.150:0/1319452246 >> v1:172.21.15.157:6789/0 conn(0x7f67ec0d3910 legacy=0x7f67ec0d0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:55.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.265+0000 7f67f0de8640 1 -- 172.21.15.150:0/1319452246 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67ec0d2e30 con 0x7f67ec0d0e30 2024-03-17T23:47:55.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.265+0000 7f67eaffd640 1 -- 172.21.15.150:0/1319452246 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3878993700 0 0) 0x7f67ec0d2870 con 0x7f67ec0d3910 2024-03-17T23:47:55.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.266+0000 7f67f0de8640 1 --2- 172.21.15.150:0/1319452246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67ec0d0e30 0x7f67ec0b6e70 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f67ec06d7c0 tx=0x7f67e402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c36f748f86e078f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:55.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.266+0000 7f67eaffd640 1 -- 172.21.15.150:0/1319452246 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e402cb20 con 0x7f67ec0d0e30 2024-03-17T23:47:55.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.266+0000 7f67eaffd640 1 -- 172.21.15.150:0/1319452246 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67e402ccc0 con 0x7f67ec0d0e30 2024-03-17T23:47:55.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.266+0000 7f67eaffd640 1 -- 172.21.15.150:0/1319452246 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e4034610 con 0x7f67ec0d0e30 2024-03-17T23:47:55.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.266+0000 7f67f284b640 1 -- 172.21.15.150:0/1319452246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67ec0d0e30 msgr2=0x7f67ec0b6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:55.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.266+0000 7f67f284b640 1 --2- 172.21.15.150:0/1319452246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67ec0d0e30 0x7f67ec0b6e70 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f67ec06d7c0 tx=0x7f67e402bf80 comp rx=0 tx=0).stop 2024-03-17T23:47:55.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.266+0000 7f67f284b640 1 -- 172.21.15.150:0/1319452246 shutdown_connections 2024-03-17T23:47:55.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.266+0000 7f67f284b640 1 --2- 172.21.15.150:0/1319452246 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67ec0d0e30 0x7f67ec0b6e70 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.266+0000 7f67f284b640 1 --2- 172.21.15.150:0/1319452246 >> v2:172.21.15.157:3300/0 conn(0x7f67ec1615d0 0x7f67ec0d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.266+0000 7f67f284b640 1 -- 172.21.15.150:0/1319452246 >> 172.21.15.150:0/1319452246 conn(0x7f67ec0730b0 msgr2=0x7f67ec0b5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:55.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.266+0000 7f67f284b640 1 -- 172.21.15.150:0/1319452246 shutdown_connections 2024-03-17T23:47:55.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.266+0000 7f67f284b640 1 -- 172.21.15.150:0/1319452246 wait complete. 2024-03-17T23:47:55.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.268+0000 7f67f284b640 1 Processor -- start 2024-03-17T23:47:55.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.268+0000 7f67f284b640 1 -- start start 2024-03-17T23:47:55.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.268+0000 7f67f284b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67ec0d0e30 0x7f67ec14f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:55.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.268+0000 7f67f284b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67ec0d3910 0x7f67ec0ac5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:55.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.268+0000 7f67f284b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67ec1615d0 0x7f67ec0ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:55.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.268+0000 7f67f284b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f67ec0d2560 con 0x7f67ec0d3910 2024-03-17T23:47:55.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.268+0000 7f67f284b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f67ec0d33f0 con 0x7f67ec0d0e30 2024-03-17T23:47:55.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.268+0000 7f67f284b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f67ec0b9660 con 0x7f67ec1615d0 2024-03-17T23:47:55.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.269+0000 7f67ebfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67ec0d0e30 0x7f67ec14f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:55.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.269+0000 7f67eb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67ec0d3910 0x7f67ec0ac5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:55.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.269+0000 7f67eb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67ec0d3910 0x7f67ec0ac5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50674/0 (socket says 172.21.15.150:50674) 2024-03-17T23:47:55.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.269+0000 7f67ebfff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67ec0d0e30 0x7f67ec14f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:58914/0 (socket says 172.21.15.150:58914) 2024-03-17T23:47:55.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.269+0000 7f67eb7fe640 1 -- 172.21.15.150:0/1159556522 learned_addr learned my addr 172.21.15.150:0/1159556522 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:55.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.269+0000 7f67f0de8640 1 --2- 172.21.15.150:0/1159556522 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67ec1615d0 0x7f67ec0ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:55.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.269+0000 7f67eb7fe640 1 -- 172.21.15.150:0/1159556522 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67ec1615d0 msgr2=0x7f67ec0ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:55.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.269+0000 7f67eb7fe640 1 --2- 172.21.15.150:0/1159556522 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67ec1615d0 0x7f67ec0ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.269+0000 7f67eb7fe640 1 -- 172.21.15.150:0/1159556522 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67ec0d0e30 msgr2=0x7f67ec14f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:55.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.269+0000 7f67eb7fe640 1 --2- 172.21.15.150:0/1159556522 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67ec0d0e30 0x7f67ec14f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.269+0000 7f67eb7fe640 1 -- 172.21.15.150:0/1159556522 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67ec0ab510 con 0x7f67ec0d3910 2024-03-17T23:47:55.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.269+0000 7f67eb7fe640 1 --2- 172.21.15.150:0/1159556522 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67ec0d3910 0x7f67ec0ac5b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f67e000c490 tx=0x7f67e000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:55.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.269+0000 7f67e8ff9640 1 -- 172.21.15.150:0/1159556522 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e0005480 con 0x7f67ec0d3910 2024-03-17T23:47:55.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.269+0000 7f67e8ff9640 1 -- 172.21.15.150:0/1159556522 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67e00035f0 con 0x7f67ec0d3910 2024-03-17T23:47:55.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.269+0000 7f67f284b640 1 -- 172.21.15.150:0/1159556522 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67ec0ab7f0 con 0x7f67ec0d3910 2024-03-17T23:47:55.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.269+0000 7f67e8ff9640 1 -- 172.21.15.150:0/1159556522 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e00039c0 con 0x7f67ec0d3910 2024-03-17T23:47:55.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.269+0000 7f67f284b640 1 -- 172.21.15.150:0/1159556522 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67ec0abdb0 con 0x7f67ec0d3910 2024-03-17T23:47:55.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.270+0000 7f67f284b640 1 -- 172.21.15.150:0/1159556522 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67ec067c80 con 0x7f67ec0d3910 2024-03-17T23:47:55.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.271+0000 7f67e8ff9640 1 -- 172.21.15.150:0/1159556522 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f67e0003b60 con 0x7f67ec0d3910 2024-03-17T23:47:55.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.271+0000 7f67e8ff9640 1 --2- 172.21.15.150:0/1159556522 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f67d806bd70 0x7f67d806e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:55.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.271+0000 7f67e8ff9640 1 -- 172.21.15.150:0/1159556522 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f67e008a930 con 0x7f67ec0d3910 2024-03-17T23:47:55.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.271+0000 7f67ebfff640 1 --2- 172.21.15.150:0/1159556522 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f67d806bd70 0x7f67d806e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:55.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.272+0000 7f67ebfff640 1 --2- 172.21.15.150:0/1159556522 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f67d806bd70 0x7f67d806e230 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f67ec15cf20 tx=0x7f67dc009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:55.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.275+0000 7f67e8ff9640 1 -- 172.21.15.150:0/1159556522 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f67ec067c80 con 0x7f67ec0d3910 2024-03-17T23:47:55.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.447+0000 7f67f284b640 1 -- 172.21.15.150:0/1159556522 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f67ec0688b0 con 0x7f67ec0d3910 2024-03-17T23:47:55.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.448+0000 7f67e8ff9640 1 -- 172.21.15.150:0/1159556522 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7f67e0058a60 con 0x7f67ec0d3910 2024-03-17T23:47:55.449 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:47:55.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.449+0000 7f67f284b640 1 -- 172.21.15.150:0/1159556522 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f67d806bd70 msgr2=0x7f67d806e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:55.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.449+0000 7f67f284b640 1 --2- 172.21.15.150:0/1159556522 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f67d806bd70 0x7f67d806e230 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f67ec15cf20 tx=0x7f67dc009190 comp rx=0 tx=0).stop 2024-03-17T23:47:55.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.449+0000 7f67f284b640 1 -- 172.21.15.150:0/1159556522 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67ec0d3910 msgr2=0x7f67ec0ac5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:55.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.449+0000 7f67f284b640 1 --2- 172.21.15.150:0/1159556522 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67ec0d3910 0x7f67ec0ac5b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f67e000c490 tx=0x7f67e000c950 comp rx=0 tx=0).stop 2024-03-17T23:47:55.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.450+0000 7f67f284b640 1 -- 172.21.15.150:0/1159556522 shutdown_connections 2024-03-17T23:47:55.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.450+0000 7f67f284b640 1 --2- 172.21.15.150:0/1159556522 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67ec1615d0 0x7f67ec0ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.450+0000 7f67f284b640 1 --2- 172.21.15.150:0/1159556522 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67ec0d3910 0x7f67ec0ac5b0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.450+0000 7f67f284b640 1 --2- 172.21.15.150:0/1159556522 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f67d806bd70 0x7f67d806e230 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.450+0000 7f67f284b640 1 --2- 172.21.15.150:0/1159556522 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67ec0d0e30 0x7f67ec14f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.450+0000 7f67f284b640 1 -- 172.21.15.150:0/1159556522 >> 172.21.15.150:0/1159556522 conn(0x7f67ec0730b0 msgr2=0x7f67ec15ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:55.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.450+0000 7f67f284b640 1 -- 172.21.15.150:0/1159556522 shutdown_connections 2024-03-17T23:47:55.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.450+0000 7f67f284b640 1 -- 172.21.15.150:0/1159556522 wait complete. 2024-03-17T23:47:55.462 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-17T23:47:55.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.565+0000 7fd950fc4640 1 Processor -- start 2024-03-17T23:47:55.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.565+0000 7fd950fc4640 1 -- start start 2024-03-17T23:47:55.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.565+0000 7fd950fc4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd94c0ce6a0 0x7fd94c166a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:55.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.565+0000 7fd950fc4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd94c145170 con 0x7fd94c144bb0 2024-03-17T23:47:55.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.565+0000 7fd950fc4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd94c145450 con 0x7fd94c06f400 2024-03-17T23:47:55.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.565+0000 7fd950fc4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd94c145730 con 0x7fd94c0ce6a0 2024-03-17T23:47:55.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.565+0000 7fd94a59c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd94c144bb0 0x7fd94c15ce50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33468/0 (socket says 172.21.15.150:33468) 2024-03-17T23:47:55.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.565+0000 7fd94a59c640 1 -- 172.21.15.150:0/1424294151 learned_addr learned my addr 172.21.15.150:0/1424294151 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:55.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.565+0000 7fd94ad9d640 1 --2- 172.21.15.150:0/1424294151 >> v2:172.21.15.157:3300/0 conn(0x7fd94c0ce6a0 0x7fd94c166a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:55.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.565+0000 7fd949d9b640 1 -- 172.21.15.150:0/1424294151 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1523792690 0 0) 0x7fd94c145450 con 0x7fd94c06f400 2024-03-17T23:47:55.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.565+0000 7fd949d9b640 1 -- 172.21.15.150:0/1424294151 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd938003520 con 0x7fd94c06f400 2024-03-17T23:47:55.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.565+0000 7fd949d9b640 1 -- 172.21.15.150:0/1424294151 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1723742242 0 0) 0x7fd94c145170 con 0x7fd94c144bb0 2024-03-17T23:47:55.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.565+0000 7fd949d9b640 1 -- 172.21.15.150:0/1424294151 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd94c145450 con 0x7fd94c144bb0 2024-03-17T23:47:55.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.565+0000 7fd94ad9d640 1 -- 172.21.15.150:0/1424294151 >> v1:172.21.15.157:6789/0 conn(0x7fd94c06f400 legacy=0x7fd94c0cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:55.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.565+0000 7fd94ad9d640 1 -- 172.21.15.150:0/1424294151 >> v1:172.21.15.150:6789/0 conn(0x7fd94c144bb0 legacy=0x7fd94c15ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:55.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.565+0000 7fd94ad9d640 1 -- 172.21.15.150:0/1424294151 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd94c15d560 con 0x7fd94c0ce6a0 2024-03-17T23:47:55.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.566+0000 7fd94ad9d640 1 --2- 172.21.15.150:0/1424294151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd94c0ce6a0 0x7fd94c166a40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd944009d90 tx=0x7fd94402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfc28c0c9bea46b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:55.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.566+0000 7fd949d9b640 1 -- 172.21.15.150:0/1424294151 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd944039070 con 0x7fd94c0ce6a0 2024-03-17T23:47:55.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.566+0000 7fd949d9b640 1 -- 172.21.15.150:0/1424294151 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd94402cc60 con 0x7fd94c0ce6a0 2024-03-17T23:47:55.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.566+0000 7fd949d9b640 1 -- 172.21.15.150:0/1424294151 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd944034660 con 0x7fd94c0ce6a0 2024-03-17T23:47:55.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.567+0000 7fd950fc4640 1 -- 172.21.15.150:0/1424294151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd94c0ce6a0 msgr2=0x7fd94c166a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:55.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.567+0000 7fd950fc4640 1 --2- 172.21.15.150:0/1424294151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd94c0ce6a0 0x7fd94c166a40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd944009d90 tx=0x7fd94402beb0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.567+0000 7fd950fc4640 1 -- 172.21.15.150:0/1424294151 shutdown_connections 2024-03-17T23:47:55.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.567+0000 7fd950fc4640 1 --2- 172.21.15.150:0/1424294151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd94c0ce6a0 0x7fd94c166a40 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.567+0000 7fd950fc4640 1 -- 172.21.15.150:0/1424294151 >> 172.21.15.150:0/1424294151 conn(0x7fd94c06b190 msgr2=0x7fd94c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:55.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.567+0000 7fd950fc4640 1 -- 172.21.15.150:0/1424294151 shutdown_connections 2024-03-17T23:47:55.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.567+0000 7fd950fc4640 1 -- 172.21.15.150:0/1424294151 wait complete. 2024-03-17T23:47:55.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.568+0000 7fd950fc4640 1 Processor -- start 2024-03-17T23:47:55.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.568+0000 7fd950fc4640 1 -- start start 2024-03-17T23:47:55.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.569+0000 7fd950fc4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd94c06f400 0x7fd94c0b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:55.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.569+0000 7fd950fc4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd94c0ce6a0 0x7fd94c15a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:55.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.569+0000 7fd950fc4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd94c144bb0 0x7fd94c15ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:55.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.569+0000 7fd950fc4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd94c167100 con 0x7fd94c0ce6a0 2024-03-17T23:47:55.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.569+0000 7fd950fc4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd94c166f80 con 0x7fd94c144bb0 2024-03-17T23:47:55.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.569+0000 7fd950fc4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd94c167280 con 0x7fd94c06f400 2024-03-17T23:47:55.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.569+0000 7fd943fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd94c0ce6a0 0x7fd94c15a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:55.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.569+0000 7fd943fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd94c0ce6a0 0x7fd94c15a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45578/0 (socket says 172.21.15.150:45578) 2024-03-17T23:47:55.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.569+0000 7fd943fff640 1 -- 172.21.15.150:0/619295205 learned_addr learned my addr 172.21.15.150:0/619295205 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:55.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.569+0000 7fd94ad9d640 1 --2- 172.21.15.150:0/619295205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd94c144bb0 0x7fd94c15ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:55.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.569+0000 7fd94a59c640 1 --2- 172.21.15.150:0/619295205 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd94c06f400 0x7fd94c0b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:55.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.569+0000 7fd943fff640 1 -- 172.21.15.150:0/619295205 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd94c06f400 msgr2=0x7fd94c0b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:55.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.569+0000 7fd943fff640 1 --2- 172.21.15.150:0/619295205 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd94c06f400 0x7fd94c0b0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.569+0000 7fd943fff640 1 -- 172.21.15.150:0/619295205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd94c144bb0 msgr2=0x7fd94c15ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:55.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.569+0000 7fd943fff640 1 --2- 172.21.15.150:0/619295205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd94c144bb0 0x7fd94c15ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.569+0000 7fd943fff640 1 -- 172.21.15.150:0/619295205 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd94c0aad70 con 0x7fd94c0ce6a0 2024-03-17T23:47:55.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.569+0000 7fd943fff640 1 --2- 172.21.15.150:0/619295205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd94c0ce6a0 0x7fd94c15a6d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd93400c430 tx=0x7fd93400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:55.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.569+0000 7fd942ffd640 1 -- 172.21.15.150:0/619295205 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd934017070 con 0x7fd94c0ce6a0 2024-03-17T23:47:55.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.569+0000 7fd942ffd640 1 -- 172.21.15.150:0/619295205 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9340052d0 con 0x7fd94c0ce6a0 2024-03-17T23:47:55.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.569+0000 7fd942ffd640 1 -- 172.21.15.150:0/619295205 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd934003830 con 0x7fd94c0ce6a0 2024-03-17T23:47:55.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.569+0000 7fd950fc4640 1 -- 172.21.15.150:0/619295205 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd94c0ab050 con 0x7fd94c0ce6a0 2024-03-17T23:47:55.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.570+0000 7fd950fc4640 1 -- 172.21.15.150:0/619295205 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd94c13a420 con 0x7fd94c0ce6a0 2024-03-17T23:47:55.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.570+0000 7fd950fc4640 1 -- 172.21.15.150:0/619295205 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd94c0681a0 con 0x7fd94c0ce6a0 2024-03-17T23:47:55.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.571+0000 7fd942ffd640 1 -- 172.21.15.150:0/619295205 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd93401a070 con 0x7fd94c0ce6a0 2024-03-17T23:47:55.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.571+0000 7fd942ffd640 1 --2- 172.21.15.150:0/619295205 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd92c06bbd0 0x7fd92c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:55.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.571+0000 7fd942ffd640 1 -- 172.21.15.150:0/619295205 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fd93408a160 con 0x7fd94c0ce6a0 2024-03-17T23:47:55.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.571+0000 7fd94a59c640 1 --2- 172.21.15.150:0/619295205 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd92c06bbd0 0x7fd92c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:55.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.572+0000 7fd94a59c640 1 --2- 172.21.15.150:0/619295205 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd92c06bbd0 0x7fd92c06e090 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd93c000f60 tx=0x7fd93c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:55.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.576+0000 7fd942ffd640 1 -- 172.21.15.150:0/619295205 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd934058290 con 0x7fd94c0ce6a0 2024-03-17T23:47:55.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.748+0000 7fd950fc4640 1 -- 172.21.15.150:0/619295205 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd94c0cf3c0 con 0x7fd94c0ce6a0 2024-03-17T23:47:55.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.749+0000 7fd942ffd640 1 -- 172.21.15.150:0/619295205 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7fd93405c330 con 0x7fd94c0ce6a0 2024-03-17T23:47:55.750 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:55.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.751+0000 7fd950fc4640 1 -- 172.21.15.150:0/619295205 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd92c06bbd0 msgr2=0x7fd92c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:55.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.751+0000 7fd950fc4640 1 --2- 172.21.15.150:0/619295205 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd92c06bbd0 0x7fd92c06e090 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd93c000f60 tx=0x7fd93c009040 comp rx=0 tx=0).stop 2024-03-17T23:47:55.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.751+0000 7fd950fc4640 1 -- 172.21.15.150:0/619295205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd94c0ce6a0 msgr2=0x7fd94c15a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:55.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.751+0000 7fd950fc4640 1 --2- 172.21.15.150:0/619295205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd94c0ce6a0 0x7fd94c15a6d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd93400c430 tx=0x7fd93400c8f0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.751+0000 7fd950fc4640 1 -- 172.21.15.150:0/619295205 shutdown_connections 2024-03-17T23:47:55.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.751+0000 7fd950fc4640 1 --2- 172.21.15.150:0/619295205 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd92c06bbd0 0x7fd92c06e090 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.751+0000 7fd950fc4640 1 --2- 172.21.15.150:0/619295205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd94c144bb0 0x7fd94c15ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.751+0000 7fd950fc4640 1 --2- 172.21.15.150:0/619295205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd94c0ce6a0 0x7fd94c15a6d0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.751+0000 7fd950fc4640 1 --2- 172.21.15.150:0/619295205 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd94c06f400 0x7fd94c0b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.751+0000 7fd950fc4640 1 -- 172.21.15.150:0/619295205 >> 172.21.15.150:0/619295205 conn(0x7fd94c06b190 msgr2=0x7fd94c0cdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:55.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.751+0000 7fd950fc4640 1 -- 172.21.15.150:0/619295205 shutdown_connections 2024-03-17T23:47:55.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.751+0000 7fd950fc4640 1 -- 172.21.15.150:0/619295205 wait complete. 2024-03-17T23:47:55.764 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-17T23:47:55.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.866+0000 7f5b32d28640 1 Processor -- start 2024-03-17T23:47:55.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.866+0000 7f5b32d28640 1 -- start start 2024-03-17T23:47:55.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.867+0000 7f5b32d28640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5b2c0ccf90 0x7f5b2c0cf100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:55.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.867+0000 7f5b32d28640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b2c15bce0 0x7f5b2c0cd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:55.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.867+0000 7f5b32d28640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b2c0cd8b0 con 0x7f5b2c15e830 2024-03-17T23:47:55.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.867+0000 7f5b32d28640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f5b2c06e420 con 0x7f5b2c0ccf90 2024-03-17T23:47:55.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.867+0000 7f5b32d28640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5b2c0cf8a0 con 0x7f5b2c15bce0 2024-03-17T23:47:55.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.867+0000 7f5b30ac4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5b2c0ccf90 0x7f5b2c0cf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:55.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.867+0000 7f5b312c5640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f5b2c15e830 0x7f5b2c15ecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:45686/0 (socket says 172.21.15.150:45686) 2024-03-17T23:47:55.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.867+0000 7f5b312c5640 1 -- 172.21.15.150:0/503471408 learned_addr learned my addr 172.21.15.150:0/503471408 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:55.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.867+0000 7f5b2bfff640 1 --2- 172.21.15.150:0/503471408 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b2c15bce0 0x7f5b2c0cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:55.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.867+0000 7f5b2b7fe640 1 -- 172.21.15.150:0/503471408 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3649998858 0 0) 0x7f5b2c0cd8b0 con 0x7f5b2c15e830 2024-03-17T23:47:55.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.867+0000 7f5b2b7fe640 1 -- 172.21.15.150:0/503471408 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b1c003520 con 0x7f5b2c15e830 2024-03-17T23:47:55.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.867+0000 7f5b30ac4640 1 -- 172.21.15.150:0/503471408 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b2c15bce0 msgr2=0x7f5b2c0cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:55.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.867+0000 7f5b30ac4640 1 --2- 172.21.15.150:0/503471408 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b2c15bce0 0x7f5b2c0cd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.867+0000 7f5b30ac4640 1 -- 172.21.15.150:0/503471408 >> v1:172.21.15.157:6789/0 conn(0x7f5b2c15e830 legacy=0x7f5b2c15ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:55.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.867+0000 7f5b30ac4640 1 -- 172.21.15.150:0/503471408 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b2c06de60 con 0x7f5b2c0ccf90 2024-03-17T23:47:55.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.867+0000 7f5b2bfff640 1 --2- 172.21.15.150:0/503471408 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b2c15bce0 0x7f5b2c0cd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:55.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.868+0000 7f5b30ac4640 1 --2- 172.21.15.150:0/503471408 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b2c0ccf90 0x7f5b2c0cf100 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5b200097a0 tx=0x7f5b2002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cde2d4c69676788e server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:55.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.868+0000 7f5b2b7fe640 1 -- 172.21.15.150:0/503471408 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b200315a0 con 0x7f5b2c0ccf90 2024-03-17T23:47:55.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.868+0000 7f5b2b7fe640 1 -- 172.21.15.150:0/503471408 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b20037070 con 0x7f5b2c0ccf90 2024-03-17T23:47:55.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.868+0000 7f5b2b7fe640 1 -- 172.21.15.150:0/503471408 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b20031940 con 0x7f5b2c0ccf90 2024-03-17T23:47:55.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.869+0000 7f5b32d28640 1 -- 172.21.15.150:0/503471408 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b2c0ccf90 msgr2=0x7f5b2c0cf100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:55.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.869+0000 7f5b32d28640 1 --2- 172.21.15.150:0/503471408 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b2c0ccf90 0x7f5b2c0cf100 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5b200097a0 tx=0x7f5b2002b840 comp rx=0 tx=0).stop 2024-03-17T23:47:55.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.869+0000 7f5b32d28640 1 -- 172.21.15.150:0/503471408 shutdown_connections 2024-03-17T23:47:55.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.869+0000 7f5b32d28640 1 --2- 172.21.15.150:0/503471408 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b2c15bce0 0x7f5b2c0cd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.869+0000 7f5b32d28640 1 --2- 172.21.15.150:0/503471408 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b2c0ccf90 0x7f5b2c0cf100 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.869+0000 7f5b32d28640 1 -- 172.21.15.150:0/503471408 >> 172.21.15.150:0/503471408 conn(0x7f5b2c06b190 msgr2=0x7f5b2c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:55.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.869+0000 7f5b32d28640 1 -- 172.21.15.150:0/503471408 shutdown_connections 2024-03-17T23:47:55.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.869+0000 7f5b32d28640 1 -- 172.21.15.150:0/503471408 wait complete. 2024-03-17T23:47:55.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.870+0000 7f5b32d28640 1 Processor -- start 2024-03-17T23:47:55.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.870+0000 7f5b32d28640 1 -- start start 2024-03-17T23:47:55.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.871+0000 7f5b32d28640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b2c0ccf90 0x7f5b2c139c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:55.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.871+0000 7f5b32d28640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b2c15bce0 0x7f5b2c14def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:55.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.871+0000 7f5b32d28640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b2c15e830 0x7f5b2c0b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:55.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.871+0000 7f5b32d28640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2c15c5a0 con 0x7f5b2c0ccf90 2024-03-17T23:47:55.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.871+0000 7f5b32d28640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5b2c0cf640 con 0x7f5b2c15bce0 2024-03-17T23:47:55.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.871+0000 7f5b32d28640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5b2c15c160 con 0x7f5b2c15e830 2024-03-17T23:47:55.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.871+0000 7f5b30ac4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b2c0ccf90 0x7f5b2c139c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:55.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.871+0000 7f5b30ac4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b2c0ccf90 0x7f5b2c139c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45586/0 (socket says 172.21.15.150:45586) 2024-03-17T23:47:55.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.871+0000 7f5b30ac4640 1 -- 172.21.15.150:0/265745090 learned_addr learned my addr 172.21.15.150:0/265745090 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:55.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.871+0000 7f5b2bfff640 1 --2- 172.21.15.150:0/265745090 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b2c15bce0 0x7f5b2c14def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:55.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.871+0000 7f5b312c5640 1 --2- 172.21.15.150:0/265745090 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b2c15e830 0x7f5b2c0b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:55.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.871+0000 7f5b30ac4640 1 -- 172.21.15.150:0/265745090 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b2c15e830 msgr2=0x7f5b2c0b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:55.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.871+0000 7f5b30ac4640 1 --2- 172.21.15.150:0/265745090 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b2c15e830 0x7f5b2c0b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.871+0000 7f5b30ac4640 1 -- 172.21.15.150:0/265745090 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b2c15bce0 msgr2=0x7f5b2c14def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:55.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.871+0000 7f5b30ac4640 1 --2- 172.21.15.150:0/265745090 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b2c15bce0 0x7f5b2c14def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:55.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.871+0000 7f5b30ac4640 1 -- 172.21.15.150:0/265745090 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b2c1423b0 con 0x7f5b2c0ccf90 2024-03-17T23:47:55.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.871+0000 7f5b2bfff640 1 --2- 172.21.15.150:0/265745090 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b2c15bce0 0x7f5b2c14def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:55.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.871+0000 7f5b30ac4640 1 --2- 172.21.15.150:0/265745090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b2c0ccf90 0x7f5b2c139c90 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5b20034040 tx=0x7f5b200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:55.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.871+0000 7f5b297fa640 1 -- 172.21.15.150:0/265745090 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b20032630 con 0x7f5b2c0ccf90 2024-03-17T23:47:55.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.871+0000 7f5b297fa640 1 -- 172.21.15.150:0/265745090 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b200327f0 con 0x7f5b2c0ccf90 2024-03-17T23:47:55.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.871+0000 7f5b32d28640 1 -- 172.21.15.150:0/265745090 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b2c142690 con 0x7f5b2c0ccf90 2024-03-17T23:47:55.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.871+0000 7f5b297fa640 1 -- 172.21.15.150:0/265745090 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b20032be0 con 0x7f5b2c0ccf90 2024-03-17T23:47:55.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.871+0000 7f5b32d28640 1 -- 172.21.15.150:0/265745090 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b2c142c50 con 0x7f5b2c0ccf90 2024-03-17T23:47:55.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.873+0000 7f5b32d28640 1 -- 172.21.15.150:0/265745090 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b2c0681a0 con 0x7f5b2c0ccf90 2024-03-17T23:47:55.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.873+0000 7f5b297fa640 1 -- 172.21.15.150:0/265745090 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5b2003b3f0 con 0x7f5b2c0ccf90 2024-03-17T23:47:55.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.873+0000 7f5b297fa640 1 --2- 172.21.15.150:0/265745090 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5b1006bc00 0x7f5b1006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:55.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.873+0000 7f5b297fa640 1 -- 172.21.15.150:0/265745090 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f5b2002f030 con 0x7f5b2c0ccf90 2024-03-17T23:47:55.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.873+0000 7f5b2bfff640 1 --2- 172.21.15.150:0/265745090 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5b1006bc00 0x7f5b1006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:55.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.874+0000 7f5b2bfff640 1 --2- 172.21.15.150:0/265745090 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5b1006bc00 0x7f5b1006e0c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c158a90 tx=0x7f5b18009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:55.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:55.877+0000 7f5b297fa640 1 -- 172.21.15.150:0/265745090 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5b20040070 con 0x7f5b2c0ccf90 2024-03-17T23:47:56.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.076+0000 7f5b32d28640 1 -- 172.21.15.150:0/265745090 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5b2c06fff0 con 0x7f5b2c0ccf90 2024-03-17T23:47:56.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.077+0000 7f5b297fa640 1 -- 172.21.15.150:0/265745090 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) ==== 80+0+0 (secure 0 0 0) 0x7f5b2007aac0 con 0x7f5b2c0ccf90 2024-03-17T23:47:56.077 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-03-17T23:47:56.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.080+0000 7f5b32d28640 1 -- 172.21.15.150:0/265745090 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5b1006bc00 msgr2=0x7f5b1006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:56.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.080+0000 7f5b32d28640 1 --2- 172.21.15.150:0/265745090 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5b1006bc00 0x7f5b1006e0c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c158a90 tx=0x7f5b18009190 comp rx=0 tx=0).stop 2024-03-17T23:47:56.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.080+0000 7f5b32d28640 1 -- 172.21.15.150:0/265745090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b2c0ccf90 msgr2=0x7f5b2c139c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:56.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.080+0000 7f5b32d28640 1 --2- 172.21.15.150:0/265745090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b2c0ccf90 0x7f5b2c139c90 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5b20034040 tx=0x7f5b200046c0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.080+0000 7f5b32d28640 1 -- 172.21.15.150:0/265745090 shutdown_connections 2024-03-17T23:47:56.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.080+0000 7f5b32d28640 1 --2- 172.21.15.150:0/265745090 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5b2c15e830 0x7f5b2c0b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.080+0000 7f5b32d28640 1 --2- 172.21.15.150:0/265745090 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5b1006bc00 0x7f5b1006e0c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.080+0000 7f5b32d28640 1 --2- 172.21.15.150:0/265745090 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5b2c15bce0 0x7f5b2c14def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.080+0000 7f5b32d28640 1 --2- 172.21.15.150:0/265745090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5b2c0ccf90 0x7f5b2c139c90 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.080+0000 7f5b32d28640 1 -- 172.21.15.150:0/265745090 >> 172.21.15.150:0/265745090 conn(0x7f5b2c06b190 msgr2=0x7f5b2c159640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:56.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.080+0000 7f5b32d28640 1 -- 172.21.15.150:0/265745090 shutdown_connections 2024-03-17T23:47:56.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.080+0000 7f5b32d28640 1 -- 172.21.15.150:0/265745090 wait complete. 2024-03-17T23:47:56.095 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi157.front.sepia.ceph.com... 2024-03-17T23:47:56.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:56.095 DEBUG:teuthology.orchestra.run.smithi157:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-03-17T23:47:56.122 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:47:56.123 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi157.front.sepia.ceph.com 2024-03-17T23:47:56.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:47:56.123 DEBUG:teuthology.orchestra.run.smithi157:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-17T23:47:56.178 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-17T23:47:56.178 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:47:56.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.277+0000 7f02cb7bc640 1 Processor -- start 2024-03-17T23:47:56.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.277+0000 7f02cb7bc640 1 -- start start 2024-03-17T23:47:56.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.278+0000 7f02cb7bc640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f02c41514f0 0x7f02c415bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:56.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.278+0000 7f02cb7bc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f02c415ab90 0x7f02c406f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:56.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.278+0000 7f02cb7bc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f02c406e5c0 0x7f02c406ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:56.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.278+0000 7f02cb7bc640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f02c406fba0 con 0x7f02c415ab90 2024-03-17T23:47:56.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.278+0000 7f02cb7bc640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f02c406efb0 con 0x7f02c41514f0 2024-03-17T23:47:56.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.278+0000 7f02cb7bc640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f02c40ce790 con 0x7f02c406e5c0 2024-03-17T23:47:56.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.278+0000 7f02c8d57640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f02c415ab90 0x7f02c406f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:56.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.278+0000 7f02c8d57640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f02c415ab90 0x7f02c406f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45594/0 (socket says 172.21.15.150:45594) 2024-03-17T23:47:56.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.278+0000 7f02c8d57640 1 -- 172.21.15.150:0/2511492859 learned_addr learned my addr 172.21.15.150:0/2511492859 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:56.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.278+0000 7f02c9d59640 1 --2- 172.21.15.150:0/2511492859 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f02c406e5c0 0x7f02c406ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:56.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.278+0000 7f02c9558640 1 --2- 172.21.15.150:0/2511492859 >> v2:172.21.15.157:3300/0 conn(0x7f02c41514f0 0x7f02c415bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:56.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.278+0000 7f02c8d57640 1 -- 172.21.15.150:0/2511492859 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f02c406e5c0 msgr2=0x7f02c406ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:56.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.278+0000 7f02c8d57640 1 --2- 172.21.15.150:0/2511492859 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f02c406e5c0 0x7f02c406ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.279+0000 7f02c8d57640 1 -- 172.21.15.150:0/2511492859 >> v2:172.21.15.157:3300/0 conn(0x7f02c41514f0 msgr2=0x7f02c415bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:56.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.279+0000 7f02c8d57640 1 --2- 172.21.15.150:0/2511492859 >> v2:172.21.15.157:3300/0 conn(0x7f02c41514f0 0x7f02c415bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.279+0000 7f02c8d57640 1 -- 172.21.15.150:0/2511492859 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02c415b050 con 0x7f02c415ab90 2024-03-17T23:47:56.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.279+0000 7f02c8d57640 1 --2- 172.21.15.150:0/2511492859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02c415ab90 0x7f02c406f660 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f02c4072d90 tx=0x7f02b402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f258567141d9c8e server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:56.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.279+0000 7f02bbfff640 1 -- 172.21.15.150:0/2511492859 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b4035020 con 0x7f02c415ab90 2024-03-17T23:47:56.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.279+0000 7f02bbfff640 1 -- 172.21.15.150:0/2511492859 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02b4031870 con 0x7f02c415ab90 2024-03-17T23:47:56.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.279+0000 7f02bbfff640 1 -- 172.21.15.150:0/2511492859 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b4031c40 con 0x7f02c415ab90 2024-03-17T23:47:56.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.280+0000 7f02cb7bc640 1 -- 172.21.15.150:0/2511492859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02c415ab90 msgr2=0x7f02c406f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:56.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.280+0000 7f02cb7bc640 1 --2- 172.21.15.150:0/2511492859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02c415ab90 0x7f02c406f660 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f02c4072d90 tx=0x7f02b402b910 comp rx=0 tx=0).stop 2024-03-17T23:47:56.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.280+0000 7f02cb7bc640 1 -- 172.21.15.150:0/2511492859 shutdown_connections 2024-03-17T23:47:56.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.280+0000 7f02cb7bc640 1 --2- 172.21.15.150:0/2511492859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02c415ab90 0x7f02c406f660 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.280+0000 7f02cb7bc640 1 --2- 172.21.15.150:0/2511492859 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f02c406e5c0 0x7f02c406ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.280+0000 7f02cb7bc640 1 --2- 172.21.15.150:0/2511492859 >> v2:172.21.15.157:3300/0 conn(0x7f02c41514f0 0x7f02c415bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.280+0000 7f02cb7bc640 1 -- 172.21.15.150:0/2511492859 >> 172.21.15.150:0/2511492859 conn(0x7f02c406b190 msgr2=0x7f02c406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:56.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.280+0000 7f02cb7bc640 1 -- 172.21.15.150:0/2511492859 shutdown_connections 2024-03-17T23:47:56.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.280+0000 7f02cb7bc640 1 -- 172.21.15.150:0/2511492859 wait complete. 2024-03-17T23:47:56.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.281+0000 7f02cb7bc640 1 Processor -- start 2024-03-17T23:47:56.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.281+0000 7f02cb7bc640 1 -- start start 2024-03-17T23:47:56.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.282+0000 7f02cb7bc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02c406e5c0 0x7f02c4135070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:56.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.282+0000 7f02cb7bc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02c41514f0 0x7f02c40ae760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:56.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.282+0000 7f02cb7bc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f02c415ab90 0x7f02c40aeca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:56.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.282+0000 7f02cb7bc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f02c415b8f0 con 0x7f02c406e5c0 2024-03-17T23:47:56.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.282+0000 7f02cb7bc640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f02c415b610 con 0x7f02c41514f0 2024-03-17T23:47:56.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.282+0000 7f02cb7bc640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f02c40af510 con 0x7f02c415ab90 2024-03-17T23:47:56.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.282+0000 7f02c9558640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02c406e5c0 0x7f02c4135070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:56.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.282+0000 7f02c9558640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02c406e5c0 0x7f02c4135070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45602/0 (socket says 172.21.15.150:45602) 2024-03-17T23:47:56.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.282+0000 7f02c9558640 1 -- 172.21.15.150:0/297966994 learned_addr learned my addr 172.21.15.150:0/297966994 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:56.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.282+0000 7f02c8d57640 1 --2- 172.21.15.150:0/297966994 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02c41514f0 0x7f02c40ae760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:56.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.282+0000 7f02c9d59640 1 --2- 172.21.15.150:0/297966994 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f02c415ab90 0x7f02c40aeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:56.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.282+0000 7f02c9558640 1 -- 172.21.15.150:0/297966994 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f02c415ab90 msgr2=0x7f02c40aeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:56.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.282+0000 7f02c9558640 1 --2- 172.21.15.150:0/297966994 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f02c415ab90 0x7f02c40aeca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.282+0000 7f02c9558640 1 -- 172.21.15.150:0/297966994 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02c41514f0 msgr2=0x7f02c40ae760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:56.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.282+0000 7f02c9558640 1 --2- 172.21.15.150:0/297966994 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02c41514f0 0x7f02c40ae760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.282+0000 7f02c9558640 1 -- 172.21.15.150:0/297966994 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02c4151a00 con 0x7f02c406e5c0 2024-03-17T23:47:56.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.282+0000 7f02c9558640 1 --2- 172.21.15.150:0/297966994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02c406e5c0 0x7f02c4135070 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f02c0009db0 tx=0x7f02c000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:56.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.283+0000 7f02b9ffb640 1 -- 172.21.15.150:0/297966994 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c0018020 con 0x7f02c406e5c0 2024-03-17T23:47:56.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.283+0000 7f02b9ffb640 1 -- 172.21.15.150:0/297966994 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02c0014480 con 0x7f02c406e5c0 2024-03-17T23:47:56.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.283+0000 7f02b9ffb640 1 -- 172.21.15.150:0/297966994 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c0014850 con 0x7f02c406e5c0 2024-03-17T23:47:56.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.283+0000 7f02cb7bc640 1 -- 172.21.15.150:0/297966994 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02c4151ce0 con 0x7f02c406e5c0 2024-03-17T23:47:56.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.283+0000 7f02cb7bc640 1 -- 172.21.15.150:0/297966994 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02c41522a0 con 0x7f02c406e5c0 2024-03-17T23:47:56.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.284+0000 7f02cb7bc640 1 -- 172.21.15.150:0/297966994 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02c40681a0 con 0x7f02c406e5c0 2024-03-17T23:47:56.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.284+0000 7f02b9ffb640 1 -- 172.21.15.150:0/297966994 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f02c0005d10 con 0x7f02c406e5c0 2024-03-17T23:47:56.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.284+0000 7f02b9ffb640 1 --2- 172.21.15.150:0/297966994 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f02a806bbd0 0x7f02a806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:56.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.285+0000 7f02b9ffb640 1 -- 172.21.15.150:0/297966994 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f02c008b830 con 0x7f02c406e5c0 2024-03-17T23:47:56.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.285+0000 7f02c8d57640 1 --2- 172.21.15.150:0/297966994 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f02a806bbd0 0x7f02a806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:56.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.285+0000 7f02c8d57640 1 --2- 172.21.15.150:0/297966994 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f02a806bbd0 0x7f02a806e090 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f02b4036000 tx=0x7f02b403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:56.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.289+0000 7f02b9ffb640 1 -- 172.21.15.150:0/297966994 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f02c0022070 con 0x7f02c406e5c0 2024-03-17T23:47:56.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.501+0000 7f02cb7bc640 1 -- 172.21.15.150:0/297966994 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02c406bf90 con 0x7f02c406e5c0 2024-03-17T23:47:56.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.501+0000 7f02b9ffb640 1 -- 172.21.15.150:0/297966994 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5186 (secure 0 0 0) 0x7f02c0010040 con 0x7f02c406e5c0 2024-03-17T23:47:56.502 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:56.502 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/1949609985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6833","nonce":1949609985},{"type":"v1","addr":"172.21.15.157:6835","nonce":1949609985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/758264728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":758264728},{"type":"v1","addr":"172.21.15.150:6835","nonce":758264728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3655589010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3655589010},{"type":"v1","addr":"172.21.15.150:6837","nonce":3655589010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-17T23:46:55.253739+0000","modified":"2024-03-17T23:47:11.316249+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6834/1625058792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1625058792},{"type":"v1","addr":"172.21.15.157:6834","nonce":1625058792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4297,"qdb_cluster":[4297]},"id":1}]} 2024-03-17T23:47:56.503 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 8 2024-03-17T23:47:56.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.503+0000 7f02cb7bc640 1 -- 172.21.15.150:0/297966994 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f02a806bbd0 msgr2=0x7f02a806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:56.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.503+0000 7f02cb7bc640 1 --2- 172.21.15.150:0/297966994 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f02a806bbd0 0x7f02a806e090 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f02b4036000 tx=0x7f02b403c040 comp rx=0 tx=0).stop 2024-03-17T23:47:56.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.503+0000 7f02cb7bc640 1 -- 172.21.15.150:0/297966994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02c406e5c0 msgr2=0x7f02c4135070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:56.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.503+0000 7f02cb7bc640 1 --2- 172.21.15.150:0/297966994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02c406e5c0 0x7f02c4135070 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f02c0009db0 tx=0x7f02c000c490 comp rx=0 tx=0).stop 2024-03-17T23:47:56.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.503+0000 7f02cb7bc640 1 -- 172.21.15.150:0/297966994 shutdown_connections 2024-03-17T23:47:56.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.503+0000 7f02cb7bc640 1 --2- 172.21.15.150:0/297966994 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f02c415ab90 0x7f02c40aeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.503+0000 7f02cb7bc640 1 --2- 172.21.15.150:0/297966994 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f02a806bbd0 0x7f02a806e090 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.503+0000 7f02cb7bc640 1 --2- 172.21.15.150:0/297966994 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02c41514f0 0x7f02c40ae760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.504+0000 7f02cb7bc640 1 --2- 172.21.15.150:0/297966994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02c406e5c0 0x7f02c4135070 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.504+0000 7f02cb7bc640 1 -- 172.21.15.150:0/297966994 >> 172.21.15.150:0/297966994 conn(0x7f02c406b190 msgr2=0x7f02c40b4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:56.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.504+0000 7f02cb7bc640 1 -- 172.21.15.150:0/297966994 shutdown_connections 2024-03-17T23:47:56.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.504+0000 7f02cb7bc640 1 -- 172.21.15.150:0/297966994 wait complete. 2024-03-17T23:47:56.515 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:56.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.618+0000 7f7000d1b640 1 Processor -- start 2024-03-17T23:47:56.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.618+0000 7f7000d1b640 1 -- start start 2024-03-17T23:47:56.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.619+0000 7f7000d1b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ffc15de30 0x7f6ffc1619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:56.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.619+0000 7f7000d1b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6ffc0b9860 0x7f6ffc0cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:56.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.619+0000 7f7000d1b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6ffc0cef50 0x7f6ffc06c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:56.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.619+0000 7f7000d1b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f6ffc0cf3c0 con 0x7f6ffc0b9860 2024-03-17T23:47:56.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.619+0000 7f7000d1b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f6ffc162940 con 0x7f6ffc0cef50 2024-03-17T23:47:56.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.619+0000 7f7000d1b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6ffc0b9cc0 con 0x7f6ffc15de30 2024-03-17T23:47:56.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.620+0000 7f6ff9d9b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6ffc0b9860 0x7f6ffc0cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:56.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.620+0000 7f6ff9d9b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6ffc0b9860 0x7f6ffc0cea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45618/0 (socket says 172.21.15.150:45618) 2024-03-17T23:47:56.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.620+0000 7f6ff9d9b640 1 -- 172.21.15.150:0/1246134446 learned_addr learned my addr 172.21.15.150:0/1246134446 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:56.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.620+0000 7f6ffad9d640 1 --2- 172.21.15.150:0/1246134446 >> v2:172.21.15.157:3300/0 conn(0x7f6ffc0cef50 0x7f6ffc06c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:56.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.620+0000 7f6ffa59c640 1 --2- 172.21.15.150:0/1246134446 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ffc15de30 0x7f6ffc1619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:56.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.620+0000 7f6ff9d9b640 1 -- 172.21.15.150:0/1246134446 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ffc15de30 msgr2=0x7f6ffc1619a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:56.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.620+0000 7f6ff9d9b640 1 --2- 172.21.15.150:0/1246134446 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ffc15de30 0x7f6ffc1619a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.620+0000 7f6ff9d9b640 1 -- 172.21.15.150:0/1246134446 >> v2:172.21.15.157:3300/0 conn(0x7f6ffc0cef50 msgr2=0x7f6ffc06c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:56.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.620+0000 7f6ff9d9b640 1 --2- 172.21.15.150:0/1246134446 >> v2:172.21.15.157:3300/0 conn(0x7f6ffc0cef50 0x7f6ffc06c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.620+0000 7f6ff9d9b640 1 -- 172.21.15.150:0/1246134446 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ffc1620a0 con 0x7f6ffc0b9860 2024-03-17T23:47:56.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.621+0000 7f6ff9d9b640 1 --2- 172.21.15.150:0/1246134446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ffc0b9860 0x7f6ffc0cea10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc06dca0 tx=0x7f6ff002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f922b561c6dd6604 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:56.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.621+0000 7f6ff959a640 1 -- 172.21.15.150:0/1246134446 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff002e070 con 0x7f6ffc0b9860 2024-03-17T23:47:56.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.621+0000 7f6ff959a640 1 -- 172.21.15.150:0/1246134446 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ff0037070 con 0x7f6ffc0b9860 2024-03-17T23:47:56.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.621+0000 7f6ff959a640 1 -- 172.21.15.150:0/1246134446 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff0031a00 con 0x7f6ffc0b9860 2024-03-17T23:47:56.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.622+0000 7f7000d1b640 1 -- 172.21.15.150:0/1246134446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ffc0b9860 msgr2=0x7f6ffc0cea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:56.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.622+0000 7f7000d1b640 1 --2- 172.21.15.150:0/1246134446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ffc0b9860 0x7f6ffc0cea10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc06dca0 tx=0x7f6ff002b880 comp rx=0 tx=0).stop 2024-03-17T23:47:56.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.622+0000 7f7000d1b640 1 -- 172.21.15.150:0/1246134446 shutdown_connections 2024-03-17T23:47:56.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.622+0000 7f7000d1b640 1 --2- 172.21.15.150:0/1246134446 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ffc0b9860 0x7f6ffc0cea10 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.622+0000 7f7000d1b640 1 --2- 172.21.15.150:0/1246134446 >> v2:172.21.15.157:3300/0 conn(0x7f6ffc0cef50 0x7f6ffc06c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.622+0000 7f7000d1b640 1 --2- 172.21.15.150:0/1246134446 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ffc15de30 0x7f6ffc1619a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.622+0000 7f7000d1b640 1 -- 172.21.15.150:0/1246134446 >> 172.21.15.150:0/1246134446 conn(0x7f6ffc0730b0 msgr2=0x7f6ffc0d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:56.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.622+0000 7f7000d1b640 1 -- 172.21.15.150:0/1246134446 shutdown_connections 2024-03-17T23:47:56.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.622+0000 7f7000d1b640 1 -- 172.21.15.150:0/1246134446 wait complete. 2024-03-17T23:47:56.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.623+0000 7f7000d1b640 1 Processor -- start 2024-03-17T23:47:56.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.623+0000 7f7000d1b640 1 -- start start 2024-03-17T23:47:56.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.624+0000 7f7000d1b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ffc0b9860 0x7f6ffc1593c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:56.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.624+0000 7f7000d1b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ffc0cef50 0x7f6ffc0b7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:56.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.624+0000 7f7000d1b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ffc15de30 0x7f6ffc0b8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:56.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.624+0000 7f7000d1b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6ffc0d2170 con 0x7f6ffc0cef50 2024-03-17T23:47:56.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.624+0000 7f7000d1b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6ffc162660 con 0x7f6ffc0b9860 2024-03-17T23:47:56.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.624+0000 7f7000d1b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6ffc15e210 con 0x7f6ffc15de30 2024-03-17T23:47:56.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.624+0000 7f6ff9d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ffc0cef50 0x7f6ffc0b7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:56.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.624+0000 7f6ffa59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ffc0b9860 0x7f6ffc1593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:56.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.624+0000 7f6ffad9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ffc15de30 0x7f6ffc0b8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:56.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.624+0000 7f6ffa59c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ffc0b9860 0x7f6ffc1593c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:55716/0 (socket says 172.21.15.150:55716) 2024-03-17T23:47:56.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.624+0000 7f6ff9d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ffc0cef50 0x7f6ffc0b7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45630/0 (socket says 172.21.15.150:45630) 2024-03-17T23:47:56.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.624+0000 7f6ffa59c640 1 -- 172.21.15.150:0/2746464050 learned_addr learned my addr 172.21.15.150:0/2746464050 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:56.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.624+0000 7f6ffad9d640 1 -- 172.21.15.150:0/2746464050 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ffc0b9860 msgr2=0x7f6ffc1593c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:56.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.624+0000 7f6ffad9d640 1 --2- 172.21.15.150:0/2746464050 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ffc0b9860 0x7f6ffc1593c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.624+0000 7f6ffad9d640 1 -- 172.21.15.150:0/2746464050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ffc0cef50 msgr2=0x7f6ffc0b7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:56.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.624+0000 7f6ffad9d640 1 --2- 172.21.15.150:0/2746464050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ffc0cef50 0x7f6ffc0b7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.624+0000 7f6ffad9d640 1 -- 172.21.15.150:0/2746464050 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ffc1f3800 con 0x7f6ffc15de30 2024-03-17T23:47:56.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.624+0000 7f6ffa59c640 1 --2- 172.21.15.150:0/2746464050 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ffc0b9860 0x7f6ffc1593c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:56.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.625+0000 7f6ffad9d640 1 --2- 172.21.15.150:0/2746464050 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ffc15de30 0x7f6ffc0b8310 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6ff400eed0 tx=0x7f6ff400b480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:56.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.625+0000 7f6feaffd640 1 -- 172.21.15.150:0/2746464050 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4018020 con 0x7f6ffc15de30 2024-03-17T23:47:56.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.625+0000 7f6feaffd640 1 -- 172.21.15.150:0/2746464050 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ff4014400 con 0x7f6ffc15de30 2024-03-17T23:47:56.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.625+0000 7f7000d1b640 1 -- 172.21.15.150:0/2746464050 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ffc1f3ae0 con 0x7f6ffc15de30 2024-03-17T23:47:56.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.625+0000 7f6feaffd640 1 -- 172.21.15.150:0/2746464050 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4009e60 con 0x7f6ffc15de30 2024-03-17T23:47:56.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.625+0000 7f7000d1b640 1 -- 172.21.15.150:0/2746464050 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ffc1f40a0 con 0x7f6ffc15de30 2024-03-17T23:47:56.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.626+0000 7f7000d1b640 1 -- 172.21.15.150:0/2746464050 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ffc0681a0 con 0x7f6ffc15de30 2024-03-17T23:47:56.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.626+0000 7f6feaffd640 1 -- 172.21.15.150:0/2746464050 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6ff4005d10 con 0x7f6ffc15de30 2024-03-17T23:47:56.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.626+0000 7f6feaffd640 1 --2- 172.21.15.150:0/2746464050 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6fdc06bbd0 0x7f6fdc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:56.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.627+0000 7f6feaffd640 1 -- 172.21.15.150:0/2746464050 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f6ff408b7e0 con 0x7f6ffc15de30 2024-03-17T23:47:56.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.627+0000 7f6ffa59c640 1 --2- 172.21.15.150:0/2746464050 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6fdc06bbd0 0x7f6fdc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:56.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.627+0000 7f6ffa59c640 1 --2- 172.21.15.150:0/2746464050 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6fdc06bbd0 0x7f6fdc06e090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc1615f0 tx=0x7f6fec009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:56.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.631+0000 7f6feaffd640 1 -- 172.21.15.150:0/2746464050 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6ff4022030 con 0x7f6ffc15de30 2024-03-17T23:47:56.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.807+0000 7f7000d1b640 1 -- 172.21.15.150:0/2746464050 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6ffc1f4870 con 0x7f6ffc15de30 2024-03-17T23:47:56.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.809+0000 7f6feaffd640 1 -- 172.21.15.150:0/2746464050 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18881 (secure 0 0 0) 0x7f6ff40597e0 con 0x7f6ffc15de30 2024-03-17T23:47:56.809 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:56.810 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":28,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:37.031864+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.e","mappings":[{"from":7,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:56.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.811+0000 7f7000d1b640 1 -- 172.21.15.150:0/2746464050 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6fdc06bbd0 msgr2=0x7f6fdc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:56.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.811+0000 7f7000d1b640 1 --2- 172.21.15.150:0/2746464050 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6fdc06bbd0 0x7f6fdc06e090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc1615f0 tx=0x7f6fec009190 comp rx=0 tx=0).stop 2024-03-17T23:47:56.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.811+0000 7f7000d1b640 1 -- 172.21.15.150:0/2746464050 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ffc15de30 msgr2=0x7f6ffc0b8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:56.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.811+0000 7f7000d1b640 1 --2- 172.21.15.150:0/2746464050 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ffc15de30 0x7f6ffc0b8310 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6ff400eed0 tx=0x7f6ff400b480 comp rx=0 tx=0).stop 2024-03-17T23:47:56.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.811+0000 7f7000d1b640 1 -- 172.21.15.150:0/2746464050 shutdown_connections 2024-03-17T23:47:56.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.811+0000 7f7000d1b640 1 --2- 172.21.15.150:0/2746464050 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6ffc15de30 0x7f6ffc0b8310 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.811+0000 7f7000d1b640 1 --2- 172.21.15.150:0/2746464050 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ffc0cef50 0x7f6ffc0b7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.811+0000 7f7000d1b640 1 --2- 172.21.15.150:0/2746464050 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6fdc06bbd0 0x7f6fdc06e090 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.811+0000 7f7000d1b640 1 --2- 172.21.15.150:0/2746464050 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ffc0b9860 0x7f6ffc1593c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.811+0000 7f7000d1b640 1 -- 172.21.15.150:0/2746464050 >> 172.21.15.150:0/2746464050 conn(0x7f6ffc0730b0 msgr2=0x7f6ffc15bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:56.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.811+0000 7f7000d1b640 1 -- 172.21.15.150:0/2746464050 shutdown_connections 2024-03-17T23:47:56.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.811+0000 7f7000d1b640 1 -- 172.21.15.150:0/2746464050 wait complete. 2024-03-17T23:47:56.823 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:46:53.610678+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:46:54.227335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-17T23:46:57.949235+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-17T23:47:56.823 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:47:56.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.927+0000 7fd619685640 1 Processor -- start 2024-03-17T23:47:56.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.927+0000 7fd619685640 1 -- start start 2024-03-17T23:47:56.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.928+0000 7fd619685640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd6140b2bf0 0x7fd6140b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:56.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.928+0000 7fd619685640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd614161d30 con 0x7fd6140d0ce0 2024-03-17T23:47:56.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.928+0000 7fd619685640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd614162010 con 0x7fd614161520 2024-03-17T23:47:56.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.928+0000 7fd619685640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd6140b9bf0 con 0x7fd6140b2bf0 2024-03-17T23:47:56.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.928+0000 7fd6127fc640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd6140d0ce0 0x7fd6140d1fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33484/0 (socket says 172.21.15.150:33484) 2024-03-17T23:47:56.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.928+0000 7fd6127fc640 1 -- 172.21.15.150:0/2743998596 learned_addr learned my addr 172.21.15.150:0/2743998596 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:56.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.928+0000 7fd6137fe640 1 --2- 172.21.15.150:0/2743998596 >> v2:172.21.15.157:3300/0 conn(0x7fd6140b2bf0 0x7fd6140b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:56.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.928+0000 7fd611ffb640 1 -- 172.21.15.150:0/2743998596 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2239951641 0 0) 0x7fd614161d30 con 0x7fd6140d0ce0 2024-03-17T23:47:56.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.928+0000 7fd611ffb640 1 -- 172.21.15.150:0/2743998596 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5fc003520 con 0x7fd6140d0ce0 2024-03-17T23:47:56.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.928+0000 7fd611ffb640 1 -- 172.21.15.150:0/2743998596 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3775213900 0 0) 0x7fd614162010 con 0x7fd614161520 2024-03-17T23:47:56.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.928+0000 7fd611ffb640 1 -- 172.21.15.150:0/2743998596 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd614161d30 con 0x7fd614161520 2024-03-17T23:47:56.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.928+0000 7fd6137fe640 1 -- 172.21.15.150:0/2743998596 >> v1:172.21.15.157:6789/0 conn(0x7fd614161520 legacy=0x7fd6140d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:56.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.928+0000 7fd6137fe640 1 -- 172.21.15.150:0/2743998596 >> v1:172.21.15.150:6789/0 conn(0x7fd6140d0ce0 legacy=0x7fd6140d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:56.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.928+0000 7fd6137fe640 1 -- 172.21.15.150:0/2743998596 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6141622f0 con 0x7fd6140b2bf0 2024-03-17T23:47:56.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.929+0000 7fd6137fe640 1 --2- 172.21.15.150:0/2743998596 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6140b2bf0 0x7fd6140b96b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd60c009d90 tx=0x7fd60c02be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=515307b67ef3f845 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:56.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.929+0000 7fd611ffb640 1 -- 172.21.15.150:0/2743998596 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60c039070 con 0x7fd6140b2bf0 2024-03-17T23:47:56.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.929+0000 7fd611ffb640 1 -- 172.21.15.150:0/2743998596 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd60c02cbd0 con 0x7fd6140b2bf0 2024-03-17T23:47:56.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.929+0000 7fd611ffb640 1 -- 172.21.15.150:0/2743998596 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60c034440 con 0x7fd6140b2bf0 2024-03-17T23:47:56.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.930+0000 7fd619685640 1 -- 172.21.15.150:0/2743998596 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6140b2bf0 msgr2=0x7fd6140b96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:56.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.930+0000 7fd619685640 1 --2- 172.21.15.150:0/2743998596 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6140b2bf0 0x7fd6140b96b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd60c009d90 tx=0x7fd60c02be20 comp rx=0 tx=0).stop 2024-03-17T23:47:56.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.930+0000 7fd619685640 1 -- 172.21.15.150:0/2743998596 shutdown_connections 2024-03-17T23:47:56.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.930+0000 7fd619685640 1 --2- 172.21.15.150:0/2743998596 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6140b2bf0 0x7fd6140b96b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.930+0000 7fd619685640 1 -- 172.21.15.150:0/2743998596 >> 172.21.15.150:0/2743998596 conn(0x7fd61406ca00 msgr2=0x7fd61406ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:56.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.930+0000 7fd619685640 1 -- 172.21.15.150:0/2743998596 shutdown_connections 2024-03-17T23:47:56.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.930+0000 7fd619685640 1 -- 172.21.15.150:0/2743998596 wait complete. 2024-03-17T23:47:56.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.931+0000 7fd619685640 1 Processor -- start 2024-03-17T23:47:56.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.931+0000 7fd619685640 1 -- start start 2024-03-17T23:47:56.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.932+0000 7fd619685640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6140b2bf0 0x7fd61415b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:56.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.932+0000 7fd619685640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6140d0ce0 0x7fd6141e8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:56.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.932+0000 7fd619685640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd614161520 0x7fd6140b3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:56.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.932+0000 7fd619685640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd614161900 con 0x7fd6140b2bf0 2024-03-17T23:47:56.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.932+0000 7fd619685640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd614162940 con 0x7fd6140d0ce0 2024-03-17T23:47:56.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.932+0000 7fd619685640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd6140d26b0 con 0x7fd614161520 2024-03-17T23:47:56.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.932+0000 7fd612ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6140b2bf0 0x7fd61415b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:56.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.932+0000 7fd612ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6140b2bf0 0x7fd61415b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45644/0 (socket says 172.21.15.150:45644) 2024-03-17T23:47:56.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.932+0000 7fd612ffd640 1 -- 172.21.15.150:0/3652386335 learned_addr learned my addr 172.21.15.150:0/3652386335 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:56.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.932+0000 7fd6127fc640 1 --2- 172.21.15.150:0/3652386335 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6140d0ce0 0x7fd6141e8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:56.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.932+0000 7fd6137fe640 1 --2- 172.21.15.150:0/3652386335 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd614161520 0x7fd6140b3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:56.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.932+0000 7fd612ffd640 1 -- 172.21.15.150:0/3652386335 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd614161520 msgr2=0x7fd6140b3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:56.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.932+0000 7fd612ffd640 1 --2- 172.21.15.150:0/3652386335 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd614161520 0x7fd6140b3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.932+0000 7fd612ffd640 1 -- 172.21.15.150:0/3652386335 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6140d0ce0 msgr2=0x7fd6141e8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:56.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.932+0000 7fd612ffd640 1 --2- 172.21.15.150:0/3652386335 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6140d0ce0 0x7fd6141e8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:56.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.932+0000 7fd612ffd640 1 -- 172.21.15.150:0/3652386335 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6141e90f0 con 0x7fd6140b2bf0 2024-03-17T23:47:56.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.932+0000 7fd612ffd640 1 --2- 172.21.15.150:0/3652386335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6140b2bf0 0x7fd61415b100 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd60400c430 tx=0x7fd60400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:56.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.932+0000 7fd5fb7fe640 1 -- 172.21.15.150:0/3652386335 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd604017070 con 0x7fd6140b2bf0 2024-03-17T23:47:56.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.933+0000 7fd5fb7fe640 1 -- 172.21.15.150:0/3652386335 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6040052f0 con 0x7fd6140b2bf0 2024-03-17T23:47:56.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.933+0000 7fd619685640 1 -- 172.21.15.150:0/3652386335 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd61415ee80 con 0x7fd6140b2bf0 2024-03-17T23:47:56.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.933+0000 7fd5fb7fe640 1 -- 172.21.15.150:0/3652386335 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd604003830 con 0x7fd6140b2bf0 2024-03-17T23:47:56.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.933+0000 7fd619685640 1 -- 172.21.15.150:0/3652386335 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd61415b640 con 0x7fd6140b2bf0 2024-03-17T23:47:56.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.934+0000 7fd619685640 1 -- 172.21.15.150:0/3652386335 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6140681a0 con 0x7fd6140b2bf0 2024-03-17T23:47:56.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.934+0000 7fd5fb7fe640 1 -- 172.21.15.150:0/3652386335 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd60401a070 con 0x7fd6140b2bf0 2024-03-17T23:47:56.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.934+0000 7fd5fb7fe640 1 --2- 172.21.15.150:0/3652386335 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd60006bbd0 0x7fd60006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:56.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.935+0000 7fd6127fc640 1 --2- 172.21.15.150:0/3652386335 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd60006bbd0 0x7fd60006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:56.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.935+0000 7fd5fb7fe640 1 -- 172.21.15.150:0/3652386335 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fd60408a1a0 con 0x7fd6140b2bf0 2024-03-17T23:47:56.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.935+0000 7fd6127fc640 1 --2- 172.21.15.150:0/3652386335 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd60006bbd0 0x7fd60006e090 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd6141e8080 tx=0x7fd608007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:56.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:56.939+0000 7fd5fb7fe640 1 -- 172.21.15.150:0/3652386335 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd604058240 con 0x7fd6140b2bf0 2024-03-17T23:47:57.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.113+0000 7fd619685640 1 -- 172.21.15.150:0/3652386335 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd6140c7680 con 0x7fd6140b2bf0 2024-03-17T23:47:57.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.113+0000 7fd5fb7fe640 1 -- 172.21.15.150:0/3652386335 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7fd60405c2e0 con 0x7fd6140b2bf0 2024-03-17T23:47:57.114 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:47:57.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.115+0000 7fd619685640 1 -- 172.21.15.150:0/3652386335 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd60006bbd0 msgr2=0x7fd60006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:57.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.115+0000 7fd619685640 1 --2- 172.21.15.150:0/3652386335 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd60006bbd0 0x7fd60006e090 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd6141e8080 tx=0x7fd608007d30 comp rx=0 tx=0).stop 2024-03-17T23:47:57.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.115+0000 7fd619685640 1 -- 172.21.15.150:0/3652386335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6140b2bf0 msgr2=0x7fd61415b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:57.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.115+0000 7fd619685640 1 --2- 172.21.15.150:0/3652386335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6140b2bf0 0x7fd61415b100 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd60400c430 tx=0x7fd60400c8f0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.115+0000 7fd619685640 1 -- 172.21.15.150:0/3652386335 shutdown_connections 2024-03-17T23:47:57.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.115+0000 7fd619685640 1 --2- 172.21.15.150:0/3652386335 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd614161520 0x7fd6140b3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.115+0000 7fd619685640 1 --2- 172.21.15.150:0/3652386335 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd60006bbd0 0x7fd60006e090 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.115+0000 7fd619685640 1 --2- 172.21.15.150:0/3652386335 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd6140d0ce0 0x7fd6141e8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.115+0000 7fd619685640 1 --2- 172.21.15.150:0/3652386335 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6140b2bf0 0x7fd61415b100 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.115+0000 7fd619685640 1 -- 172.21.15.150:0/3652386335 >> 172.21.15.150:0/3652386335 conn(0x7fd61406ca00 msgr2=0x7fd6140cf750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:57.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.116+0000 7fd619685640 1 -- 172.21.15.150:0/3652386335 shutdown_connections 2024-03-17T23:47:57.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.116+0000 7fd619685640 1 -- 172.21.15.150:0/3652386335 wait complete. 2024-03-17T23:47:57.127 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:47:57.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.230+0000 7f79d4052640 1 Processor -- start 2024-03-17T23:47:57.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.230+0000 7f79d4052640 1 -- start start 2024-03-17T23:47:57.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.231+0000 7f79d4052640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f79cc15f3d0 0x7f79cc0cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:57.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.231+0000 7f79d4052640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79cc0d0880 0x7f79cc1603c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:57.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.231+0000 7f79d4052640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79cc0ccc50 con 0x7f79cc0cc770 2024-03-17T23:47:57.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.231+0000 7f79d4052640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f79cc15f7b0 con 0x7f79cc15f3d0 2024-03-17T23:47:57.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.231+0000 7f79d4052640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f79cc160900 con 0x7f79cc0d0880 2024-03-17T23:47:57.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.231+0000 7f79d15ed640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f79cc0cc770 0x7f79cc0d0170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33498/0 (socket says 172.21.15.150:33498) 2024-03-17T23:47:57.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.231+0000 7f79d15ed640 1 -- 172.21.15.150:0/474295813 learned_addr learned my addr 172.21.15.150:0/474295813 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:57.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.231+0000 7f79d1dee640 1 --2- 172.21.15.150:0/474295813 >> v2:172.21.15.157:3300/0 conn(0x7f79cc15f3d0 0x7f79cc0cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:57.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.231+0000 7f79d25ef640 1 --2- 172.21.15.150:0/474295813 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79cc0d0880 0x7f79cc1603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:57.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.232+0000 7f79d0dec640 1 -- 172.21.15.150:0/474295813 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 25149993 0 0) 0x7f79cc0ccc50 con 0x7f79cc0cc770 2024-03-17T23:47:57.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.232+0000 7f79d0dec640 1 -- 172.21.15.150:0/474295813 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79b4003520 con 0x7f79cc0cc770 2024-03-17T23:47:57.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.232+0000 7f79d25ef640 1 -- 172.21.15.150:0/474295813 >> v2:172.21.15.157:3300/0 conn(0x7f79cc15f3d0 msgr2=0x7f79cc0cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:57.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.232+0000 7f79d25ef640 1 --2- 172.21.15.150:0/474295813 >> v2:172.21.15.157:3300/0 conn(0x7f79cc15f3d0 0x7f79cc0cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.232+0000 7f79d25ef640 1 -- 172.21.15.150:0/474295813 >> v1:172.21.15.150:6789/0 conn(0x7f79cc0cc770 legacy=0x7f79cc0d0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:57.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.232+0000 7f79d25ef640 1 -- 172.21.15.150:0/474295813 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79cc0ccf30 con 0x7f79cc0d0880 2024-03-17T23:47:57.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.232+0000 7f79d1dee640 1 --2- 172.21.15.150:0/474295813 >> v2:172.21.15.157:3300/0 conn(0x7f79cc15f3d0 0x7f79cc0cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:57.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.232+0000 7f79d25ef640 1 --2- 172.21.15.150:0/474295813 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79cc0d0880 0x7f79cc1603c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f79c40097a0 tx=0x7f79c402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd55ecfa331d7f50 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:57.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.233+0000 7f79d0dec640 1 -- 172.21.15.150:0/474295813 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c4035020 con 0x7f79cc0d0880 2024-03-17T23:47:57.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.233+0000 7f79d0dec640 1 -- 172.21.15.150:0/474295813 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79c4031830 con 0x7f79cc0d0880 2024-03-17T23:47:57.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.233+0000 7f79d0dec640 1 -- 172.21.15.150:0/474295813 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c4031c00 con 0x7f79cc0d0880 2024-03-17T23:47:57.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.233+0000 7f79d4052640 1 -- 172.21.15.150:0/474295813 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79cc0d0880 msgr2=0x7f79cc1603c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:57.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.233+0000 7f79d4052640 1 --2- 172.21.15.150:0/474295813 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79cc0d0880 0x7f79cc1603c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f79c40097a0 tx=0x7f79c402b8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.233+0000 7f79d4052640 1 -- 172.21.15.150:0/474295813 shutdown_connections 2024-03-17T23:47:57.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.233+0000 7f79d4052640 1 --2- 172.21.15.150:0/474295813 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79cc0d0880 0x7f79cc1603c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.233+0000 7f79d4052640 1 --2- 172.21.15.150:0/474295813 >> v2:172.21.15.157:3300/0 conn(0x7f79cc15f3d0 0x7f79cc0cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.233+0000 7f79d4052640 1 -- 172.21.15.150:0/474295813 >> 172.21.15.150:0/474295813 conn(0x7f79cc0b36c0 msgr2=0x7f79cc0b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:57.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.233+0000 7f79d4052640 1 -- 172.21.15.150:0/474295813 shutdown_connections 2024-03-17T23:47:57.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.233+0000 7f79d4052640 1 -- 172.21.15.150:0/474295813 wait complete. 2024-03-17T23:47:57.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.234+0000 7f79d4052640 1 Processor -- start 2024-03-17T23:47:57.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.234+0000 7f79d4052640 1 -- start start 2024-03-17T23:47:57.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.235+0000 7f79d4052640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79cc0cc770 0x7f79cc14ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:57.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.235+0000 7f79d4052640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79cc0d0880 0x7f79cc14d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:57.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.235+0000 7f79d4052640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79cc15f3d0 0x7f79cc14da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:57.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.235+0000 7f79d4052640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc0cd7d0 con 0x7f79cc0d0880 2024-03-17T23:47:57.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.235+0000 7f79d4052640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f79cc0cd4f0 con 0x7f79cc0cc770 2024-03-17T23:47:57.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.235+0000 7f79d4052640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f79cc0cfa30 con 0x7f79cc15f3d0 2024-03-17T23:47:57.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.235+0000 7f79d15ed640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79cc0d0880 0x7f79cc14d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:57.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.235+0000 7f79d1dee640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79cc0cc770 0x7f79cc14ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:57.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.235+0000 7f79d15ed640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79cc0d0880 0x7f79cc14d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45646/0 (socket says 172.21.15.150:45646) 2024-03-17T23:47:57.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.235+0000 7f79d15ed640 1 -- 172.21.15.150:0/2809731047 learned_addr learned my addr 172.21.15.150:0/2809731047 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:57.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.235+0000 7f79d25ef640 1 --2- 172.21.15.150:0/2809731047 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79cc15f3d0 0x7f79cc14da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:57.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.236+0000 7f79d1dee640 1 -- 172.21.15.150:0/2809731047 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79cc15f3d0 msgr2=0x7f79cc14da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:57.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.236+0000 7f79d1dee640 1 --2- 172.21.15.150:0/2809731047 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79cc15f3d0 0x7f79cc14da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.236+0000 7f79d1dee640 1 -- 172.21.15.150:0/2809731047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79cc0d0880 msgr2=0x7f79cc14d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:57.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.236+0000 7f79d1dee640 1 --2- 172.21.15.150:0/2809731047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79cc0d0880 0x7f79cc14d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.236+0000 7f79d1dee640 1 -- 172.21.15.150:0/2809731047 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79cc0aeb90 con 0x7f79cc0cc770 2024-03-17T23:47:57.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.236+0000 7f79d15ed640 1 --2- 172.21.15.150:0/2809731047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79cc0d0880 0x7f79cc14d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:57.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.236+0000 7f79d1dee640 1 --2- 172.21.15.150:0/2809731047 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79cc0cc770 0x7f79cc14ac20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f79bc00efc0 tx=0x7f79bc00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:57.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.236+0000 7f79ca7fc640 1 -- 172.21.15.150:0/2809731047 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79bc018020 con 0x7f79cc0cc770 2024-03-17T23:47:57.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.236+0000 7f79ca7fc640 1 -- 172.21.15.150:0/2809731047 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79bc014480 con 0x7f79cc0cc770 2024-03-17T23:47:57.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.236+0000 7f79d4052640 1 -- 172.21.15.150:0/2809731047 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79cc14b160 con 0x7f79cc0cc770 2024-03-17T23:47:57.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.236+0000 7f79ca7fc640 1 -- 172.21.15.150:0/2809731047 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79bc014870 con 0x7f79cc0cc770 2024-03-17T23:47:57.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.236+0000 7f79d4052640 1 -- 172.21.15.150:0/2809731047 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79cc0cb5c0 con 0x7f79cc0cc770 2024-03-17T23:47:57.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.237+0000 7f79d4052640 1 -- 172.21.15.150:0/2809731047 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79cc0688b0 con 0x7f79cc0cc770 2024-03-17T23:47:57.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.237+0000 7f79ca7fc640 1 -- 172.21.15.150:0/2809731047 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f79bc005d10 con 0x7f79cc0cc770 2024-03-17T23:47:57.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.238+0000 7f79ca7fc640 1 --2- 172.21.15.150:0/2809731047 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f79b806bbd0 0x7f79b806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:57.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.238+0000 7f79ca7fc640 1 -- 172.21.15.150:0/2809731047 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f79bc08ba90 con 0x7f79cc0cc770 2024-03-17T23:47:57.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.238+0000 7f79d15ed640 1 --2- 172.21.15.150:0/2809731047 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f79b806bbd0 0x7f79b806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:57.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.238+0000 7f79d15ed640 1 --2- 172.21.15.150:0/2809731047 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f79b806bbd0 0x7f79b806e090 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f79cc0b33b0 tx=0x7f79c0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:57.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.247+0000 7f79ca7fc640 1 -- 172.21.15.150:0/2809731047 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f79bc022030 con 0x7f79cc0cc770 2024-03-17T23:47:57.289 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-17T23:47:57.291 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-17T23:47:57.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.423+0000 7f79d4052640 1 -- 172.21.15.150:0/2809731047 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f79cc0b7b80 con 0x7f79cc0cc770 2024-03-17T23:47:57.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.423+0000 7f79ca7fc640 1 -- 172.21.15.150:0/2809731047 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) ==== 104+0+11 (secure 0 0 0) 0x7f79cc0aeb90 con 0x7f79cc0cc770 2024-03-17T23:47:57.424 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:57.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.425+0000 7f79d4052640 1 -- 172.21.15.150:0/2809731047 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f79b806bbd0 msgr2=0x7f79b806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:57.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.425+0000 7f79d4052640 1 --2- 172.21.15.150:0/2809731047 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f79b806bbd0 0x7f79b806e090 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f79cc0b33b0 tx=0x7f79c0009040 comp rx=0 tx=0).stop 2024-03-17T23:47:57.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.425+0000 7f79d4052640 1 -- 172.21.15.150:0/2809731047 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79cc0cc770 msgr2=0x7f79cc14ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:57.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.425+0000 7f79d4052640 1 --2- 172.21.15.150:0/2809731047 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79cc0cc770 0x7f79cc14ac20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f79bc00efc0 tx=0x7f79bc00c490 comp rx=0 tx=0).stop 2024-03-17T23:47:57.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.425+0000 7f79d4052640 1 -- 172.21.15.150:0/2809731047 shutdown_connections 2024-03-17T23:47:57.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.425+0000 7f79d4052640 1 --2- 172.21.15.150:0/2809731047 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f79cc15f3d0 0x7f79cc14da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.425+0000 7f79d4052640 1 --2- 172.21.15.150:0/2809731047 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79cc0d0880 0x7f79cc14d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.425+0000 7f79d4052640 1 --2- 172.21.15.150:0/2809731047 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f79b806bbd0 0x7f79b806e090 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.425+0000 7f79d4052640 1 --2- 172.21.15.150:0/2809731047 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79cc0cc770 0x7f79cc14ac20 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.425+0000 7f79d4052640 1 -- 172.21.15.150:0/2809731047 >> 172.21.15.150:0/2809731047 conn(0x7f79cc0b36c0 msgr2=0x7f79cc0c7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:57.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.425+0000 7f79d4052640 1 -- 172.21.15.150:0/2809731047 shutdown_connections 2024-03-17T23:47:57.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.425+0000 7f79d4052640 1 -- 172.21.15.150:0/2809731047 wait complete. 2024-03-17T23:47:57.437 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:47:57.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.541+0000 7fe707f7e640 1 Processor -- start 2024-03-17T23:47:57.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.541+0000 7fe707f7e640 1 -- start start 2024-03-17T23:47:57.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.542+0000 7fe707f7e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe700144bb0 0x7fe70015ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:57.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.542+0000 7fe707f7e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe7000ce6a0 0x7fe700165a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:57.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.542+0000 7fe707f7e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe700145170 con 0x7fe70015d390 2024-03-17T23:47:57.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.542+0000 7fe707f7e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fe700145450 con 0x7fe7000ce6a0 2024-03-17T23:47:57.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.542+0000 7fe707f7e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe700145730 con 0x7fe700144bb0 2024-03-17T23:47:57.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.542+0000 7fe70651b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe7000ce6a0 0x7fe700165a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:57.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.542+0000 7fe705519640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fe70015d390 0x7fe7000cdf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33514/0 (socket says 172.21.15.150:33514) 2024-03-17T23:47:57.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.542+0000 7fe705519640 1 -- 172.21.15.150:0/2008092573 learned_addr learned my addr 172.21.15.150:0/2008092573 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:57.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.542+0000 7fe70651b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe7000ce6a0 0x7fe700165a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45654/0 (socket says 172.21.15.150:45654) 2024-03-17T23:47:57.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.542+0000 7fe705d1a640 1 --2- 172.21.15.150:0/2008092573 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe700144bb0 0x7fe70015ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:57.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.542+0000 7fe70651b640 1 -- 172.21.15.150:0/2008092573 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe700144bb0 msgr2=0x7fe70015ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:57.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.542+0000 7fe70651b640 1 --2- 172.21.15.150:0/2008092573 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe700144bb0 0x7fe70015ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.542+0000 7fe70651b640 1 -- 172.21.15.150:0/2008092573 >> v1:172.21.15.150:6789/0 conn(0x7fe70015d390 legacy=0x7fe7000cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:57.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.542+0000 7fe70651b640 1 -- 172.21.15.150:0/2008092573 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe70006bd60 con 0x7fe7000ce6a0 2024-03-17T23:47:57.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.542+0000 7fe704d18640 1 -- 172.21.15.150:0/2008092573 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3245865151 0 0) 0x7fe700145170 con 0x7fe70015d390 2024-03-17T23:47:57.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.543+0000 7fe70651b640 1 --2- 172.21.15.150:0/2008092573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7000ce6a0 0x7fe700165a30 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8009d90 tx=0x7fe6f802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bf5013ff4788804 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:57.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.543+0000 7fe704d18640 1 -- 172.21.15.150:0/2008092573 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f8039070 con 0x7fe7000ce6a0 2024-03-17T23:47:57.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.543+0000 7fe704d18640 1 -- 172.21.15.150:0/2008092573 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6f802cbc0 con 0x7fe7000ce6a0 2024-03-17T23:47:57.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.543+0000 7fe704d18640 1 -- 172.21.15.150:0/2008092573 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f8034440 con 0x7fe7000ce6a0 2024-03-17T23:47:57.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.544+0000 7fe707f7e640 1 -- 172.21.15.150:0/2008092573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7000ce6a0 msgr2=0x7fe700165a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:57.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.544+0000 7fe707f7e640 1 --2- 172.21.15.150:0/2008092573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7000ce6a0 0x7fe700165a30 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8009d90 tx=0x7fe6f802be10 comp rx=0 tx=0).stop 2024-03-17T23:47:57.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.544+0000 7fe707f7e640 1 -- 172.21.15.150:0/2008092573 shutdown_connections 2024-03-17T23:47:57.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.544+0000 7fe707f7e640 1 --2- 172.21.15.150:0/2008092573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7000ce6a0 0x7fe700165a30 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.544+0000 7fe707f7e640 1 --2- 172.21.15.150:0/2008092573 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe700144bb0 0x7fe70015ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.544+0000 7fe707f7e640 1 -- 172.21.15.150:0/2008092573 >> 172.21.15.150:0/2008092573 conn(0x7fe70006b190 msgr2=0x7fe70006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:57.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.544+0000 7fe707f7e640 1 -- 172.21.15.150:0/2008092573 shutdown_connections 2024-03-17T23:47:57.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.544+0000 7fe707f7e640 1 -- 172.21.15.150:0/2008092573 wait complete. 2024-03-17T23:47:57.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.545+0000 7fe707f7e640 1 Processor -- start 2024-03-17T23:47:57.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.545+0000 7fe707f7e640 1 -- start start 2024-03-17T23:47:57.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.546+0000 7fe707f7e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7000ce6a0 0x7fe700159c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:57.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.546+0000 7fe707f7e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe700144bb0 0x7fe70015b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:57.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.546+0000 7fe707f7e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe70015d390 0x7fe700136200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:57.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.546+0000 7fe707f7e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe700165f70 con 0x7fe700144bb0 2024-03-17T23:47:57.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.546+0000 7fe707f7e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe70006f650 con 0x7fe7000ce6a0 2024-03-17T23:47:57.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.546+0000 7fe707f7e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe7001660f0 con 0x7fe70015d390 2024-03-17T23:47:57.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.546+0000 7fe705519640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe700144bb0 0x7fe70015b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:57.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.546+0000 7fe705519640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe700144bb0 0x7fe70015b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45662/0 (socket says 172.21.15.150:45662) 2024-03-17T23:47:57.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.546+0000 7fe705519640 1 -- 172.21.15.150:0/2855897759 learned_addr learned my addr 172.21.15.150:0/2855897759 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:57.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.546+0000 7fe70651b640 1 --2- 172.21.15.150:0/2855897759 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe70015d390 0x7fe700136200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:57.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.546+0000 7fe705d1a640 1 --2- 172.21.15.150:0/2855897759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7000ce6a0 0x7fe700159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:57.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.546+0000 7fe705519640 1 -- 172.21.15.150:0/2855897759 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe70015d390 msgr2=0x7fe700136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:57.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.546+0000 7fe705519640 1 --2- 172.21.15.150:0/2855897759 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe70015d390 0x7fe700136200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.546+0000 7fe705519640 1 -- 172.21.15.150:0/2855897759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7000ce6a0 msgr2=0x7fe700159c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:57.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.546+0000 7fe705519640 1 --2- 172.21.15.150:0/2855897759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7000ce6a0 0x7fe700159c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.546+0000 7fe705519640 1 -- 172.21.15.150:0/2855897759 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe70013bd50 con 0x7fe700144bb0 2024-03-17T23:47:57.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.546+0000 7fe705519640 1 --2- 172.21.15.150:0/2855897759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe700144bb0 0x7fe70015b940 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fe6f000c410 tx=0x7fe6f000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:57.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.546+0000 7fe6f67fc640 1 -- 172.21.15.150:0/2855897759 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f00074e0 con 0x7fe700144bb0 2024-03-17T23:47:57.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.546+0000 7fe6f67fc640 1 -- 172.21.15.150:0/2855897759 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6f00047e0 con 0x7fe700144bb0 2024-03-17T23:47:57.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.546+0000 7fe707f7e640 1 -- 172.21.15.150:0/2855897759 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe70013bff0 con 0x7fe700144bb0 2024-03-17T23:47:57.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.546+0000 7fe6f67fc640 1 -- 172.21.15.150:0/2855897759 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0004bd0 con 0x7fe700144bb0 2024-03-17T23:47:57.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.546+0000 7fe707f7e640 1 -- 172.21.15.150:0/2855897759 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe70013c5b0 con 0x7fe700144bb0 2024-03-17T23:47:57.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.547+0000 7fe707f7e640 1 -- 172.21.15.150:0/2855897759 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7000681a0 con 0x7fe700144bb0 2024-03-17T23:47:57.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.548+0000 7fe6f67fc640 1 -- 172.21.15.150:0/2855897759 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe6f0011070 con 0x7fe700144bb0 2024-03-17T23:47:57.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.548+0000 7fe6f67fc640 1 --2- 172.21.15.150:0/2855897759 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe6e406bbd0 0x7fe6e406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:57.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.548+0000 7fe6f67fc640 1 -- 172.21.15.150:0/2855897759 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fe6f008a920 con 0x7fe700144bb0 2024-03-17T23:47:57.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.548+0000 7fe705d1a640 1 --2- 172.21.15.150:0/2855897759 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe6e406bbd0 0x7fe6e406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:57.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.549+0000 7fe705d1a640 1 --2- 172.21.15.150:0/2855897759 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe6e406bbd0 0x7fe6e406e090 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc009b40 tx=0x7fe6fc009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:57.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.552+0000 7fe6f67fc640 1 -- 172.21.15.150:0/2855897759 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe6f00589c0 con 0x7fe700144bb0 2024-03-17T23:47:57.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.726+0000 7fe707f7e640 1 -- 172.21.15.150:0/2855897759 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe700138260 con 0x7fe700144bb0 2024-03-17T23:47:57.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.726+0000 7fe6f67fc640 1 -- 172.21.15.150:0/2855897759 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) ==== 100+0+11 (secure 0 0 0) 0x7fe6f005ca60 con 0x7fe700144bb0 2024-03-17T23:47:57.727 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:47:57.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.728+0000 7fe707f7e640 1 -- 172.21.15.150:0/2855897759 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe6e406bbd0 msgr2=0x7fe6e406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:57.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.728+0000 7fe707f7e640 1 --2- 172.21.15.150:0/2855897759 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe6e406bbd0 0x7fe6e406e090 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc009b40 tx=0x7fe6fc009190 comp rx=0 tx=0).stop 2024-03-17T23:47:57.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.728+0000 7fe707f7e640 1 -- 172.21.15.150:0/2855897759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe700144bb0 msgr2=0x7fe70015b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:57.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.728+0000 7fe707f7e640 1 --2- 172.21.15.150:0/2855897759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe700144bb0 0x7fe70015b940 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fe6f000c410 tx=0x7fe6f000c8d0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.728+0000 7fe707f7e640 1 -- 172.21.15.150:0/2855897759 shutdown_connections 2024-03-17T23:47:57.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.728+0000 7fe707f7e640 1 --2- 172.21.15.150:0/2855897759 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe70015d390 0x7fe700136200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.728+0000 7fe707f7e640 1 --2- 172.21.15.150:0/2855897759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe700144bb0 0x7fe70015b940 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.728+0000 7fe707f7e640 1 --2- 172.21.15.150:0/2855897759 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe6e406bbd0 0x7fe6e406e090 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.728+0000 7fe707f7e640 1 --2- 172.21.15.150:0/2855897759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7000ce6a0 0x7fe700159c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.728+0000 7fe707f7e640 1 -- 172.21.15.150:0/2855897759 >> 172.21.15.150:0/2855897759 conn(0x7fe70006b190 msgr2=0x7fe700071cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:57.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.728+0000 7fe707f7e640 1 -- 172.21.15.150:0/2855897759 shutdown_connections 2024-03-17T23:47:57.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.728+0000 7fe707f7e640 1 -- 172.21.15.150:0/2855897759 wait complete. 2024-03-17T23:47:57.740 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-17T23:47:57.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.842+0000 7f6893073640 1 Processor -- start 2024-03-17T23:47:57.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.842+0000 7f6893073640 1 -- start start 2024-03-17T23:47:57.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.843+0000 7f6893073640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f688c161520 0x7f688c0b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:57.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.843+0000 7f6893073640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f688c0b2bf0 0x7f688c0d05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:57.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.843+0000 7f6893073640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f688c161d30 con 0x7f688c0d0b10 2024-03-17T23:47:57.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.843+0000 7f6893073640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f688c0b9bf0 con 0x7f688c0b2bf0 2024-03-17T23:47:57.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.843+0000 7f6893073640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f688c162940 con 0x7f688c161520 2024-03-17T23:47:57.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.843+0000 7f6891610640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f688c0d0b10 0x7f688c0b57a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33530/0 (socket says 172.21.15.150:33530) 2024-03-17T23:47:57.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.843+0000 7f6891610640 1 -- 172.21.15.150:0/1601891703 learned_addr learned my addr 172.21.15.150:0/1601891703 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:57.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.843+0000 7f688bfff640 1 --2- 172.21.15.150:0/1601891703 >> v2:172.21.15.157:3300/0 conn(0x7f688c0b2bf0 0x7f688c0d05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:57.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.843+0000 7f6890e0f640 1 --2- 172.21.15.150:0/1601891703 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f688c161520 0x7f688c0b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:57.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.843+0000 7f688b7fe640 1 -- 172.21.15.150:0/1601891703 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 497725251 0 0) 0x7f688c161d30 con 0x7f688c0d0b10 2024-03-17T23:47:57.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.843+0000 7f688b7fe640 1 -- 172.21.15.150:0/1601891703 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6874003520 con 0x7f688c0d0b10 2024-03-17T23:47:57.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.843+0000 7f6890e0f640 1 -- 172.21.15.150:0/1601891703 >> v2:172.21.15.157:3300/0 conn(0x7f688c0b2bf0 msgr2=0x7f688c0d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:57.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.844+0000 7f6890e0f640 1 --2- 172.21.15.150:0/1601891703 >> v2:172.21.15.157:3300/0 conn(0x7f688c0b2bf0 0x7f688c0d05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.844+0000 7f6890e0f640 1 -- 172.21.15.150:0/1601891703 >> v1:172.21.15.150:6789/0 conn(0x7f688c0d0b10 legacy=0x7f688c0b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:57.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.844+0000 7f6890e0f640 1 -- 172.21.15.150:0/1601891703 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f688c162010 con 0x7f688c161520 2024-03-17T23:47:57.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.844+0000 7f6890e0f640 1 --2- 172.21.15.150:0/1601891703 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f688c161520 0x7f688c0b96b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f687c0097a0 tx=0x7f687c02b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b50925e7030cf0a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:57.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.844+0000 7f688b7fe640 1 -- 172.21.15.150:0/1601891703 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687c038070 con 0x7f688c161520 2024-03-17T23:47:57.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.845+0000 7f688b7fe640 1 -- 172.21.15.150:0/1601891703 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f687c031700 con 0x7f688c161520 2024-03-17T23:47:57.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.845+0000 7f688b7fe640 1 -- 172.21.15.150:0/1601891703 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687c031ab0 con 0x7f688c161520 2024-03-17T23:47:57.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.845+0000 7f6893073640 1 -- 172.21.15.150:0/1601891703 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f688c161520 msgr2=0x7f688c0b96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:57.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.845+0000 7f6893073640 1 --2- 172.21.15.150:0/1601891703 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f688c161520 0x7f688c0b96b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f687c0097a0 tx=0x7f687c02b7a0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.845+0000 7f6893073640 1 -- 172.21.15.150:0/1601891703 shutdown_connections 2024-03-17T23:47:57.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.845+0000 7f6893073640 1 --2- 172.21.15.150:0/1601891703 >> v2:172.21.15.157:3300/0 conn(0x7f688c0b2bf0 0x7f688c0d05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.845+0000 7f6893073640 1 --2- 172.21.15.150:0/1601891703 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f688c161520 0x7f688c0b96b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.845+0000 7f6893073640 1 -- 172.21.15.150:0/1601891703 >> 172.21.15.150:0/1601891703 conn(0x7f688c06ca00 msgr2=0x7f688c06ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:57.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.845+0000 7f6893073640 1 -- 172.21.15.150:0/1601891703 shutdown_connections 2024-03-17T23:47:57.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.845+0000 7f6893073640 1 -- 172.21.15.150:0/1601891703 wait complete. 2024-03-17T23:47:57.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.846+0000 7f6893073640 1 Processor -- start 2024-03-17T23:47:57.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.846+0000 7f6893073640 1 -- start start 2024-03-17T23:47:57.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.847+0000 7f6893073640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f688c0b2bf0 0x7f688c151b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:57.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.847+0000 7f6893073640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f688c0d0b10 0x7f688c0b83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:57.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.847+0000 7f6893073640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f688c161520 0x7f688c0b8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:57.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.847+0000 7f6893073640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f688c1625d0 con 0x7f688c0d0b10 2024-03-17T23:47:57.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.847+0000 7f6893073640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f688c0d0f60 con 0x7f688c0b2bf0 2024-03-17T23:47:57.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.847+0000 7f6893073640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f688c161900 con 0x7f688c161520 2024-03-17T23:47:57.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.847+0000 7f688bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f688c0d0b10 0x7f688c0b83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:57.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.847+0000 7f688bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f688c0d0b10 0x7f688c0b83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45678/0 (socket says 172.21.15.150:45678) 2024-03-17T23:47:57.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.847+0000 7f688bfff640 1 -- 172.21.15.150:0/2362338336 learned_addr learned my addr 172.21.15.150:0/2362338336 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:57.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.847+0000 7f6891610640 1 --2- 172.21.15.150:0/2362338336 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f688c161520 0x7f688c0b8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:57.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.847+0000 7f6890e0f640 1 --2- 172.21.15.150:0/2362338336 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f688c0b2bf0 0x7f688c151b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:57.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.848+0000 7f688bfff640 1 -- 172.21.15.150:0/2362338336 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f688c161520 msgr2=0x7f688c0b8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:57.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.848+0000 7f688bfff640 1 --2- 172.21.15.150:0/2362338336 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f688c161520 0x7f688c0b8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.848+0000 7f688bfff640 1 -- 172.21.15.150:0/2362338336 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f688c0b2bf0 msgr2=0x7f688c151b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:57.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.848+0000 7f688bfff640 1 --2- 172.21.15.150:0/2362338336 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f688c0b2bf0 0x7f688c151b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:57.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.848+0000 7f688bfff640 1 -- 172.21.15.150:0/2362338336 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f688c1f7850 con 0x7f688c0d0b10 2024-03-17T23:47:57.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.848+0000 7f688bfff640 1 --2- 172.21.15.150:0/2362338336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f688c0d0b10 0x7f688c0b83e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f688000e870 tx=0x7f688000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:57.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.848+0000 7f68897fa640 1 -- 172.21.15.150:0/2362338336 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6880017070 con 0x7f688c0d0b10 2024-03-17T23:47:57.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.848+0000 7f68897fa640 1 -- 172.21.15.150:0/2362338336 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f688000c940 con 0x7f688c0d0b10 2024-03-17T23:47:57.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.848+0000 7f6893073640 1 -- 172.21.15.150:0/2362338336 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f688c1f7b30 con 0x7f688c0d0b10 2024-03-17T23:47:57.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.848+0000 7f68897fa640 1 -- 172.21.15.150:0/2362338336 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688000cd30 con 0x7f688c0d0b10 2024-03-17T23:47:57.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.848+0000 7f6893073640 1 -- 172.21.15.150:0/2362338336 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f688c1f80f0 con 0x7f688c0d0b10 2024-03-17T23:47:57.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.849+0000 7f6893073640 1 -- 172.21.15.150:0/2362338336 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f688c0681a0 con 0x7f688c0d0b10 2024-03-17T23:47:57.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.849+0000 7f68897fa640 1 -- 172.21.15.150:0/2362338336 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f688001a070 con 0x7f688c0d0b10 2024-03-17T23:47:57.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.850+0000 7f68897fa640 1 --2- 172.21.15.150:0/2362338336 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f687806bbd0 0x7f687806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:57.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.850+0000 7f68897fa640 1 -- 172.21.15.150:0/2362338336 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f688008afd0 con 0x7f688c0d0b10 2024-03-17T23:47:57.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.850+0000 7f6890e0f640 1 --2- 172.21.15.150:0/2362338336 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f687806bbd0 0x7f687806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:57.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.850+0000 7f6890e0f640 1 --2- 172.21.15.150:0/2362338336 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f687806bbd0 0x7f687806e090 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f687c035000 tx=0x7f687c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:57.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:57.854+0000 7f68897fa640 1 -- 172.21.15.150:0/2362338336 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6880059100 con 0x7f688c0d0b10 2024-03-17T23:47:58.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.025+0000 7f6893073640 1 -- 172.21.15.150:0/2362338336 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f688c0d19c0 con 0x7f688c0d0b10 2024-03-17T23:47:58.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.025+0000 7f68897fa640 1 -- 172.21.15.150:0/2362338336 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) ==== 103+0+11 (secure 0 0 0) 0x7f688005d1a0 con 0x7f688c0d0b10 2024-03-17T23:47:58.026 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:47:58.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.027+0000 7f6893073640 1 -- 172.21.15.150:0/2362338336 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f687806bbd0 msgr2=0x7f687806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:58.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.027+0000 7f6893073640 1 --2- 172.21.15.150:0/2362338336 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f687806bbd0 0x7f687806e090 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f687c035000 tx=0x7f687c008030 comp rx=0 tx=0).stop 2024-03-17T23:47:58.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.027+0000 7f6893073640 1 -- 172.21.15.150:0/2362338336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f688c0d0b10 msgr2=0x7f688c0b83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:58.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.027+0000 7f6893073640 1 --2- 172.21.15.150:0/2362338336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f688c0d0b10 0x7f688c0b83e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f688000e870 tx=0x7f688000ed30 comp rx=0 tx=0).stop 2024-03-17T23:47:58.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.027+0000 7f6893073640 1 -- 172.21.15.150:0/2362338336 shutdown_connections 2024-03-17T23:47:58.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.027+0000 7f6893073640 1 --2- 172.21.15.150:0/2362338336 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f688c161520 0x7f688c0b8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.027+0000 7f6893073640 1 --2- 172.21.15.150:0/2362338336 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f688c0d0b10 0x7f688c0b83e0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.027+0000 7f6893073640 1 --2- 172.21.15.150:0/2362338336 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f687806bbd0 0x7f687806e090 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.027+0000 7f6893073640 1 --2- 172.21.15.150:0/2362338336 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f688c0b2bf0 0x7f688c151b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.027+0000 7f6893073640 1 -- 172.21.15.150:0/2362338336 >> 172.21.15.150:0/2362338336 conn(0x7f688c06ca00 msgr2=0x7f688c0b73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:58.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.027+0000 7f6893073640 1 -- 172.21.15.150:0/2362338336 shutdown_connections 2024-03-17T23:47:58.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.027+0000 7f6893073640 1 -- 172.21.15.150:0/2362338336 wait complete. 2024-03-17T23:47:58.039 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:47:58.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.139+0000 7f9b9d9c4640 1 Processor -- start 2024-03-17T23:47:58.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.139+0000 7f9b9d9c4640 1 -- start start 2024-03-17T23:47:58.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.140+0000 7f9b9d9c4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9b980c8210 0x7f9b980cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:58.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.140+0000 7f9b9d9c4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9b980ccb80 0x7f9b980cd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:58.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.140+0000 7f9b9d9c4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b980cd550 con 0x7f9b9815be40 2024-03-17T23:47:58.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.140+0000 7f9b9d9c4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f9b980c5d70 con 0x7f9b980ccb80 2024-03-17T23:47:58.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.140+0000 7f9b9d9c4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9b9809b260 con 0x7f9b980c8210 2024-03-17T23:47:58.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.140+0000 7f9b977fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9b980ccb80 0x7f9b980cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:58.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.140+0000 7f9b977fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9b980ccb80 0x7f9b980cd010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45692/0 (socket says 172.21.15.150:45692) 2024-03-17T23:47:58.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.140+0000 7f9b977fe640 1 -- 172.21.15.150:0/272263808 learned_addr learned my addr 172.21.15.150:0/272263808 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:58.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.140+0000 7f9b8e7fc640 1 --2- 172.21.15.150:0/272263808 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9b980c8210 0x7f9b980cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:58.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.141+0000 7f9b977fe640 1 -- 172.21.15.150:0/272263808 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9b980c8210 msgr2=0x7f9b980cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:58.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.141+0000 7f9b977fe640 1 --2- 172.21.15.150:0/272263808 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9b980c8210 0x7f9b980cc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.141+0000 7f9b977fe640 1 -- 172.21.15.150:0/272263808 >> v1:172.21.15.157:6789/0 conn(0x7f9b9815be40 legacy=0x7f9b9809a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:58.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.141+0000 7f9b977fe640 1 -- 172.21.15.150:0/272263808 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b9809af80 con 0x7f9b980ccb80 2024-03-17T23:47:58.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.141+0000 7f9b977fe640 1 --2- 172.21.15.150:0/272263808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b980ccb80 0x7f9b980cd010 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f9b90009e50 tx=0x7f9b9002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d583d3632d4a70c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:58.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.141+0000 7f9b967fc640 1 -- 172.21.15.150:0/272263808 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9002cb00 con 0x7f9b980ccb80 2024-03-17T23:47:58.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.141+0000 7f9b967fc640 1 -- 172.21.15.150:0/272263808 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b9002cca0 con 0x7f9b980ccb80 2024-03-17T23:47:58.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.141+0000 7f9b967fc640 1 -- 172.21.15.150:0/272263808 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b90034610 con 0x7f9b980ccb80 2024-03-17T23:47:58.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.142+0000 7f9b9d9c4640 1 -- 172.21.15.150:0/272263808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b980ccb80 msgr2=0x7f9b980cd010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:58.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.142+0000 7f9b9d9c4640 1 --2- 172.21.15.150:0/272263808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b980ccb80 0x7f9b980cd010 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f9b90009e50 tx=0x7f9b9002bf60 comp rx=0 tx=0).stop 2024-03-17T23:47:58.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.142+0000 7f9b9d9c4640 1 -- 172.21.15.150:0/272263808 shutdown_connections 2024-03-17T23:47:58.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.142+0000 7f9b9d9c4640 1 --2- 172.21.15.150:0/272263808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b980ccb80 0x7f9b980cd010 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.142+0000 7f9b9d9c4640 1 --2- 172.21.15.150:0/272263808 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9b980c8210 0x7f9b980cc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.142+0000 7f9b9d9c4640 1 -- 172.21.15.150:0/272263808 >> 172.21.15.150:0/272263808 conn(0x7f9b980b3620 msgr2=0x7f9b980b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:58.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.142+0000 7f9b9d9c4640 1 -- 172.21.15.150:0/272263808 shutdown_connections 2024-03-17T23:47:58.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.142+0000 7f9b9d9c4640 1 -- 172.21.15.150:0/272263808 wait complete. 2024-03-17T23:47:58.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.143+0000 7f9b9d9c4640 1 Processor -- start 2024-03-17T23:47:58.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.144+0000 7f9b9d9c4640 1 -- start start 2024-03-17T23:47:58.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.144+0000 7f9b9d9c4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b980c8210 0x7f9b981653b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:58.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.144+0000 7f9b9d9c4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b980ccb80 0x7f9b9814adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:58.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.144+0000 7f9b9d9c4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9b9815be40 0x7f9b980c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:58.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.144+0000 7f9b9d9c4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9815c220 con 0x7f9b980c8210 2024-03-17T23:47:58.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.144+0000 7f9b9d9c4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9b980c5a00 con 0x7f9b980ccb80 2024-03-17T23:47:58.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.144+0000 7f9b9d9c4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9b981659e0 con 0x7f9b9815be40 2024-03-17T23:47:58.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.144+0000 7f9b96ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b980c8210 0x7f9b981653b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:58.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.144+0000 7f9b96ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b980c8210 0x7f9b981653b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45706/0 (socket says 172.21.15.150:45706) 2024-03-17T23:47:58.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.144+0000 7f9b96ffd640 1 -- 172.21.15.150:0/1214270516 learned_addr learned my addr 172.21.15.150:0/1214270516 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:58.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.144+0000 7f9b8e7fc640 1 --2- 172.21.15.150:0/1214270516 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b980ccb80 0x7f9b9814adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:58.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.144+0000 7f9b977fe640 1 --2- 172.21.15.150:0/1214270516 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9b9815be40 0x7f9b980c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:58.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.145+0000 7f9b96ffd640 1 -- 172.21.15.150:0/1214270516 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9b9815be40 msgr2=0x7f9b980c8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:58.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.145+0000 7f9b96ffd640 1 --2- 172.21.15.150:0/1214270516 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9b9815be40 0x7f9b980c8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.145+0000 7f9b96ffd640 1 -- 172.21.15.150:0/1214270516 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b980ccb80 msgr2=0x7f9b9814adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:58.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.145+0000 7f9b96ffd640 1 --2- 172.21.15.150:0/1214270516 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b980ccb80 0x7f9b9814adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.145+0000 7f9b96ffd640 1 -- 172.21.15.150:0/1214270516 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b9815ec60 con 0x7f9b980c8210 2024-03-17T23:47:58.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.145+0000 7f9b96ffd640 1 --2- 172.21.15.150:0/1214270516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b980c8210 0x7f9b981653b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800c400 tx=0x7f9b8800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:58.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.145+0000 7f9b94ff9640 1 -- 172.21.15.150:0/1214270516 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b880052d0 con 0x7f9b980c8210 2024-03-17T23:47:58.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.145+0000 7f9b94ff9640 1 -- 172.21.15.150:0/1214270516 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b88005490 con 0x7f9b980c8210 2024-03-17T23:47:58.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.145+0000 7f9b9d9c4640 1 -- 172.21.15.150:0/1214270516 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b9815ef40 con 0x7f9b980c8210 2024-03-17T23:47:58.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.145+0000 7f9b94ff9640 1 -- 172.21.15.150:0/1214270516 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b88007260 con 0x7f9b980c8210 2024-03-17T23:47:58.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.145+0000 7f9b9d9c4640 1 -- 172.21.15.150:0/1214270516 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b9815d0f0 con 0x7f9b980c8210 2024-03-17T23:47:58.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.146+0000 7f9b9d9c4640 1 -- 172.21.15.150:0/1214270516 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b980688b0 con 0x7f9b980c8210 2024-03-17T23:47:58.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.146+0000 7f9b94ff9640 1 -- 172.21.15.150:0/1214270516 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9b88007400 con 0x7f9b980c8210 2024-03-17T23:47:58.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.147+0000 7f9b94ff9640 1 --2- 172.21.15.150:0/1214270516 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f9b7806bc00 0x7f9b7806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:58.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.147+0000 7f9b94ff9640 1 -- 172.21.15.150:0/1214270516 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f9b8808a2b0 con 0x7f9b980c8210 2024-03-17T23:47:58.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.147+0000 7f9b8e7fc640 1 --2- 172.21.15.150:0/1214270516 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f9b7806bc00 0x7f9b7806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:58.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.147+0000 7f9b8e7fc640 1 --2- 172.21.15.150:0/1214270516 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f9b7806bc00 0x7f9b7806e0c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9b9814f190 tx=0x7f9b80009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:58.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.151+0000 7f9b94ff9640 1 -- 172.21.15.150:0/1214270516 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9b880583e0 con 0x7f9b980c8210 2024-03-17T23:47:58.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.359+0000 7f9b9d9c4640 1 -- 172.21.15.150:0/1214270516 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b98070e50 con 0x7f9b980c8210 2024-03-17T23:47:58.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.359+0000 7f9b94ff9640 1 -- 172.21.15.150:0/1214270516 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5186 (secure 0 0 0) 0x7f9b8805c480 con 0x7f9b980c8210 2024-03-17T23:47:58.360 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:58.360 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/1949609985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6833","nonce":1949609985},{"type":"v1","addr":"172.21.15.157:6835","nonce":1949609985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/758264728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":758264728},{"type":"v1","addr":"172.21.15.150:6835","nonce":758264728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3655589010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3655589010},{"type":"v1","addr":"172.21.15.150:6837","nonce":3655589010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-17T23:46:55.253739+0000","modified":"2024-03-17T23:47:11.316249+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6834/1625058792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1625058792},{"type":"v1","addr":"172.21.15.157:6834","nonce":1625058792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4297,"qdb_cluster":[4297]},"id":1}]} 2024-03-17T23:47:58.361 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 8 2024-03-17T23:47:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.362+0000 7f9b9d9c4640 1 -- 172.21.15.150:0/1214270516 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f9b7806bc00 msgr2=0x7f9b7806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.362+0000 7f9b9d9c4640 1 --2- 172.21.15.150:0/1214270516 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f9b7806bc00 0x7f9b7806e0c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9b9814f190 tx=0x7f9b80009190 comp rx=0 tx=0).stop 2024-03-17T23:47:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.362+0000 7f9b9d9c4640 1 -- 172.21.15.150:0/1214270516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b980c8210 msgr2=0x7f9b981653b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.362+0000 7f9b9d9c4640 1 --2- 172.21.15.150:0/1214270516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b980c8210 0x7f9b981653b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800c400 tx=0x7f9b8800c8c0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.362+0000 7f9b9d9c4640 1 -- 172.21.15.150:0/1214270516 shutdown_connections 2024-03-17T23:47:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.362+0000 7f9b9d9c4640 1 --2- 172.21.15.150:0/1214270516 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9b9815be40 0x7f9b980c8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.362+0000 7f9b9d9c4640 1 --2- 172.21.15.150:0/1214270516 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f9b7806bc00 0x7f9b7806e0c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.362+0000 7f9b9d9c4640 1 --2- 172.21.15.150:0/1214270516 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b980ccb80 0x7f9b9814adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.362+0000 7f9b9d9c4640 1 --2- 172.21.15.150:0/1214270516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9b980c8210 0x7f9b981653b0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.362+0000 7f9b9d9c4640 1 -- 172.21.15.150:0/1214270516 >> 172.21.15.150:0/1214270516 conn(0x7f9b980b3620 msgr2=0x7f9b98071070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:58.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.362+0000 7f9b9d9c4640 1 -- 172.21.15.150:0/1214270516 shutdown_connections 2024-03-17T23:47:58.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.362+0000 7f9b9d9c4640 1 -- 172.21.15.150:0/1214270516 wait complete. 2024-03-17T23:47:58.373 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:58.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.474+0000 7f2d246e8640 1 Processor -- start 2024-03-17T23:47:58.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.474+0000 7f2d246e8640 1 -- start start 2024-03-17T23:47:58.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.475+0000 7f2d246e8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2d1c15bfd0 0x7f2d1c0ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:58.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.475+0000 7f2d246e8640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2d1c0cd250 0x7f2d1c15ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:58.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.475+0000 7f2d246e8640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d1c0cf3b0 con 0x7f2d1c155320 2024-03-17T23:47:58.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.475+0000 7f2d246e8640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f2d1c15c550 con 0x7f2d1c15bfd0 2024-03-17T23:47:58.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.475+0000 7f2d246e8640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f2d1c0cf690 con 0x7f2d1c0cd250 2024-03-17T23:47:58.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.475+0000 7f2d21c83640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2d1c15bfd0 0x7f2d1c0ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:58.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.475+0000 7f2d22484640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f2d1c155320 0x7f2d1c06dd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33534/0 (socket says 172.21.15.150:33534) 2024-03-17T23:47:58.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.475+0000 7f2d22484640 1 -- 172.21.15.150:0/652417265 learned_addr learned my addr 172.21.15.150:0/652417265 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:58.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.475+0000 7f2d22c85640 1 --2- 172.21.15.150:0/652417265 >> v2:172.21.15.157:3300/0 conn(0x7f2d1c0cd250 0x7f2d1c15ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:58.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.475+0000 7f2d21c83640 1 -- 172.21.15.150:0/652417265 >> v2:172.21.15.157:3300/0 conn(0x7f2d1c0cd250 msgr2=0x7f2d1c15ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:58.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.475+0000 7f2d21c83640 1 --2- 172.21.15.150:0/652417265 >> v2:172.21.15.157:3300/0 conn(0x7f2d1c0cd250 0x7f2d1c15ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.475+0000 7f2d21c83640 1 -- 172.21.15.150:0/652417265 >> v1:172.21.15.150:6789/0 conn(0x7f2d1c155320 legacy=0x7f2d1c06dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:58.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.475+0000 7f2d21c83640 1 -- 172.21.15.150:0/652417265 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d1c15c7a0 con 0x7f2d1c15bfd0 2024-03-17T23:47:58.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.475+0000 7f2d21482640 1 -- 172.21.15.150:0/652417265 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1037970230 0 0) 0x7f2d1c0cf3b0 con 0x7f2d1c155320 2024-03-17T23:47:58.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.476+0000 7f2d21c83640 1 --2- 172.21.15.150:0/652417265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d1c15bfd0 0x7f2d1c0ccd10 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c06c0f0 tx=0x7f2d1802e0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b95330b8a3fd8bc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:58.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.476+0000 7f2d21482640 1 -- 172.21.15.150:0/652417265 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1803b070 con 0x7f2d1c15bfd0 2024-03-17T23:47:58.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.476+0000 7f2d21482640 1 -- 172.21.15.150:0/652417265 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d1802ee50 con 0x7f2d1c15bfd0 2024-03-17T23:47:58.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.476+0000 7f2d21482640 1 -- 172.21.15.150:0/652417265 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d18036660 con 0x7f2d1c15bfd0 2024-03-17T23:47:58.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.476+0000 7f2d246e8640 1 -- 172.21.15.150:0/652417265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d1c15bfd0 msgr2=0x7f2d1c0ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:58.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.476+0000 7f2d246e8640 1 --2- 172.21.15.150:0/652417265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d1c15bfd0 0x7f2d1c0ccd10 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c06c0f0 tx=0x7f2d1802e0a0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.477+0000 7f2d246e8640 1 -- 172.21.15.150:0/652417265 shutdown_connections 2024-03-17T23:47:58.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.477+0000 7f2d246e8640 1 --2- 172.21.15.150:0/652417265 >> v2:172.21.15.157:3300/0 conn(0x7f2d1c0cd250 0x7f2d1c15ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.477+0000 7f2d246e8640 1 --2- 172.21.15.150:0/652417265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d1c15bfd0 0x7f2d1c0ccd10 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.477+0000 7f2d246e8640 1 -- 172.21.15.150:0/652417265 >> 172.21.15.150:0/652417265 conn(0x7f2d1c06b190 msgr2=0x7f2d1c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:58.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.477+0000 7f2d246e8640 1 -- 172.21.15.150:0/652417265 shutdown_connections 2024-03-17T23:47:58.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.477+0000 7f2d246e8640 1 -- 172.21.15.150:0/652417265 wait complete. 2024-03-17T23:47:58.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.478+0000 7f2d246e8640 1 Processor -- start 2024-03-17T23:47:58.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.478+0000 7f2d246e8640 1 -- start start 2024-03-17T23:47:58.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.479+0000 7f2d246e8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d1c0cd250 0x7f2d1c145550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:58.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.479+0000 7f2d246e8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d1c155320 0x7f2d1c0cb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:58.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.479+0000 7f2d246e8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d1c15bfd0 0x7f2d1c06e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:58.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.479+0000 7f2d246e8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1c155880 con 0x7f2d1c155320 2024-03-17T23:47:58.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.479+0000 7f2d246e8640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1c155700 con 0x7f2d1c15bfd0 2024-03-17T23:47:58.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.479+0000 7f2d246e8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2d1c15d200 con 0x7f2d1c0cd250 2024-03-17T23:47:58.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.479+0000 7f2d21c83640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d1c155320 0x7f2d1c0cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:58.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.479+0000 7f2d21c83640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d1c155320 0x7f2d1c0cb470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45714/0 (socket says 172.21.15.150:45714) 2024-03-17T23:47:58.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.479+0000 7f2d21c83640 1 -- 172.21.15.150:0/2684820428 learned_addr learned my addr 172.21.15.150:0/2684820428 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:58.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.479+0000 7f2d22484640 1 --2- 172.21.15.150:0/2684820428 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d1c0cd250 0x7f2d1c145550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:58.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.479+0000 7f2d22c85640 1 --2- 172.21.15.150:0/2684820428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d1c15bfd0 0x7f2d1c06e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:58.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.479+0000 7f2d21c83640 1 -- 172.21.15.150:0/2684820428 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d1c0cd250 msgr2=0x7f2d1c145550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:58.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.479+0000 7f2d21c83640 1 --2- 172.21.15.150:0/2684820428 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d1c0cd250 0x7f2d1c145550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.479+0000 7f2d21c83640 1 -- 172.21.15.150:0/2684820428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d1c15bfd0 msgr2=0x7f2d1c06e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:58.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.479+0000 7f2d21c83640 1 --2- 172.21.15.150:0/2684820428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d1c15bfd0 0x7f2d1c06e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.479+0000 7f2d21c83640 1 -- 172.21.15.150:0/2684820428 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d1c157fc0 con 0x7f2d1c155320 2024-03-17T23:47:58.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.479+0000 7f2d22c85640 1 --2- 172.21.15.150:0/2684820428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d1c15bfd0 0x7f2d1c06e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:47:58.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.479+0000 7f2d21c83640 1 --2- 172.21.15.150:0/2684820428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d1c155320 0x7f2d1c0cb470 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f2d18000c00 tx=0x7f2d18004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:58.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.479+0000 7f2d12ffd640 1 -- 172.21.15.150:0/2684820428 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d18046070 con 0x7f2d1c155320 2024-03-17T23:47:58.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.479+0000 7f2d12ffd640 1 -- 172.21.15.150:0/2684820428 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d18004030 con 0x7f2d1c155320 2024-03-17T23:47:58.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.479+0000 7f2d246e8640 1 -- 172.21.15.150:0/2684820428 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d1c1582a0 con 0x7f2d1c155320 2024-03-17T23:47:58.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.479+0000 7f2d12ffd640 1 -- 172.21.15.150:0/2684820428 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1803b040 con 0x7f2d1c155320 2024-03-17T23:47:58.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.479+0000 7f2d246e8640 1 -- 172.21.15.150:0/2684820428 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d1c158860 con 0x7f2d1c155320 2024-03-17T23:47:58.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.480+0000 7f2d246e8640 1 -- 172.21.15.150:0/2684820428 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d1c0681a0 con 0x7f2d1c155320 2024-03-17T23:47:58.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.481+0000 7f2d12ffd640 1 -- 172.21.15.150:0/2684820428 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2d18049070 con 0x7f2d1c155320 2024-03-17T23:47:58.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.481+0000 7f2d12ffd640 1 --2- 172.21.15.150:0/2684820428 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2d0006bbd0 0x7f2d0006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:58.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.481+0000 7f2d22484640 1 --2- 172.21.15.150:0/2684820428 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2d0006bbd0 0x7f2d0006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:58.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.481+0000 7f2d12ffd640 1 -- 172.21.15.150:0/2684820428 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f2d180aca10 con 0x7f2d1c155320 2024-03-17T23:47:58.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.482+0000 7f2d22484640 1 --2- 172.21.15.150:0/2684820428 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2d0006bbd0 0x7f2d0006e090 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2d140012e0 tx=0x7f2d14001700 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:58.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.486+0000 7f2d12ffd640 1 -- 172.21.15.150:0/2684820428 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2d1807ab40 con 0x7f2d1c155320 2024-03-17T23:47:58.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.663+0000 7f2d246e8640 1 -- 172.21.15.150:0/2684820428 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2d1c072900 con 0x7f2d1c155320 2024-03-17T23:47:58.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.665+0000 7f2d12ffd640 1 -- 172.21.15.150:0/2684820428 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18881 (secure 0 0 0) 0x7f2d1807ebe0 con 0x7f2d1c155320 2024-03-17T23:47:58.665 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:58.666 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":28,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:37.031864+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.e","mappings":[{"from":7,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:58.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.667+0000 7f2d246e8640 1 -- 172.21.15.150:0/2684820428 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2d0006bbd0 msgr2=0x7f2d0006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:58.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.667+0000 7f2d246e8640 1 --2- 172.21.15.150:0/2684820428 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2d0006bbd0 0x7f2d0006e090 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2d140012e0 tx=0x7f2d14001700 comp rx=0 tx=0).stop 2024-03-17T23:47:58.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.667+0000 7f2d246e8640 1 -- 172.21.15.150:0/2684820428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d1c155320 msgr2=0x7f2d1c0cb470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:58.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.667+0000 7f2d246e8640 1 --2- 172.21.15.150:0/2684820428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d1c155320 0x7f2d1c0cb470 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f2d18000c00 tx=0x7f2d18004730 comp rx=0 tx=0).stop 2024-03-17T23:47:58.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.667+0000 7f2d246e8640 1 -- 172.21.15.150:0/2684820428 shutdown_connections 2024-03-17T23:47:58.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.667+0000 7f2d246e8640 1 --2- 172.21.15.150:0/2684820428 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2d0006bbd0 0x7f2d0006e090 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.667+0000 7f2d246e8640 1 --2- 172.21.15.150:0/2684820428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d1c15bfd0 0x7f2d1c06e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.667+0000 7f2d246e8640 1 --2- 172.21.15.150:0/2684820428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d1c155320 0x7f2d1c0cb470 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.667+0000 7f2d246e8640 1 --2- 172.21.15.150:0/2684820428 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d1c0cd250 0x7f2d1c145550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.667+0000 7f2d246e8640 1 -- 172.21.15.150:0/2684820428 >> 172.21.15.150:0/2684820428 conn(0x7f2d1c06b190 msgr2=0x7f2d1c153b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:58.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.667+0000 7f2d246e8640 1 -- 172.21.15.150:0/2684820428 shutdown_connections 2024-03-17T23:47:58.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.667+0000 7f2d246e8640 1 -- 172.21.15.150:0/2684820428 wait complete. 2024-03-17T23:47:58.679 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-17T23:47:58.679 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:47:58.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.780+0000 7f87c26e8640 1 Processor -- start 2024-03-17T23:47:58.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.780+0000 7f87c26e8640 1 -- start start 2024-03-17T23:47:58.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.780+0000 7f87c26e8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc1619a0 0x7f87bc0b9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:58.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.780+0000 7f87c26e8640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87bc1620a0 con 0x7f87bc0cf120 2024-03-17T23:47:58.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.780+0000 7f87c26e8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87bc162380 con 0x7f87bc15de30 2024-03-17T23:47:58.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.780+0000 7f87c26e8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f87bc162940 con 0x7f87bc1619a0 2024-03-17T23:47:58.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.781+0000 7f87bb7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f87bc0cf120 0x7f87bc0d2690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33546/0 (socket says 172.21.15.150:33546) 2024-03-17T23:47:58.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.781+0000 7f87bb7fe640 1 -- 172.21.15.150:0/3414424682 learned_addr learned my addr 172.21.15.150:0/3414424682 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:58.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.781+0000 7f87c0c85640 1 --2- 172.21.15.150:0/3414424682 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc1619a0 0x7f87bc0b9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:58.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.781+0000 7f87baffd640 1 -- 172.21.15.150:0/3414424682 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2232452075 0 0) 0x7f87bc1620a0 con 0x7f87bc0cf120 2024-03-17T23:47:58.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.781+0000 7f87baffd640 1 -- 172.21.15.150:0/3414424682 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87a4003520 con 0x7f87bc0cf120 2024-03-17T23:47:58.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.781+0000 7f87baffd640 1 -- 172.21.15.150:0/3414424682 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3660537656 0 0) 0x7f87bc162380 con 0x7f87bc15de30 2024-03-17T23:47:58.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.781+0000 7f87baffd640 1 -- 172.21.15.150:0/3414424682 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87bc1620a0 con 0x7f87bc15de30 2024-03-17T23:47:58.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.781+0000 7f87c0c85640 1 -- 172.21.15.150:0/3414424682 >> v1:172.21.15.157:6789/0 conn(0x7f87bc15de30 legacy=0x7f87bc0cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:58.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.781+0000 7f87c0c85640 1 -- 172.21.15.150:0/3414424682 >> v1:172.21.15.150:6789/0 conn(0x7f87bc0cf120 legacy=0x7f87bc0d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:58.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.781+0000 7f87c0c85640 1 -- 172.21.15.150:0/3414424682 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87bc162660 con 0x7f87bc1619a0 2024-03-17T23:47:58.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.782+0000 7f87c0c85640 1 --2- 172.21.15.150:0/3414424682 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc1619a0 0x7f87bc0b9860 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f87b4009dc0 tx=0x7f87b402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ee283ead6fbc443 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:58.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.782+0000 7f87baffd640 1 -- 172.21.15.150:0/3414424682 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4039070 con 0x7f87bc1619a0 2024-03-17T23:47:58.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.782+0000 7f87baffd640 1 -- 172.21.15.150:0/3414424682 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87b402cc40 con 0x7f87bc1619a0 2024-03-17T23:47:58.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.782+0000 7f87baffd640 1 -- 172.21.15.150:0/3414424682 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4034440 con 0x7f87bc1619a0 2024-03-17T23:47:58.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.782+0000 7f87c26e8640 1 -- 172.21.15.150:0/3414424682 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc1619a0 msgr2=0x7f87bc0b9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:58.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.782+0000 7f87c26e8640 1 --2- 172.21.15.150:0/3414424682 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc1619a0 0x7f87bc0b9860 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f87b4009dc0 tx=0x7f87b402be90 comp rx=0 tx=0).stop 2024-03-17T23:47:58.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.782+0000 7f87c26e8640 1 -- 172.21.15.150:0/3414424682 shutdown_connections 2024-03-17T23:47:58.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.782+0000 7f87c26e8640 1 --2- 172.21.15.150:0/3414424682 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc1619a0 0x7f87bc0b9860 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.782+0000 7f87c26e8640 1 -- 172.21.15.150:0/3414424682 >> 172.21.15.150:0/3414424682 conn(0x7f87bc0730b0 msgr2=0x7f87bc0d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:58.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.782+0000 7f87c26e8640 1 -- 172.21.15.150:0/3414424682 shutdown_connections 2024-03-17T23:47:58.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.782+0000 7f87c26e8640 1 -- 172.21.15.150:0/3414424682 wait complete. 2024-03-17T23:47:58.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.784+0000 7f87c26e8640 1 Processor -- start 2024-03-17T23:47:58.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.784+0000 7f87c26e8640 1 -- start start 2024-03-17T23:47:58.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.784+0000 7f87c26e8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87bc0cf120 0x7f87bc157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:58.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.784+0000 7f87c26e8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc15de30 0x7f87bc157be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:58.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.784+0000 7f87c26e8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87bc1619a0 0x7f87bc160770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:58.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.785+0000 7f87c26e8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc15e210 con 0x7f87bc1619a0 2024-03-17T23:47:58.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.785+0000 7f87c26e8640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc161da0 con 0x7f87bc0cf120 2024-03-17T23:47:58.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.785+0000 7f87c26e8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f87bc0d2da0 con 0x7f87bc15de30 2024-03-17T23:47:58.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.785+0000 7f87c0c85640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87bc1619a0 0x7f87bc160770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:58.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.785+0000 7f87c0c85640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87bc1619a0 0x7f87bc160770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45726/0 (socket says 172.21.15.150:45726) 2024-03-17T23:47:58.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.785+0000 7f87c0c85640 1 -- 172.21.15.150:0/1426029245 learned_addr learned my addr 172.21.15.150:0/1426029245 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:58.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.785+0000 7f87bbfff640 1 --2- 172.21.15.150:0/1426029245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87bc0cf120 0x7f87bc157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:58.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.785+0000 7f87bb7fe640 1 --2- 172.21.15.150:0/1426029245 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc15de30 0x7f87bc157be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:58.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.785+0000 7f87c0c85640 1 -- 172.21.15.150:0/1426029245 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc15de30 msgr2=0x7f87bc157be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:58.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.785+0000 7f87c0c85640 1 --2- 172.21.15.150:0/1426029245 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc15de30 0x7f87bc157be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.785+0000 7f87c0c85640 1 -- 172.21.15.150:0/1426029245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87bc0cf120 msgr2=0x7f87bc157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:58.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.785+0000 7f87c0c85640 1 --2- 172.21.15.150:0/1426029245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87bc0cf120 0x7f87bc157180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:58.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.785+0000 7f87c0c85640 1 -- 172.21.15.150:0/1426029245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87bc1584e0 con 0x7f87bc1619a0 2024-03-17T23:47:58.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.785+0000 7f87c0c85640 1 --2- 172.21.15.150:0/1426029245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87bc1619a0 0x7f87bc160770 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f87b4009ee0 tx=0x7f87b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:58.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.785+0000 7f87b8ff9640 1 -- 172.21.15.150:0/1426029245 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4039050 con 0x7f87bc1619a0 2024-03-17T23:47:58.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.785+0000 7f87b8ff9640 1 -- 172.21.15.150:0/1426029245 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87b4045070 con 0x7f87bc1619a0 2024-03-17T23:47:58.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.785+0000 7f87c26e8640 1 -- 172.21.15.150:0/1426029245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87bc1587c0 con 0x7f87bc1619a0 2024-03-17T23:47:58.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.785+0000 7f87b8ff9640 1 -- 172.21.15.150:0/1426029245 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4040690 con 0x7f87bc1619a0 2024-03-17T23:47:58.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.785+0000 7f87c26e8640 1 -- 172.21.15.150:0/1426029245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87bc158d80 con 0x7f87bc1619a0 2024-03-17T23:47:58.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.786+0000 7f87c26e8640 1 -- 172.21.15.150:0/1426029245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87bc0681a0 con 0x7f87bc1619a0 2024-03-17T23:47:58.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.787+0000 7f87b8ff9640 1 -- 172.21.15.150:0/1426029245 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f87b4040830 con 0x7f87bc1619a0 2024-03-17T23:47:58.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.787+0000 7f87b8ff9640 1 --2- 172.21.15.150:0/1426029245 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f87a806bbd0 0x7f87a806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:58.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.787+0000 7f87b8ff9640 1 -- 172.21.15.150:0/1426029245 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f87b40ad6d0 con 0x7f87bc1619a0 2024-03-17T23:47:58.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.787+0000 7f87bbfff640 1 --2- 172.21.15.150:0/1426029245 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f87a806bbd0 0x7f87a806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:58.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.787+0000 7f87bbfff640 1 --2- 172.21.15.150:0/1426029245 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f87a806bbd0 0x7f87a806e090 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f87ac007fd0 tx=0x7f87ac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:58.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:58.791+0000 7f87b8ff9640 1 -- 172.21.15.150:0/1426029245 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f87b407b770 con 0x7f87bc1619a0 2024-03-17T23:47:59.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.003+0000 7f87c26e8640 1 -- 172.21.15.150:0/1426029245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87bc0ca6a0 con 0x7f87bc1619a0 2024-03-17T23:47:59.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.004+0000 7f87b8ff9640 1 -- 172.21.15.150:0/1426029245 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5186 (secure 0 0 0) 0x7f87b407f810 con 0x7f87bc1619a0 2024-03-17T23:47:59.004 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:59.005 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/1949609985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6833","nonce":1949609985},{"type":"v1","addr":"172.21.15.157:6835","nonce":1949609985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/758264728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":758264728},{"type":"v1","addr":"172.21.15.150:6835","nonce":758264728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3655589010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3655589010},{"type":"v1","addr":"172.21.15.150:6837","nonce":3655589010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-17T23:46:55.253739+0000","modified":"2024-03-17T23:47:11.316249+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6834/1625058792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1625058792},{"type":"v1","addr":"172.21.15.157:6834","nonce":1625058792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4297,"qdb_cluster":[4297]},"id":1}]} 2024-03-17T23:47:59.005 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 8 2024-03-17T23:47:59.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.006+0000 7f87c26e8640 1 -- 172.21.15.150:0/1426029245 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f87a806bbd0 msgr2=0x7f87a806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:59.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.006+0000 7f87c26e8640 1 --2- 172.21.15.150:0/1426029245 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f87a806bbd0 0x7f87a806e090 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f87ac007fd0 tx=0x7f87ac009040 comp rx=0 tx=0).stop 2024-03-17T23:47:59.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.006+0000 7f87c26e8640 1 -- 172.21.15.150:0/1426029245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87bc1619a0 msgr2=0x7f87bc160770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:59.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.006+0000 7f87c26e8640 1 --2- 172.21.15.150:0/1426029245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87bc1619a0 0x7f87bc160770 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f87b4009ee0 tx=0x7f87b4004a40 comp rx=0 tx=0).stop 2024-03-17T23:47:59.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.006+0000 7f87c26e8640 1 -- 172.21.15.150:0/1426029245 shutdown_connections 2024-03-17T23:47:59.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.006+0000 7f87c26e8640 1 --2- 172.21.15.150:0/1426029245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87bc1619a0 0x7f87bc160770 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.006+0000 7f87c26e8640 1 --2- 172.21.15.150:0/1426029245 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc15de30 0x7f87bc157be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.006+0000 7f87c26e8640 1 --2- 172.21.15.150:0/1426029245 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f87a806bbd0 0x7f87a806e090 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.006+0000 7f87c26e8640 1 --2- 172.21.15.150:0/1426029245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87bc0cf120 0x7f87bc157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.006+0000 7f87c26e8640 1 -- 172.21.15.150:0/1426029245 >> 172.21.15.150:0/1426029245 conn(0x7f87bc0730b0 msgr2=0x7f87bc0d31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:59.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.006+0000 7f87c26e8640 1 -- 172.21.15.150:0/1426029245 shutdown_connections 2024-03-17T23:47:59.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.006+0000 7f87c26e8640 1 -- 172.21.15.150:0/1426029245 wait complete. 2024-03-17T23:47:59.018 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:47:59.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.119+0000 7fb93d762640 1 Processor -- start 2024-03-17T23:47:59.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.119+0000 7fb93d762640 1 -- start start 2024-03-17T23:47:59.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.120+0000 7fb93d762640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb93815de30 0x7fb9381619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:59.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.120+0000 7fb93d762640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9380b9860 0x7fb9380cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:59.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.120+0000 7fb93d762640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9380b9cc0 con 0x7fb9380cef50 2024-03-17T23:47:59.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.120+0000 7fb93d762640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fb9380cf3c0 con 0x7fb93815de30 2024-03-17T23:47:59.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.120+0000 7fb93d762640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb938162940 con 0x7fb9380b9860 2024-03-17T23:47:59.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.120+0000 7fb9367fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9380b9860 0x7fb9380cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:59.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.120+0000 7fb936ffd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb93815de30 0x7fb9381619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:59.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.120+0000 7fb9377fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fb9380cef50 0x7fb93806c790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:45714/0 (socket says 172.21.15.150:45714) 2024-03-17T23:47:59.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.120+0000 7fb9377fe640 1 -- 172.21.15.150:0/1716890531 learned_addr learned my addr 172.21.15.150:0/1716890531 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:59.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.120+0000 7fb936ffd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb93815de30 0x7fb9381619a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:55812/0 (socket says 172.21.15.150:55812) 2024-03-17T23:47:59.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.120+0000 7fb9367fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9380b9860 0x7fb9380cea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:38722/0 (socket says 172.21.15.150:38722) 2024-03-17T23:47:59.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.120+0000 7fb936ffd640 1 -- 172.21.15.150:0/1716890531 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9380b9860 msgr2=0x7fb9380cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:59.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.120+0000 7fb936ffd640 1 --2- 172.21.15.150:0/1716890531 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9380b9860 0x7fb9380cea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.120+0000 7fb936ffd640 1 -- 172.21.15.150:0/1716890531 >> v1:172.21.15.157:6789/0 conn(0x7fb9380cef50 legacy=0x7fb93806c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:59.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.120+0000 7fb936ffd640 1 -- 172.21.15.150:0/1716890531 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9381620a0 con 0x7fb93815de30 2024-03-17T23:47:59.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.121+0000 7fb936ffd640 1 --2- 172.21.15.150:0/1716890531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb93815de30 0x7fb9381619a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb9280097a0 tx=0x7fb92802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c86689cbe62a9705 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:59.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.121+0000 7fb935ffb640 1 -- 172.21.15.150:0/1716890531 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9280315a0 con 0x7fb93815de30 2024-03-17T23:47:59.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.121+0000 7fb935ffb640 1 -- 172.21.15.150:0/1716890531 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb928037070 con 0x7fb93815de30 2024-03-17T23:47:59.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.121+0000 7fb935ffb640 1 -- 172.21.15.150:0/1716890531 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb928031960 con 0x7fb93815de30 2024-03-17T23:47:59.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.122+0000 7fb93d762640 1 -- 172.21.15.150:0/1716890531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb93815de30 msgr2=0x7fb9381619a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:59.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.122+0000 7fb93d762640 1 --2- 172.21.15.150:0/1716890531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb93815de30 0x7fb9381619a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb9280097a0 tx=0x7fb92802b840 comp rx=0 tx=0).stop 2024-03-17T23:47:59.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.122+0000 7fb93d762640 1 -- 172.21.15.150:0/1716890531 shutdown_connections 2024-03-17T23:47:59.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.122+0000 7fb93d762640 1 --2- 172.21.15.150:0/1716890531 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9380b9860 0x7fb9380cea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.122+0000 7fb93d762640 1 --2- 172.21.15.150:0/1716890531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb93815de30 0x7fb9381619a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.122+0000 7fb93d762640 1 -- 172.21.15.150:0/1716890531 >> 172.21.15.150:0/1716890531 conn(0x7fb9380730b0 msgr2=0x7fb9380d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:59.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.122+0000 7fb93d762640 1 -- 172.21.15.150:0/1716890531 shutdown_connections 2024-03-17T23:47:59.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.122+0000 7fb93d762640 1 -- 172.21.15.150:0/1716890531 wait complete. 2024-03-17T23:47:59.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.123+0000 7fb93d762640 1 Processor -- start 2024-03-17T23:47:59.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.123+0000 7fb93d762640 1 -- start start 2024-03-17T23:47:59.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.124+0000 7fb93d762640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9380b9860 0x7fb9380b5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:59.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.124+0000 7fb93d762640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9380cef50 0x7fb9380cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:59.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.124+0000 7fb93d762640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb93815de30 0x7fb9380cb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:59.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.124+0000 7fb93d762640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb9380d2170 con 0x7fb9380b9860 2024-03-17T23:47:59.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.124+0000 7fb93d762640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb938162660 con 0x7fb93815de30 2024-03-17T23:47:59.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.124+0000 7fb93d762640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb93815e210 con 0x7fb9380cef50 2024-03-17T23:47:59.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.124+0000 7fb936ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9380b9860 0x7fb9380b5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:59.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.124+0000 7fb9377fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb93815de30 0x7fb9380cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:59.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.124+0000 7fb936ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9380b9860 0x7fb9380b5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45736/0 (socket says 172.21.15.150:45736) 2024-03-17T23:47:59.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.124+0000 7fb936ffd640 1 -- 172.21.15.150:0/630938133 learned_addr learned my addr 172.21.15.150:0/630938133 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:59.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.124+0000 7fb9367fc640 1 --2- 172.21.15.150:0/630938133 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9380cef50 0x7fb9380cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:59.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.124+0000 7fb9377fe640 1 -- 172.21.15.150:0/630938133 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9380cef50 msgr2=0x7fb9380cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:59.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.124+0000 7fb9377fe640 1 --2- 172.21.15.150:0/630938133 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9380cef50 0x7fb9380cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.124+0000 7fb9377fe640 1 -- 172.21.15.150:0/630938133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9380b9860 msgr2=0x7fb9380b5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:59.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.124+0000 7fb9377fe640 1 --2- 172.21.15.150:0/630938133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9380b9860 0x7fb9380b5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.124+0000 7fb9377fe640 1 -- 172.21.15.150:0/630938133 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb938160e40 con 0x7fb93815de30 2024-03-17T23:47:59.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.124+0000 7fb936ffd640 1 --2- 172.21.15.150:0/630938133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9380b9860 0x7fb9380b5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:59.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.125+0000 7fb9377fe640 1 --2- 172.21.15.150:0/630938133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb93815de30 0x7fb9380cb350 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb93000ca70 tx=0x7fb93000cf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:59.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.125+0000 7fb91f7fe640 1 -- 172.21.15.150:0/630938133 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb930017070 con 0x7fb93815de30 2024-03-17T23:47:59.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.125+0000 7fb91f7fe640 1 -- 172.21.15.150:0/630938133 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb93000aa90 con 0x7fb93815de30 2024-03-17T23:47:59.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.125+0000 7fb93d762640 1 -- 172.21.15.150:0/630938133 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9380cb980 con 0x7fb93815de30 2024-03-17T23:47:59.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.125+0000 7fb91f7fe640 1 -- 172.21.15.150:0/630938133 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb930012400 con 0x7fb93815de30 2024-03-17T23:47:59.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.125+0000 7fb93d762640 1 -- 172.21.15.150:0/630938133 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9380cbf40 con 0x7fb93815de30 2024-03-17T23:47:59.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.126+0000 7fb93d762640 1 -- 172.21.15.150:0/630938133 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9380681a0 con 0x7fb93815de30 2024-03-17T23:47:59.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.126+0000 7fb91f7fe640 1 -- 172.21.15.150:0/630938133 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb93001a070 con 0x7fb93815de30 2024-03-17T23:47:59.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.126+0000 7fb91f7fe640 1 --2- 172.21.15.150:0/630938133 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb92406bbd0 0x7fb92406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:59.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.127+0000 7fb936ffd640 1 --2- 172.21.15.150:0/630938133 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb92406bbd0 0x7fb92406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:59.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.127+0000 7fb91f7fe640 1 -- 172.21.15.150:0/630938133 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7fb93008aa40 con 0x7fb93815de30 2024-03-17T23:47:59.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.127+0000 7fb936ffd640 1 --2- 172.21.15.150:0/630938133 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb92406bbd0 0x7fb92406e090 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb9280097a0 tx=0x7fb92803d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:59.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.131+0000 7fb91f7fe640 1 -- 172.21.15.150:0/630938133 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb930058ae0 con 0x7fb93815de30 2024-03-17T23:47:59.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.340+0000 7fb93d762640 1 -- 172.21.15.150:0/630938133 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9380ca5b0 con 0x7fb93815de30 2024-03-17T23:47:59.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.341+0000 7fb91f7fe640 1 -- 172.21.15.150:0/630938133 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5186 (secure 0 0 0) 0x7fb93008ae20 con 0x7fb93815de30 2024-03-17T23:47:59.342 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:59.342 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4253,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/1949609985","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6833","nonce":1949609985},{"type":"v1","addr":"172.21.15.157:6835","nonce":1949609985}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/758264728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":758264728},{"type":"v1","addr":"172.21.15.150:6835","nonce":758264728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3655589010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":3655589010},{"type":"v1","addr":"172.21.15.150:6837","nonce":3655589010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-17T23:46:55.253739+0000","modified":"2024-03-17T23:47:11.316249+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4297},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4297":{"gid":4297,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6834/1625058792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1625058792},{"type":"v1","addr":"172.21.15.157:6834","nonce":1625058792}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4297,"qdb_cluster":[4297]},"id":1}]} 2024-03-17T23:47:59.342 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 8 2024-03-17T23:47:59.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.343+0000 7fb93d762640 1 -- 172.21.15.150:0/630938133 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb92406bbd0 msgr2=0x7fb92406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:59.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.343+0000 7fb93d762640 1 --2- 172.21.15.150:0/630938133 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb92406bbd0 0x7fb92406e090 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb9280097a0 tx=0x7fb92803d040 comp rx=0 tx=0).stop 2024-03-17T23:47:59.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.343+0000 7fb93d762640 1 -- 172.21.15.150:0/630938133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb93815de30 msgr2=0x7fb9380cb350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:59.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.343+0000 7fb93d762640 1 --2- 172.21.15.150:0/630938133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb93815de30 0x7fb9380cb350 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb93000ca70 tx=0x7fb93000cf30 comp rx=0 tx=0).stop 2024-03-17T23:47:59.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.343+0000 7fb93d762640 1 -- 172.21.15.150:0/630938133 shutdown_connections 2024-03-17T23:47:59.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.343+0000 7fb93d762640 1 --2- 172.21.15.150:0/630938133 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb92406bbd0 0x7fb92406e090 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.343+0000 7fb93d762640 1 --2- 172.21.15.150:0/630938133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb93815de30 0x7fb9380cb350 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.343+0000 7fb93d762640 1 --2- 172.21.15.150:0/630938133 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb9380cef50 0x7fb9380cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.343+0000 7fb93d762640 1 --2- 172.21.15.150:0/630938133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9380b9860 0x7fb9380b5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.343+0000 7fb93d762640 1 -- 172.21.15.150:0/630938133 >> 172.21.15.150:0/630938133 conn(0x7fb9380730b0 msgr2=0x7fb9380b44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:59.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.343+0000 7fb93d762640 1 -- 172.21.15.150:0/630938133 shutdown_connections 2024-03-17T23:47:59.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.343+0000 7fb93d762640 1 -- 172.21.15.150:0/630938133 wait complete. 2024-03-17T23:47:59.355 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:47:59.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.455+0000 7f9085473640 1 Processor -- start 2024-03-17T23:47:59.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.455+0000 7f9085473640 1 -- start start 2024-03-17T23:47:59.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.456+0000 7f9085473640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90801619a0 0x7f90800b9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:59.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.456+0000 7f9085473640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90801620a0 con 0x7f90800cf120 2024-03-17T23:47:59.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.456+0000 7f9085473640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9080162380 con 0x7f908015de30 2024-03-17T23:47:59.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.456+0000 7f9085473640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9080162940 con 0x7f90801619a0 2024-03-17T23:47:59.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.456+0000 7f907e7fc640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f90800cf120 0x7f90800d2690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33550/0 (socket says 172.21.15.150:33550) 2024-03-17T23:47:59.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.456+0000 7f907e7fc640 1 -- 172.21.15.150:0/3153250176 learned_addr learned my addr 172.21.15.150:0/3153250176 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:47:59.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.456+0000 7f907f7fe640 1 --2- 172.21.15.150:0/3153250176 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90801619a0 0x7f90800b9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:59.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.456+0000 7f907dffb640 1 -- 172.21.15.150:0/3153250176 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1726054754 0 0) 0x7f90801620a0 con 0x7f90800cf120 2024-03-17T23:47:59.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.456+0000 7f907dffb640 1 -- 172.21.15.150:0/3153250176 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9068003520 con 0x7f90800cf120 2024-03-17T23:47:59.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.456+0000 7f907dffb640 1 -- 172.21.15.150:0/3153250176 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 999796814 0 0) 0x7f9080162380 con 0x7f908015de30 2024-03-17T23:47:59.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.456+0000 7f907dffb640 1 -- 172.21.15.150:0/3153250176 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90801620a0 con 0x7f908015de30 2024-03-17T23:47:59.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.456+0000 7f907f7fe640 1 -- 172.21.15.150:0/3153250176 >> v1:172.21.15.157:6789/0 conn(0x7f908015de30 legacy=0x7f90800cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:59.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.456+0000 7f907f7fe640 1 -- 172.21.15.150:0/3153250176 >> v1:172.21.15.150:6789/0 conn(0x7f90800cf120 legacy=0x7f90800d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:59.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.456+0000 7f907f7fe640 1 -- 172.21.15.150:0/3153250176 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9080162660 con 0x7f90801619a0 2024-03-17T23:47:59.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.457+0000 7f907f7fe640 1 --2- 172.21.15.150:0/3153250176 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90801619a0 0x7f90800b9860 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9078009dc0 tx=0x7f907802be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47403023f131a617 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:59.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.457+0000 7f907dffb640 1 -- 172.21.15.150:0/3153250176 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9078039070 con 0x7f90801619a0 2024-03-17T23:47:59.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.457+0000 7f907dffb640 1 -- 172.21.15.150:0/3153250176 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f907802cc40 con 0x7f90801619a0 2024-03-17T23:47:59.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.457+0000 7f907dffb640 1 -- 172.21.15.150:0/3153250176 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9078034440 con 0x7f90801619a0 2024-03-17T23:47:59.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.458+0000 7f9085473640 1 -- 172.21.15.150:0/3153250176 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90801619a0 msgr2=0x7f90800b9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:59.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.458+0000 7f9085473640 1 --2- 172.21.15.150:0/3153250176 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90801619a0 0x7f90800b9860 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9078009dc0 tx=0x7f907802be90 comp rx=0 tx=0).stop 2024-03-17T23:47:59.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.458+0000 7f9085473640 1 -- 172.21.15.150:0/3153250176 shutdown_connections 2024-03-17T23:47:59.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.458+0000 7f9085473640 1 --2- 172.21.15.150:0/3153250176 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90801619a0 0x7f90800b9860 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.458+0000 7f9085473640 1 -- 172.21.15.150:0/3153250176 >> 172.21.15.150:0/3153250176 conn(0x7f90800730b0 msgr2=0x7f90800d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:59.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.458+0000 7f9085473640 1 -- 172.21.15.150:0/3153250176 shutdown_connections 2024-03-17T23:47:59.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.458+0000 7f9085473640 1 -- 172.21.15.150:0/3153250176 wait complete. 2024-03-17T23:47:59.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.459+0000 7f9085473640 1 Processor -- start 2024-03-17T23:47:59.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.459+0000 7f9085473640 1 -- start start 2024-03-17T23:47:59.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.460+0000 7f9085473640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90800cf120 0x7f9080157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:59.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.460+0000 7f9085473640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f908015de30 0x7f90800cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:59.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.460+0000 7f9085473640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90801619a0 0x7f9080157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:59.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.460+0000 7f9085473640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f908015e210 con 0x7f90801619a0 2024-03-17T23:47:59.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.460+0000 7f9085473640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9080161da0 con 0x7f908015de30 2024-03-17T23:47:59.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.460+0000 7f9085473640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f90800d2da0 con 0x7f90800cf120 2024-03-17T23:47:59.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.460+0000 7f907e7fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f908015de30 0x7f90800cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:59.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.460+0000 7f907effd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90800cf120 0x7f9080157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:59.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.460+0000 7f907f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90801619a0 0x7f9080157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:59.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.460+0000 7f907e7fc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f908015de30 0x7f90800cbf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:55832/0 (socket says 172.21.15.150:55832) 2024-03-17T23:47:59.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.460+0000 7f907e7fc640 1 -- 172.21.15.150:0/3758773037 learned_addr learned my addr 172.21.15.150:0/3758773037 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:59.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.460+0000 7f907f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90801619a0 0x7f9080157a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45744/0 (socket says 172.21.15.150:45744) 2024-03-17T23:47:59.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.460+0000 7f907f7fe640 1 -- 172.21.15.150:0/3758773037 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90800cf120 msgr2=0x7f9080157310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:59.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.460+0000 7f907f7fe640 1 --2- 172.21.15.150:0/3758773037 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90800cf120 0x7f9080157310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.460+0000 7f907f7fe640 1 -- 172.21.15.150:0/3758773037 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f908015de30 msgr2=0x7f90800cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:59.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.460+0000 7f907f7fe640 1 --2- 172.21.15.150:0/3758773037 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f908015de30 0x7f90800cbf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.460+0000 7f907f7fe640 1 -- 172.21.15.150:0/3758773037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90800cc4d0 con 0x7f90801619a0 2024-03-17T23:47:59.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.460+0000 7f907effd640 1 --2- 172.21.15.150:0/3758773037 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90800cf120 0x7f9080157310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:59.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.461+0000 7f907f7fe640 1 --2- 172.21.15.150:0/3758773037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90801619a0 0x7f9080157a50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9078033000 tx=0x7f9078004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:59.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.461+0000 7f90677fe640 1 -- 172.21.15.150:0/3758773037 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9078039050 con 0x7f90801619a0 2024-03-17T23:47:59.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.461+0000 7f90677fe640 1 -- 172.21.15.150:0/3758773037 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9078045070 con 0x7f90801619a0 2024-03-17T23:47:59.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.461+0000 7f9085473640 1 -- 172.21.15.150:0/3758773037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9080158550 con 0x7f90801619a0 2024-03-17T23:47:59.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.461+0000 7f90677fe640 1 -- 172.21.15.150:0/3758773037 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9078009450 con 0x7f90801619a0 2024-03-17T23:47:59.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.461+0000 7f9085473640 1 -- 172.21.15.150:0/3758773037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f908015f1e0 con 0x7f90801619a0 2024-03-17T23:47:59.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.462+0000 7f9085473640 1 -- 172.21.15.150:0/3758773037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9080158a70 con 0x7f90801619a0 2024-03-17T23:47:59.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.462+0000 7f90677fe640 1 -- 172.21.15.150:0/3758773037 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9078040670 con 0x7f90801619a0 2024-03-17T23:47:59.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.463+0000 7f90677fe640 1 --2- 172.21.15.150:0/3758773037 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f906c06bbd0 0x7f906c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:59.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.463+0000 7f90677fe640 1 -- 172.21.15.150:0/3758773037 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f90780ad620 con 0x7f90801619a0 2024-03-17T23:47:59.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.463+0000 7f907effd640 1 --2- 172.21.15.150:0/3758773037 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f906c06bbd0 0x7f906c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:59.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.463+0000 7f907effd640 1 --2- 172.21.15.150:0/3758773037 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f906c06bbd0 0x7f906c06e090 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9070007fd0 tx=0x7f9070009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:59.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.467+0000 7f90677fe640 1 -- 172.21.15.150:0/3758773037 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9080158a70 con 0x7f90801619a0 2024-03-17T23:47:59.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.642+0000 7f9085473640 1 -- 172.21.15.150:0/3758773037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f90800681a0 con 0x7f90801619a0 2024-03-17T23:47:59.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.643+0000 7f90677fe640 1 -- 172.21.15.150:0/3758773037 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) ==== 74+0+18881 (secure 0 0 0) 0x7f907807b750 con 0x7f90801619a0 2024-03-17T23:47:59.644 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:47:59.644 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":28,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:47:37.031864+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:46:53.610678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-17T23:46:54.227335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-17T23:46:57.949235+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.e","mappings":[{"from":7,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:47:59.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.645+0000 7f9085473640 1 -- 172.21.15.150:0/3758773037 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f906c06bbd0 msgr2=0x7f906c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:59.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.645+0000 7f9085473640 1 --2- 172.21.15.150:0/3758773037 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f906c06bbd0 0x7f906c06e090 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9070007fd0 tx=0x7f9070009040 comp rx=0 tx=0).stop 2024-03-17T23:47:59.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.645+0000 7f9085473640 1 -- 172.21.15.150:0/3758773037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90801619a0 msgr2=0x7f9080157a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:59.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.645+0000 7f9085473640 1 --2- 172.21.15.150:0/3758773037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90801619a0 0x7f9080157a50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9078033000 tx=0x7f9078004a40 comp rx=0 tx=0).stop 2024-03-17T23:47:59.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.645+0000 7f9085473640 1 -- 172.21.15.150:0/3758773037 shutdown_connections 2024-03-17T23:47:59.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.645+0000 7f9085473640 1 --2- 172.21.15.150:0/3758773037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f90801619a0 0x7f9080157a50 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.645+0000 7f9085473640 1 --2- 172.21.15.150:0/3758773037 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f906c06bbd0 0x7f906c06e090 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.645+0000 7f9085473640 1 --2- 172.21.15.150:0/3758773037 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f908015de30 0x7f90800cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.645+0000 7f9085473640 1 --2- 172.21.15.150:0/3758773037 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f90800cf120 0x7f9080157310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.645+0000 7f9085473640 1 -- 172.21.15.150:0/3758773037 >> 172.21.15.150:0/3758773037 conn(0x7f90800730b0 msgr2=0x7f90800d3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:59.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.645+0000 7f9085473640 1 -- 172.21.15.150:0/3758773037 shutdown_connections 2024-03-17T23:47:59.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.645+0000 7f9085473640 1 -- 172.21.15.150:0/3758773037 wait complete. 2024-03-17T23:47:59.657 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-17T23:47:59.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.758+0000 7f49dc10a640 1 Processor -- start 2024-03-17T23:47:59.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.758+0000 7f49dc10a640 1 -- start start 2024-03-17T23:47:59.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.759+0000 7f49dc10a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f49d4161530 0x7f49d40d3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:59.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.759+0000 7f49dc10a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f49d406cda0 0x7f49d40d1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:59.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.759+0000 7f49dc10a640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49d40ce6a0 con 0x7f49d4161910 2024-03-17T23:47:59.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.759+0000 7f49dc10a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f49d406db30 con 0x7f49d406cda0 2024-03-17T23:47:59.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.759+0000 7f49dc10a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f49d406d470 con 0x7f49d4161530 2024-03-17T23:47:59.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.759+0000 7f49da6a7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f49d406cda0 0x7f49d40d1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:59.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.759+0000 7f49da6a7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f49d406cda0 0x7f49d40d1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45760/0 (socket says 172.21.15.150:45760) 2024-03-17T23:47:59.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.759+0000 7f49d9ea6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f49d4161530 0x7f49d40d3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:59.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.759+0000 7f49da6a7640 1 -- 172.21.15.150:0/3673140943 learned_addr learned my addr 172.21.15.150:0/3673140943 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:59.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.760+0000 7f49d8ea4640 1 -- 172.21.15.150:0/3673140943 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1132417029 0 0) 0x7f49d40ce6a0 con 0x7f49d4161910 2024-03-17T23:47:59.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.760+0000 7f49d8ea4640 1 -- 172.21.15.150:0/3673140943 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49c0003520 con 0x7f49d4161910 2024-03-17T23:47:59.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.760+0000 7f49d9ea6640 1 -- 172.21.15.150:0/3673140943 >> v2:172.21.15.150:3300/0 conn(0x7f49d406cda0 msgr2=0x7f49d40d1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:47:59.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.760+0000 7f49d9ea6640 1 --2- 172.21.15.150:0/3673140943 >> v2:172.21.15.150:3300/0 conn(0x7f49d406cda0 0x7f49d40d1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.760+0000 7f49d9ea6640 1 -- 172.21.15.150:0/3673140943 >> v1:172.21.15.157:6789/0 conn(0x7f49d4161910 legacy=0x7f49d406c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:59.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.760+0000 7f49d9ea6640 1 -- 172.21.15.150:0/3673140943 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d406d750 con 0x7f49d4161530 2024-03-17T23:47:59.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.760+0000 7f49da6a7640 1 --2- 172.21.15.150:0/3673140943 >> v2:172.21.15.150:3300/0 conn(0x7f49d406cda0 0x7f49d40d1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:47:59.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.760+0000 7f49d9ea6640 1 --2- 172.21.15.150:0/3673140943 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f49d4161530 0x7f49d40d3690 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f49d0009d90 tx=0x7f49d002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9c507bd25ed5316 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:59.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.761+0000 7f49d8ea4640 1 -- 172.21.15.150:0/3673140943 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d0039070 con 0x7f49d4161530 2024-03-17T23:47:59.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.761+0000 7f49d8ea4640 1 -- 172.21.15.150:0/3673140943 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49d002cc50 con 0x7f49d4161530 2024-03-17T23:47:59.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.761+0000 7f49d8ea4640 1 -- 172.21.15.150:0/3673140943 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d0034440 con 0x7f49d4161530 2024-03-17T23:47:59.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.761+0000 7f49dc10a640 1 -- 172.21.15.150:0/3673140943 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f49d4161530 msgr2=0x7f49d40d3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:59.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.761+0000 7f49dc10a640 1 --2- 172.21.15.150:0/3673140943 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f49d4161530 0x7f49d40d3690 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f49d0009d90 tx=0x7f49d002bea0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.761+0000 7f49dc10a640 1 -- 172.21.15.150:0/3673140943 shutdown_connections 2024-03-17T23:47:59.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.761+0000 7f49dc10a640 1 --2- 172.21.15.150:0/3673140943 >> v2:172.21.15.150:3300/0 conn(0x7f49d406cda0 0x7f49d40d1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.761+0000 7f49dc10a640 1 --2- 172.21.15.150:0/3673140943 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f49d4161530 0x7f49d40d3690 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.761+0000 7f49dc10a640 1 -- 172.21.15.150:0/3673140943 >> 172.21.15.150:0/3673140943 conn(0x7f49d40d0d50 msgr2=0x7f49d40d3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:59.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.761+0000 7f49dc10a640 1 -- 172.21.15.150:0/3673140943 shutdown_connections 2024-03-17T23:47:59.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.761+0000 7f49dc10a640 1 -- 172.21.15.150:0/3673140943 wait complete. 2024-03-17T23:47:59.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.762+0000 7f49dc10a640 1 Processor -- start 2024-03-17T23:47:59.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.762+0000 7f49dc10a640 1 -- start start 2024-03-17T23:47:59.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.763+0000 7f49dc10a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49d406cda0 0x7f49d41570f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:59.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.763+0000 7f49dc10a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49d4161530 0x7f49d4157630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:59.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.763+0000 7f49dc10a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f49d4161910 0x7f49d40cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:59.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.763+0000 7f49dc10a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f49d40b3210 con 0x7f49d406cda0 2024-03-17T23:47:59.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.763+0000 7f49dc10a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f49d40b3090 con 0x7f49d4161530 2024-03-17T23:47:59.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.763+0000 7f49dc10a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f49d40b3390 con 0x7f49d4161910 2024-03-17T23:47:59.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.763+0000 7f49d9ea6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49d406cda0 0x7f49d41570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:59.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.763+0000 7f49d9ea6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49d406cda0 0x7f49d41570f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45772/0 (socket says 172.21.15.150:45772) 2024-03-17T23:47:59.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.763+0000 7f49d9ea6640 1 -- 172.21.15.150:0/2940341803 learned_addr learned my addr 172.21.15.150:0/2940341803 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:47:59.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.763+0000 7f49da6a7640 1 --2- 172.21.15.150:0/2940341803 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f49d4161910 0x7f49d40cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:59.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.763+0000 7f49d96a5640 1 --2- 172.21.15.150:0/2940341803 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49d4161530 0x7f49d4157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:59.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.764+0000 7f49d9ea6640 1 -- 172.21.15.150:0/2940341803 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f49d4161910 msgr2=0x7f49d40cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:59.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.764+0000 7f49d9ea6640 1 --2- 172.21.15.150:0/2940341803 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f49d4161910 0x7f49d40cc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.764+0000 7f49d9ea6640 1 -- 172.21.15.150:0/2940341803 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49d4161530 msgr2=0x7f49d4157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:59.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.764+0000 7f49d9ea6640 1 --2- 172.21.15.150:0/2940341803 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49d4161530 0x7f49d4157630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.764+0000 7f49d9ea6640 1 -- 172.21.15.150:0/2940341803 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d41f3f60 con 0x7f49d406cda0 2024-03-17T23:47:59.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.764+0000 7f49d9ea6640 1 --2- 172.21.15.150:0/2940341803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49d406cda0 0x7f49d41570f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f49d0037040 tx=0x7f49d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:59.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.764+0000 7f49ca7fc640 1 -- 172.21.15.150:0/2940341803 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d0039050 con 0x7f49d406cda0 2024-03-17T23:47:59.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.764+0000 7f49ca7fc640 1 -- 172.21.15.150:0/2940341803 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49d0045070 con 0x7f49d406cda0 2024-03-17T23:47:59.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.764+0000 7f49ca7fc640 1 -- 172.21.15.150:0/2940341803 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d0009450 con 0x7f49d406cda0 2024-03-17T23:47:59.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.764+0000 7f49dc10a640 1 -- 172.21.15.150:0/2940341803 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49d40cd500 con 0x7f49d406cda0 2024-03-17T23:47:59.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.764+0000 7f49dc10a640 1 -- 172.21.15.150:0/2940341803 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49d40c8640 con 0x7f49d406cda0 2024-03-17T23:47:59.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.765+0000 7f49dc10a640 1 -- 172.21.15.150:0/2940341803 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49d40681a0 con 0x7f49d406cda0 2024-03-17T23:47:59.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.765+0000 7f49ca7fc640 1 -- 172.21.15.150:0/2940341803 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f49d003e610 con 0x7f49d406cda0 2024-03-17T23:47:59.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.766+0000 7f49ca7fc640 1 --2- 172.21.15.150:0/2940341803 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f49b806bbd0 0x7f49b806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:47:59.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.766+0000 7f49ca7fc640 1 -- 172.21.15.150:0/2940341803 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6735+0+0 (secure 0 0 0) 0x7f49d00ad780 con 0x7f49d406cda0 2024-03-17T23:47:59.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.766+0000 7f49d96a5640 1 --2- 172.21.15.150:0/2940341803 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f49b806bbd0 0x7f49b806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:47:59.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.766+0000 7f49d96a5640 1 --2- 172.21.15.150:0/2940341803 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f49b806bbd0 0x7f49b806e090 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f49c4007fd0 tx=0x7f49c4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:47:59.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.770+0000 7f49ca7fc640 1 -- 172.21.15.150:0/2940341803 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f49d007b8b0 con 0x7f49d406cda0 2024-03-17T23:47:59.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.982+0000 7f49dc10a640 1 -- 172.21.15.150:0/2940341803 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f49d40d4a30 con 0x7f49d406cda0 2024-03-17T23:47:59.983 INFO:tasks.ceph.mon.a.smithi150.stderr:2024-03-17T23:47:59.982+0000 7fd9a9062640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-17T23:47:59.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.984+0000 7f49ca7fc640 1 -- 172.21.15.150:0/2940341803 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) ==== 161+0+0 (secure 0 0 0) 0x7f49d007f950 con 0x7f49d406cda0 2024-03-17T23:47:59.985 INFO:teuthology.orchestra.run.smithi150.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-17T23:47:59.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.986+0000 7f49dc10a640 1 -- 172.21.15.150:0/2940341803 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f49b806bbd0 msgr2=0x7f49b806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:59.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.986+0000 7f49dc10a640 1 --2- 172.21.15.150:0/2940341803 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f49b806bbd0 0x7f49b806e090 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f49c4007fd0 tx=0x7f49c4009040 comp rx=0 tx=0).stop 2024-03-17T23:47:59.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.986+0000 7f49dc10a640 1 -- 172.21.15.150:0/2940341803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49d406cda0 msgr2=0x7f49d41570f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:47:59.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.986+0000 7f49dc10a640 1 --2- 172.21.15.150:0/2940341803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49d406cda0 0x7f49d41570f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f49d0037040 tx=0x7f49d0004a40 comp rx=0 tx=0).stop 2024-03-17T23:47:59.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.986+0000 7f49dc10a640 1 -- 172.21.15.150:0/2940341803 shutdown_connections 2024-03-17T23:47:59.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.986+0000 7f49dc10a640 1 --2- 172.21.15.150:0/2940341803 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f49d4161910 0x7f49d40cc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.986+0000 7f49dc10a640 1 --2- 172.21.15.150:0/2940341803 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f49b806bbd0 0x7f49b806e090 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.986+0000 7f49dc10a640 1 --2- 172.21.15.150:0/2940341803 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49d4161530 0x7f49d4157630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.986+0000 7f49dc10a640 1 --2- 172.21.15.150:0/2940341803 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f49d406cda0 0x7f49d41570f0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:47:59.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.986+0000 7f49dc10a640 1 -- 172.21.15.150:0/2940341803 >> 172.21.15.150:0/2940341803 conn(0x7f49d40d0d50 msgr2=0x7f49d40cebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:47:59.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.986+0000 7f49dc10a640 1 -- 172.21.15.150:0/2940341803 shutdown_connections 2024-03-17T23:47:59.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:47:59.986+0000 7f49dc10a640 1 -- 172.21.15.150:0/2940341803 wait complete. 2024-03-17T23:47:59.990 INFO:tasks.ceph.mds.b.smithi157.stderr: -4482> 2024-03-17T23:47:00.203+0000 7f61d3871640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-17T23:47:59.998 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-17T23:48:00.019 INFO:tasks.ceph.mds.b.smithi157.stdout:starting mds.b at 2024-03-17T23:48:00.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.101+0000 7f3a926ff640 1 Processor -- start 2024-03-17T23:48:00.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.101+0000 7f3a926ff640 1 -- start start 2024-03-17T23:48:00.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.102+0000 7f3a926ff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a8c15bac0 0x7f3a8c067570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:00.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.102+0000 7f3a926ff640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a8c099c10 con 0x7f3a8c15f210 2024-03-17T23:48:00.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.102+0000 7f3a926ff640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a8c15e730 con 0x7f3a8c06ec90 2024-03-17T23:48:00.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.102+0000 7f3a926ff640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3a8c099ef0 con 0x7f3a8c15bac0 2024-03-17T23:48:00.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.102+0000 7f3a8b7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f3a8c15f210 0x7f3a8c06e580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33560/0 (socket says 172.21.15.150:33560) 2024-03-17T23:48:00.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.102+0000 7f3a8b7fe640 1 -- 172.21.15.150:0/3344178767 learned_addr learned my addr 172.21.15.150:0/3344178767 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:00.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.102+0000 7f3a8bfff640 1 --2- 172.21.15.150:0/3344178767 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a8c15bac0 0x7f3a8c067570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:00.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.102+0000 7f3a8affd640 1 -- 172.21.15.150:0/3344178767 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 976431949 0 0) 0x7f3a8c099c10 con 0x7f3a8c15f210 2024-03-17T23:48:00.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.103+0000 7f3a8affd640 1 -- 172.21.15.150:0/3344178767 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a74003520 con 0x7f3a8c15f210 2024-03-17T23:48:00.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.103+0000 7f3a8bfff640 1 -- 172.21.15.150:0/3344178767 >> v1:172.21.15.157:6789/0 conn(0x7f3a8c06ec90 legacy=0x7f3a8c06b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:00.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.103+0000 7f3a8bfff640 1 -- 172.21.15.150:0/3344178767 >> v1:172.21.15.150:6789/0 conn(0x7f3a8c15f210 legacy=0x7f3a8c06e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:00.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.103+0000 7f3a8bfff640 1 -- 172.21.15.150:0/3344178767 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a8c15ea10 con 0x7f3a8c15bac0 2024-03-17T23:48:00.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.103+0000 7f3a8bfff640 1 --2- 172.21.15.150:0/3344178767 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a8c15bac0 0x7f3a8c067570 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c0097d0 tx=0x7f3a7c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccfc0553b796a763 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:00.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.104+0000 7f3a8affd640 1 -- 172.21.15.150:0/3344178767 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c035020 con 0x7f3a8c15bac0 2024-03-17T23:48:00.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.104+0000 7f3a8affd640 1 -- 172.21.15.150:0/3344178767 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a7c031800 con 0x7f3a8c15bac0 2024-03-17T23:48:00.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.104+0000 7f3a8affd640 1 -- 172.21.15.150:0/3344178767 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c031bd0 con 0x7f3a8c15bac0 2024-03-17T23:48:00.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.104+0000 7f3a926ff640 1 -- 172.21.15.150:0/3344178767 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a8c15bac0 msgr2=0x7f3a8c067570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:00.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.104+0000 7f3a926ff640 1 --2- 172.21.15.150:0/3344178767 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a8c15bac0 0x7f3a8c067570 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c0097d0 tx=0x7f3a7c02b8a0 comp rx=0 tx=0).stop 2024-03-17T23:48:00.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.104+0000 7f3a926ff640 1 -- 172.21.15.150:0/3344178767 shutdown_connections 2024-03-17T23:48:00.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.104+0000 7f3a926ff640 1 --2- 172.21.15.150:0/3344178767 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a8c15bac0 0x7f3a8c067570 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:00.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.104+0000 7f3a926ff640 1 -- 172.21.15.150:0/3344178767 >> 172.21.15.150:0/3344178767 conn(0x7f3a8c0b3da0 msgr2=0x7f3a8c0b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:00.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.104+0000 7f3a926ff640 1 -- 172.21.15.150:0/3344178767 shutdown_connections 2024-03-17T23:48:00.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.104+0000 7f3a926ff640 1 -- 172.21.15.150:0/3344178767 wait complete. 2024-03-17T23:48:00.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.105+0000 7f3a926ff640 1 Processor -- start 2024-03-17T23:48:00.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.106+0000 7f3a926ff640 1 -- start start 2024-03-17T23:48:00.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.106+0000 7f3a926ff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a8c06ec90 0x7f3a8c09bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:00.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.106+0000 7f3a926ff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a8c15bac0 0x7f3a8c14abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:00.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.106+0000 7f3a926ff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a8c15f210 0x7f3a8c14b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:00.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.106+0000 7f3a926ff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3a8c072ba0 con 0x7f3a8c15bac0 2024-03-17T23:48:00.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.106+0000 7f3a926ff640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3a8c072a20 con 0x7f3a8c06ec90 2024-03-17T23:48:00.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.106+0000 7f3a926ff640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3a8c072d20 con 0x7f3a8c15f210 2024-03-17T23:48:00.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.106+0000 7f3a8b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a8c15bac0 0x7f3a8c14abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:00.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.107+0000 7f3a8b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a8c15bac0 0x7f3a8c14abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45780/0 (socket says 172.21.15.150:45780) 2024-03-17T23:48:00.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.107+0000 7f3a90c9c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a8c15f210 0x7f3a8c14b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:00.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.107+0000 7f3a8b7fe640 1 -- 172.21.15.150:0/3494253966 learned_addr learned my addr 172.21.15.150:0/3494253966 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:00.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.107+0000 7f3a8bfff640 1 --2- 172.21.15.150:0/3494253966 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a8c06ec90 0x7f3a8c09bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:00.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.107+0000 7f3a8b7fe640 1 -- 172.21.15.150:0/3494253966 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a8c15f210 msgr2=0x7f3a8c14b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:00.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.107+0000 7f3a8b7fe640 1 --2- 172.21.15.150:0/3494253966 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a8c15f210 0x7f3a8c14b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:00.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.107+0000 7f3a8b7fe640 1 -- 172.21.15.150:0/3494253966 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a8c06ec90 msgr2=0x7f3a8c09bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:00.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.107+0000 7f3a8b7fe640 1 --2- 172.21.15.150:0/3494253966 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a8c06ec90 0x7f3a8c09bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:00.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.107+0000 7f3a8b7fe640 1 -- 172.21.15.150:0/3494253966 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a8c09c790 con 0x7f3a8c15bac0 2024-03-17T23:48:00.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.107+0000 7f3a90c9c640 1 --2- 172.21.15.150:0/3494253966 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a8c15f210 0x7f3a8c14b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:48:00.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.107+0000 7f3a8b7fe640 1 --2- 172.21.15.150:0/3494253966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a8c15bac0 0x7f3a8c14abe0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3a8000c410 tx=0x7f3a8000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:00.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.107+0000 7f3a88ff9640 1 -- 172.21.15.150:0/3494253966 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a80017070 con 0x7f3a8c15bac0 2024-03-17T23:48:00.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.107+0000 7f3a88ff9640 1 -- 172.21.15.150:0/3494253966 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a80005ce0 con 0x7f3a8c15bac0 2024-03-17T23:48:00.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.107+0000 7f3a926ff640 1 -- 172.21.15.150:0/3494253966 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a8c09ca70 con 0x7f3a8c15bac0 2024-03-17T23:48:00.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.107+0000 7f3a88ff9640 1 -- 172.21.15.150:0/3494253966 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a800072f0 con 0x7f3a8c15bac0 2024-03-17T23:48:00.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.107+0000 7f3a926ff640 1 -- 172.21.15.150:0/3494253966 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a8c09a680 con 0x7f3a8c15bac0 2024-03-17T23:48:00.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.108+0000 7f3a926ff640 1 -- 172.21.15.150:0/3494253966 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a8c0681a0 con 0x7f3a8c15bac0 2024-03-17T23:48:00.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.109+0000 7f3a88ff9640 1 -- 172.21.15.150:0/3494253966 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3a8001a070 con 0x7f3a8c15bac0 2024-03-17T23:48:00.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.109+0000 7f3a88ff9640 1 --2- 172.21.15.150:0/3494253966 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3a7806bbd0 0x7f3a7806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:00.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.109+0000 7f3a8bfff640 1 --2- 172.21.15.150:0/3494253966 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3a7806bbd0 0x7f3a7806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:00.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.109+0000 7f3a88ff9640 1 -- 172.21.15.150:0/3494253966 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6821+0+0 (secure 0 0 0) 0x7f3a8008a280 con 0x7f3a8c15bac0 2024-03-17T23:48:00.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.110+0000 7f3a8bfff640 1 --2- 172.21.15.150:0/3494253966 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3a7806bbd0 0x7f3a7806e090 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c036000 tx=0x7f3a7c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:00.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.114+0000 7f3a88ff9640 1 -- 172.21.15.150:0/3494253966 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3a80058380 con 0x7f3a8c15bac0 2024-03-17T23:48:00.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.327+0000 7f3a926ff640 1 -- 172.21.15.150:0/3494253966 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f3a8c06cac0 con 0x7f3a8c15bac0 2024-03-17T23:48:00.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.985+0000 7f3a88ff9640 1 -- 172.21.15.150:0/3494253966 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) ==== 104+0+0 (secure 0 0 0) 0x7f3a8005c420 con 0x7f3a8c15bac0 2024-03-17T23:48:00.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.987+0000 7f3a926ff640 1 -- 172.21.15.150:0/3494253966 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3a7806bbd0 msgr2=0x7f3a7806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:00.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.987+0000 7f3a926ff640 1 --2- 172.21.15.150:0/3494253966 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3a7806bbd0 0x7f3a7806e090 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c036000 tx=0x7f3a7c008030 comp rx=0 tx=0).stop 2024-03-17T23:48:00.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.987+0000 7f3a926ff640 1 -- 172.21.15.150:0/3494253966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a8c15bac0 msgr2=0x7f3a8c14abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:00.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.987+0000 7f3a926ff640 1 --2- 172.21.15.150:0/3494253966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a8c15bac0 0x7f3a8c14abe0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f3a8000c410 tx=0x7f3a8000c8d0 comp rx=0 tx=0).stop 2024-03-17T23:48:00.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.988+0000 7f3a926ff640 1 -- 172.21.15.150:0/3494253966 shutdown_connections 2024-03-17T23:48:00.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.988+0000 7f3a926ff640 1 --2- 172.21.15.150:0/3494253966 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a8c15f210 0x7f3a8c14b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:00.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.988+0000 7f3a926ff640 1 --2- 172.21.15.150:0/3494253966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a8c15bac0 0x7f3a8c14abe0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:00.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.988+0000 7f3a926ff640 1 --2- 172.21.15.150:0/3494253966 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3a7806bbd0 0x7f3a7806e090 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:00.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.988+0000 7f3a926ff640 1 --2- 172.21.15.150:0/3494253966 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a8c06ec90 0x7f3a8c09bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:00.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.988+0000 7f3a926ff640 1 -- 172.21.15.150:0/3494253966 >> 172.21.15.150:0/3494253966 conn(0x7f3a8c0b3da0 msgr2=0x7f3a8c06e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:00.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.988+0000 7f3a926ff640 1 -- 172.21.15.150:0/3494253966 shutdown_connections 2024-03-17T23:48:00.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:00.988+0000 7f3a926ff640 1 -- 172.21.15.150:0/3494253966 wait complete. 2024-03-17T23:48:01.001 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-17T23:48:01.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.109+0000 7fc003e85640 1 Processor -- start 2024-03-17T23:48:01.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.109+0000 7fc003e85640 1 -- start start 2024-03-17T23:48:01.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.110+0000 7fc003e85640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fbffc15e0f0 0x7fbffc0d2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:01.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.110+0000 7fc003e85640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbffc0d37c0 0x7fbffc0b9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:01.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.110+0000 7fc003e85640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbffc0d2720 con 0x7fbffc0d0a70 2024-03-17T23:48:01.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.110+0000 7fc003e85640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fbffc0b9c90 con 0x7fbffc0d37c0 2024-03-17T23:48:01.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.110+0000 7fc003e85640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fbffc0d2a00 con 0x7fbffc15e0f0 2024-03-17T23:48:01.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.110+0000 7fc001420640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbffc0d37c0 0x7fbffc0b9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:01.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.110+0000 7fc002422640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fbffc0d0a70 0x7fbffc0b5760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33576/0 (socket says 172.21.15.150:33576) 2024-03-17T23:48:01.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.110+0000 7fc002422640 1 -- 172.21.15.150:0/4105966401 learned_addr learned my addr 172.21.15.150:0/4105966401 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:01.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.111+0000 7fc001c21640 1 --2- 172.21.15.150:0/4105966401 >> v2:172.21.15.157:3300/0 conn(0x7fbffc15e0f0 0x7fbffc0d2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:01.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.111+0000 7fc001420640 1 -- 172.21.15.150:0/4105966401 >> v2:172.21.15.157:3300/0 conn(0x7fbffc15e0f0 msgr2=0x7fbffc0d2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:01.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.111+0000 7fc001420640 1 --2- 172.21.15.150:0/4105966401 >> v2:172.21.15.157:3300/0 conn(0x7fbffc15e0f0 0x7fbffc0d2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:01.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.111+0000 7fc001420640 1 -- 172.21.15.150:0/4105966401 >> v1:172.21.15.150:6789/0 conn(0x7fbffc0d0a70 legacy=0x7fbffc0b5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:01.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.111+0000 7fc001420640 1 -- 172.21.15.150:0/4105966401 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbffc0d2ce0 con 0x7fbffc0d37c0 2024-03-17T23:48:01.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.111+0000 7fc000c1f640 1 -- 172.21.15.150:0/4105966401 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 265401951 0 0) 0x7fbffc0d2720 con 0x7fbffc0d0a70 2024-03-17T23:48:01.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.111+0000 7fc001c21640 1 --2- 172.21.15.150:0/4105966401 >> v2:172.21.15.157:3300/0 conn(0x7fbffc15e0f0 0x7fbffc0d2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:01.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.111+0000 7fc001420640 1 --2- 172.21.15.150:0/4105966401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbffc0d37c0 0x7fbffc0b9750 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fbffc0d4dc0 tx=0x7fbfec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b94043916f3b799 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:01.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.112+0000 7fc000c1f640 1 -- 172.21.15.150:0/4105966401 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec035020 con 0x7fbffc0d37c0 2024-03-17T23:48:01.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.112+0000 7fc000c1f640 1 -- 172.21.15.150:0/4105966401 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfec031830 con 0x7fbffc0d37c0 2024-03-17T23:48:01.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.112+0000 7fc000c1f640 1 -- 172.21.15.150:0/4105966401 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec031c00 con 0x7fbffc0d37c0 2024-03-17T23:48:01.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.112+0000 7fc003e85640 1 -- 172.21.15.150:0/4105966401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbffc0d37c0 msgr2=0x7fbffc0b9750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:01.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.112+0000 7fc003e85640 1 --2- 172.21.15.150:0/4105966401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbffc0d37c0 0x7fbffc0b9750 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fbffc0d4dc0 tx=0x7fbfec02b8d0 comp rx=0 tx=0).stop 2024-03-17T23:48:01.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.112+0000 7fc003e85640 1 -- 172.21.15.150:0/4105966401 shutdown_connections 2024-03-17T23:48:01.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.112+0000 7fc003e85640 1 --2- 172.21.15.150:0/4105966401 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbffc0d37c0 0x7fbffc0b9750 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:01.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.112+0000 7fc003e85640 1 --2- 172.21.15.150:0/4105966401 >> v2:172.21.15.157:3300/0 conn(0x7fbffc15e0f0 0x7fbffc0d2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:01.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.112+0000 7fc003e85640 1 -- 172.21.15.150:0/4105966401 >> 172.21.15.150:0/4105966401 conn(0x7fbffc06cbd0 msgr2=0x7fbffc06cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:01.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.112+0000 7fc003e85640 1 -- 172.21.15.150:0/4105966401 shutdown_connections 2024-03-17T23:48:01.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.112+0000 7fc003e85640 1 -- 172.21.15.150:0/4105966401 wait complete. 2024-03-17T23:48:01.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.113+0000 7fc003e85640 1 Processor -- start 2024-03-17T23:48:01.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.113+0000 7fc003e85640 1 -- start start 2024-03-17T23:48:01.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.114+0000 7fc003e85640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbffc0d0a70 0x7fbffc15e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:01.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.114+0000 7fc003e85640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbffc0d37c0 0x7fbffc1f6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:01.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.114+0000 7fc003e85640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbffc15e0f0 0x7fbffc1f7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:01.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.114+0000 7fc003e85640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbffc0d0f00 con 0x7fbffc0d0a70 2024-03-17T23:48:01.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.114+0000 7fc003e85640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbffc0d32a0 con 0x7fbffc15e0f0 2024-03-17T23:48:01.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.114+0000 7fc003e85640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbffc0d1080 con 0x7fbffc0d37c0 2024-03-17T23:48:01.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.114+0000 7fc001c21640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbffc0d0a70 0x7fbffc15e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:01.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.114+0000 7fc001c21640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbffc0d0a70 0x7fbffc15e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45794/0 (socket says 172.21.15.150:45794) 2024-03-17T23:48:01.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.114+0000 7fc001c21640 1 -- 172.21.15.150:0/227244819 learned_addr learned my addr 172.21.15.150:0/227244819 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:01.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.114+0000 7fc001420640 1 --2- 172.21.15.150:0/227244819 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbffc0d37c0 0x7fbffc1f6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:01.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.114+0000 7fc002422640 1 --2- 172.21.15.150:0/227244819 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbffc15e0f0 0x7fbffc1f7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:01.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.115+0000 7fc001c21640 1 -- 172.21.15.150:0/227244819 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbffc0d37c0 msgr2=0x7fbffc1f6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:01.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.115+0000 7fc001c21640 1 --2- 172.21.15.150:0/227244819 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbffc0d37c0 0x7fbffc1f6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:01.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.115+0000 7fc001c21640 1 -- 172.21.15.150:0/227244819 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbffc15e0f0 msgr2=0x7fbffc1f7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:01.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.115+0000 7fc001c21640 1 --2- 172.21.15.150:0/227244819 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbffc15e0f0 0x7fbffc1f7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:01.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.115+0000 7fc001c21640 1 -- 172.21.15.150:0/227244819 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbffc160190 con 0x7fbffc0d0a70 2024-03-17T23:48:01.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.115+0000 7fc001c21640 1 --2- 172.21.15.150:0/227244819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbffc0d0a70 0x7fbffc15e4d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fbff8009fa0 tx=0x7fbff800d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:01.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.115+0000 7fbff27fc640 1 -- 172.21.15.150:0/227244819 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff801a070 con 0x7fbffc0d0a70 2024-03-17T23:48:01.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.115+0000 7fbff27fc640 1 -- 172.21.15.150:0/227244819 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbff8003bb0 con 0x7fbffc0d0a70 2024-03-17T23:48:01.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.115+0000 7fc003e85640 1 -- 172.21.15.150:0/227244819 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbffc160470 con 0x7fbffc0d0a70 2024-03-17T23:48:01.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.115+0000 7fbff27fc640 1 -- 172.21.15.150:0/227244819 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff8004030 con 0x7fbffc0d0a70 2024-03-17T23:48:01.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.115+0000 7fc003e85640 1 -- 172.21.15.150:0/227244819 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbffc160a30 con 0x7fbffc0d0a70 2024-03-17T23:48:01.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.116+0000 7fc003e85640 1 -- 172.21.15.150:0/227244819 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbffc0688b0 con 0x7fbffc0d0a70 2024-03-17T23:48:01.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.116+0000 7fbff27fc640 1 -- 172.21.15.150:0/227244819 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbff801d070 con 0x7fbffc0d0a70 2024-03-17T23:48:01.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.117+0000 7fbff27fc640 1 --2- 172.21.15.150:0/227244819 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbfe006bbd0 0x7fbfe006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:01.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.117+0000 7fbff27fc640 1 -- 172.21.15.150:0/227244819 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6821+0+0 (secure 0 0 0) 0x7fbff808b020 con 0x7fbffc0d0a70 2024-03-17T23:48:01.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.117+0000 7fc001420640 1 --2- 172.21.15.150:0/227244819 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbfe006bbd0 0x7fbfe006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:01.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.117+0000 7fc001420640 1 --2- 172.21.15.150:0/227244819 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbfe006bbd0 0x7fbfe006e090 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fbfec036000 tx=0x7fbfec03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:01.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.125+0000 7fbff27fc640 1 -- 172.21.15.150:0/227244819 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbff8010070 con 0x7fbffc0d0a70 2024-03-17T23:48:01.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:01.297+0000 7fc003e85640 1 -- 172.21.15.150:0/227244819 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fbffc160dd0 con 0x7fbffc0d0a70 2024-03-17T23:48:02.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.123+0000 7fbff27fc640 1 -- 172.21.15.150:0/227244819 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v30) ==== 181+0+0 (secure 0 0 0) 0x7fbff8059060 con 0x7fbffc0d0a70 2024-03-17T23:48:02.124 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_metadata' removed 2024-03-17T23:48:02.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.126+0000 7fc003e85640 1 -- 172.21.15.150:0/227244819 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbfe006bbd0 msgr2=0x7fbfe006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:02.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.126+0000 7fc003e85640 1 --2- 172.21.15.150:0/227244819 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbfe006bbd0 0x7fbfe006e090 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fbfec036000 tx=0x7fbfec03d040 comp rx=0 tx=0).stop 2024-03-17T23:48:02.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.126+0000 7fc003e85640 1 -- 172.21.15.150:0/227244819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbffc0d0a70 msgr2=0x7fbffc15e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:02.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.126+0000 7fc003e85640 1 --2- 172.21.15.150:0/227244819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbffc0d0a70 0x7fbffc15e4d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fbff8009fa0 tx=0x7fbff800d560 comp rx=0 tx=0).stop 2024-03-17T23:48:02.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.126+0000 7fc003e85640 1 -- 172.21.15.150:0/227244819 shutdown_connections 2024-03-17T23:48:02.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.126+0000 7fc003e85640 1 --2- 172.21.15.150:0/227244819 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbfe006bbd0 0x7fbfe006e090 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:02.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.126+0000 7fc003e85640 1 --2- 172.21.15.150:0/227244819 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbffc15e0f0 0x7fbffc1f7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:02.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.126+0000 7fc003e85640 1 --2- 172.21.15.150:0/227244819 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbffc0d37c0 0x7fbffc1f6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:02.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.126+0000 7fc003e85640 1 --2- 172.21.15.150:0/227244819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbffc0d0a70 0x7fbffc15e4d0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:02.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.126+0000 7fc003e85640 1 -- 172.21.15.150:0/227244819 >> 172.21.15.150:0/227244819 conn(0x7fbffc06cbd0 msgr2=0x7fbffc0cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:02.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.127+0000 7fc003e85640 1 -- 172.21.15.150:0/227244819 shutdown_connections 2024-03-17T23:48:02.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.127+0000 7fc003e85640 1 -- 172.21.15.150:0/227244819 wait complete. 2024-03-17T23:48:02.142 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-17T23:48:02.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.252+0000 7f8d911a9640 1 Processor -- start 2024-03-17T23:48:02.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.252+0000 7f8d911a9640 1 -- start start 2024-03-17T23:48:02.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.253+0000 7f8d911a9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8d8c072780 0x7f8d8c16d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:02.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.253+0000 7f8d911a9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d8c165f30 con 0x7f8d8c15fcf0 2024-03-17T23:48:02.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.253+0000 7f8d911a9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d8c166210 con 0x7f8d8c1632b0 2024-03-17T23:48:02.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.253+0000 7f8d911a9640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f8d8c163740 con 0x7f8d8c072780 2024-03-17T23:48:02.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.253+0000 7f8d8ad9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f8d8c15fcf0 0x7f8d8c0b4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33582/0 (socket says 172.21.15.150:33582) 2024-03-17T23:48:02.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.253+0000 7f8d8ad9d640 1 -- 172.21.15.150:0/2249524776 learned_addr learned my addr 172.21.15.150:0/2249524776 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:02.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.253+0000 7f8d8b59e640 1 --2- 172.21.15.150:0/2249524776 >> v2:172.21.15.157:3300/0 conn(0x7f8d8c072780 0x7f8d8c16d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:02.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.253+0000 7f8d8a59c640 1 -- 172.21.15.150:0/2249524776 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4141747756 0 0) 0x7f8d8c165f30 con 0x7f8d8c15fcf0 2024-03-17T23:48:02.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.253+0000 7f8d8a59c640 1 -- 172.21.15.150:0/2249524776 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d78003520 con 0x7f8d8c15fcf0 2024-03-17T23:48:02.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.253+0000 7f8d8a59c640 1 -- 172.21.15.150:0/2249524776 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1949718401 0 0) 0x7f8d78003520 con 0x7f8d8c15fcf0 2024-03-17T23:48:02.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.253+0000 7f8d8a59c640 1 -- 172.21.15.150:0/2249524776 >> v2:172.21.15.157:3300/0 conn(0x7f8d8c072780 msgr2=0x7f8d8c16d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:02.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.253+0000 7f8d8a59c640 1 --2- 172.21.15.150:0/2249524776 >> v2:172.21.15.157:3300/0 conn(0x7f8d8c072780 0x7f8d8c16d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:02.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.253+0000 7f8d8a59c640 1 -- 172.21.15.150:0/2249524776 >> v1:172.21.15.157:6789/0 conn(0x7f8d8c1632b0 legacy=0x7f8d8c072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:02.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.253+0000 7f8d8a59c640 1 -- 172.21.15.150:0/2249524776 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d8c1664f0 con 0x7f8d8c15fcf0 2024-03-17T23:48:02.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.253+0000 7f8d8a59c640 1 -- 172.21.15.150:0/2249524776 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f8d7c001670 con 0x7f8d8c15fcf0 2024-03-17T23:48:02.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.253+0000 7f8d8a59c640 1 -- 172.21.15.150:0/2249524776 >> v1:172.21.15.150:6789/0 conn(0x7f8d8c15fcf0 legacy=0x7f8d8c0b4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:02.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.253+0000 7f8d8a59c640 1 --2- 172.21.15.150:0/2249524776 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d780039b0 0x7f8d78003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:02.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.253+0000 7f8d8a59c640 1 --2- 172.21.15.150:0/2249524776 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d78004800 0x7f8d78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:02.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.253+0000 7f8d8a59c640 1 --2- 172.21.15.150:0/2249524776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d780051c0 0x7f8d7800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:02.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.254+0000 7f8d8a59c640 1 -- 172.21.15.150:0/2249524776 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8d7800c030 con 0x7f8d780051c0 2024-03-17T23:48:02.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.254+0000 7f8d8a59c640 1 -- 172.21.15.150:0/2249524776 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8d7800c1e0 con 0x7f8d780039b0 2024-03-17T23:48:02.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.254+0000 7f8d8a59c640 1 -- 172.21.15.150:0/2249524776 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8d7800c3e0 con 0x7f8d78004800 2024-03-17T23:48:02.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.254+0000 7f8d8b59e640 1 --2- 172.21.15.150:0/2249524776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d780051c0 0x7f8d7800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:02.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.254+0000 7f8d8ad9d640 1 --2- 172.21.15.150:0/2249524776 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d780039b0 0x7f8d78003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:02.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.254+0000 7f8d83fff640 1 --2- 172.21.15.150:0/2249524776 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d78004800 0x7f8d78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:02.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.254+0000 7f8d8b59e640 1 -- 172.21.15.150:0/2249524776 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d78004800 msgr2=0x7f8d78004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:02.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.254+0000 7f8d8b59e640 1 --2- 172.21.15.150:0/2249524776 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d78004800 0x7f8d78004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:02.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.254+0000 7f8d8b59e640 1 -- 172.21.15.150:0/2249524776 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d780039b0 msgr2=0x7f8d78003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:02.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.254+0000 7f8d8b59e640 1 --2- 172.21.15.150:0/2249524776 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d780039b0 0x7f8d78003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:02.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.254+0000 7f8d8b59e640 1 -- 172.21.15.150:0/2249524776 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d78003520 con 0x7f8d780051c0 2024-03-17T23:48:02.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.255+0000 7f8d8b59e640 1 -- 172.21.15.150:0/2249524776 reap_dead start 2024-03-17T23:48:02.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.255+0000 7f8d8b59e640 1 --2- 172.21.15.150:0/2249524776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d780051c0 0x7f8d7800b9c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8d84009fc0 tx=0x7f8d78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e44e1f2df8b06b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:02.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.255+0000 7f8d8a59c640 1 -- 172.21.15.150:0/2249524776 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d84009c10 con 0x7f8d780051c0 2024-03-17T23:48:02.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.255+0000 7f8d8a59c640 1 -- 172.21.15.150:0/2249524776 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d8402cca0 con 0x7f8d780051c0 2024-03-17T23:48:02.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.255+0000 7f8d8a59c640 1 -- 172.21.15.150:0/2249524776 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d84003bd0 con 0x7f8d780051c0 2024-03-17T23:48:02.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.255+0000 7f8d911a9640 1 -- 172.21.15.150:0/2249524776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d780051c0 msgr2=0x7f8d7800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:02.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.255+0000 7f8d911a9640 1 --2- 172.21.15.150:0/2249524776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d780051c0 0x7f8d7800b9c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8d84009fc0 tx=0x7f8d78005830 comp rx=0 tx=0).stop 2024-03-17T23:48:02.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.255+0000 7f8d911a9640 1 -- 172.21.15.150:0/2249524776 shutdown_connections 2024-03-17T23:48:02.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.255+0000 7f8d911a9640 1 --2- 172.21.15.150:0/2249524776 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d780051c0 0x7f8d7800b9c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:02.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.255+0000 7f8d911a9640 1 -- 172.21.15.150:0/2249524776 >> 172.21.15.150:0/2249524776 conn(0x7f8d8c0b1be0 msgr2=0x7f8d8c0b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:02.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.255+0000 7f8d911a9640 1 -- 172.21.15.150:0/2249524776 shutdown_connections 2024-03-17T23:48:02.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.255+0000 7f8d911a9640 1 -- 172.21.15.150:0/2249524776 wait complete. 2024-03-17T23:48:02.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.257+0000 7f8d911a9640 1 Processor -- start 2024-03-17T23:48:02.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.257+0000 7f8d911a9640 1 -- start start 2024-03-17T23:48:02.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.257+0000 7f8d911a9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d780051c0 0x7f8d8c1f7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:02.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.257+0000 7f8d911a9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d8c1fae50 0x7f8d8c1646e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:02.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.257+0000 7f8d911a9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d8c164c20 0x7f8d8c0b0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:02.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.257+0000 7f8d911a9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c072070 con 0x7f8d8c164c20 2024-03-17T23:48:02.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.257+0000 7f8d911a9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8c1600d0 con 0x7f8d8c1fae50 2024-03-17T23:48:02.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.257+0000 7f8d911a9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8d8c0721f0 con 0x7f8d780051c0 2024-03-17T23:48:02.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.258+0000 7f8d8b59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d8c164c20 0x7f8d8c0b0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:02.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.258+0000 7f8d8b59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d8c164c20 0x7f8d8c0b0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45814/0 (socket says 172.21.15.150:45814) 2024-03-17T23:48:02.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.258+0000 7f8d8b59e640 1 -- 172.21.15.150:0/3958953547 learned_addr learned my addr 172.21.15.150:0/3958953547 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:02.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.258+0000 7f8d8ad9d640 1 --2- 172.21.15.150:0/3958953547 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d780051c0 0x7f8d8c1f7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:02.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.258+0000 7f8d83fff640 1 --2- 172.21.15.150:0/3958953547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d8c1fae50 0x7f8d8c1646e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:02.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.258+0000 7f8d8b59e640 1 -- 172.21.15.150:0/3958953547 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d780051c0 msgr2=0x7f8d8c1f7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:02.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.258+0000 7f8d8b59e640 1 --2- 172.21.15.150:0/3958953547 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d780051c0 0x7f8d8c1f7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:02.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.258+0000 7f8d8b59e640 1 -- 172.21.15.150:0/3958953547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d8c1fae50 msgr2=0x7f8d8c1646e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:02.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.258+0000 7f8d8b59e640 1 --2- 172.21.15.150:0/3958953547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d8c1fae50 0x7f8d8c1646e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:02.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.258+0000 7f8d8b59e640 1 -- 172.21.15.150:0/3958953547 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d8c1650c0 con 0x7f8d8c164c20 2024-03-17T23:48:02.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.258+0000 7f8d83fff640 1 --2- 172.21.15.150:0/3958953547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d8c1fae50 0x7f8d8c1646e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:02.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.258+0000 7f8d8b59e640 1 --2- 172.21.15.150:0/3958953547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d8c164c20 0x7f8d8c0b0b30 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f8d8402c6d0 tx=0x7f8d84009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:02.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.258+0000 7f8d837fe640 1 -- 172.21.15.150:0/3958953547 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d84042020 con 0x7f8d8c164c20 2024-03-17T23:48:02.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.258+0000 7f8d837fe640 1 -- 172.21.15.150:0/3958953547 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d8402ce50 con 0x7f8d8c164c20 2024-03-17T23:48:02.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.258+0000 7f8d911a9640 1 -- 172.21.15.150:0/3958953547 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d8c1653a0 con 0x7f8d8c164c20 2024-03-17T23:48:02.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.258+0000 7f8d837fe640 1 -- 172.21.15.150:0/3958953547 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d84009c10 con 0x7f8d8c164c20 2024-03-17T23:48:02.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.258+0000 7f8d911a9640 1 -- 172.21.15.150:0/3958953547 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d8c1610a0 con 0x7f8d8c164c20 2024-03-17T23:48:02.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.259+0000 7f8d911a9640 1 -- 172.21.15.150:0/3958953547 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d8c0681a0 con 0x7f8d8c164c20 2024-03-17T23:48:02.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.260+0000 7f8d837fe640 1 -- 172.21.15.150:0/3958953547 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8d84045020 con 0x7f8d8c164c20 2024-03-17T23:48:02.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.260+0000 7f8d837fe640 1 --2- 172.21.15.150:0/3958953547 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8d6c06bbd0 0x7f8d6c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:02.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.260+0000 7f8d837fe640 1 -- 172.21.15.150:0/3958953547 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6383+0+0 (secure 0 0 0) 0x7f8d840ad580 con 0x7f8d8c164c20 2024-03-17T23:48:02.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.260+0000 7f8d8ad9d640 1 --2- 172.21.15.150:0/3958953547 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8d6c06bbd0 0x7f8d6c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:02.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.261+0000 7f8d8ad9d640 1 --2- 172.21.15.150:0/3958953547 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8d6c06bbd0 0x7f8d6c06e090 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c001050 tx=0x7f8d7c0078f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:02.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.265+0000 7f8d837fe640 1 -- 172.21.15.150:0/3958953547 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8d8404c020 con 0x7f8d8c164c20 2024-03-17T23:48:02.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:02.437+0000 7f8d911a9640 1 -- 172.21.15.150:0/3958953547 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8d8c06f2b0 con 0x7f8d8c164c20 2024-03-17T23:48:03.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.128+0000 7f8d837fe640 1 -- 172.21.15.150:0/3958953547 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v31) ==== 169+0+0 (secure 0 0 0) 0x7f8d8407bf80 con 0x7f8d8c164c20 2024-03-17T23:48:03.130 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_data' removed 2024-03-17T23:48:03.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.130+0000 7f8d911a9640 1 -- 172.21.15.150:0/3958953547 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8d6c06bbd0 msgr2=0x7f8d6c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:03.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.130+0000 7f8d911a9640 1 --2- 172.21.15.150:0/3958953547 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8d6c06bbd0 0x7f8d6c06e090 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c001050 tx=0x7f8d7c0078f0 comp rx=0 tx=0).stop 2024-03-17T23:48:03.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.130+0000 7f8d911a9640 1 -- 172.21.15.150:0/3958953547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d8c164c20 msgr2=0x7f8d8c0b0b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:03.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.130+0000 7f8d911a9640 1 --2- 172.21.15.150:0/3958953547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d8c164c20 0x7f8d8c0b0b30 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f8d8402c6d0 tx=0x7f8d84009750 comp rx=0 tx=0).stop 2024-03-17T23:48:03.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.131+0000 7f8d911a9640 1 -- 172.21.15.150:0/3958953547 shutdown_connections 2024-03-17T23:48:03.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.131+0000 7f8d911a9640 1 --2- 172.21.15.150:0/3958953547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d8c164c20 0x7f8d8c0b0b30 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:03.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.131+0000 7f8d911a9640 1 --2- 172.21.15.150:0/3958953547 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8d6c06bbd0 0x7f8d6c06e090 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:03.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.131+0000 7f8d911a9640 1 --2- 172.21.15.150:0/3958953547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d8c1fae50 0x7f8d8c1646e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:03.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.131+0000 7f8d911a9640 1 --2- 172.21.15.150:0/3958953547 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d780051c0 0x7f8d8c1f7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:03.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.131+0000 7f8d911a9640 1 -- 172.21.15.150:0/3958953547 >> 172.21.15.150:0/3958953547 conn(0x7f8d8c0b1be0 msgr2=0x7f8d8c0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:03.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.131+0000 7f8d911a9640 1 -- 172.21.15.150:0/3958953547 shutdown_connections 2024-03-17T23:48:03.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.131+0000 7f8d911a9640 1 -- 172.21.15.150:0/3958953547 wait complete. 2024-03-17T23:48:03.145 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-17T23:48:03.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.250+0000 7fab7c800640 1 Processor -- start 2024-03-17T23:48:03.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.250+0000 7fab7c800640 1 -- start start 2024-03-17T23:48:03.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.250+0000 7fab7c800640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fab7815dd70 0x7fab780708c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:03.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.251+0000 7fab7c800640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fab7806b190 0x7fab7806b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:03.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.251+0000 7fab7c800640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab780cfa90 con 0x7fab780cae50 2024-03-17T23:48:03.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.251+0000 7fab7c800640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fab7815e150 con 0x7fab7806b190 2024-03-17T23:48:03.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.251+0000 7fab7c800640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fab780d0380 con 0x7fab7815dd70 2024-03-17T23:48:03.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.251+0000 7fab7759e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fab7806b190 0x7fab7806b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:03.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.251+0000 7fab7759e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fab7806b190 0x7fab7806b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45828/0 (socket says 172.21.15.150:45828) 2024-03-17T23:48:03.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.251+0000 7fab7759e640 1 -- 172.21.15.150:0/3310943436 learned_addr learned my addr 172.21.15.150:0/3310943436 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:03.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.251+0000 7fab76d9d640 1 --2- 172.21.15.150:0/3310943436 >> v2:172.21.15.157:3300/0 conn(0x7fab7815dd70 0x7fab780708c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:03.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.251+0000 7fab7759e640 1 -- 172.21.15.150:0/3310943436 >> v2:172.21.15.157:3300/0 conn(0x7fab7815dd70 msgr2=0x7fab780708c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:03.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.251+0000 7fab7759e640 1 --2- 172.21.15.150:0/3310943436 >> v2:172.21.15.157:3300/0 conn(0x7fab7815dd70 0x7fab780708c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:03.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.251+0000 7fab7759e640 1 -- 172.21.15.150:0/3310943436 >> v1:172.21.15.157:6789/0 conn(0x7fab780cae50 legacy=0x7fab7806f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:03.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.251+0000 7fab7759e640 1 -- 172.21.15.150:0/3310943436 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab7806fda0 con 0x7fab7806b190 2024-03-17T23:48:03.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.252+0000 7fab7759e640 1 --2- 172.21.15.150:0/3310943436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab7806b190 0x7fab7806b620 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fab780d02f0 tx=0x7fab7002bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=295a1c041062d2df server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:03.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.252+0000 7fab7659c640 1 -- 172.21.15.150:0/3310943436 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab7002c930 con 0x7fab7806b190 2024-03-17T23:48:03.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.252+0000 7fab7659c640 1 -- 172.21.15.150:0/3310943436 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab7002cad0 con 0x7fab7806b190 2024-03-17T23:48:03.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.252+0000 7fab7659c640 1 -- 172.21.15.150:0/3310943436 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab700343f0 con 0x7fab7806b190 2024-03-17T23:48:03.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.252+0000 7fab7c800640 1 -- 172.21.15.150:0/3310943436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab7806b190 msgr2=0x7fab7806b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:03.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.252+0000 7fab7c800640 1 --2- 172.21.15.150:0/3310943436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab7806b190 0x7fab7806b620 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fab780d02f0 tx=0x7fab7002bd90 comp rx=0 tx=0).stop 2024-03-17T23:48:03.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.252+0000 7fab7c800640 1 -- 172.21.15.150:0/3310943436 shutdown_connections 2024-03-17T23:48:03.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.252+0000 7fab7c800640 1 --2- 172.21.15.150:0/3310943436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab7806b190 0x7fab7806b620 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:03.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.252+0000 7fab7c800640 1 --2- 172.21.15.150:0/3310943436 >> v2:172.21.15.157:3300/0 conn(0x7fab7815dd70 0x7fab780708c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:03.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.252+0000 7fab7c800640 1 -- 172.21.15.150:0/3310943436 >> 172.21.15.150:0/3310943436 conn(0x7fab78099a70 msgr2=0x7fab78099e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:03.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.252+0000 7fab7c800640 1 -- 172.21.15.150:0/3310943436 shutdown_connections 2024-03-17T23:48:03.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.253+0000 7fab7c800640 1 -- 172.21.15.150:0/3310943436 wait complete. 2024-03-17T23:48:03.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.254+0000 7fab7c800640 1 Processor -- start 2024-03-17T23:48:03.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.254+0000 7fab7c800640 1 -- start start 2024-03-17T23:48:03.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.254+0000 7fab7c800640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fab7806b190 0x7fab78156890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:03.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.255+0000 7fab7c800640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fab780cae50 0x7fab780b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:03.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.255+0000 7fab7c800640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab7815dd70 0x7fab7815c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:03.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.255+0000 7fab7c800640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fab780d0810 con 0x7fab7815dd70 2024-03-17T23:48:03.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.255+0000 7fab7c800640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fab780d0010 con 0x7fab780cae50 2024-03-17T23:48:03.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.255+0000 7fab7c800640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fab780d0990 con 0x7fab7806b190 2024-03-17T23:48:03.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.255+0000 7fab7759e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab7815dd70 0x7fab7815c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:03.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.255+0000 7fab7759e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab7815dd70 0x7fab7815c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45842/0 (socket says 172.21.15.150:45842) 2024-03-17T23:48:03.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.255+0000 7fab7759e640 1 -- 172.21.15.150:0/3509346371 learned_addr learned my addr 172.21.15.150:0/3509346371 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:03.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.255+0000 7fab6ffff640 1 --2- 172.21.15.150:0/3509346371 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fab780cae50 0x7fab780b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:03.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.255+0000 7fab76d9d640 1 --2- 172.21.15.150:0/3509346371 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fab7806b190 0x7fab78156890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:03.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.255+0000 7fab7759e640 1 -- 172.21.15.150:0/3509346371 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fab7806b190 msgr2=0x7fab78156890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:03.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.255+0000 7fab7759e640 1 --2- 172.21.15.150:0/3509346371 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fab7806b190 0x7fab78156890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:03.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.255+0000 7fab7759e640 1 -- 172.21.15.150:0/3509346371 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fab780cae50 msgr2=0x7fab780b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:03.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.255+0000 7fab7759e640 1 --2- 172.21.15.150:0/3509346371 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fab780cae50 0x7fab780b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:03.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.255+0000 7fab7759e640 1 -- 172.21.15.150:0/3509346371 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab78156dd0 con 0x7fab7815dd70 2024-03-17T23:48:03.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.255+0000 7fab6ffff640 1 --2- 172.21.15.150:0/3509346371 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fab780cae50 0x7fab780b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:03.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.255+0000 7fab7759e640 1 --2- 172.21.15.150:0/3509346371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab7815dd70 0x7fab7815c350 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fab70036040 tx=0x7fab70004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:03.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.255+0000 7fab6f7fe640 1 -- 172.21.15.150:0/3509346371 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab70043020 con 0x7fab7815dd70 2024-03-17T23:48:03.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.255+0000 7fab6f7fe640 1 -- 172.21.15.150:0/3509346371 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab7003e450 con 0x7fab7815dd70 2024-03-17T23:48:03.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.255+0000 7fab6f7fe640 1 -- 172.21.15.150:0/3509346371 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab7003e820 con 0x7fab7815dd70 2024-03-17T23:48:03.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.255+0000 7fab7c800640 1 -- 172.21.15.150:0/3509346371 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab781e7650 con 0x7fab7815dd70 2024-03-17T23:48:03.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.255+0000 7fab7c800640 1 -- 172.21.15.150:0/3509346371 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab781e7c10 con 0x7fab7815dd70 2024-03-17T23:48:03.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.256+0000 7fab7c800640 1 -- 172.21.15.150:0/3509346371 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab78074c40 con 0x7fab7815dd70 2024-03-17T23:48:03.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.257+0000 7fab6f7fe640 1 -- 172.21.15.150:0/3509346371 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fab70046020 con 0x7fab7815dd70 2024-03-17T23:48:03.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.257+0000 7fab6f7fe640 1 --2- 172.21.15.150:0/3509346371 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fab5806bbd0 0x7fab5806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:03.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.257+0000 7fab6f7fe640 1 -- 172.21.15.150:0/3509346371 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5853+0+0 (secure 0 0 0) 0x7fab700ad3e0 con 0x7fab7815dd70 2024-03-17T23:48:03.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.257+0000 7fab76d9d640 1 --2- 172.21.15.150:0/3509346371 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fab5806bbd0 0x7fab5806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:03.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.258+0000 7fab76d9d640 1 --2- 172.21.15.150:0/3509346371 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fab5806bbd0 0x7fab5806e090 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fab68006fd0 tx=0x7fab68008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:03.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.262+0000 7fab6f7fe640 1 -- 172.21.15.150:0/3509346371 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fab7004d030 con 0x7fab7815dd70 2024-03-17T23:48:03.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:03.436+0000 7fab7c800640 1 -- 172.21.15.150:0/3509346371 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7fab780ccf60 con 0x7fab7815dd70 2024-03-17T23:48:04.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.133+0000 7fab6f7fe640 1 -- 172.21.15.150:0/3509346371 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v32) ==== 178+0+0 (secure 0 0 0) 0x7fab78156dd0 con 0x7fab7815dd70 2024-03-17T23:48:04.134 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_data_ec' removed 2024-03-17T23:48:04.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.136+0000 7fab7c800640 1 -- 172.21.15.150:0/3509346371 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fab5806bbd0 msgr2=0x7fab5806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:04.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.136+0000 7fab7c800640 1 --2- 172.21.15.150:0/3509346371 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fab5806bbd0 0x7fab5806e090 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fab68006fd0 tx=0x7fab68008040 comp rx=0 tx=0).stop 2024-03-17T23:48:04.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.136+0000 7fab7c800640 1 -- 172.21.15.150:0/3509346371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab7815dd70 msgr2=0x7fab7815c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:04.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.136+0000 7fab7c800640 1 --2- 172.21.15.150:0/3509346371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab7815dd70 0x7fab7815c350 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fab70036040 tx=0x7fab70004a40 comp rx=0 tx=0).stop 2024-03-17T23:48:04.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.136+0000 7fab7c800640 1 -- 172.21.15.150:0/3509346371 shutdown_connections 2024-03-17T23:48:04.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.137+0000 7fab7c800640 1 --2- 172.21.15.150:0/3509346371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab7815dd70 0x7fab7815c350 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:04.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.137+0000 7fab7c800640 1 --2- 172.21.15.150:0/3509346371 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fab5806bbd0 0x7fab5806e090 secure :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fab68006fd0 tx=0x7fab68008040 comp rx=0 tx=0).stop 2024-03-17T23:48:04.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.137+0000 7fab7c800640 1 --2- 172.21.15.150:0/3509346371 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fab780cae50 0x7fab780b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:04.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.137+0000 7fab7c800640 1 --2- 172.21.15.150:0/3509346371 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fab7806b190 0x7fab78156890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:04.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.137+0000 7fab7c800640 1 -- 172.21.15.150:0/3509346371 >> 172.21.15.150:0/3509346371 conn(0x7fab78099a70 msgr2=0x7fab780d0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:04.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.137+0000 7fab7c800640 1 -- 172.21.15.150:0/3509346371 shutdown_connections 2024-03-17T23:48:04.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.137+0000 7fab7c800640 1 -- 172.21.15.150:0/3509346371 wait complete. 2024-03-17T23:48:04.151 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-17T23:48:04.151 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-17T23:48:04.151 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-17T23:48:04.152 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:48:04.152 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-17T23:48:04.152 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-17T23:48:04.152 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-17T23:48:04.152 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:48:04.152 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-17T23:48:04.152 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-17T23:48:04.153 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-17T23:48:04.153 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:48:04.153 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-17T23:48:04.153 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-17T23:48:04.153 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-17T23:48:04.153 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:48:04.161 INFO:tasks.ceph.mds.a:Stopped 2024-03-17T23:48:04.161 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-17T23:48:04.162 INFO:tasks.ceph.mds.c:Stopped 2024-03-17T23:48:04.162 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-17T23:48:04.164 INFO:tasks.ceph.mds.b:Stopped 2024-03-17T23:48:04.164 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-17T23:48:04.165 INFO:tasks.ceph.mds.d:Stopped 2024-03-17T23:48:04.165 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-17T23:48:04.204 INFO:tasks.ceph.mds.a:Started 2024-03-17T23:48:04.204 DEBUG:teuthology.parallel:result is None 2024-03-17T23:48:04.204 INFO:tasks.ceph.mds.c:Started 2024-03-17T23:48:04.205 DEBUG:teuthology.parallel:result is None 2024-03-17T23:48:04.207 INFO:tasks.ceph.mds.b:Started 2024-03-17T23:48:04.207 DEBUG:teuthology.parallel:result is None 2024-03-17T23:48:04.208 INFO:tasks.ceph.mds.d:Started 2024-03-17T23:48:04.208 DEBUG:teuthology.parallel:result is None 2024-03-17T23:48:04.208 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-17T23:48:04.280 INFO:tasks.ceph.mds.a.smithi150.stdout:starting mds.a at 2024-03-17T23:48:04.281 INFO:tasks.ceph.mds.c.smithi150.stdout:starting mds.c at 2024-03-17T23:48:04.291 INFO:tasks.ceph.mds.d.smithi157.stdout:starting mds.d at 2024-03-17T23:48:04.291 INFO:tasks.ceph.mds.b.smithi157.stdout:starting mds.b at 2024-03-17T23:48:04.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.313+0000 7fd60fade640 1 Processor -- start 2024-03-17T23:48:04.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.314+0000 7fd60fade640 1 -- start start 2024-03-17T23:48:04.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.314+0000 7fd60fade640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd60815e2a0 0x7fd60809a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:04.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.314+0000 7fd60fade640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd60806ccb0 0x7fd60806d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:04.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.314+0000 7fd60fade640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6080d03b0 con 0x7fd6080cfdf0 2024-03-17T23:48:04.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.314+0000 7fd60fade640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fd6080d0690 con 0x7fd60815e2a0 2024-03-17T23:48:04.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.314+0000 7fd60fade640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd6080d0970 con 0x7fd60806ccb0 2024-03-17T23:48:04.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.314+0000 7fd60d87a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd60815e2a0 0x7fd60809a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:04.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.314+0000 7fd60d87a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd60815e2a0 0x7fd60809a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45872/0 (socket says 172.21.15.150:45872) 2024-03-17T23:48:04.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.314+0000 7fd60d87a640 1 -- 172.21.15.150:0/2194005862 learned_addr learned my addr 172.21.15.150:0/2194005862 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:04.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.314+0000 7fd60e07b640 1 --2- 172.21.15.150:0/2194005862 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd60806ccb0 0x7fd60806d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:04.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.315+0000 7fd60d87a640 1 -- 172.21.15.150:0/2194005862 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd60806ccb0 msgr2=0x7fd60806d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:04.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.315+0000 7fd60d87a640 1 --2- 172.21.15.150:0/2194005862 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd60806ccb0 0x7fd60806d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:04.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.315+0000 7fd60d87a640 1 -- 172.21.15.150:0/2194005862 >> v1:172.21.15.157:6789/0 conn(0x7fd6080cfdf0 legacy=0x7fd6080d1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:04.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.315+0000 7fd60d87a640 1 -- 172.21.15.150:0/2194005862 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6080d0c50 con 0x7fd60815e2a0 2024-03-17T23:48:04.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.315+0000 7fd60d87a640 1 --2- 172.21.15.150:0/2194005862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd60815e2a0 0x7fd60809a790 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd604009e50 tx=0x7fd60402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=472b38f8b0a6f0ee server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:04.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.316+0000 7fd60c878640 1 -- 172.21.15.150:0/2194005862 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60402ca70 con 0x7fd60815e2a0 2024-03-17T23:48:04.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.316+0000 7fd60c878640 1 -- 172.21.15.150:0/2194005862 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd60402cc10 con 0x7fd60815e2a0 2024-03-17T23:48:04.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.316+0000 7fd60c878640 1 -- 172.21.15.150:0/2194005862 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6040343f0 con 0x7fd60815e2a0 2024-03-17T23:48:04.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.316+0000 7fd60fade640 1 -- 172.21.15.150:0/2194005862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd60815e2a0 msgr2=0x7fd60809a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:04.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.316+0000 7fd60fade640 1 --2- 172.21.15.150:0/2194005862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd60815e2a0 0x7fd60809a790 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd604009e50 tx=0x7fd60402bed0 comp rx=0 tx=0).stop 2024-03-17T23:48:04.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.316+0000 7fd60fade640 1 -- 172.21.15.150:0/2194005862 shutdown_connections 2024-03-17T23:48:04.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.316+0000 7fd60fade640 1 --2- 172.21.15.150:0/2194005862 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd60806ccb0 0x7fd60806d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:04.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.316+0000 7fd60fade640 1 --2- 172.21.15.150:0/2194005862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd60815e2a0 0x7fd60809a790 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:04.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.316+0000 7fd60fade640 1 -- 172.21.15.150:0/2194005862 >> 172.21.15.150:0/2194005862 conn(0x7fd6080b36a0 msgr2=0x7fd6080b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:04.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.316+0000 7fd60fade640 1 -- 172.21.15.150:0/2194005862 shutdown_connections 2024-03-17T23:48:04.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.316+0000 7fd60fade640 1 -- 172.21.15.150:0/2194005862 wait complete. 2024-03-17T23:48:04.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.317+0000 7fd60fade640 1 Processor -- start 2024-03-17T23:48:04.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.317+0000 7fd60fade640 1 -- start start 2024-03-17T23:48:04.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.318+0000 7fd60fade640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd60806ccb0 0x7fd6081f7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:04.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.318+0000 7fd60fade640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6080cfdf0 0x7fd6080ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:04.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.318+0000 7fd60fade640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd60815e2a0 0x7fd6080b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:04.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.318+0000 7fd60fade640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd6080cc200 con 0x7fd6080cfdf0 2024-03-17T23:48:04.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.318+0000 7fd60fade640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd60806c740 con 0x7fd60806ccb0 2024-03-17T23:48:04.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.318+0000 7fd60fade640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd6080cc380 con 0x7fd60815e2a0 2024-03-17T23:48:04.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.318+0000 7fd60d079640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6080cfdf0 0x7fd6080ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:04.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.318+0000 7fd60d079640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6080cfdf0 0x7fd6080ca6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45880/0 (socket says 172.21.15.150:45880) 2024-03-17T23:48:04.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.318+0000 7fd60d079640 1 -- 172.21.15.150:0/4219943573 learned_addr learned my addr 172.21.15.150:0/4219943573 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:04.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.318+0000 7fd60e07b640 1 --2- 172.21.15.150:0/4219943573 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd60815e2a0 0x7fd6080b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:04.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.318+0000 7fd60d87a640 1 --2- 172.21.15.150:0/4219943573 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd60806ccb0 0x7fd6081f7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:04.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.319+0000 7fd60d079640 1 -- 172.21.15.150:0/4219943573 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd60815e2a0 msgr2=0x7fd6080b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:04.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.319+0000 7fd60d079640 1 --2- 172.21.15.150:0/4219943573 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd60815e2a0 0x7fd6080b0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:04.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.319+0000 7fd60d079640 1 -- 172.21.15.150:0/4219943573 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd60806ccb0 msgr2=0x7fd6081f7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:04.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.319+0000 7fd60d079640 1 --2- 172.21.15.150:0/4219943573 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd60806ccb0 0x7fd6081f7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:04.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.319+0000 7fd60d079640 1 -- 172.21.15.150:0/4219943573 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6080cb610 con 0x7fd6080cfdf0 2024-03-17T23:48:04.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.319+0000 7fd60e07b640 1 --2- 172.21.15.150:0/4219943573 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd60815e2a0 0x7fd6080b0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:04.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.319+0000 7fd60d87a640 1 --2- 172.21.15.150:0/4219943573 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd60806ccb0 0x7fd6081f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:04.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.319+0000 7fd60d079640 1 --2- 172.21.15.150:0/4219943573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6080cfdf0 0x7fd6080ca6e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd5f800c430 tx=0x7fd5f800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:04.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.319+0000 7fd5fe7fc640 1 -- 172.21.15.150:0/4219943573 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f80052d0 con 0x7fd6080cfdf0 2024-03-17T23:48:04.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.319+0000 7fd5fe7fc640 1 -- 172.21.15.150:0/4219943573 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5f8005490 con 0x7fd6080cfdf0 2024-03-17T23:48:04.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.319+0000 7fd60fade640 1 -- 172.21.15.150:0/4219943573 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6080cb050 con 0x7fd6080cfdf0 2024-03-17T23:48:04.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.319+0000 7fd5fe7fc640 1 -- 172.21.15.150:0/4219943573 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5f8007260 con 0x7fd6080cfdf0 2024-03-17T23:48:04.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.319+0000 7fd60fade640 1 -- 172.21.15.150:0/4219943573 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6080b0750 con 0x7fd6080cfdf0 2024-03-17T23:48:04.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.320+0000 7fd60fade640 1 -- 172.21.15.150:0/4219943573 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd60814f8b0 con 0x7fd6080cfdf0 2024-03-17T23:48:04.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.320+0000 7fd5fe7fc640 1 -- 172.21.15.150:0/4219943573 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd5f8007400 con 0x7fd6080cfdf0 2024-03-17T23:48:04.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.321+0000 7fd5fe7fc640 1 --2- 172.21.15.150:0/4219943573 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd5ec06bbd0 0x7fd5ec06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:04.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.321+0000 7fd5fe7fc640 1 -- 172.21.15.150:0/4219943573 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5335+0+0 (secure 0 0 0) 0x7fd5f8089e80 con 0x7fd6080cfdf0 2024-03-17T23:48:04.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.321+0000 7fd60d87a640 1 --2- 172.21.15.150:0/4219943573 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd5ec06bbd0 0x7fd5ec06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:04.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.321+0000 7fd60d87a640 1 --2- 172.21.15.150:0/4219943573 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd5ec06bbd0 0x7fd5ec06e090 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd604035000 tx=0x7fd604030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:04.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.325+0000 7fd5fe7fc640 1 -- 172.21.15.150:0/4219943573 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd60814f8b0 con 0x7fd6080cfdf0 2024-03-17T23:48:04.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:04.519+0000 7fd60fade640 1 -- 172.21.15.150:0/4219943573 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fd6080688b0 con 0x7fd6080cfdf0 2024-03-17T23:48:05.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.142+0000 7fd5fe7fc640 1 -- 172.21.15.150:0/4219943573 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v34) ==== 97+0+0 (secure 0 0 0) 0x7fd5f8058ac0 con 0x7fd6080cfdf0 2024-03-17T23:48:05.143 INFO:teuthology.orchestra.run.smithi150.stderr: removed all blocklist entries 2024-03-17T23:48:05.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.145+0000 7fd60fade640 1 -- 172.21.15.150:0/4219943573 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd5ec06bbd0 msgr2=0x7fd5ec06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:05.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.145+0000 7fd60fade640 1 --2- 172.21.15.150:0/4219943573 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd5ec06bbd0 0x7fd5ec06e090 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd604035000 tx=0x7fd604030000 comp rx=0 tx=0).stop 2024-03-17T23:48:05.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.145+0000 7fd60fade640 1 -- 172.21.15.150:0/4219943573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6080cfdf0 msgr2=0x7fd6080ca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:05.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.145+0000 7fd60fade640 1 --2- 172.21.15.150:0/4219943573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6080cfdf0 0x7fd6080ca6e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd5f800c430 tx=0x7fd5f800c8f0 comp rx=0 tx=0).stop 2024-03-17T23:48:05.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.146+0000 7fd60fade640 1 -- 172.21.15.150:0/4219943573 shutdown_connections 2024-03-17T23:48:05.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.146+0000 7fd60fade640 1 --2- 172.21.15.150:0/4219943573 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd60815e2a0 0x7fd6080b0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:05.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.146+0000 7fd60fade640 1 --2- 172.21.15.150:0/4219943573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd6080cfdf0 0x7fd6080ca6e0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:05.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.146+0000 7fd60fade640 1 --2- 172.21.15.150:0/4219943573 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd5ec06bbd0 0x7fd5ec06e090 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:05.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.146+0000 7fd60fade640 1 --2- 172.21.15.150:0/4219943573 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd60806ccb0 0x7fd6081f7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:05.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.146+0000 7fd60fade640 1 -- 172.21.15.150:0/4219943573 >> 172.21.15.150:0/4219943573 conn(0x7fd6080b36a0 msgr2=0x7fd6080b6770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:05.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.146+0000 7fd60fade640 1 -- 172.21.15.150:0/4219943573 shutdown_connections 2024-03-17T23:48:05.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.146+0000 7fd60fade640 1 -- 172.21.15.150:0/4219943573 wait complete. 2024-03-17T23:48:05.160 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-17T23:48:05.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.263+0000 7f0b1e60b640 1 Processor -- start 2024-03-17T23:48:05.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.263+0000 7f0b1e60b640 1 -- start start 2024-03-17T23:48:05.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.264+0000 7f0b1e60b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b1806e580 0x7f0b180ae070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:05.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.264+0000 7f0b1e60b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b1806ea90 con 0x7f0b180d0850 2024-03-17T23:48:05.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.264+0000 7f0b1e60b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b1806ed30 con 0x7f0b1815a4b0 2024-03-17T23:48:05.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.264+0000 7f0b1e60b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0b1806c340 con 0x7f0b1806e580 2024-03-17T23:48:05.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.264+0000 7f0b177fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f0b180d0850 0x7f0b180ad960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33600/0 (socket says 172.21.15.150:33600) 2024-03-17T23:48:05.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.264+0000 7f0b177fe640 1 -- 172.21.15.150:0/752893078 learned_addr learned my addr 172.21.15.150:0/752893078 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:05.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.264+0000 7f0b1cba8640 1 --2- 172.21.15.150:0/752893078 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b1806e580 0x7f0b180ae070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:05.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.264+0000 7f0b16ffd640 1 -- 172.21.15.150:0/752893078 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1785895689 0 0) 0x7f0b1806ea90 con 0x7f0b180d0850 2024-03-17T23:48:05.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.264+0000 7f0b16ffd640 1 -- 172.21.15.150:0/752893078 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b00003520 con 0x7f0b180d0850 2024-03-17T23:48:05.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.264+0000 7f0b16ffd640 1 -- 172.21.15.150:0/752893078 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3460595462 0 0) 0x7f0b1806ed30 con 0x7f0b1815a4b0 2024-03-17T23:48:05.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.264+0000 7f0b16ffd640 1 -- 172.21.15.150:0/752893078 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b1806ea90 con 0x7f0b1815a4b0 2024-03-17T23:48:05.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.264+0000 7f0b1cba8640 1 -- 172.21.15.150:0/752893078 >> v1:172.21.15.157:6789/0 conn(0x7f0b1815a4b0 legacy=0x7f0b180d0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:05.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.264+0000 7f0b16ffd640 1 -- 172.21.15.150:0/752893078 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1018611920 0 0) 0x7f0b00003520 con 0x7f0b180d0850 2024-03-17T23:48:05.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.264+0000 7f0b1cba8640 1 -- 172.21.15.150:0/752893078 >> v1:172.21.15.150:6789/0 conn(0x7f0b180d0850 legacy=0x7f0b180ad960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:05.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.264+0000 7f0b1cba8640 1 -- 172.21.15.150:0/752893078 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b1806c620 con 0x7f0b1806e580 2024-03-17T23:48:05.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.265+0000 7f0b1cba8640 1 --2- 172.21.15.150:0/752893078 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b1806e580 0x7f0b180ae070 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0b1000aed0 tx=0x7f0b1002be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6dbed1445e782b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:05.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.265+0000 7f0b16ffd640 1 -- 172.21.15.150:0/752893078 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c001670 con 0x7f0b1806e580 2024-03-17T23:48:05.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.265+0000 7f0b16ffd640 1 -- 172.21.15.150:0/752893078 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b1002c7a0 con 0x7f0b1806e580 2024-03-17T23:48:05.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.265+0000 7f0b16ffd640 1 -- 172.21.15.150:0/752893078 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b1002cb70 con 0x7f0b1806e580 2024-03-17T23:48:05.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.266+0000 7f0b1e60b640 1 -- 172.21.15.150:0/752893078 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b1806e580 msgr2=0x7f0b180ae070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:05.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.266+0000 7f0b1e60b640 1 --2- 172.21.15.150:0/752893078 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b1806e580 0x7f0b180ae070 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0b1000aed0 tx=0x7f0b1002be80 comp rx=0 tx=0).stop 2024-03-17T23:48:05.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.266+0000 7f0b1e60b640 1 -- 172.21.15.150:0/752893078 shutdown_connections 2024-03-17T23:48:05.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.266+0000 7f0b1e60b640 1 --2- 172.21.15.150:0/752893078 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b1806e580 0x7f0b180ae070 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:05.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.266+0000 7f0b1e60b640 1 -- 172.21.15.150:0/752893078 >> 172.21.15.150:0/752893078 conn(0x7f0b180b3da0 msgr2=0x7f0b180b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:05.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.266+0000 7f0b1e60b640 1 -- 172.21.15.150:0/752893078 shutdown_connections 2024-03-17T23:48:05.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.266+0000 7f0b1e60b640 1 -- 172.21.15.150:0/752893078 wait complete. 2024-03-17T23:48:05.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.267+0000 7f0b1e60b640 1 Processor -- start 2024-03-17T23:48:05.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.267+0000 7f0b1e60b640 1 -- start start 2024-03-17T23:48:05.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.268+0000 7f0b1e60b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b1806e580 0x7f0b18153600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:05.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.268+0000 7f0b1e60b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b180d0850 0x7f0b18149b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:05.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.268+0000 7f0b1e60b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b1815a4b0 0x7f0b181509f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:05.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.268+0000 7f0b1e60b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0b18072dc0 con 0x7f0b1815a4b0 2024-03-17T23:48:05.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.268+0000 7f0b1e60b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0b18072c40 con 0x7f0b1806e580 2024-03-17T23:48:05.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.268+0000 7f0b1e60b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0b1815a890 con 0x7f0b180d0850 2024-03-17T23:48:05.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.268+0000 7f0b1cba8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b1815a4b0 0x7f0b181509f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:05.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.268+0000 7f0b1cba8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b1815a4b0 0x7f0b181509f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45884/0 (socket says 172.21.15.150:45884) 2024-03-17T23:48:05.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.268+0000 7f0b1cba8640 1 -- 172.21.15.150:0/530360502 learned_addr learned my addr 172.21.15.150:0/530360502 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:05.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.268+0000 7f0b177fe640 1 --2- 172.21.15.150:0/530360502 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b180d0850 0x7f0b18149b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:05.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.268+0000 7f0b17fff640 1 --2- 172.21.15.150:0/530360502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b1806e580 0x7f0b18153600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:05.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.268+0000 7f0b1cba8640 1 -- 172.21.15.150:0/530360502 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b180d0850 msgr2=0x7f0b18149b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:05.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.268+0000 7f0b1cba8640 1 --2- 172.21.15.150:0/530360502 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b180d0850 0x7f0b18149b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:05.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.268+0000 7f0b1cba8640 1 -- 172.21.15.150:0/530360502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b1806e580 msgr2=0x7f0b18153600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:05.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.268+0000 7f0b1cba8640 1 --2- 172.21.15.150:0/530360502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b1806e580 0x7f0b18153600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:05.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.268+0000 7f0b1cba8640 1 -- 172.21.15.150:0/530360502 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b181513d0 con 0x7f0b1815a4b0 2024-03-17T23:48:05.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.269+0000 7f0b1cba8640 1 --2- 172.21.15.150:0/530360502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b1815a4b0 0x7f0b181509f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0b1002ba60 tx=0x7f0b10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:05.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.269+0000 7f0b14ff9640 1 -- 172.21.15.150:0/530360502 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b10043020 con 0x7f0b1815a4b0 2024-03-17T23:48:05.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.269+0000 7f0b14ff9640 1 -- 172.21.15.150:0/530360502 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b10008a00 con 0x7f0b1815a4b0 2024-03-17T23:48:05.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.269+0000 7f0b14ff9640 1 -- 172.21.15.150:0/530360502 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b1000a560 con 0x7f0b1815a4b0 2024-03-17T23:48:05.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.269+0000 7f0b1e60b640 1 -- 172.21.15.150:0/530360502 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b181516b0 con 0x7f0b1815a4b0 2024-03-17T23:48:05.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.269+0000 7f0b1e60b640 1 -- 172.21.15.150:0/530360502 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b1814ee90 con 0x7f0b1815a4b0 2024-03-17T23:48:05.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.270+0000 7f0b1e60b640 1 -- 172.21.15.150:0/530360502 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b180681a0 con 0x7f0b1815a4b0 2024-03-17T23:48:05.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.270+0000 7f0b14ff9640 1 -- 172.21.15.150:0/530360502 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0b10046020 con 0x7f0b1815a4b0 2024-03-17T23:48:05.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.271+0000 7f0b14ff9640 1 --2- 172.21.15.150:0/530360502 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b0406bbd0 0x7f0b0406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:05.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.271+0000 7f0b17fff640 1 --2- 172.21.15.150:0/530360502 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b0406bbd0 0x7f0b0406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:05.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.271+0000 7f0b14ff9640 1 -- 172.21.15.150:0/530360502 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5249+0+0 (secure 0 0 0) 0x7f0b100acc10 con 0x7f0b1815a4b0 2024-03-17T23:48:05.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.271+0000 7f0b17fff640 1 --2- 172.21.15.150:0/530360502 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b0406bbd0 0x7f0b0406e090 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0b080011e0 tx=0x7f0b08009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:05.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.275+0000 7f0b14ff9640 1 -- 172.21.15.150:0/530360502 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0b1004d030 con 0x7f0b1815a4b0 2024-03-17T23:48:05.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.514+0000 7f0b1e60b640 1 -- 172.21.15.150:0/530360502 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f0b1814f060 con 0x7f0b1815a4b0 2024-03-17T23:48:05.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.514+0000 7f0b14ff9640 1 -- 172.21.15.150:0/530360502 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) ==== 80+0+6645 (secure 0 0 0) 0x7f0b10081090 con 0x7f0b1815a4b0 2024-03-17T23:48:05.515 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:05.515 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-17T23:48:05.515 INFO:teuthology.orchestra.run.smithi150.stdout: "auth_dump": [ 2024-03-17T23:48:05.515 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:48:05.515 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "mds.a", 2024-03-17T23:48:05.515 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQC3gPdl0bOfHRAAqIf9ExwsoDeTZAbP6JhU/w==", 2024-03-17T23:48:05.515 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:48:05.515 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-03-17T23:48:05.515 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow *", 2024-03-17T23:48:05.515 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow *", 2024-03-17T23:48:05.515 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:48:05.516 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:05.516 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:05.516 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:48:05.516 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "mds.b", 2024-03-17T23:48:05.516 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQC3gPdlSlEVKhAAAScr6WsFt5/c2N1ns90pOg==", 2024-03-17T23:48:05.516 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:48:05.516 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-03-17T23:48:05.516 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow *", 2024-03-17T23:48:05.516 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow *", 2024-03-17T23:48:05.517 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:48:05.517 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:05.517 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:05.517 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:48:05.517 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "mds.c", 2024-03-17T23:48:05.517 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQC3gPdlJtcBJRAAsxXTt1otrV0PIJnP9p6CBQ==", 2024-03-17T23:48:05.517 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:48:05.517 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-03-17T23:48:05.517 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow *", 2024-03-17T23:48:05.517 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow *", 2024-03-17T23:48:05.517 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:48:05.517 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:05.517 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:05.517 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:48:05.518 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "mds.d", 2024-03-17T23:48:05.518 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQC3gPdlt7+DMRAAzv9aOR4QsJBCNHpUgSNQPg==", 2024-03-17T23:48:05.518 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:48:05.518 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-03-17T23:48:05.518 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow *", 2024-03-17T23:48:05.518 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow *", 2024-03-17T23:48:05.518 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:48:05.518 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:05.518 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:05.518 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:48:05.518 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.0", 2024-03-17T23:48:05.518 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQC7gPdlymEuGRAA0wyRX+r9Pocq1nrDOjfTfQ==", 2024-03-17T23:48:05.518 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:48:05.518 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-03-17T23:48:05.519 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-03-17T23:48:05.519 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:48:05.519 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:05.519 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:05.519 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:48:05.519 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.1", 2024-03-17T23:48:05.519 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQC+gPdl87WwFxAAO0iNw6UDAJ1I3WxmTusxFQ==", 2024-03-17T23:48:05.519 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:48:05.519 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-03-17T23:48:05.519 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-03-17T23:48:05.519 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:48:05.519 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:05.519 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:05.519 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:48:05.520 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.2", 2024-03-17T23:48:05.520 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQDBgPdlYVQIFxAANc97g+jl1RHahJS9BS2Kgw==", 2024-03-17T23:48:05.520 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:48:05.520 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-03-17T23:48:05.520 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-03-17T23:48:05.520 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:48:05.520 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:05.520 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:05.520 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:48:05.520 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.3", 2024-03-17T23:48:05.520 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQDEgPdlmaOqFRAAoYhWrnWdTC4LK0LB1SfSwg==", 2024-03-17T23:48:05.520 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:48:05.520 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-03-17T23:48:05.520 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-03-17T23:48:05.520 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:48:05.521 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:05.521 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:05.521 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:48:05.521 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.4", 2024-03-17T23:48:05.521 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQDIgPdlxX5rIhAAXHKRWgfegyAAfb/57c/xvQ==", 2024-03-17T23:48:05.521 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:48:05.521 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-03-17T23:48:05.521 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-03-17T23:48:05.521 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:48:05.521 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:05.521 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:05.521 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:48:05.521 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.5", 2024-03-17T23:48:05.521 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQDLgPdlf6xmIBAAK1d5IdCTjOHaR6yf41n/pQ==", 2024-03-17T23:48:05.522 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:48:05.522 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-03-17T23:48:05.522 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-03-17T23:48:05.522 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:48:05.522 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:05.522 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:05.522 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:48:05.522 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.6", 2024-03-17T23:48:05.522 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQDOgPdljVrnHhAAtIBm9CDQeN44WpOpUF3g3Q==", 2024-03-17T23:48:05.522 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:48:05.522 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-03-17T23:48:05.522 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-03-17T23:48:05.522 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:48:05.522 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:05.522 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:48:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.7", 2024-03-17T23:48:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQDRgPdlZDqfHRAAkkbz3Blr04H2uQsXjrQgRA==", 2024-03-17T23:48:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:48:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-03-17T23:48:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-03-17T23:48:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:48:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:48:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.0", 2024-03-17T23:48:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQC3gPdlvlPJNRAAkUfetPJK4hZQG5rNiSDozw==", 2024-03-17T23:48:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:48:05.523 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-03-17T23:48:05.524 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow r", 2024-03-17T23:48:05.524 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow rw", 2024-03-17T23:48:05.524 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow rwx" 2024-03-17T23:48:05.524 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:05.524 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:05.524 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:48:05.524 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.1", 2024-03-17T23:48:05.524 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQC4gPdlNqN0AxAA4FezHbC/cqNvSbsaThwfNw==", 2024-03-17T23:48:05.524 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:48:05.525 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-03-17T23:48:05.525 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow r", 2024-03-17T23:48:05.525 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow rw", 2024-03-17T23:48:05.525 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow rwx" 2024-03-17T23:48:05.525 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:05.525 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:05.525 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:48:05.525 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.2", 2024-03-17T23:48:05.525 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQC4gPdlvBwIABAA9kdWclUPy/mGtbT4dxfoxA==", 2024-03-17T23:48:05.525 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:48:05.525 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-03-17T23:48:05.525 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow r", 2024-03-17T23:48:05.525 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow rw", 2024-03-17T23:48:05.525 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow rwx" 2024-03-17T23:48:05.526 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:05.526 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:05.526 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:48:05.526 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.3", 2024-03-17T23:48:05.526 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQC4gPdlbNtBCRAADYXS1k+3rScOubs0QA7doQ==", 2024-03-17T23:48:05.526 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:48:05.526 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-03-17T23:48:05.526 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow r", 2024-03-17T23:48:05.526 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow rw", 2024-03-17T23:48:05.526 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow rwx" 2024-03-17T23:48:05.526 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:05.526 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:05.526 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:48:05.526 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.admin", 2024-03-17T23:48:05.527 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQC3gPdldgyOABAA2MdA5GGq5XNBML+72T9WSw==", 2024-03-17T23:48:05.527 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:48:05.527 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow *", 2024-03-17T23:48:05.527 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow *", 2024-03-17T23:48:05.527 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow *", 2024-03-17T23:48:05.527 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:48:05.527 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:05.527 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:05.527 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:48:05.527 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.bootstrap-mds", 2024-03-17T23:48:05.527 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQDXgPdlTNAiGBAA4b4U/+56Isb25h2JlDtLkQ==", 2024-03-17T23:48:05.527 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:48:05.527 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile bootstrap-mds" 2024-03-17T23:48:05.528 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:05.528 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:05.528 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:48:05.528 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.bootstrap-mgr", 2024-03-17T23:48:05.528 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQDXgPdledgiGBAAAcW4/gHeg8BH0+h8RIdZ9Q==", 2024-03-17T23:48:05.528 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:48:05.528 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-17T23:48:05.528 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:05.528 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:05.528 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:48:05.528 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.bootstrap-osd", 2024-03-17T23:48:05.528 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQDXgPdlL98iGBAAMq2dLiEukT41VuY4SoL03A==", 2024-03-17T23:48:05.529 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:48:05.529 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile bootstrap-osd" 2024-03-17T23:48:05.529 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:05.529 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:05.529 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:48:05.529 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.bootstrap-rbd", 2024-03-17T23:48:05.529 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQDXgPdl3eUiGBAAhYZHFD7PmEmgSSr+kD3jXQ==", 2024-03-17T23:48:05.529 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:48:05.529 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-17T23:48:05.529 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:05.529 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:05.529 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:48:05.529 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-17T23:48:05.529 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQDXgPdliuwiGBAAVOTTo5a3E808zNFfAAyv9Q==", 2024-03-17T23:48:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:48:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-17T23:48:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:48:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.bootstrap-rgw", 2024-03-17T23:48:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQDXgPdlY/MiGBAAAPYTOWiB6WuMcIyMr1o/dQ==", 2024-03-17T23:48:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:48:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-17T23:48:05.530 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:48:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "mgr.x", 2024-03-17T23:48:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQC3gPdlPDo1GhAAO8CxqquWsZg4j7EipcmOcQ==", 2024-03-17T23:48:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:48:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow *", 2024-03-17T23:48:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile mgr", 2024-03-17T23:48:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:48:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:48:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "mgr.y", 2024-03-17T23:48:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQC3gPdlQ9jOFRAAo0eZ4CMoH7heIX+sa4IuCw==", 2024-03-17T23:48:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:48:05.531 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow *", 2024-03-17T23:48:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile mgr", 2024-03-17T23:48:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:48:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:05.532 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-03-17T23:48:05.532 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-17T23:48:05.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.516+0000 7f0b1e60b640 1 -- 172.21.15.150:0/530360502 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b0406bbd0 msgr2=0x7f0b0406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:05.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.516+0000 7f0b1e60b640 1 --2- 172.21.15.150:0/530360502 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b0406bbd0 0x7f0b0406e090 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0b080011e0 tx=0x7f0b08009040 comp rx=0 tx=0).stop 2024-03-17T23:48:05.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.516+0000 7f0b1e60b640 1 -- 172.21.15.150:0/530360502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b1815a4b0 msgr2=0x7f0b181509f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:05.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.516+0000 7f0b1e60b640 1 --2- 172.21.15.150:0/530360502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b1815a4b0 0x7f0b181509f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0b1002ba60 tx=0x7f0b10004a40 comp rx=0 tx=0).stop 2024-03-17T23:48:05.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.516+0000 7f0b1e60b640 1 -- 172.21.15.150:0/530360502 shutdown_connections 2024-03-17T23:48:05.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.516+0000 7f0b1e60b640 1 --2- 172.21.15.150:0/530360502 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b1815a4b0 0x7f0b181509f0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:05.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.516+0000 7f0b1e60b640 1 --2- 172.21.15.150:0/530360502 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b180d0850 0x7f0b18149b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:05.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.516+0000 7f0b1e60b640 1 --2- 172.21.15.150:0/530360502 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b0406bbd0 0x7f0b0406e090 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:05.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.516+0000 7f0b1e60b640 1 --2- 172.21.15.150:0/530360502 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b1806e580 0x7f0b18153600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:05.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.516+0000 7f0b1e60b640 1 -- 172.21.15.150:0/530360502 >> 172.21.15.150:0/530360502 conn(0x7f0b180b3da0 msgr2=0x7f0b181565d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:05.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.516+0000 7f0b1e60b640 1 -- 172.21.15.150:0/530360502 shutdown_connections 2024-03-17T23:48:05.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.516+0000 7f0b1e60b640 1 -- 172.21.15.150:0/530360502 wait complete. 2024-03-17T23:48:05.536 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-17T23:48:05.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.637+0000 7f000eb74640 1 Processor -- start 2024-03-17T23:48:05.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.637+0000 7f000eb74640 1 -- start start 2024-03-17T23:48:05.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.638+0000 7f000eb74640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f00080cf100 0x7f000815bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:05.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.638+0000 7f000eb74640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f000815cac0 0x7f000815cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:05.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.638+0000 7f000eb74640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f000806de60 con 0x7f00080ccf90 2024-03-17T23:48:05.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.638+0000 7f000eb74640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f000806e420 con 0x7f00080cf100 2024-03-17T23:48:05.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.638+0000 7f000eb74640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f00080cf8a0 con 0x7f000815cac0 2024-03-17T23:48:05.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.638+0000 7f0007fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f00080cf100 0x7f000815bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:05.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.638+0000 7f000c910640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f00080ccf90 0x7f00080cd370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37500/0 (socket says 172.21.15.150:37500) 2024-03-17T23:48:05.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.638+0000 7f000c910640 1 -- 172.21.15.150:0/1493263313 learned_addr learned my addr 172.21.15.150:0/1493263313 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:05.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.638+0000 7f000d111640 1 --2- 172.21.15.150:0/1493263313 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f000815cac0 0x7f000815cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:05.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.638+0000 7f0007fff640 1 -- 172.21.15.150:0/1493263313 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f000815cac0 msgr2=0x7f000815cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:05.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.638+0000 7f0007fff640 1 --2- 172.21.15.150:0/1493263313 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f000815cac0 0x7f000815cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:05.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.638+0000 7f0007fff640 1 -- 172.21.15.150:0/1493263313 >> v1:172.21.15.150:6789/0 conn(0x7f00080ccf90 legacy=0x7f00080cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:05.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.638+0000 7f00077fe640 1 -- 172.21.15.150:0/1493263313 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 318116044 0 0) 0x7f000806de60 con 0x7f00080ccf90 2024-03-17T23:48:05.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.638+0000 7f0007fff640 1 -- 172.21.15.150:0/1493263313 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f000806e140 con 0x7f00080cf100 2024-03-17T23:48:05.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.639+0000 7f0007fff640 1 --2- 172.21.15.150:0/1493263313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00080cf100 0x7f000815bce0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7efffc0097a0 tx=0x7efffc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2f4293a0bb053b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:05.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.639+0000 7f00077fe640 1 -- 172.21.15.150:0/1493263313 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efffc0315a0 con 0x7f00080cf100 2024-03-17T23:48:05.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.639+0000 7f00077fe640 1 -- 172.21.15.150:0/1493263313 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efffc037070 con 0x7f00080cf100 2024-03-17T23:48:05.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.639+0000 7f00077fe640 1 -- 172.21.15.150:0/1493263313 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efffc031960 con 0x7f00080cf100 2024-03-17T23:48:05.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.640+0000 7f000eb74640 1 -- 172.21.15.150:0/1493263313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00080cf100 msgr2=0x7f000815bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:05.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.640+0000 7f000eb74640 1 --2- 172.21.15.150:0/1493263313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00080cf100 0x7f000815bce0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7efffc0097a0 tx=0x7efffc02b840 comp rx=0 tx=0).stop 2024-03-17T23:48:05.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.640+0000 7f000eb74640 1 -- 172.21.15.150:0/1493263313 shutdown_connections 2024-03-17T23:48:05.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.640+0000 7f000eb74640 1 --2- 172.21.15.150:0/1493263313 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f000815cac0 0x7f000815cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:05.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.640+0000 7f000eb74640 1 --2- 172.21.15.150:0/1493263313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00080cf100 0x7f000815bce0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:05.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.640+0000 7f000eb74640 1 -- 172.21.15.150:0/1493263313 >> 172.21.15.150:0/1493263313 conn(0x7f000806b190 msgr2=0x7f000806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:05.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.640+0000 7f000eb74640 1 -- 172.21.15.150:0/1493263313 shutdown_connections 2024-03-17T23:48:05.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.640+0000 7f000eb74640 1 -- 172.21.15.150:0/1493263313 wait complete. 2024-03-17T23:48:05.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.641+0000 7f000eb74640 1 Processor -- start 2024-03-17T23:48:05.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.641+0000 7f000eb74640 1 -- start start 2024-03-17T23:48:05.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.642+0000 7f000eb74640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00080ccf90 0x7f0008144340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:05.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.642+0000 7f000eb74640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00080cf100 0x7f00081e5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:05.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.642+0000 7f000eb74640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f000815cac0 0x7f00080b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:05.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.642+0000 7f000eb74640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f00080cf5b0 con 0x7f00080ccf90 2024-03-17T23:48:05.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.642+0000 7f000eb74640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f000815c220 con 0x7f000815cac0 2024-03-17T23:48:05.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.642+0000 7f000eb74640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0008155530 con 0x7f00080cf100 2024-03-17T23:48:05.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.642+0000 7f000c910640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00080ccf90 0x7f0008144340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:05.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.642+0000 7f000c910640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00080ccf90 0x7f0008144340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43422/0 (socket says 172.21.15.150:43422) 2024-03-17T23:48:05.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.642+0000 7f000c910640 1 -- 172.21.15.150:0/2641785217 learned_addr learned my addr 172.21.15.150:0/2641785217 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:05.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.642+0000 7f0007fff640 1 --2- 172.21.15.150:0/2641785217 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00080cf100 0x7f00081e5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:05.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.642+0000 7f000d111640 1 --2- 172.21.15.150:0/2641785217 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f000815cac0 0x7f00080b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:05.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.642+0000 7f000c910640 1 -- 172.21.15.150:0/2641785217 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00080cf100 msgr2=0x7f00081e5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:05.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.642+0000 7f000c910640 1 --2- 172.21.15.150:0/2641785217 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00080cf100 0x7f00081e5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:05.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.642+0000 7f000c910640 1 -- 172.21.15.150:0/2641785217 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f000815cac0 msgr2=0x7f00080b3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:05.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.642+0000 7f000c910640 1 --2- 172.21.15.150:0/2641785217 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f000815cac0 0x7f00080b3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:05.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.642+0000 7f000c910640 1 -- 172.21.15.150:0/2641785217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00080b2ac0 con 0x7f00080ccf90 2024-03-17T23:48:05.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.642+0000 7f0007fff640 1 --2- 172.21.15.150:0/2641785217 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00080cf100 0x7f00081e5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:05.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.642+0000 7f000c910640 1 --2- 172.21.15.150:0/2641785217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00080ccf90 0x7f0008144340 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7efff800c430 tx=0x7efff800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:05.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.643+0000 7f00057fa640 1 -- 172.21.15.150:0/2641785217 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff80074e0 con 0x7f00080ccf90 2024-03-17T23:48:05.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.643+0000 7f00057fa640 1 -- 172.21.15.150:0/2641785217 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efff80047e0 con 0x7f00080ccf90 2024-03-17T23:48:05.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.643+0000 7f000eb74640 1 -- 172.21.15.150:0/2641785217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00080b2da0 con 0x7f00080ccf90 2024-03-17T23:48:05.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.643+0000 7f00057fa640 1 -- 172.21.15.150:0/2641785217 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff8004bd0 con 0x7f00080ccf90 2024-03-17T23:48:05.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.643+0000 7f000eb74640 1 -- 172.21.15.150:0/2641785217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00080b3360 con 0x7f00080ccf90 2024-03-17T23:48:05.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.644+0000 7f000eb74640 1 -- 172.21.15.150:0/2641785217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00080681a0 con 0x7f00080ccf90 2024-03-17T23:48:05.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.644+0000 7f00057fa640 1 -- 172.21.15.150:0/2641785217 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7efff8011070 con 0x7f00080ccf90 2024-03-17T23:48:05.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.644+0000 7f00057fa640 1 --2- 172.21.15.150:0/2641785217 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7efff406bbd0 0x7efff406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:05.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.645+0000 7f00057fa640 1 -- 172.21.15.150:0/2641785217 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5249+0+0 (secure 0 0 0) 0x7efff808a5a0 con 0x7f00080ccf90 2024-03-17T23:48:05.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.645+0000 7f0007fff640 1 --2- 172.21.15.150:0/2641785217 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7efff406bbd0 0x7efff406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:05.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.645+0000 7f0007fff640 1 --2- 172.21.15.150:0/2641785217 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7efff406bbd0 0x7efff406e090 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7efffc036000 tx=0x7efffc03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:05.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.649+0000 7f00057fa640 1 -- 172.21.15.150:0/2641785217 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7efff8059250 con 0x7f00080ccf90 2024-03-17T23:48:05.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.892+0000 7f000eb74640 1 -- 172.21.15.150:0/2641785217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f000806bf90 con 0x7f00080ccf90 2024-03-17T23:48:05.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.894+0000 7f00057fa640 1 -- 172.21.15.150:0/2641785217 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7efff8058b50 con 0x7f00080ccf90 2024-03-17T23:48:05.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.896+0000 7f000eb74640 1 -- 172.21.15.150:0/2641785217 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7efff406bbd0 msgr2=0x7efff406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:05.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.896+0000 7f000eb74640 1 --2- 172.21.15.150:0/2641785217 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7efff406bbd0 0x7efff406e090 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7efffc036000 tx=0x7efffc03d040 comp rx=0 tx=0).stop 2024-03-17T23:48:05.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.896+0000 7f000eb74640 1 -- 172.21.15.150:0/2641785217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00080ccf90 msgr2=0x7f0008144340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:05.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.896+0000 7f000eb74640 1 --2- 172.21.15.150:0/2641785217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00080ccf90 0x7f0008144340 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7efff800c430 tx=0x7efff800c8f0 comp rx=0 tx=0).stop 2024-03-17T23:48:05.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.896+0000 7f000eb74640 1 -- 172.21.15.150:0/2641785217 shutdown_connections 2024-03-17T23:48:05.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.896+0000 7f000eb74640 1 --2- 172.21.15.150:0/2641785217 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7efff406bbd0 0x7efff406e090 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:05.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.896+0000 7f000eb74640 1 --2- 172.21.15.150:0/2641785217 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f000815cac0 0x7f00080b3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:05.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.896+0000 7f000eb74640 1 --2- 172.21.15.150:0/2641785217 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00080cf100 0x7f00081e5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:05.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.896+0000 7f000eb74640 1 --2- 172.21.15.150:0/2641785217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00080ccf90 0x7f0008144340 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:05.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.896+0000 7f000eb74640 1 -- 172.21.15.150:0/2641785217 >> 172.21.15.150:0/2641785217 conn(0x7f000806b190 msgr2=0x7f000806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:05.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.896+0000 7f000eb74640 1 -- 172.21.15.150:0/2641785217 shutdown_connections 2024-03-17T23:48:05.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:05.896+0000 7f000eb74640 1 -- 172.21.15.150:0/2641785217 wait complete. 2024-03-17T23:48:05.908 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-17T23:48:06.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.008+0000 7f82010bf640 1 Processor -- start 2024-03-17T23:48:06.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.008+0000 7f82010bf640 1 -- start start 2024-03-17T23:48:06.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.009+0000 7f82010bf640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f81fc147830 0x7f81fc099a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.009+0000 7f82010bf640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f81fc0b59e0 0x7f81fc0d1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.009+0000 7f82010bf640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81fc15e440 con 0x7f81fc0d1580 2024-03-17T23:48:06.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.009+0000 7f82010bf640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f81fc06fe90 con 0x7f81fc0b59e0 2024-03-17T23:48:06.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.009+0000 7f82010bf640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f81fc09a310 con 0x7f81fc147830 2024-03-17T23:48:06.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.009+0000 7f81fa59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f81fc0b59e0 0x7f81fc0d1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:06.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.009+0000 7f81fa59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f81fc0b59e0 0x7f81fc0d1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43424/0 (socket says 172.21.15.150:43424) 2024-03-17T23:48:06.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.009+0000 7f81fa59c640 1 -- 172.21.15.150:0/2129725438 learned_addr learned my addr 172.21.15.150:0/2129725438 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:06.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.009+0000 7f81fad9d640 1 --2- 172.21.15.150:0/2129725438 >> v2:172.21.15.157:3300/0 conn(0x7f81fc147830 0x7f81fc099a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:06.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.009+0000 7f81fa59c640 1 -- 172.21.15.150:0/2129725438 >> v2:172.21.15.157:3300/0 conn(0x7f81fc147830 msgr2=0x7f81fc099a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:06.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.009+0000 7f81fa59c640 1 --2- 172.21.15.150:0/2129725438 >> v2:172.21.15.157:3300/0 conn(0x7f81fc147830 0x7f81fc099a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.009+0000 7f81fa59c640 1 -- 172.21.15.150:0/2129725438 >> v1:172.21.15.157:6789/0 conn(0x7f81fc0d1580 legacy=0x7f81fc06b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:06.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.010+0000 7f81fa59c640 1 -- 172.21.15.150:0/2129725438 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81fc15e700 con 0x7f81fc0b59e0 2024-03-17T23:48:06.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.010+0000 7f81fa59c640 1 --2- 172.21.15.150:0/2129725438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81fc0b59e0 0x7f81fc0d1040 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f81fc070010 tx=0x7f81f002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e894945ac2e0f25d server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:06.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.010+0000 7f81f9d9b640 1 -- 172.21.15.150:0/2129725438 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f0035020 con 0x7f81fc0b59e0 2024-03-17T23:48:06.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.010+0000 7f81f9d9b640 1 -- 172.21.15.150:0/2129725438 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81f0031870 con 0x7f81fc0b59e0 2024-03-17T23:48:06.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.010+0000 7f81f9d9b640 1 -- 172.21.15.150:0/2129725438 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f0031c40 con 0x7f81fc0b59e0 2024-03-17T23:48:06.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.011+0000 7f82010bf640 1 -- 172.21.15.150:0/2129725438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81fc0b59e0 msgr2=0x7f81fc0d1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:06.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.011+0000 7f82010bf640 1 --2- 172.21.15.150:0/2129725438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81fc0b59e0 0x7f81fc0d1040 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f81fc070010 tx=0x7f81f002b910 comp rx=0 tx=0).stop 2024-03-17T23:48:06.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.011+0000 7f82010bf640 1 -- 172.21.15.150:0/2129725438 shutdown_connections 2024-03-17T23:48:06.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.011+0000 7f82010bf640 1 --2- 172.21.15.150:0/2129725438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81fc0b59e0 0x7f81fc0d1040 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.011+0000 7f82010bf640 1 --2- 172.21.15.150:0/2129725438 >> v2:172.21.15.157:3300/0 conn(0x7f81fc147830 0x7f81fc099a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.011+0000 7f82010bf640 1 -- 172.21.15.150:0/2129725438 >> 172.21.15.150:0/2129725438 conn(0x7f81fc0b5ec0 msgr2=0x7f81fc0b62d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:06.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.011+0000 7f82010bf640 1 -- 172.21.15.150:0/2129725438 shutdown_connections 2024-03-17T23:48:06.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.011+0000 7f82010bf640 1 -- 172.21.15.150:0/2129725438 wait complete. 2024-03-17T23:48:06.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.012+0000 7f82010bf640 1 Processor -- start 2024-03-17T23:48:06.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.012+0000 7f82010bf640 1 -- start start 2024-03-17T23:48:06.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.013+0000 7f82010bf640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81fc0b59e0 0x7f81fc09b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.013+0000 7f82010bf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81fc0d1580 0x7f81fc133d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.013+0000 7f82010bf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81fc147830 0x7f81fc0aef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.013+0000 7f82010bf640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f81fc0d1960 con 0x7f81fc0d1580 2024-03-17T23:48:06.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.013+0000 7f82010bf640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f81fc15ecc0 con 0x7f81fc0b59e0 2024-03-17T23:48:06.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.013+0000 7f82010bf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f81fc099fb0 con 0x7f81fc147830 2024-03-17T23:48:06.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.013+0000 7f81fa59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81fc0d1580 0x7f81fc133d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:06.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.013+0000 7f81fa59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81fc0d1580 0x7f81fc133d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43438/0 (socket says 172.21.15.150:43438) 2024-03-17T23:48:06.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.013+0000 7f81fa59c640 1 -- 172.21.15.150:0/864345428 learned_addr learned my addr 172.21.15.150:0/864345428 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:06.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.014+0000 7f81fb59e640 1 --2- 172.21.15.150:0/864345428 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81fc147830 0x7f81fc0aef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:06.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.014+0000 7f81fa59c640 1 -- 172.21.15.150:0/864345428 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81fc147830 msgr2=0x7f81fc0aef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:06.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.014+0000 7f81fa59c640 1 --2- 172.21.15.150:0/864345428 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81fc147830 0x7f81fc0aef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.014+0000 7f81fad9d640 1 --2- 172.21.15.150:0/864345428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81fc0b59e0 0x7f81fc09b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:06.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.014+0000 7f81fa59c640 1 -- 172.21.15.150:0/864345428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81fc0b59e0 msgr2=0x7f81fc09b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:06.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.014+0000 7f81fa59c640 1 --2- 172.21.15.150:0/864345428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81fc0b59e0 0x7f81fc09b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.014+0000 7f81fa59c640 1 -- 172.21.15.150:0/864345428 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81fc134850 con 0x7f81fc0d1580 2024-03-17T23:48:06.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.014+0000 7f81fa59c640 1 --2- 172.21.15.150:0/864345428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81fc0d1580 0x7f81fc133d60 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f81f002b950 tx=0x7f81f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:06.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.014+0000 7f81e37fe640 1 -- 172.21.15.150:0/864345428 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f0040070 con 0x7f81fc0d1580 2024-03-17T23:48:06.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.014+0000 7f81e37fe640 1 -- 172.21.15.150:0/864345428 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81f0030d00 con 0x7f81fc0d1580 2024-03-17T23:48:06.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.014+0000 7f82010bf640 1 -- 172.21.15.150:0/864345428 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81fc0ca5a0 con 0x7f81fc0d1580 2024-03-17T23:48:06.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.014+0000 7f81e37fe640 1 -- 172.21.15.150:0/864345428 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f0035040 con 0x7f81fc0d1580 2024-03-17T23:48:06.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.014+0000 7f82010bf640 1 -- 172.21.15.150:0/864345428 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81fc134af0 con 0x7f81fc0d1580 2024-03-17T23:48:06.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.015+0000 7f82010bf640 1 -- 172.21.15.150:0/864345428 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81fc0681a0 con 0x7f81fc0d1580 2024-03-17T23:48:06.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.016+0000 7f81e37fe640 1 -- 172.21.15.150:0/864345428 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f81f0038020 con 0x7f81fc0d1580 2024-03-17T23:48:06.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.016+0000 7f81e37fe640 1 --2- 172.21.15.150:0/864345428 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f81e806bbd0 0x7f81e806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.016+0000 7f81fad9d640 1 --2- 172.21.15.150:0/864345428 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f81e806bbd0 0x7f81e806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:06.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.016+0000 7f81e37fe640 1 -- 172.21.15.150:0/864345428 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5249+0+0 (secure 0 0 0) 0x7f81f00ac6a0 con 0x7f81fc0d1580 2024-03-17T23:48:06.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.017+0000 7f81fad9d640 1 --2- 172.21.15.150:0/864345428 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f81e806bbd0 0x7f81e806e090 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f81ec006fd0 tx=0x7f81ec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:06.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.021+0000 7f81e37fe640 1 -- 172.21.15.150:0/864345428 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f81f007b3e0 con 0x7f81fc0d1580 2024-03-17T23:48:06.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.263+0000 7f82010bf640 1 -- 172.21.15.150:0/864345428 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7f81fc0b3100 con 0x7f81fc0d1580 2024-03-17T23:48:06.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.265+0000 7f81e37fe640 1 -- 172.21.15.150:0/864345428 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7f81f00ac030 con 0x7f81fc0d1580 2024-03-17T23:48:06.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.267+0000 7f82010bf640 1 -- 172.21.15.150:0/864345428 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f81e806bbd0 msgr2=0x7f81e806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:06.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.267+0000 7f82010bf640 1 --2- 172.21.15.150:0/864345428 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f81e806bbd0 0x7f81e806e090 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f81ec006fd0 tx=0x7f81ec008040 comp rx=0 tx=0).stop 2024-03-17T23:48:06.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.267+0000 7f82010bf640 1 -- 172.21.15.150:0/864345428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81fc0d1580 msgr2=0x7f81fc133d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:06.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.267+0000 7f82010bf640 1 --2- 172.21.15.150:0/864345428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81fc0d1580 0x7f81fc133d60 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f81f002b950 tx=0x7f81f00046c0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.268+0000 7f82010bf640 1 -- 172.21.15.150:0/864345428 shutdown_connections 2024-03-17T23:48:06.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.268+0000 7f82010bf640 1 --2- 172.21.15.150:0/864345428 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81fc147830 0x7f81fc0aef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.268+0000 7f82010bf640 1 --2- 172.21.15.150:0/864345428 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81fc0d1580 0x7f81fc133d60 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.268+0000 7f82010bf640 1 --2- 172.21.15.150:0/864345428 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f81e806bbd0 0x7f81e806e090 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.268+0000 7f82010bf640 1 --2- 172.21.15.150:0/864345428 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81fc0b59e0 0x7f81fc09b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.268+0000 7f82010bf640 1 -- 172.21.15.150:0/864345428 >> 172.21.15.150:0/864345428 conn(0x7f81fc0b5ec0 msgr2=0x7f81fc146160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:06.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.268+0000 7f82010bf640 1 -- 172.21.15.150:0/864345428 shutdown_connections 2024-03-17T23:48:06.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.268+0000 7f82010bf640 1 -- 172.21.15.150:0/864345428 wait complete. 2024-03-17T23:48:06.279 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-17T23:48:06.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.379+0000 7f598f39d640 1 Processor -- start 2024-03-17T23:48:06.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.379+0000 7f598f39d640 1 -- start start 2024-03-17T23:48:06.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.380+0000 7f598f39d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f598806b7e0 0x7f598815ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.380+0000 7f598f39d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f598815f550 con 0x7f598815f120 2024-03-17T23:48:06.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.380+0000 7f598f39d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59880ce410 con 0x7f598806b400 2024-03-17T23:48:06.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.380+0000 7f598f39d640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f59880cd0a0 con 0x7f598806b7e0 2024-03-17T23:48:06.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.380+0000 7f598eb9c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f598815f120 0x7f598815f7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37514/0 (socket says 172.21.15.150:37514) 2024-03-17T23:48:06.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.380+0000 7f598eb9c640 1 -- 172.21.15.150:0/2719580816 learned_addr learned my addr 172.21.15.150:0/2719580816 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:06.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.380+0000 7f598db9a640 1 --2- 172.21.15.150:0/2719580816 >> v2:172.21.15.157:3300/0 conn(0x7f598806b7e0 0x7f598815ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:06.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.380+0000 7f598d399640 1 -- 172.21.15.150:0/2719580816 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1877143218 0 0) 0x7f598815f550 con 0x7f598815f120 2024-03-17T23:48:06.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.380+0000 7f598d399640 1 -- 172.21.15.150:0/2719580816 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5970003520 con 0x7f598815f120 2024-03-17T23:48:06.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.381+0000 7f598d399640 1 -- 172.21.15.150:0/2719580816 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1103033005 0 0) 0x7f5970003520 con 0x7f598815f120 2024-03-17T23:48:06.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.381+0000 7f598d399640 1 -- 172.21.15.150:0/2719580816 >> v2:172.21.15.157:3300/0 conn(0x7f598806b7e0 msgr2=0x7f598815ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:06.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.381+0000 7f598d399640 1 --2- 172.21.15.150:0/2719580816 >> v2:172.21.15.157:3300/0 conn(0x7f598806b7e0 0x7f598815ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.381+0000 7f598d399640 1 -- 172.21.15.150:0/2719580816 >> v1:172.21.15.157:6789/0 conn(0x7f598806b400 legacy=0x7f59880cc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:06.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.381+0000 7f598d399640 1 -- 172.21.15.150:0/2719580816 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59880ce6d0 con 0x7f598815f120 2024-03-17T23:48:06.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.381+0000 7f598d399640 1 -- 172.21.15.150:0/2719580816 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f5978001670 con 0x7f598815f120 2024-03-17T23:48:06.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.381+0000 7f598d399640 1 -- 172.21.15.150:0/2719580816 >> v1:172.21.15.150:6789/0 conn(0x7f598815f120 legacy=0x7f598815f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:06.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.381+0000 7f598d399640 1 --2- 172.21.15.150:0/2719580816 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59700039b0 0x7f5970003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.381+0000 7f598d399640 1 --2- 172.21.15.150:0/2719580816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5970004800 0x7f5970004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.381+0000 7f598d399640 1 --2- 172.21.15.150:0/2719580816 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f59700051c0 0x7f597000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.381+0000 7f598d399640 1 -- 172.21.15.150:0/2719580816 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f59880cd0a0 con 0x7f5970004800 2024-03-17T23:48:06.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.381+0000 7f598d399640 1 -- 172.21.15.150:0/2719580816 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f597000c060 con 0x7f59700039b0 2024-03-17T23:48:06.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.381+0000 7f598d399640 1 -- 172.21.15.150:0/2719580816 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f597000c260 con 0x7f59700051c0 2024-03-17T23:48:06.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.381+0000 7f598db9a640 1 --2- 172.21.15.150:0/2719580816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5970004800 0x7f5970004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:06.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.381+0000 7f598e39b640 1 --2- 172.21.15.150:0/2719580816 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59700039b0 0x7f5970003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:06.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.381+0000 7f598eb9c640 1 --2- 172.21.15.150:0/2719580816 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f59700051c0 0x7f597000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:06.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.381+0000 7f598db9a640 1 -- 172.21.15.150:0/2719580816 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f59700051c0 msgr2=0x7f597000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:06.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.381+0000 7f598db9a640 1 --2- 172.21.15.150:0/2719580816 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f59700051c0 0x7f597000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.381+0000 7f598db9a640 1 -- 172.21.15.150:0/2719580816 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59700039b0 msgr2=0x7f5970003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:06.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.381+0000 7f598db9a640 1 --2- 172.21.15.150:0/2719580816 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59700039b0 0x7f5970003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.381+0000 7f598eb9c640 1 -- 172.21.15.150:0/2719580816 reap_dead start 2024-03-17T23:48:06.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.381+0000 7f598db9a640 1 -- 172.21.15.150:0/2719580816 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5970003520 con 0x7f5970004800 2024-03-17T23:48:06.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.382+0000 7f598db9a640 1 --2- 172.21.15.150:0/2719580816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5970004800 0x7f5970004c80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f597c009ac0 tx=0x7f5970005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58cedca506b59863 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:06.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.382+0000 7f598d399640 1 -- 172.21.15.150:0/2719580816 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f597c039070 con 0x7f5970004800 2024-03-17T23:48:06.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.382+0000 7f598d399640 1 -- 172.21.15.150:0/2719580816 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f597c02c6b0 con 0x7f5970004800 2024-03-17T23:48:06.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.382+0000 7f598d399640 1 -- 172.21.15.150:0/2719580816 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f597c009690 con 0x7f5970004800 2024-03-17T23:48:06.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.382+0000 7f598f39d640 1 -- 172.21.15.150:0/2719580816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5970004800 msgr2=0x7f5970004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:06.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.382+0000 7f598f39d640 1 --2- 172.21.15.150:0/2719580816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5970004800 0x7f5970004c80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f597c009ac0 tx=0x7f5970005830 comp rx=0 tx=0).stop 2024-03-17T23:48:06.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.383+0000 7f598f39d640 1 -- 172.21.15.150:0/2719580816 shutdown_connections 2024-03-17T23:48:06.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.383+0000 7f598f39d640 1 --2- 172.21.15.150:0/2719580816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5970004800 0x7f5970004c80 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.383+0000 7f598f39d640 1 -- 172.21.15.150:0/2719580816 >> 172.21.15.150:0/2719580816 conn(0x7f5988099a70 msgr2=0x7f5988099e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:06.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.383+0000 7f598f39d640 1 -- 172.21.15.150:0/2719580816 shutdown_connections 2024-03-17T23:48:06.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.383+0000 7f598f39d640 1 -- 172.21.15.150:0/2719580816 wait complete. 2024-03-17T23:48:06.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.384+0000 7f598f39d640 1 Processor -- start 2024-03-17T23:48:06.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.384+0000 7f598f39d640 1 -- start start 2024-03-17T23:48:06.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.384+0000 7f598f39d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5970004800 0x7f5988157d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.384+0000 7f598f39d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59880c4d50 0x7f59881f4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.384+0000 7f598f39d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59880c51d0 0x7f59881f53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.384+0000 7f598f39d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f59880d15e0 con 0x7f59880c4d50 2024-03-17T23:48:06.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.384+0000 7f598f39d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f59880cec90 con 0x7f59880c51d0 2024-03-17T23:48:06.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.384+0000 7f598f39d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f59880cf1b0 con 0x7f5970004800 2024-03-17T23:48:06.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.385+0000 7f598db9a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59880c4d50 0x7f59881f4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:06.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.385+0000 7f598db9a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59880c4d50 0x7f59881f4eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43452/0 (socket says 172.21.15.150:43452) 2024-03-17T23:48:06.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.385+0000 7f598db9a640 1 -- 172.21.15.150:0/376839234 learned_addr learned my addr 172.21.15.150:0/376839234 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:06.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.385+0000 7f598e39b640 1 --2- 172.21.15.150:0/376839234 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5970004800 0x7f5988157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:06.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.385+0000 7f598eb9c640 1 --2- 172.21.15.150:0/376839234 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59880c51d0 0x7f59881f53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:06.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.385+0000 7f598db9a640 1 -- 172.21.15.150:0/376839234 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5970004800 msgr2=0x7f5988157d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:06.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.385+0000 7f598db9a640 1 --2- 172.21.15.150:0/376839234 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5970004800 0x7f5988157d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.385+0000 7f598db9a640 1 -- 172.21.15.150:0/376839234 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59880c51d0 msgr2=0x7f59881f53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:06.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.385+0000 7f598db9a640 1 --2- 172.21.15.150:0/376839234 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59880c51d0 0x7f59881f53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.385+0000 7f598db9a640 1 -- 172.21.15.150:0/376839234 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59880c57a0 con 0x7f59880c4d50 2024-03-17T23:48:06.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.385+0000 7f598eb9c640 1 --2- 172.21.15.150:0/376839234 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59880c51d0 0x7f59881f53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:06.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.385+0000 7f598db9a640 1 --2- 172.21.15.150:0/376839234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59880c4d50 0x7f59881f4eb0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f597c009ac0 tx=0x7f597c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:06.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.385+0000 7f5986ffd640 1 -- 172.21.15.150:0/376839234 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f597c039050 con 0x7f59880c4d50 2024-03-17T23:48:06.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.385+0000 7f5986ffd640 1 -- 172.21.15.150:0/376839234 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f597c045070 con 0x7f59880c4d50 2024-03-17T23:48:06.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.385+0000 7f598f39d640 1 -- 172.21.15.150:0/376839234 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59880c5a80 con 0x7f59880c4d50 2024-03-17T23:48:06.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.385+0000 7f5986ffd640 1 -- 172.21.15.150:0/376839234 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f597c0044e0 con 0x7f59880c4d50 2024-03-17T23:48:06.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.385+0000 7f598f39d640 1 -- 172.21.15.150:0/376839234 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5988158270 con 0x7f59880c4d50 2024-03-17T23:48:06.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.386+0000 7f598f39d640 1 -- 172.21.15.150:0/376839234 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5988074bd0 con 0x7f59880c4d50 2024-03-17T23:48:06.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.387+0000 7f5986ffd640 1 -- 172.21.15.150:0/376839234 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f597c003a50 con 0x7f59880c4d50 2024-03-17T23:48:06.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.387+0000 7f5986ffd640 1 --2- 172.21.15.150:0/376839234 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f597406bbd0 0x7f597406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.387+0000 7f5986ffd640 1 -- 172.21.15.150:0/376839234 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5249+0+0 (secure 0 0 0) 0x7f597c0ac7b0 con 0x7f59880c4d50 2024-03-17T23:48:06.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.387+0000 7f598e39b640 1 --2- 172.21.15.150:0/376839234 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f597406bbd0 0x7f597406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:06.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.388+0000 7f598e39b640 1 --2- 172.21.15.150:0/376839234 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f597406bbd0 0x7f597406e090 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5980000f80 tx=0x7f5980009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:06.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.392+0000 7f5986ffd640 1 -- 172.21.15.150:0/376839234 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f597c04f020 con 0x7f59880c4d50 2024-03-17T23:48:06.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.630+0000 7f598f39d640 1 -- 172.21.15.150:0/376839234 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f59880afd40 con 0x7f59880c4d50 2024-03-17T23:48:06.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.632+0000 7f5986ffd640 1 -- 172.21.15.150:0/376839234 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) ==== 90+0+0 (secure 0 0 0) 0x7f597c07ae80 con 0x7f59880c4d50 2024-03-17T23:48:06.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.635+0000 7f598f39d640 1 -- 172.21.15.150:0/376839234 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f597406bbd0 msgr2=0x7f597406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:06.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.635+0000 7f598f39d640 1 --2- 172.21.15.150:0/376839234 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f597406bbd0 0x7f597406e090 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5980000f80 tx=0x7f5980009040 comp rx=0 tx=0).stop 2024-03-17T23:48:06.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.635+0000 7f598f39d640 1 -- 172.21.15.150:0/376839234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59880c4d50 msgr2=0x7f59881f4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:06.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.635+0000 7f598f39d640 1 --2- 172.21.15.150:0/376839234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59880c4d50 0x7f59881f4eb0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f597c009ac0 tx=0x7f597c004730 comp rx=0 tx=0).stop 2024-03-17T23:48:06.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.635+0000 7f598f39d640 1 -- 172.21.15.150:0/376839234 shutdown_connections 2024-03-17T23:48:06.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.635+0000 7f598f39d640 1 --2- 172.21.15.150:0/376839234 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f597406bbd0 0x7f597406e090 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.635+0000 7f598f39d640 1 --2- 172.21.15.150:0/376839234 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59880c51d0 0x7f59881f53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.635+0000 7f598f39d640 1 --2- 172.21.15.150:0/376839234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59880c4d50 0x7f59881f4eb0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.635+0000 7f598f39d640 1 --2- 172.21.15.150:0/376839234 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5970004800 0x7f5988157d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.635+0000 7f598f39d640 1 -- 172.21.15.150:0/376839234 >> 172.21.15.150:0/376839234 conn(0x7f5988099a70 msgr2=0x7f59880b13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:06.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.635+0000 7f598f39d640 1 -- 172.21.15.150:0/376839234 shutdown_connections 2024-03-17T23:48:06.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.635+0000 7f598f39d640 1 -- 172.21.15.150:0/376839234 wait complete. 2024-03-17T23:48:06.646 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-17T23:48:06.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.748+0000 7f745bf1b640 1 Processor -- start 2024-03-17T23:48:06.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.748+0000 7f745bf1b640 1 -- start start 2024-03-17T23:48:06.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.749+0000 7f745bf1b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f74540ccf90 0x7f74540cf100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.749+0000 7f745bf1b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f745415cac0 0x7f745415cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.749+0000 7f745bf1b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f745406de60 con 0x7f745415bce0 2024-03-17T23:48:06.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.749+0000 7f745bf1b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f745406e420 con 0x7f74540ccf90 2024-03-17T23:48:06.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.749+0000 7f745bf1b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f74540cf8a0 con 0x7f745415cac0 2024-03-17T23:48:06.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.749+0000 7f74594b6640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f745415bce0 0x7f74540cd370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37516/0 (socket says 172.21.15.150:37516) 2024-03-17T23:48:06.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.749+0000 7f74594b6640 1 -- 172.21.15.150:0/3849491892 learned_addr learned my addr 172.21.15.150:0/3849491892 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:06.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.749+0000 7f7458cb5640 1 -- 172.21.15.150:0/3849491892 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2388867816 0 0) 0x7f745406de60 con 0x7f745415bce0 2024-03-17T23:48:06.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.749+0000 7f7458cb5640 1 -- 172.21.15.150:0/3849491892 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7440003520 con 0x7f745415bce0 2024-03-17T23:48:06.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.749+0000 7f745a4b8640 1 --2- 172.21.15.150:0/3849491892 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f745415cac0 0x7f745415cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:06.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.749+0000 7f7459cb7640 1 --2- 172.21.15.150:0/3849491892 >> v2:172.21.15.157:3300/0 conn(0x7f74540ccf90 0x7f74540cf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:06.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.749+0000 7f7458cb5640 1 -- 172.21.15.150:0/3849491892 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4249555857 0 0) 0x7f7440003520 con 0x7f745415bce0 2024-03-17T23:48:06.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.749+0000 7f7458cb5640 1 -- 172.21.15.150:0/3849491892 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f745415cac0 msgr2=0x7f745415cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:06.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.749+0000 7f7458cb5640 1 --2- 172.21.15.150:0/3849491892 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f745415cac0 0x7f745415cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.749+0000 7f7458cb5640 1 -- 172.21.15.150:0/3849491892 >> v2:172.21.15.157:3300/0 conn(0x7f74540ccf90 msgr2=0x7f74540cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:06.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.749+0000 7f7458cb5640 1 --2- 172.21.15.150:0/3849491892 >> v2:172.21.15.157:3300/0 conn(0x7f74540ccf90 0x7f74540cf100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.749+0000 7f7458cb5640 1 -- 172.21.15.150:0/3849491892 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f745406e140 con 0x7f745415bce0 2024-03-17T23:48:06.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.749+0000 7f7458cb5640 1 -- 172.21.15.150:0/3849491892 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f7444001670 con 0x7f745415bce0 2024-03-17T23:48:06.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.749+0000 7f7458cb5640 1 -- 172.21.15.150:0/3849491892 >> v1:172.21.15.150:6789/0 conn(0x7f745415bce0 legacy=0x7f74540cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:06.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.749+0000 7f7458cb5640 1 --2- 172.21.15.150:0/3849491892 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f74400039b0 0x7f7440003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.750+0000 7f7458cb5640 1 --2- 172.21.15.150:0/3849491892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7440004800 0x7f7440004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.750+0000 7f7458cb5640 1 --2- 172.21.15.150:0/3849491892 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f745415cac0 0x7f74400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.750+0000 7f7458cb5640 1 -- 172.21.15.150:0/3849491892 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f745406e420 con 0x7f7440004800 2024-03-17T23:48:06.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.750+0000 7f7458cb5640 1 -- 172.21.15.150:0/3849491892 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f74540cf8a0 con 0x7f74400039b0 2024-03-17T23:48:06.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.750+0000 7f7458cb5640 1 -- 172.21.15.150:0/3849491892 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f744000dc00 con 0x7f745415cac0 2024-03-17T23:48:06.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.750+0000 7f74594b6640 1 --2- 172.21.15.150:0/3849491892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7440004800 0x7f7440004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:06.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.750+0000 7f7459cb7640 1 --2- 172.21.15.150:0/3849491892 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f74400039b0 0x7f7440003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:06.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.750+0000 7f745a4b8640 1 --2- 172.21.15.150:0/3849491892 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f745415cac0 0x7f74400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:06.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.750+0000 7f74594b6640 1 -- 172.21.15.150:0/3849491892 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f745415cac0 msgr2=0x7f74400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:06.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.750+0000 7f74594b6640 1 --2- 172.21.15.150:0/3849491892 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f745415cac0 0x7f74400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.750+0000 7f74594b6640 1 -- 172.21.15.150:0/3849491892 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f74400039b0 msgr2=0x7f7440003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:06.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.750+0000 7f74594b6640 1 --2- 172.21.15.150:0/3849491892 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f74400039b0 0x7f7440003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.750+0000 7f74594b6640 1 -- 172.21.15.150:0/3849491892 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7440003520 con 0x7f7440004800 2024-03-17T23:48:06.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.751+0000 7f74594b6640 1 --2- 172.21.15.150:0/3849491892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7440004800 0x7f7440004c80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7444007fd0 tx=0x7f7440005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38de46efae8e0f02 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:06.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.751+0000 7f7458cb5640 1 -- 172.21.15.150:0/3849491892 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7444007d50 con 0x7f7440004800 2024-03-17T23:48:06.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.751+0000 7f7458cb5640 1 -- 172.21.15.150:0/3849491892 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7444030070 con 0x7f7440004800 2024-03-17T23:48:06.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.751+0000 7f7458cb5640 1 -- 172.21.15.150:0/3849491892 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744402cd20 con 0x7f7440004800 2024-03-17T23:48:06.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.751+0000 7f745bf1b640 1 -- 172.21.15.150:0/3849491892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7440004800 msgr2=0x7f7440004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:06.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.751+0000 7f745bf1b640 1 --2- 172.21.15.150:0/3849491892 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7440004800 0x7f7440004c80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7444007fd0 tx=0x7f7440005830 comp rx=0 tx=0).stop 2024-03-17T23:48:06.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.751+0000 7f745a4b8640 1 -- 172.21.15.150:0/3849491892 reap_dead start 2024-03-17T23:48:06.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.751+0000 7f745bf1b640 1 -- 172.21.15.150:0/3849491892 shutdown_connections 2024-03-17T23:48:06.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.751+0000 7f745bf1b640 1 -- 172.21.15.150:0/3849491892 >> 172.21.15.150:0/3849491892 conn(0x7f745406b190 msgr2=0x7f745406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:06.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.751+0000 7f745bf1b640 1 -- 172.21.15.150:0/3849491892 shutdown_connections 2024-03-17T23:48:06.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.751+0000 7f745bf1b640 1 -- 172.21.15.150:0/3849491892 wait complete. 2024-03-17T23:48:06.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.753+0000 7f745bf1b640 1 Processor -- start 2024-03-17T23:48:06.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.753+0000 7f745bf1b640 1 -- start start 2024-03-17T23:48:06.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.753+0000 7f745bf1b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7440004800 0x7f7454155900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.753+0000 7f745bf1b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f745406eef0 0x7f7454141250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.753+0000 7f745bf1b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f74541e57f0 0x7f74541e5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.753+0000 7f745bf1b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f745415c190 con 0x7f7440004800 2024-03-17T23:48:06.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.753+0000 7f745bf1b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f74540cf640 con 0x7f74541e57f0 2024-03-17T23:48:06.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.753+0000 7f745bf1b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7454155530 con 0x7f745406eef0 2024-03-17T23:48:06.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.753+0000 7f7459cb7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7440004800 0x7f7454155900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:06.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.753+0000 7f7459cb7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7440004800 0x7f7454155900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43478/0 (socket says 172.21.15.150:43478) 2024-03-17T23:48:06.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.753+0000 7f7459cb7640 1 -- 172.21.15.150:0/3931220948 learned_addr learned my addr 172.21.15.150:0/3931220948 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:06.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.754+0000 7f7459cb7640 1 -- 172.21.15.150:0/3931220948 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f745406eef0 msgr2=0x7f7454141250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:06.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.754+0000 7f7459cb7640 1 --2- 172.21.15.150:0/3931220948 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f745406eef0 0x7f7454141250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.754+0000 7f7459cb7640 1 -- 172.21.15.150:0/3931220948 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f74541e57f0 msgr2=0x7f74541e5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:06.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.754+0000 7f7459cb7640 1 --2- 172.21.15.150:0/3931220948 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f74541e57f0 0x7f74541e5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:06.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.754+0000 7f7459cb7640 1 -- 172.21.15.150:0/3931220948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f745414e570 con 0x7f7440004800 2024-03-17T23:48:06.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.754+0000 7f7459cb7640 1 --2- 172.21.15.150:0/3931220948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7440004800 0x7f7454155900 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f7450011940 tx=0x7f7450011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:06.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.754+0000 7f744a7fc640 1 -- 172.21.15.150:0/3931220948 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745000fa70 con 0x7f7440004800 2024-03-17T23:48:06.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.754+0000 7f744a7fc640 1 -- 172.21.15.150:0/3931220948 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f745000fc10 con 0x7f7440004800 2024-03-17T23:48:06.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.754+0000 7f745bf1b640 1 -- 172.21.15.150:0/3931220948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74541572b0 con 0x7f7440004800 2024-03-17T23:48:06.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.754+0000 7f744a7fc640 1 -- 172.21.15.150:0/3931220948 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7450014440 con 0x7f7440004800 2024-03-17T23:48:06.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.754+0000 7f745bf1b640 1 -- 172.21.15.150:0/3931220948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7454141790 con 0x7f7440004800 2024-03-17T23:48:06.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.755+0000 7f745bf1b640 1 -- 172.21.15.150:0/3931220948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74540681a0 con 0x7f7440004800 2024-03-17T23:48:06.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.755+0000 7f744a7fc640 1 -- 172.21.15.150:0/3931220948 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7450012040 con 0x7f7440004800 2024-03-17T23:48:06.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.756+0000 7f744a7fc640 1 --2- 172.21.15.150:0/3931220948 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f743806bbd0 0x7f743806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:06.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.756+0000 7f744a7fc640 1 -- 172.21.15.150:0/3931220948 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5249+0+0 (secure 0 0 0) 0x7f745008ae60 con 0x7f7440004800 2024-03-17T23:48:06.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.756+0000 7f74594b6640 1 --2- 172.21.15.150:0/3931220948 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f743806bbd0 0x7f743806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:06.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.756+0000 7f74594b6640 1 --2- 172.21.15.150:0/3931220948 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f743806bbd0 0x7f743806e090 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7444034000 tx=0x7f744402e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:06.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.760+0000 7f744a7fc640 1 -- 172.21.15.150:0/3931220948 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f74541572b0 con 0x7f7440004800 2024-03-17T23:48:06.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:06.998+0000 7f745bf1b640 1 -- 172.21.15.150:0/3931220948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f74540cf320 con 0x7f7440004800 2024-03-17T23:48:07.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.000+0000 7f744a7fc640 1 -- 172.21.15.150:0/3931220948 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7f7450019030 con 0x7f7440004800 2024-03-17T23:48:07.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.002+0000 7f745bf1b640 1 -- 172.21.15.150:0/3931220948 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f743806bbd0 msgr2=0x7f743806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:07.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.002+0000 7f745bf1b640 1 --2- 172.21.15.150:0/3931220948 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f743806bbd0 0x7f743806e090 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7444034000 tx=0x7f744402e000 comp rx=0 tx=0).stop 2024-03-17T23:48:07.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.002+0000 7f745bf1b640 1 -- 172.21.15.150:0/3931220948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7440004800 msgr2=0x7f7454155900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:07.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.002+0000 7f745bf1b640 1 --2- 172.21.15.150:0/3931220948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7440004800 0x7f7454155900 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f7450011940 tx=0x7f7450011e00 comp rx=0 tx=0).stop 2024-03-17T23:48:07.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.003+0000 7f745bf1b640 1 -- 172.21.15.150:0/3931220948 shutdown_connections 2024-03-17T23:48:07.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.003+0000 7f745bf1b640 1 --2- 172.21.15.150:0/3931220948 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f743806bbd0 0x7f743806e090 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.003+0000 7f745bf1b640 1 --2- 172.21.15.150:0/3931220948 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f74541e57f0 0x7f74541e5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.003+0000 7f745bf1b640 1 --2- 172.21.15.150:0/3931220948 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f745406eef0 0x7f7454141250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.003+0000 7f745bf1b640 1 --2- 172.21.15.150:0/3931220948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7440004800 0x7f7454155900 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.003+0000 7f745bf1b640 1 -- 172.21.15.150:0/3931220948 >> 172.21.15.150:0/3931220948 conn(0x7f745406b190 msgr2=0x7f745415a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:07.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.003+0000 7f745bf1b640 1 -- 172.21.15.150:0/3931220948 shutdown_connections 2024-03-17T23:48:07.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.003+0000 7f745bf1b640 1 -- 172.21.15.150:0/3931220948 wait complete. 2024-03-17T23:48:07.014 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-17T23:48:07.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.114+0000 7f55676ff640 1 Processor -- start 2024-03-17T23:48:07.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.114+0000 7f55676ff640 1 -- start start 2024-03-17T23:48:07.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.115+0000 7f55676ff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f556006f390 0x7f55600cdf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:07.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.115+0000 7f55676ff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f55600ce440 0x7f5560165b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:07.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.115+0000 7f55676ff640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f556006bf60 con 0x7f5560142520 2024-03-17T23:48:07.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.115+0000 7f55676ff640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f5560142900 con 0x7f55600ce440 2024-03-17T23:48:07.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.115+0000 7f55676ff640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f55600cd900 con 0x7f556006f390 2024-03-17T23:48:07.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.115+0000 7f5565c9c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f55600ce440 0x7f5560165b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:07.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.115+0000 7f5565c9c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f55600ce440 0x7f5560165b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43492/0 (socket says 172.21.15.150:43492) 2024-03-17T23:48:07.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.115+0000 7f5565c9c640 1 -- 172.21.15.150:0/574981932 learned_addr learned my addr 172.21.15.150:0/574981932 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:07.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.115+0000 7f5565c9c640 1 -- 172.21.15.150:0/574981932 >> v2:172.21.15.157:3300/0 conn(0x7f556006f390 msgr2=0x7f55600cdf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:07.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.115+0000 7f5565c9c640 1 --2- 172.21.15.150:0/574981932 >> v2:172.21.15.157:3300/0 conn(0x7f556006f390 0x7f55600cdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.115+0000 7f5565c9c640 1 -- 172.21.15.150:0/574981932 >> v1:172.21.15.157:6789/0 conn(0x7f5560142520 legacy=0x7f55600cc510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:07.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.115+0000 7f5565c9c640 1 -- 172.21.15.150:0/574981932 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55600cd060 con 0x7f55600ce440 2024-03-17T23:48:07.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.116+0000 7f5565c9c640 1 --2- 172.21.15.150:0/574981932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55600ce440 0x7f5560165b60 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5558009870 tx=0x7f555802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15c1119c15d2430f server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:07.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.116+0000 7f5557fff640 1 -- 172.21.15.150:0/574981932 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555802e070 con 0x7f55600ce440 2024-03-17T23:48:07.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.116+0000 7f5557fff640 1 -- 172.21.15.150:0/574981932 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5558037070 con 0x7f55600ce440 2024-03-17T23:48:07.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.116+0000 7f5557fff640 1 -- 172.21.15.150:0/574981932 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5558031a00 con 0x7f55600ce440 2024-03-17T23:48:07.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.117+0000 7f55676ff640 1 -- 172.21.15.150:0/574981932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55600ce440 msgr2=0x7f5560165b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:07.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.117+0000 7f55676ff640 1 --2- 172.21.15.150:0/574981932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55600ce440 0x7f5560165b60 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5558009870 tx=0x7f555802b880 comp rx=0 tx=0).stop 2024-03-17T23:48:07.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.117+0000 7f55676ff640 1 -- 172.21.15.150:0/574981932 shutdown_connections 2024-03-17T23:48:07.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.117+0000 7f55676ff640 1 --2- 172.21.15.150:0/574981932 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55600ce440 0x7f5560165b60 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.117+0000 7f55676ff640 1 --2- 172.21.15.150:0/574981932 >> v2:172.21.15.157:3300/0 conn(0x7f556006f390 0x7f55600cdf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.117+0000 7f55676ff640 1 -- 172.21.15.150:0/574981932 >> 172.21.15.150:0/574981932 conn(0x7f556006b190 msgr2=0x7f556006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:07.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.117+0000 7f55676ff640 1 -- 172.21.15.150:0/574981932 shutdown_connections 2024-03-17T23:48:07.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.117+0000 7f55676ff640 1 -- 172.21.15.150:0/574981932 wait complete. 2024-03-17T23:48:07.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.118+0000 7f55676ff640 1 Processor -- start 2024-03-17T23:48:07.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.118+0000 7f55676ff640 1 -- start start 2024-03-17T23:48:07.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.119+0000 7f55676ff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f556006f390 0x7f5560137c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:07.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.119+0000 7f55676ff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55600ce440 0x7f55600ae480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:07.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.119+0000 7f55676ff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5560142520 0x7f55600aace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:07.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.119+0000 7f55676ff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f556006de60 con 0x7f55600ce440 2024-03-17T23:48:07.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.119+0000 7f55676ff640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f55600cd620 con 0x7f556006f390 2024-03-17T23:48:07.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.119+0000 7f55676ff640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f55600ccc20 con 0x7f5560142520 2024-03-17T23:48:07.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.119+0000 7f5564c9a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55600ce440 0x7f55600ae480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:07.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.119+0000 7f5564c9a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55600ce440 0x7f55600ae480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43498/0 (socket says 172.21.15.150:43498) 2024-03-17T23:48:07.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.119+0000 7f5564c9a640 1 -- 172.21.15.150:0/444864296 learned_addr learned my addr 172.21.15.150:0/444864296 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:07.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.119+0000 7f556549b640 1 --2- 172.21.15.150:0/444864296 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f556006f390 0x7f5560137c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:07.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.119+0000 7f5564c9a640 1 -- 172.21.15.150:0/444864296 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5560142520 msgr2=0x7f55600aace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:07.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.119+0000 7f5565c9c640 1 --2- 172.21.15.150:0/444864296 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5560142520 0x7f55600aace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:07.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.119+0000 7f5564c9a640 1 --2- 172.21.15.150:0/444864296 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5560142520 0x7f55600aace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.119+0000 7f5564c9a640 1 -- 172.21.15.150:0/444864296 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f556006f390 msgr2=0x7f5560137c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:07.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.119+0000 7f5564c9a640 1 --2- 172.21.15.150:0/444864296 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f556006f390 0x7f5560137c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.119+0000 7f5564c9a640 1 -- 172.21.15.150:0/444864296 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5560138150 con 0x7f55600ce440 2024-03-17T23:48:07.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.119+0000 7f556549b640 1 --2- 172.21.15.150:0/444864296 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f556006f390 0x7f5560137c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:07.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.119+0000 7f5565c9c640 1 --2- 172.21.15.150:0/444864296 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5560142520 0x7f55600aace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:07.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.119+0000 7f5564c9a640 1 --2- 172.21.15.150:0/444864296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55600ce440 0x7f55600ae480 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f555000b480 tx=0x7f555000b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:07.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.119+0000 7f5555ffb640 1 -- 172.21.15.150:0/444864296 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5550014070 con 0x7f55600ce440 2024-03-17T23:48:07.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.119+0000 7f5555ffb640 1 -- 172.21.15.150:0/444864296 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5550010870 con 0x7f55600ce440 2024-03-17T23:48:07.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.119+0000 7f55676ff640 1 -- 172.21.15.150:0/444864296 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55600b31f0 con 0x7f55600ce440 2024-03-17T23:48:07.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.119+0000 7f5555ffb640 1 -- 172.21.15.150:0/444864296 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5550010c60 con 0x7f55600ce440 2024-03-17T23:48:07.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.120+0000 7f55676ff640 1 -- 172.21.15.150:0/444864296 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55600b37b0 con 0x7f55600ce440 2024-03-17T23:48:07.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.121+0000 7f55676ff640 1 -- 172.21.15.150:0/444864296 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5560067a40 con 0x7f55600ce440 2024-03-17T23:48:07.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.121+0000 7f5555ffb640 1 -- 172.21.15.150:0/444864296 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5550017070 con 0x7f55600ce440 2024-03-17T23:48:07.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.121+0000 7f5555ffb640 1 --2- 172.21.15.150:0/444864296 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f554406bc00 0x7f554406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:07.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.121+0000 7f5555ffb640 1 -- 172.21.15.150:0/444864296 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5249+0+0 (secure 0 0 0) 0x7f555008a610 con 0x7f55600ce440 2024-03-17T23:48:07.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.122+0000 7f556549b640 1 --2- 172.21.15.150:0/444864296 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f554406bc00 0x7f554406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:07.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.122+0000 7f556549b640 1 --2- 172.21.15.150:0/444864296 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f554406bc00 0x7f554406e0c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f555c007eb0 tx=0x7f555c0073c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:07.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.126+0000 7f5555ffb640 1 -- 172.21.15.150:0/444864296 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5560067a40 con 0x7f55600ce440 2024-03-17T23:48:07.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.364+0000 7f55676ff640 1 -- 172.21.15.150:0/444864296 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7f55600681a0 con 0x7f55600ce440 2024-03-17T23:48:07.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.366+0000 7f5555ffb640 1 -- 172.21.15.150:0/444864296 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) ==== 97+0+0 (secure 0 0 0) 0x7f5550059350 con 0x7f55600ce440 2024-03-17T23:48:07.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.368+0000 7f55676ff640 1 -- 172.21.15.150:0/444864296 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f554406bc00 msgr2=0x7f554406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:07.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.368+0000 7f55676ff640 1 --2- 172.21.15.150:0/444864296 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f554406bc00 0x7f554406e0c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f555c007eb0 tx=0x7f555c0073c0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.368+0000 7f55676ff640 1 -- 172.21.15.150:0/444864296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55600ce440 msgr2=0x7f55600ae480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:07.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.368+0000 7f55676ff640 1 --2- 172.21.15.150:0/444864296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55600ce440 0x7f55600ae480 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f555000b480 tx=0x7f555000b940 comp rx=0 tx=0).stop 2024-03-17T23:48:07.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.368+0000 7f55676ff640 1 -- 172.21.15.150:0/444864296 shutdown_connections 2024-03-17T23:48:07.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.368+0000 7f55676ff640 1 --2- 172.21.15.150:0/444864296 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5560142520 0x7f55600aace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.368+0000 7f55676ff640 1 --2- 172.21.15.150:0/444864296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55600ce440 0x7f55600ae480 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.368+0000 7f55676ff640 1 --2- 172.21.15.150:0/444864296 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f554406bc00 0x7f554406e0c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.368+0000 7f55676ff640 1 --2- 172.21.15.150:0/444864296 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f556006f390 0x7f5560137c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.368+0000 7f55676ff640 1 -- 172.21.15.150:0/444864296 >> 172.21.15.150:0/444864296 conn(0x7f556006b190 msgr2=0x7f55600cc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:07.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.368+0000 7f55676ff640 1 -- 172.21.15.150:0/444864296 shutdown_connections 2024-03-17T23:48:07.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.368+0000 7f55676ff640 1 -- 172.21.15.150:0/444864296 wait complete. 2024-03-17T23:48:07.379 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-17T23:48:07.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.480+0000 7fa3abf1b640 1 Processor -- start 2024-03-17T23:48:07.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.480+0000 7fa3abf1b640 1 -- start start 2024-03-17T23:48:07.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.481+0000 7fa3abf1b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3a40cf100 0x7fa3a415bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:07.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.481+0000 7fa3abf1b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa3a415cac0 0x7fa3a415cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:07.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.481+0000 7fa3abf1b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3a406de60 con 0x7fa3a40ccf90 2024-03-17T23:48:07.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.481+0000 7fa3abf1b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fa3a40cf8a0 con 0x7fa3a415cac0 2024-03-17T23:48:07.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.481+0000 7fa3abf1b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa3a415c220 con 0x7fa3a40cf100 2024-03-17T23:48:07.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.481+0000 7fa3aa4b8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa3a415cac0 0x7fa3a415cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:07.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.481+0000 7fa3a9cb7640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa3a40ccf90 0x7fa3a40cd370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37524/0 (socket says 172.21.15.150:37524) 2024-03-17T23:48:07.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.481+0000 7fa3a9cb7640 1 -- 172.21.15.150:0/2219376817 learned_addr learned my addr 172.21.15.150:0/2219376817 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:07.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.481+0000 7fa3aa4b8640 1 -- 172.21.15.150:0/2219376817 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3a40cf100 msgr2=0x7fa3a415bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:07.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.481+0000 7fa3a94b6640 1 --2- 172.21.15.150:0/2219376817 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3a40cf100 0x7fa3a415bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:07.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.481+0000 7fa3aa4b8640 1 --2- 172.21.15.150:0/2219376817 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3a40cf100 0x7fa3a415bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.481+0000 7fa3aa4b8640 1 -- 172.21.15.150:0/2219376817 >> v1:172.21.15.150:6789/0 conn(0x7fa3a40ccf90 legacy=0x7fa3a40cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:07.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.481+0000 7fa3a8cb5640 1 -- 172.21.15.150:0/2219376817 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 474578660 0 0) 0x7fa3a406de60 con 0x7fa3a40ccf90 2024-03-17T23:48:07.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.481+0000 7fa3aa4b8640 1 -- 172.21.15.150:0/2219376817 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a406e140 con 0x7fa3a415cac0 2024-03-17T23:48:07.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.481+0000 7fa3a94b6640 1 --2- 172.21.15.150:0/2219376817 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3a40cf100 0x7fa3a415bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:07.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.482+0000 7fa3aa4b8640 1 --2- 172.21.15.150:0/2219376817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a415cac0 0x7fa3a415cf50 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa39c0097a0 tx=0x7fa39c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d42a86aa17375542 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:07.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.482+0000 7fa3a8cb5640 1 -- 172.21.15.150:0/2219376817 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa39c0315a0 con 0x7fa3a415cac0 2024-03-17T23:48:07.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.482+0000 7fa3a8cb5640 1 -- 172.21.15.150:0/2219376817 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa39c037070 con 0x7fa3a415cac0 2024-03-17T23:48:07.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.482+0000 7fa3a8cb5640 1 -- 172.21.15.150:0/2219376817 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa39c031960 con 0x7fa3a415cac0 2024-03-17T23:48:07.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.483+0000 7fa3abf1b640 1 -- 172.21.15.150:0/2219376817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a415cac0 msgr2=0x7fa3a415cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:07.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.483+0000 7fa3abf1b640 1 --2- 172.21.15.150:0/2219376817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a415cac0 0x7fa3a415cf50 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa39c0097a0 tx=0x7fa39c02b840 comp rx=0 tx=0).stop 2024-03-17T23:48:07.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.483+0000 7fa3abf1b640 1 -- 172.21.15.150:0/2219376817 shutdown_connections 2024-03-17T23:48:07.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.483+0000 7fa3abf1b640 1 --2- 172.21.15.150:0/2219376817 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3a40cf100 0x7fa3a415bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.483+0000 7fa3abf1b640 1 --2- 172.21.15.150:0/2219376817 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a415cac0 0x7fa3a415cf50 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.483+0000 7fa3abf1b640 1 -- 172.21.15.150:0/2219376817 >> 172.21.15.150:0/2219376817 conn(0x7fa3a406b190 msgr2=0x7fa3a406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:07.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.483+0000 7fa3abf1b640 1 -- 172.21.15.150:0/2219376817 shutdown_connections 2024-03-17T23:48:07.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.483+0000 7fa3abf1b640 1 -- 172.21.15.150:0/2219376817 wait complete. 2024-03-17T23:48:07.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.484+0000 7fa3abf1b640 1 Processor -- start 2024-03-17T23:48:07.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.484+0000 7fa3abf1b640 1 -- start start 2024-03-17T23:48:07.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.485+0000 7fa3abf1b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a40ccf90 0x7fa3a41494e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:07.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.485+0000 7fa3abf1b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3a40cf100 0x7fa3a40b4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:07.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.485+0000 7fa3abf1b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3a415cac0 0x7fa3a414e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:07.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.485+0000 7fa3abf1b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a4155530 con 0x7fa3a40ccf90 2024-03-17T23:48:07.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.485+0000 7fa3abf1b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a40cf580 con 0x7fa3a415cac0 2024-03-17T23:48:07.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.485+0000 7fa3abf1b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa3a41556b0 con 0x7fa3a40cf100 2024-03-17T23:48:07.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.485+0000 7fa3a9cb7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a40ccf90 0x7fa3a41494e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:07.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.485+0000 7fa3a9cb7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a40ccf90 0x7fa3a41494e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43506/0 (socket says 172.21.15.150:43506) 2024-03-17T23:48:07.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.485+0000 7fa3a9cb7640 1 -- 172.21.15.150:0/3323000024 learned_addr learned my addr 172.21.15.150:0/3323000024 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:07.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.485+0000 7fa3a9cb7640 1 -- 172.21.15.150:0/3323000024 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3a40cf100 msgr2=0x7fa3a40b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:07.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.485+0000 7fa3a94b6640 1 --2- 172.21.15.150:0/3323000024 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3a40cf100 0x7fa3a40b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:07.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.485+0000 7fa3aa4b8640 1 --2- 172.21.15.150:0/3323000024 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3a415cac0 0x7fa3a414e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:07.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.485+0000 7fa3a9cb7640 1 --2- 172.21.15.150:0/3323000024 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3a40cf100 0x7fa3a40b4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.485+0000 7fa3a9cb7640 1 -- 172.21.15.150:0/3323000024 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3a415cac0 msgr2=0x7fa3a414e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:07.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.485+0000 7fa3a9cb7640 1 --2- 172.21.15.150:0/3323000024 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3a415cac0 0x7fa3a414e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.485+0000 7fa3a9cb7640 1 -- 172.21.15.150:0/3323000024 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a4149a20 con 0x7fa3a40ccf90 2024-03-17T23:48:07.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.485+0000 7fa3a94b6640 1 --2- 172.21.15.150:0/3323000024 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3a40cf100 0x7fa3a40b4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:07.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.485+0000 7fa3aa4b8640 1 --2- 172.21.15.150:0/3323000024 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3a415cac0 0x7fa3a414e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:07.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.485+0000 7fa3a9cb7640 1 --2- 172.21.15.150:0/3323000024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a40ccf90 0x7fa3a41494e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa3a000c890 tx=0x7fa3a000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:07.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.486+0000 7fa39a7fc640 1 -- 172.21.15.150:0/3323000024 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0018070 con 0x7fa3a40ccf90 2024-03-17T23:48:07.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.486+0000 7fa39a7fc640 1 -- 172.21.15.150:0/3323000024 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3a000a640 con 0x7fa3a40ccf90 2024-03-17T23:48:07.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.486+0000 7fa39a7fc640 1 -- 172.21.15.150:0/3323000024 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a000aa30 con 0x7fa3a40ccf90 2024-03-17T23:48:07.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.486+0000 7fa3abf1b640 1 -- 172.21.15.150:0/3323000024 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3a41558c0 con 0x7fa3a40ccf90 2024-03-17T23:48:07.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.486+0000 7fa3abf1b640 1 -- 172.21.15.150:0/3323000024 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3a414c230 con 0x7fa3a40ccf90 2024-03-17T23:48:07.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.487+0000 7fa3abf1b640 1 -- 172.21.15.150:0/3323000024 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3a40681a0 con 0x7fa3a40ccf90 2024-03-17T23:48:07.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.487+0000 7fa39a7fc640 1 -- 172.21.15.150:0/3323000024 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa3a001b070 con 0x7fa3a40ccf90 2024-03-17T23:48:07.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.488+0000 7fa39a7fc640 1 --2- 172.21.15.150:0/3323000024 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa38806bbd0 0x7fa38806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:07.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.488+0000 7fa3a94b6640 1 --2- 172.21.15.150:0/3323000024 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa38806bbd0 0x7fa38806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:07.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.488+0000 7fa39a7fc640 1 -- 172.21.15.150:0/3323000024 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5249+0+0 (secure 0 0 0) 0x7fa3a0089ff0 con 0x7fa3a40ccf90 2024-03-17T23:48:07.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.488+0000 7fa3a94b6640 1 --2- 172.21.15.150:0/3323000024 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa38806bbd0 0x7fa38806e090 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa394006fd0 tx=0x7fa394008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:07.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.492+0000 7fa39a7fc640 1 -- 172.21.15.150:0/3323000024 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa3a0058c10 con 0x7fa3a40ccf90 2024-03-17T23:48:07.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.731+0000 7fa3abf1b640 1 -- 172.21.15.150:0/3323000024 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7fa3a406fff0 con 0x7fa3a40ccf90 2024-03-17T23:48:07.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.733+0000 7fa39a7fc640 1 -- 172.21.15.150:0/3323000024 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) ==== 90+0+0 (secure 0 0 0) 0x7fa3a0058510 con 0x7fa3a40ccf90 2024-03-17T23:48:07.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.735+0000 7fa3abf1b640 1 -- 172.21.15.150:0/3323000024 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa38806bbd0 msgr2=0x7fa38806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:07.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.735+0000 7fa3abf1b640 1 --2- 172.21.15.150:0/3323000024 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa38806bbd0 0x7fa38806e090 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa394006fd0 tx=0x7fa394008040 comp rx=0 tx=0).stop 2024-03-17T23:48:07.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.735+0000 7fa3abf1b640 1 -- 172.21.15.150:0/3323000024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a40ccf90 msgr2=0x7fa3a41494e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:07.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.735+0000 7fa3abf1b640 1 --2- 172.21.15.150:0/3323000024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a40ccf90 0x7fa3a41494e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa3a000c890 tx=0x7fa3a000cd50 comp rx=0 tx=0).stop 2024-03-17T23:48:07.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.735+0000 7fa3abf1b640 1 -- 172.21.15.150:0/3323000024 shutdown_connections 2024-03-17T23:48:07.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.735+0000 7fa3abf1b640 1 --2- 172.21.15.150:0/3323000024 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa38806bbd0 0x7fa38806e090 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.735+0000 7fa3abf1b640 1 --2- 172.21.15.150:0/3323000024 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3a415cac0 0x7fa3a414e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.735+0000 7fa3abf1b640 1 --2- 172.21.15.150:0/3323000024 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa3a40cf100 0x7fa3a40b4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.735+0000 7fa3abf1b640 1 --2- 172.21.15.150:0/3323000024 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa3a40ccf90 0x7fa3a41494e0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.735+0000 7fa3abf1b640 1 -- 172.21.15.150:0/3323000024 >> 172.21.15.150:0/3323000024 conn(0x7fa3a406b190 msgr2=0x7fa3a406ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:07.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.735+0000 7fa3abf1b640 1 -- 172.21.15.150:0/3323000024 shutdown_connections 2024-03-17T23:48:07.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.735+0000 7fa3abf1b640 1 -- 172.21.15.150:0/3323000024 wait complete. 2024-03-17T23:48:07.747 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:48:07.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.846+0000 7f80316c6640 1 Processor -- start 2024-03-17T23:48:07.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.846+0000 7f80316c6640 1 -- start start 2024-03-17T23:48:07.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.847+0000 7f80316c6640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f802c0b5c40 0x7f802c0b7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:07.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.847+0000 7f80316c6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f802c0d2890 con 0x7f802c15e060 2024-03-17T23:48:07.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.847+0000 7f80316c6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f802c0d2b70 con 0x7f802c0b57d0 2024-03-17T23:48:07.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.847+0000 7f80316c6640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f802c0d2630 con 0x7f802c0b5c40 2024-03-17T23:48:07.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.847+0000 7f802affd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f802c15e060 0x7f802c0d1f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37526/0 (socket says 172.21.15.150:37526) 2024-03-17T23:48:07.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.847+0000 7f802affd640 1 -- 172.21.15.150:0/3706432741 learned_addr learned my addr 172.21.15.150:0/3706432741 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:07.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.848+0000 7f8029ffb640 1 -- 172.21.15.150:0/3706432741 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3922534492 0 0) 0x7f802c0d2890 con 0x7f802c15e060 2024-03-17T23:48:07.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.848+0000 7f802b7fe640 1 --2- 172.21.15.150:0/3706432741 >> v2:172.21.15.157:3300/0 conn(0x7f802c0b5c40 0x7f802c0b7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:07.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.848+0000 7f8029ffb640 1 -- 172.21.15.150:0/3706432741 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8014003520 con 0x7f802c15e060 2024-03-17T23:48:07.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.848+0000 7f8029ffb640 1 -- 172.21.15.150:0/3706432741 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1065786305 0 0) 0x7f8014003520 con 0x7f802c15e060 2024-03-17T23:48:07.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.848+0000 7f8029ffb640 1 -- 172.21.15.150:0/3706432741 >> v2:172.21.15.157:3300/0 conn(0x7f802c0b5c40 msgr2=0x7f802c0b7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:07.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.848+0000 7f8029ffb640 1 --2- 172.21.15.150:0/3706432741 >> v2:172.21.15.157:3300/0 conn(0x7f802c0b5c40 0x7f802c0b7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.848+0000 7f8029ffb640 1 -- 172.21.15.150:0/3706432741 >> v1:172.21.15.157:6789/0 conn(0x7f802c0b57d0 legacy=0x7f802c0b6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:07.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.848+0000 7f8029ffb640 1 -- 172.21.15.150:0/3706432741 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f802c0d2e50 con 0x7f802c15e060 2024-03-17T23:48:07.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.848+0000 7f8029ffb640 1 -- 172.21.15.150:0/3706432741 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f801c0019f0 con 0x7f802c15e060 2024-03-17T23:48:07.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.848+0000 7f8029ffb640 1 -- 172.21.15.150:0/3706432741 >> v1:172.21.15.150:6789/0 conn(0x7f802c15e060 legacy=0x7f802c0d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:07.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.848+0000 7f8029ffb640 1 --2- 172.21.15.150:0/3706432741 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80140039b0 0x7f8014003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:07.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.848+0000 7f8029ffb640 1 --2- 172.21.15.150:0/3706432741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8014004800 0x7f8014004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:07.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.848+0000 7f8029ffb640 1 --2- 172.21.15.150:0/3706432741 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f80140051c0 0x7f801400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:07.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.848+0000 7f8029ffb640 1 -- 172.21.15.150:0/3706432741 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f802c0d2630 con 0x7f8014004800 2024-03-17T23:48:07.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.848+0000 7f8029ffb640 1 -- 172.21.15.150:0/3706432741 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f801400c160 con 0x7f80140051c0 2024-03-17T23:48:07.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.848+0000 7f8029ffb640 1 -- 172.21.15.150:0/3706432741 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f801400c360 con 0x7f80140039b0 2024-03-17T23:48:07.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.848+0000 7f802a7fc640 1 --2- 172.21.15.150:0/3706432741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8014004800 0x7f8014004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:07.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.848+0000 7f802affd640 1 --2- 172.21.15.150:0/3706432741 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80140039b0 0x7f8014003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:07.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.848+0000 7f802b7fe640 1 --2- 172.21.15.150:0/3706432741 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f80140051c0 0x7f801400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:07.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.848+0000 7f802a7fc640 1 -- 172.21.15.150:0/3706432741 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80140039b0 msgr2=0x7f8014003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:07.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.848+0000 7f802a7fc640 1 --2- 172.21.15.150:0/3706432741 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f80140039b0 0x7f8014003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.848+0000 7f802a7fc640 1 -- 172.21.15.150:0/3706432741 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f80140051c0 msgr2=0x7f801400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:07.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.848+0000 7f802a7fc640 1 --2- 172.21.15.150:0/3706432741 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f80140051c0 0x7f801400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.848+0000 7f802b7fe640 1 -- 172.21.15.150:0/3706432741 reap_dead start 2024-03-17T23:48:07.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.848+0000 7f802a7fc640 1 -- 172.21.15.150:0/3706432741 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8014003520 con 0x7f8014004800 2024-03-17T23:48:07.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.849+0000 7f802a7fc640 1 --2- 172.21.15.150:0/3706432741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8014004800 0x7f8014004c80 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f8020001750 tx=0x7f8014005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=480dc85661817d03 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:07.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.849+0000 7f8029ffb640 1 -- 172.21.15.150:0/3706432741 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8020001370 con 0x7f8014004800 2024-03-17T23:48:07.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.849+0000 7f8029ffb640 1 -- 172.21.15.150:0/3706432741 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80200052d0 con 0x7f8014004800 2024-03-17T23:48:07.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.849+0000 7f8029ffb640 1 -- 172.21.15.150:0/3706432741 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8020003830 con 0x7f8014004800 2024-03-17T23:48:07.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.850+0000 7f80316c6640 1 -- 172.21.15.150:0/3706432741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8014004800 msgr2=0x7f8014004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:07.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.850+0000 7f80316c6640 1 --2- 172.21.15.150:0/3706432741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8014004800 0x7f8014004c80 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f8020001750 tx=0x7f8014005830 comp rx=0 tx=0).stop 2024-03-17T23:48:07.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.850+0000 7f80316c6640 1 -- 172.21.15.150:0/3706432741 shutdown_connections 2024-03-17T23:48:07.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.850+0000 7f80316c6640 1 --2- 172.21.15.150:0/3706432741 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8014004800 0x7f8014004c80 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.850+0000 7f80316c6640 1 -- 172.21.15.150:0/3706432741 >> 172.21.15.150:0/3706432741 conn(0x7f802c073200 msgr2=0x7f802c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:07.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.850+0000 7f80316c6640 1 -- 172.21.15.150:0/3706432741 shutdown_connections 2024-03-17T23:48:07.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.850+0000 7f80316c6640 1 -- 172.21.15.150:0/3706432741 wait complete. 2024-03-17T23:48:07.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.851+0000 7f80316c6640 1 Processor -- start 2024-03-17T23:48:07.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.851+0000 7f80316c6640 1 -- start start 2024-03-17T23:48:07.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.852+0000 7f80316c6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8014004800 0x7f802c151050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:07.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.852+0000 7f80316c6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f802c0c8b60 0x7f802c0b7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:07.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.852+0000 7f80316c6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f802c0c8fe0 0x7f802c0b7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:07.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.852+0000 7f80316c6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f802c0b60b0 con 0x7f802c0c8fe0 2024-03-17T23:48:07.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.852+0000 7f80316c6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f802c0d3410 con 0x7f802c0c8b60 2024-03-17T23:48:07.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.852+0000 7f80316c6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f802c15e440 con 0x7f8014004800 2024-03-17T23:48:07.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.852+0000 7f802b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f802c0c8fe0 0x7f802c0b7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:07.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.852+0000 7f802b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f802c0c8fe0 0x7f802c0b7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43520/0 (socket says 172.21.15.150:43520) 2024-03-17T23:48:07.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.852+0000 7f802b7fe640 1 -- 172.21.15.150:0/2937414007 learned_addr learned my addr 172.21.15.150:0/2937414007 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:07.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.852+0000 7f802affd640 1 --2- 172.21.15.150:0/2937414007 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8014004800 0x7f802c151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:07.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.852+0000 7f802a7fc640 1 --2- 172.21.15.150:0/2937414007 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f802c0c8b60 0x7f802c0b7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:07.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.852+0000 7f802b7fe640 1 -- 172.21.15.150:0/2937414007 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8014004800 msgr2=0x7f802c151050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:07.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.852+0000 7f802b7fe640 1 --2- 172.21.15.150:0/2937414007 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8014004800 0x7f802c151050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.852+0000 7f802b7fe640 1 -- 172.21.15.150:0/2937414007 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f802c0c8b60 msgr2=0x7f802c0b7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:07.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.852+0000 7f802b7fe640 1 --2- 172.21.15.150:0/2937414007 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f802c0c8b60 0x7f802c0b7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:07.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.852+0000 7f802b7fe640 1 -- 172.21.15.150:0/2937414007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f802c0c9540 con 0x7f802c0c8fe0 2024-03-17T23:48:07.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.852+0000 7f802b7fe640 1 --2- 172.21.15.150:0/2937414007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f802c0c8fe0 0x7f802c0b7c90 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f802c0cf6c0 tx=0x7f8024011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:07.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.853+0000 7f80137fe640 1 -- 172.21.15.150:0/2937414007 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8024004030 con 0x7f802c0c8fe0 2024-03-17T23:48:07.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.853+0000 7f80137fe640 1 -- 172.21.15.150:0/2937414007 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f802401b070 con 0x7f802c0c8fe0 2024-03-17T23:48:07.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.853+0000 7f80316c6640 1 -- 172.21.15.150:0/2937414007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f802c0c9820 con 0x7f802c0c8fe0 2024-03-17T23:48:07.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.853+0000 7f80137fe640 1 -- 172.21.15.150:0/2937414007 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8024004430 con 0x7f802c0c8fe0 2024-03-17T23:48:07.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.853+0000 7f80316c6640 1 -- 172.21.15.150:0/2937414007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f802c0c9de0 con 0x7f802c0c8fe0 2024-03-17T23:48:07.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.854+0000 7f80316c6640 1 -- 172.21.15.150:0/2937414007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f802c0688b0 con 0x7f802c0c8fe0 2024-03-17T23:48:07.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.854+0000 7f80137fe640 1 -- 172.21.15.150:0/2937414007 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8024012040 con 0x7f802c0c8fe0 2024-03-17T23:48:07.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.854+0000 7f80137fe640 1 --2- 172.21.15.150:0/2937414007 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f801806bc00 0x7f801806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:07.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.854+0000 7f80137fe640 1 -- 172.21.15.150:0/2937414007 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5249+0+0 (secure 0 0 0) 0x7f802408a7b0 con 0x7f802c0c8fe0 2024-03-17T23:48:07.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.854+0000 7f802affd640 1 --2- 172.21.15.150:0/2937414007 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f801806bc00 0x7f801806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:07.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.855+0000 7f802affd640 1 --2- 172.21.15.150:0/2937414007 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f801806bc00 0x7f801806e0c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f801c001b50 tx=0x7f801c006040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:07.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:07.859+0000 7f80137fe640 1 -- 172.21.15.150:0/2937414007 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f802408ce20 con 0x7f802c0c8fe0 2024-03-17T23:48:08.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.033+0000 7f80316c6640 1 -- 172.21.15.150:0/2937414007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f802c0b9160 con 0x7f802c0c8fe0 2024-03-17T23:48:08.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.034+0000 7f80137fe640 1 -- 172.21.15.150:0/2937414007 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) ==== 74+0+13200 (secure 0 0 0) 0x7f8024059460 con 0x7f802c0c8fe0 2024-03-17T23:48:08.038 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:08.038 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":36,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:48:05.149206+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:48:08.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.036+0000 7f80316c6640 1 -- 172.21.15.150:0/2937414007 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f801806bc00 msgr2=0x7f801806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:08.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.036+0000 7f80316c6640 1 --2- 172.21.15.150:0/2937414007 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f801806bc00 0x7f801806e0c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f801c001b50 tx=0x7f801c006040 comp rx=0 tx=0).stop 2024-03-17T23:48:08.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.036+0000 7f80316c6640 1 -- 172.21.15.150:0/2937414007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f802c0c8fe0 msgr2=0x7f802c0b7c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:08.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.036+0000 7f80316c6640 1 --2- 172.21.15.150:0/2937414007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f802c0c8fe0 0x7f802c0b7c90 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f802c0cf6c0 tx=0x7f8024011cf0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.036+0000 7f80316c6640 1 -- 172.21.15.150:0/2937414007 shutdown_connections 2024-03-17T23:48:08.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.036+0000 7f80316c6640 1 --2- 172.21.15.150:0/2937414007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f802c0c8fe0 0x7f802c0b7c90 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.036+0000 7f80316c6640 1 --2- 172.21.15.150:0/2937414007 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f801806bc00 0x7f801806e0c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.036+0000 7f80316c6640 1 --2- 172.21.15.150:0/2937414007 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f802c0c8b60 0x7f802c0b7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.036+0000 7f80316c6640 1 --2- 172.21.15.150:0/2937414007 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8014004800 0x7f802c151050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.036+0000 7f80316c6640 1 -- 172.21.15.150:0/2937414007 >> 172.21.15.150:0/2937414007 conn(0x7f802c073200 msgr2=0x7f802c0b3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:08.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.036+0000 7f80316c6640 1 -- 172.21.15.150:0/2937414007 shutdown_connections 2024-03-17T23:48:08.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.036+0000 7f80316c6640 1 -- 172.21.15.150:0/2937414007 wait complete. 2024-03-17T23:48:08.048 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-17T23:48:08.048 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:48:08.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.152+0000 7f12ad20c640 1 Processor -- start 2024-03-17T23:48:08.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.152+0000 7f12ad20c640 1 -- start start 2024-03-17T23:48:08.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.152+0000 7f12ad20c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12a80d0e30 0x7f12a8168bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:08.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.152+0000 7f12ad20c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12a80d2870 con 0x7f12a81615d0 2024-03-17T23:48:08.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.152+0000 7f12ad20c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12a80d2b50 con 0x7f12a80d3910 2024-03-17T23:48:08.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.152+0000 7f12ad20c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f12a81619b0 con 0x7f12a80d0e30 2024-03-17T23:48:08.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.153+0000 7f12a6d9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f12a81615d0 0x7f12a80d2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37534/0 (socket says 172.21.15.150:37534) 2024-03-17T23:48:08.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.153+0000 7f12a6d9d640 1 -- 172.21.15.150:0/1203935250 learned_addr learned my addr 172.21.15.150:0/1203935250 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:08.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.153+0000 7f12a759e640 1 --2- 172.21.15.150:0/1203935250 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12a80d0e30 0x7f12a8168bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:08.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.153+0000 7f12a659c640 1 -- 172.21.15.150:0/1203935250 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3219683529 0 0) 0x7f12a80d2870 con 0x7f12a81615d0 2024-03-17T23:48:08.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.153+0000 7f12a659c640 1 -- 172.21.15.150:0/1203935250 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1294003520 con 0x7f12a81615d0 2024-03-17T23:48:08.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.153+0000 7f12a759e640 1 -- 172.21.15.150:0/1203935250 >> v1:172.21.15.157:6789/0 conn(0x7f12a80d3910 legacy=0x7f12a80d0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:08.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.153+0000 7f12a759e640 1 -- 172.21.15.150:0/1203935250 >> v1:172.21.15.150:6789/0 conn(0x7f12a81615d0 legacy=0x7f12a80d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:08.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.153+0000 7f12a659c640 1 -- 172.21.15.150:0/1203935250 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 330946512 0 0) 0x7f12a80d2b50 con 0x7f12a80d3910 2024-03-17T23:48:08.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.153+0000 7f12a759e640 1 -- 172.21.15.150:0/1203935250 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12a80d2e30 con 0x7f12a80d0e30 2024-03-17T23:48:08.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.154+0000 7f12a759e640 1 --2- 172.21.15.150:0/1203935250 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12a80d0e30 0x7f12a8168bf0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f12a0009dc0 tx=0x7f12a002bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9dc3681698fd85d server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:08.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.154+0000 7f12a659c640 1 -- 172.21.15.150:0/1203935250 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a0039070 con 0x7f12a80d0e30 2024-03-17T23:48:08.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.154+0000 7f12a659c640 1 -- 172.21.15.150:0/1203935250 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12a002ccd0 con 0x7f12a80d0e30 2024-03-17T23:48:08.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.154+0000 7f12a659c640 1 -- 172.21.15.150:0/1203935250 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a0034640 con 0x7f12a80d0e30 2024-03-17T23:48:08.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.154+0000 7f12ad20c640 1 -- 172.21.15.150:0/1203935250 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12a80d0e30 msgr2=0x7f12a8168bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:08.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.154+0000 7f12ad20c640 1 --2- 172.21.15.150:0/1203935250 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12a80d0e30 0x7f12a8168bf0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f12a0009dc0 tx=0x7f12a002bf20 comp rx=0 tx=0).stop 2024-03-17T23:48:08.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.154+0000 7f12ad20c640 1 -- 172.21.15.150:0/1203935250 shutdown_connections 2024-03-17T23:48:08.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.154+0000 7f12ad20c640 1 --2- 172.21.15.150:0/1203935250 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12a80d0e30 0x7f12a8168bf0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.154+0000 7f12ad20c640 1 -- 172.21.15.150:0/1203935250 >> 172.21.15.150:0/1203935250 conn(0x7f12a80730b0 msgr2=0x7f12a80b5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:08.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.154+0000 7f12ad20c640 1 -- 172.21.15.150:0/1203935250 shutdown_connections 2024-03-17T23:48:08.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.154+0000 7f12ad20c640 1 -- 172.21.15.150:0/1203935250 wait complete. 2024-03-17T23:48:08.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.156+0000 7f12ad20c640 1 Processor -- start 2024-03-17T23:48:08.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.156+0000 7f12ad20c640 1 -- start start 2024-03-17T23:48:08.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.156+0000 7f12ad20c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f12a80d0e30 0x7f12a80b86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:08.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.156+0000 7f12ad20c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12a80d3910 0x7f12a81f3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:08.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.156+0000 7f12ad20c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12a81615d0 0x7f12a8160500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:08.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.156+0000 7f12ad20c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f12a80d33f0 con 0x7f12a80d3910 2024-03-17T23:48:08.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.156+0000 7f12ad20c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f12a80b7e80 con 0x7f12a80d0e30 2024-03-17T23:48:08.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.156+0000 7f12ad20c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f12a80b9660 con 0x7f12a81615d0 2024-03-17T23:48:08.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.157+0000 7f129ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12a80d3910 0x7f12a81f3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:08.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.157+0000 7f129ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12a80d3910 0x7f12a81f3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43526/0 (socket says 172.21.15.150:43526) 2024-03-17T23:48:08.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.157+0000 7f129ffff640 1 -- 172.21.15.150:0/1980312057 learned_addr learned my addr 172.21.15.150:0/1980312057 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:08.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.157+0000 7f12a759e640 1 --2- 172.21.15.150:0/1980312057 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12a81615d0 0x7f12a8160500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:08.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.157+0000 7f12a6d9d640 1 --2- 172.21.15.150:0/1980312057 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f12a80d0e30 0x7f12a80b86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:08.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.157+0000 7f129ffff640 1 -- 172.21.15.150:0/1980312057 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12a81615d0 msgr2=0x7f12a8160500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:08.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.157+0000 7f129ffff640 1 --2- 172.21.15.150:0/1980312057 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12a81615d0 0x7f12a8160500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.157+0000 7f129ffff640 1 -- 172.21.15.150:0/1980312057 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f12a80d0e30 msgr2=0x7f12a80b86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:08.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.157+0000 7f129ffff640 1 --2- 172.21.15.150:0/1980312057 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f12a80d0e30 0x7f12a80b86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.157+0000 7f129ffff640 1 -- 172.21.15.150:0/1980312057 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12a80abfa0 con 0x7f12a80d3910 2024-03-17T23:48:08.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.157+0000 7f12a6d9d640 1 --2- 172.21.15.150:0/1980312057 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f12a80d0e30 0x7f12a80b86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:08.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.157+0000 7f129ffff640 1 --2- 172.21.15.150:0/1980312057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12a80d3910 0x7f12a81f3ef0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f129000c4b0 tx=0x7f129000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:08.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.157+0000 7f129f7fe640 1 -- 172.21.15.150:0/1980312057 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1290017070 con 0x7f12a80d3910 2024-03-17T23:48:08.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.157+0000 7f129f7fe640 1 -- 172.21.15.150:0/1980312057 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1290005480 con 0x7f12a80d3910 2024-03-17T23:48:08.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.157+0000 7f12ad20c640 1 -- 172.21.15.150:0/1980312057 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12a80ab9e0 con 0x7f12a80d3910 2024-03-17T23:48:08.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.157+0000 7f129f7fe640 1 -- 172.21.15.150:0/1980312057 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1290003830 con 0x7f12a80d3910 2024-03-17T23:48:08.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.157+0000 7f12ad20c640 1 -- 172.21.15.150:0/1980312057 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12a81f36f0 con 0x7f12a80d3910 2024-03-17T23:48:08.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.158+0000 7f12ad20c640 1 -- 172.21.15.150:0/1980312057 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12a80688b0 con 0x7f12a80d3910 2024-03-17T23:48:08.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.159+0000 7f129f7fe640 1 -- 172.21.15.150:0/1980312057 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f129001a070 con 0x7f12a80d3910 2024-03-17T23:48:08.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.159+0000 7f129f7fe640 1 --2- 172.21.15.150:0/1980312057 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f128806bbd0 0x7f128806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:08.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.159+0000 7f12a6d9d640 1 --2- 172.21.15.150:0/1980312057 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f128806bbd0 0x7f128806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:08.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.159+0000 7f129f7fe640 1 -- 172.21.15.150:0/1980312057 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5249+0+0 (secure 0 0 0) 0x7f1290089b00 con 0x7f12a80d3910 2024-03-17T23:48:08.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.160+0000 7f12a6d9d640 1 --2- 172.21.15.150:0/1980312057 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f128806bbd0 0x7f128806e090 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1298007fd0 tx=0x7f1298009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:08.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.163+0000 7f129f7fe640 1 -- 172.21.15.150:0/1980312057 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f12900581f0 con 0x7f12a80d3910 2024-03-17T23:48:08.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.336+0000 7f12ad20c640 1 -- 172.21.15.150:0/1980312057 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f12a81f3b90 con 0x7f12a80d3910 2024-03-17T23:48:08.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.336+0000 7f129f7fe640 1 -- 172.21.15.150:0/1980312057 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) ==== 93+0+10 (secure 0 0 0) 0x7f129005e030 con 0x7f12a80d3910 2024-03-17T23:48:08.337 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:48:08.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.338+0000 7f12ad20c640 1 -- 172.21.15.150:0/1980312057 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f128806bbd0 msgr2=0x7f128806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:08.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.338+0000 7f12ad20c640 1 --2- 172.21.15.150:0/1980312057 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f128806bbd0 0x7f128806e090 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1298007fd0 tx=0x7f1298009040 comp rx=0 tx=0).stop 2024-03-17T23:48:08.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.338+0000 7f12ad20c640 1 -- 172.21.15.150:0/1980312057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12a80d3910 msgr2=0x7f12a81f3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:08.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.338+0000 7f12ad20c640 1 --2- 172.21.15.150:0/1980312057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12a80d3910 0x7f12a81f3ef0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f129000c4b0 tx=0x7f129000c970 comp rx=0 tx=0).stop 2024-03-17T23:48:08.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.338+0000 7f12ad20c640 1 -- 172.21.15.150:0/1980312057 shutdown_connections 2024-03-17T23:48:08.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.338+0000 7f12ad20c640 1 --2- 172.21.15.150:0/1980312057 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f12a81615d0 0x7f12a8160500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.338+0000 7f12ad20c640 1 --2- 172.21.15.150:0/1980312057 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f12a80d3910 0x7f12a81f3ef0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.338+0000 7f12ad20c640 1 --2- 172.21.15.150:0/1980312057 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f128806bbd0 0x7f128806e090 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.338+0000 7f12ad20c640 1 --2- 172.21.15.150:0/1980312057 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f12a80d0e30 0x7f12a80b86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.338+0000 7f12ad20c640 1 -- 172.21.15.150:0/1980312057 >> 172.21.15.150:0/1980312057 conn(0x7f12a80730b0 msgr2=0x7f12a806c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:08.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.338+0000 7f12ad20c640 1 -- 172.21.15.150:0/1980312057 shutdown_connections 2024-03-17T23:48:08.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.338+0000 7f12ad20c640 1 -- 172.21.15.150:0/1980312057 wait complete. 2024-03-17T23:48:08.350 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-17T23:48:08.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.450+0000 7ff2943c4640 1 Processor -- start 2024-03-17T23:48:08.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.450+0000 7ff2943c4640 1 -- start start 2024-03-17T23:48:08.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.451+0000 7ff2943c4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff28c0b5730 0x7ff28c0b2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:08.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.451+0000 7ff2943c4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff28c0b3c30 0x7ff28c0b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:08.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.451+0000 7ff2943c4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff28c0d1230 con 0x7ff28c06b190 2024-03-17T23:48:08.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.451+0000 7ff2943c4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7ff28c0b2560 con 0x7ff28c0b3c30 2024-03-17T23:48:08.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.451+0000 7ff2943c4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff28c0b5c10 con 0x7ff28c0b5730 2024-03-17T23:48:08.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.451+0000 7ff292961640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff28c0b3c30 0x7ff28c0b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:08.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.451+0000 7ff292160640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7ff28c06b190 0x7ff28c06b570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37542/0 (socket says 172.21.15.150:37542) 2024-03-17T23:48:08.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.451+0000 7ff292160640 1 -- 172.21.15.150:0/3497780113 learned_addr learned my addr 172.21.15.150:0/3497780113 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:08.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.452+0000 7ff292961640 1 -- 172.21.15.150:0/3497780113 >> v2:172.21.15.157:3300/0 conn(0x7ff28c0b5730 msgr2=0x7ff28c0b2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:08.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.452+0000 7ff292961640 1 --2- 172.21.15.150:0/3497780113 >> v2:172.21.15.157:3300/0 conn(0x7ff28c0b5730 0x7ff28c0b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.452+0000 7ff292961640 1 -- 172.21.15.150:0/3497780113 >> v1:172.21.15.150:6789/0 conn(0x7ff28c06b190 legacy=0x7ff28c06b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:08.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.452+0000 7ff292961640 1 -- 172.21.15.150:0/3497780113 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff28c0d14d0 con 0x7ff28c0b3c30 2024-03-17T23:48:08.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.452+0000 7ff29115e640 1 -- 172.21.15.150:0/3497780113 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3036498033 0 0) 0x7ff28c0d1230 con 0x7ff28c06b190 2024-03-17T23:48:08.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.452+0000 7ff292961640 1 --2- 172.21.15.150:0/3497780113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff28c0b3c30 0x7ff28c0b01a0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7ff28c0d1b60 tx=0x7ff28402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=275c398592189cc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:08.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.452+0000 7ff29115e640 1 -- 172.21.15.150:0/3497780113 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2840315a0 con 0x7ff28c0b3c30 2024-03-17T23:48:08.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.452+0000 7ff29115e640 1 -- 172.21.15.150:0/3497780113 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff284037070 con 0x7ff28c0b3c30 2024-03-17T23:48:08.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.453+0000 7ff29115e640 1 -- 172.21.15.150:0/3497780113 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff284031960 con 0x7ff28c0b3c30 2024-03-17T23:48:08.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.453+0000 7ff2943c4640 1 -- 172.21.15.150:0/3497780113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff28c0b3c30 msgr2=0x7ff28c0b01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:08.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.453+0000 7ff2943c4640 1 --2- 172.21.15.150:0/3497780113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff28c0b3c30 0x7ff28c0b01a0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7ff28c0d1b60 tx=0x7ff28402b840 comp rx=0 tx=0).stop 2024-03-17T23:48:08.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.453+0000 7ff2943c4640 1 -- 172.21.15.150:0/3497780113 shutdown_connections 2024-03-17T23:48:08.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.453+0000 7ff2943c4640 1 --2- 172.21.15.150:0/3497780113 >> v2:172.21.15.157:3300/0 conn(0x7ff28c0b5730 0x7ff28c0b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.453+0000 7ff2943c4640 1 --2- 172.21.15.150:0/3497780113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff28c0b3c30 0x7ff28c0b01a0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.453+0000 7ff2943c4640 1 -- 172.21.15.150:0/3497780113 >> 172.21.15.150:0/3497780113 conn(0x7ff28c0b6170 msgr2=0x7ff28c099a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:08.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.453+0000 7ff2943c4640 1 -- 172.21.15.150:0/3497780113 shutdown_connections 2024-03-17T23:48:08.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.453+0000 7ff2943c4640 1 -- 172.21.15.150:0/3497780113 wait complete. 2024-03-17T23:48:08.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.454+0000 7ff2943c4640 1 Processor -- start 2024-03-17T23:48:08.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.454+0000 7ff2943c4640 1 -- start start 2024-03-17T23:48:08.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.455+0000 7ff2943c4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff28c06b190 0x7ff28c14e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:08.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.455+0000 7ff2943c4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff28c0b3c30 0x7ff28c09e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:08.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.455+0000 7ff2943c4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff28c0b5730 0x7ff28c0ca720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:08.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.455+0000 7ff2943c4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c09a380 con 0x7ff28c0b5730 2024-03-17T23:48:08.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.455+0000 7ff2943c4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c0b5ef0 con 0x7ff28c0b3c30 2024-03-17T23:48:08.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.455+0000 7ff2943c4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff28c15e260 con 0x7ff28c06b190 2024-03-17T23:48:08.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.455+0000 7ff292961640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff28c0b5730 0x7ff28c0ca720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:08.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.455+0000 7ff292961640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff28c0b5730 0x7ff28c0ca720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43542/0 (socket says 172.21.15.150:43542) 2024-03-17T23:48:08.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.455+0000 7ff292961640 1 -- 172.21.15.150:0/4260454578 learned_addr learned my addr 172.21.15.150:0/4260454578 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:08.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.455+0000 7ff292160640 1 --2- 172.21.15.150:0/4260454578 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff28c06b190 0x7ff28c14e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:08.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.455+0000 7ff29195f640 1 --2- 172.21.15.150:0/4260454578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff28c0b3c30 0x7ff28c09e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:08.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.455+0000 7ff292961640 1 -- 172.21.15.150:0/4260454578 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff28c06b190 msgr2=0x7ff28c14e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:08.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.455+0000 7ff292961640 1 --2- 172.21.15.150:0/4260454578 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff28c06b190 0x7ff28c14e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.455+0000 7ff292961640 1 -- 172.21.15.150:0/4260454578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff28c0b3c30 msgr2=0x7ff28c09e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:08.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.455+0000 7ff292961640 1 --2- 172.21.15.150:0/4260454578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff28c0b3c30 0x7ff28c09e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.455+0000 7ff292961640 1 -- 172.21.15.150:0/4260454578 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff28c15ad90 con 0x7ff28c0b5730 2024-03-17T23:48:08.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.456+0000 7ff292961640 1 --2- 172.21.15.150:0/4260454578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff28c0b5730 0x7ff28c0ca720 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ff284009890 tx=0x7ff2840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:08.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.456+0000 7ff282ffd640 1 -- 172.21.15.150:0/4260454578 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff284004230 con 0x7ff28c0b5730 2024-03-17T23:48:08.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.456+0000 7ff282ffd640 1 -- 172.21.15.150:0/4260454578 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2840043d0 con 0x7ff28c0b5730 2024-03-17T23:48:08.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.456+0000 7ff2943c4640 1 -- 172.21.15.150:0/4260454578 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff28c09d560 con 0x7ff28c0b5730 2024-03-17T23:48:08.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.456+0000 7ff282ffd640 1 -- 172.21.15.150:0/4260454578 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff284003830 con 0x7ff28c0b5730 2024-03-17T23:48:08.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.456+0000 7ff2943c4640 1 -- 172.21.15.150:0/4260454578 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff28c09db20 con 0x7ff28c0b5730 2024-03-17T23:48:08.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.457+0000 7ff2943c4640 1 -- 172.21.15.150:0/4260454578 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff28c0681a0 con 0x7ff28c0b5730 2024-03-17T23:48:08.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.457+0000 7ff282ffd640 1 -- 172.21.15.150:0/4260454578 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff2840039d0 con 0x7ff28c0b5730 2024-03-17T23:48:08.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.457+0000 7ff282ffd640 1 --2- 172.21.15.150:0/4260454578 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff27006bbd0 0x7ff27006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:08.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.457+0000 7ff282ffd640 1 -- 172.21.15.150:0/4260454578 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5249+0+0 (secure 0 0 0) 0x7ff28402f030 con 0x7ff28c0b5730 2024-03-17T23:48:08.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.457+0000 7ff292160640 1 --2- 172.21.15.150:0/4260454578 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff27006bbd0 0x7ff27006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:08.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.458+0000 7ff292160640 1 --2- 172.21.15.150:0/4260454578 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff27006bbd0 0x7ff27006e090 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff2880012e0 tx=0x7ff288009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:08.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.462+0000 7ff282ffd640 1 -- 172.21.15.150:0/4260454578 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff28c15ad90 con 0x7ff28c0b5730 2024-03-17T23:48:08.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.637+0000 7ff2943c4640 1 -- 172.21.15.150:0/4260454578 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7ff28c160e70 con 0x7ff28c0b5730 2024-03-17T23:48:08.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.637+0000 7ff282ffd640 1 -- 172.21.15.150:0/4260454578 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v36) ==== 84+0+65 (secure 0 0 0) 0x7ff284041070 con 0x7ff28c0b5730 2024-03-17T23:48:08.638 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:08.638 INFO:teuthology.orchestra.run.smithi150.stdout:[ 2024-03-17T23:48:08.638 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:48:08.638 INFO:teuthology.orchestra.run.smithi150.stdout: "poolnum": 1, 2024-03-17T23:48:08.638 INFO:teuthology.orchestra.run.smithi150.stdout: "poolname": ".mgr" 2024-03-17T23:48:08.638 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:08.638 INFO:teuthology.orchestra.run.smithi150.stdout:] 2024-03-17T23:48:08.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.639+0000 7ff2943c4640 1 -- 172.21.15.150:0/4260454578 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff27006bbd0 msgr2=0x7ff27006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:08.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.639+0000 7ff2943c4640 1 --2- 172.21.15.150:0/4260454578 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff27006bbd0 0x7ff27006e090 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff2880012e0 tx=0x7ff288009040 comp rx=0 tx=0).stop 2024-03-17T23:48:08.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.639+0000 7ff2943c4640 1 -- 172.21.15.150:0/4260454578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff28c0b5730 msgr2=0x7ff28c0ca720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:08.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.639+0000 7ff2943c4640 1 --2- 172.21.15.150:0/4260454578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff28c0b5730 0x7ff28c0ca720 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ff284009890 tx=0x7ff2840046c0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.640+0000 7ff2943c4640 1 -- 172.21.15.150:0/4260454578 shutdown_connections 2024-03-17T23:48:08.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.640+0000 7ff2943c4640 1 --2- 172.21.15.150:0/4260454578 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff28c0b5730 0x7ff28c0ca720 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.640+0000 7ff2943c4640 1 --2- 172.21.15.150:0/4260454578 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff27006bbd0 0x7ff27006e090 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.640+0000 7ff2943c4640 1 --2- 172.21.15.150:0/4260454578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff28c0b3c30 0x7ff28c09e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.640+0000 7ff2943c4640 1 --2- 172.21.15.150:0/4260454578 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff28c06b190 0x7ff28c14e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.640+0000 7ff2943c4640 1 -- 172.21.15.150:0/4260454578 >> 172.21.15.150:0/4260454578 conn(0x7ff28c0b6170 msgr2=0x7ff28c0d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:08.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.640+0000 7ff2943c4640 1 -- 172.21.15.150:0/4260454578 shutdown_connections 2024-03-17T23:48:08.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.640+0000 7ff2943c4640 1 -- 172.21.15.150:0/4260454578 wait complete. 2024-03-17T23:48:08.652 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-17T23:48:08.652 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-17T23:48:08.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.756+0000 7ff4ce19d640 1 Processor -- start 2024-03-17T23:48:08.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.756+0000 7ff4ce19d640 1 -- start start 2024-03-17T23:48:08.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.757+0000 7ff4ce19d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff4c8099cd0 0x7ff4c809a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:08.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.757+0000 7ff4ce19d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff4c80cd2b0 0x7ff4c8161a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:08.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.757+0000 7ff4ce19d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4c80d0aa0 con 0x7ff4c80cd720 2024-03-17T23:48:08.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.757+0000 7ff4ce19d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7ff4c809a5f0 con 0x7ff4c8099cd0 2024-03-17T23:48:08.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.757+0000 7ff4ce19d640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff4c8161770 con 0x7ff4c80cd2b0 2024-03-17T23:48:08.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.757+0000 7ff4cd19b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff4c8099cd0 0x7ff4c809a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:08.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.757+0000 7ff4cd19b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff4c8099cd0 0x7ff4c809a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43548/0 (socket says 172.21.15.150:43548) 2024-03-17T23:48:08.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.757+0000 7ff4cd19b640 1 -- 172.21.15.150:0/3474733946 learned_addr learned my addr 172.21.15.150:0/3474733946 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:08.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.757+0000 7ff4cc99a640 1 --2- 172.21.15.150:0/3474733946 >> v2:172.21.15.157:3300/0 conn(0x7ff4c80cd2b0 0x7ff4c8161a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:08.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.757+0000 7ff4cd19b640 1 -- 172.21.15.150:0/3474733946 >> v2:172.21.15.157:3300/0 conn(0x7ff4c80cd2b0 msgr2=0x7ff4c8161a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:08.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.757+0000 7ff4cd19b640 1 --2- 172.21.15.150:0/3474733946 >> v2:172.21.15.157:3300/0 conn(0x7ff4c80cd2b0 0x7ff4c8161a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.757+0000 7ff4cd19b640 1 -- 172.21.15.150:0/3474733946 >> v1:172.21.15.157:6789/0 conn(0x7ff4c80cd720 legacy=0x7ff4c8161f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:08.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.757+0000 7ff4cd19b640 1 -- 172.21.15.150:0/3474733946 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c80cfa00 con 0x7ff4c8099cd0 2024-03-17T23:48:08.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.758+0000 7ff4cd19b640 1 --2- 172.21.15.150:0/3474733946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4c8099cd0 0x7ff4c809a0b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff4c80732f0 tx=0x7ff4c402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac573700725de59d server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:08.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.758+0000 7ff4bffff640 1 -- 172.21.15.150:0/3474733946 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4034070 con 0x7ff4c8099cd0 2024-03-17T23:48:08.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.758+0000 7ff4bffff640 1 -- 172.21.15.150:0/3474733946 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4c4030860 con 0x7ff4c8099cd0 2024-03-17T23:48:08.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.758+0000 7ff4bffff640 1 -- 172.21.15.150:0/3474733946 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4030c30 con 0x7ff4c8099cd0 2024-03-17T23:48:08.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.759+0000 7ff4ce19d640 1 -- 172.21.15.150:0/3474733946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4c8099cd0 msgr2=0x7ff4c809a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:08.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.759+0000 7ff4ce19d640 1 --2- 172.21.15.150:0/3474733946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4c8099cd0 0x7ff4c809a0b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff4c80732f0 tx=0x7ff4c402b9a0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.759+0000 7ff4ce19d640 1 -- 172.21.15.150:0/3474733946 shutdown_connections 2024-03-17T23:48:08.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.759+0000 7ff4ce19d640 1 --2- 172.21.15.150:0/3474733946 >> v2:172.21.15.157:3300/0 conn(0x7ff4c80cd2b0 0x7ff4c8161a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.759+0000 7ff4ce19d640 1 --2- 172.21.15.150:0/3474733946 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4c8099cd0 0x7ff4c809a0b0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.759+0000 7ff4ce19d640 1 -- 172.21.15.150:0/3474733946 >> 172.21.15.150:0/3474733946 conn(0x7ff4c80b3560 msgr2=0x7ff4c80b3970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:08.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.759+0000 7ff4ce19d640 1 -- 172.21.15.150:0/3474733946 shutdown_connections 2024-03-17T23:48:08.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.759+0000 7ff4ce19d640 1 -- 172.21.15.150:0/3474733946 wait complete. 2024-03-17T23:48:08.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.760+0000 7ff4ce19d640 1 Processor -- start 2024-03-17T23:48:08.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.760+0000 7ff4ce19d640 1 -- start start 2024-03-17T23:48:08.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.761+0000 7ff4ce19d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4c8099cd0 0x7ff4c8155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:08.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.761+0000 7ff4ce19d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4c80cd2b0 0x7ff4c80c7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:08.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.761+0000 7ff4ce19d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4c80cd720 0x7ff4c80c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:08.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.761+0000 7ff4ce19d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c80d02a0 con 0x7ff4c80cd720 2024-03-17T23:48:08.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.761+0000 7ff4ce19d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c80cffc0 con 0x7ff4c80cd2b0 2024-03-17T23:48:08.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.761+0000 7ff4ce19d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff4c806d2b0 con 0x7ff4c8099cd0 2024-03-17T23:48:08.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.761+0000 7ff4cd99c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4c80cd720 0x7ff4c80c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:08.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.761+0000 7ff4cd99c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4c80cd720 0x7ff4c80c7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43550/0 (socket says 172.21.15.150:43550) 2024-03-17T23:48:08.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.761+0000 7ff4cd99c640 1 -- 172.21.15.150:0/1116931881 learned_addr learned my addr 172.21.15.150:0/1116931881 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:08.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.761+0000 7ff4cd19b640 1 --2- 172.21.15.150:0/1116931881 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4c8099cd0 0x7ff4c8155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:08.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.761+0000 7ff4cc99a640 1 --2- 172.21.15.150:0/1116931881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4c80cd2b0 0x7ff4c80c7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:08.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.761+0000 7ff4cd99c640 1 -- 172.21.15.150:0/1116931881 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4c8099cd0 msgr2=0x7ff4c8155fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:08.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.761+0000 7ff4cd99c640 1 --2- 172.21.15.150:0/1116931881 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4c8099cd0 0x7ff4c8155fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.761+0000 7ff4cd99c640 1 -- 172.21.15.150:0/1116931881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4c80cd2b0 msgr2=0x7ff4c80c7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:08.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.761+0000 7ff4cd99c640 1 --2- 172.21.15.150:0/1116931881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4c80cd2b0 0x7ff4c80c7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:08.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.761+0000 7ff4cd99c640 1 -- 172.21.15.150:0/1116931881 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c8156c60 con 0x7ff4c80cd720 2024-03-17T23:48:08.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.761+0000 7ff4cd99c640 1 --2- 172.21.15.150:0/1116931881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4c80cd720 0x7ff4c80c7f70 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000c7b0 tx=0x7ff4c000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:08.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.761+0000 7ff4bdffb640 1 -- 172.21.15.150:0/1116931881 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0004b60 con 0x7ff4c80cd720 2024-03-17T23:48:08.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.761+0000 7ff4bdffb640 1 -- 172.21.15.150:0/1116931881 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4c0004d00 con 0x7ff4c80cd720 2024-03-17T23:48:08.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.761+0000 7ff4bdffb640 1 -- 172.21.15.150:0/1116931881 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c000aa90 con 0x7ff4c80cd720 2024-03-17T23:48:08.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.761+0000 7ff4ce19d640 1 -- 172.21.15.150:0/1116931881 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4c8156f40 con 0x7ff4c80cd720 2024-03-17T23:48:08.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.762+0000 7ff4ce19d640 1 -- 172.21.15.150:0/1116931881 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4c8156520 con 0x7ff4c80cd720 2024-03-17T23:48:08.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.763+0000 7ff4ce19d640 1 -- 172.21.15.150:0/1116931881 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4c8074bd0 con 0x7ff4c80cd720 2024-03-17T23:48:08.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.763+0000 7ff4bdffb640 1 -- 172.21.15.150:0/1116931881 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff4c000ac30 con 0x7ff4c80cd720 2024-03-17T23:48:08.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.763+0000 7ff4bdffb640 1 --2- 172.21.15.150:0/1116931881 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff4ac06bbd0 0x7ff4ac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:08.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.763+0000 7ff4bdffb640 1 -- 172.21.15.150:0/1116931881 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5249+0+0 (secure 0 0 0) 0x7ff4c008aa30 con 0x7ff4c80cd720 2024-03-17T23:48:08.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.763+0000 7ff4cd19b640 1 --2- 172.21.15.150:0/1116931881 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff4ac06bbd0 0x7ff4ac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:08.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.764+0000 7ff4cd19b640 1 --2- 172.21.15.150:0/1116931881 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff4ac06bbd0 0x7ff4ac06e090 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4036000 tx=0x7ff4c403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:08.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.768+0000 7ff4bdffb640 1 -- 172.21.15.150:0/1116931881 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff4c0020070 con 0x7ff4c80cd720 2024-03-17T23:48:08.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:08.939+0000 7ff4ce19d640 1 -- 172.21.15.150:0/1116931881 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7ff4c80d0d60 con 0x7ff4c80cd720 2024-03-17T23:48:09.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.369+0000 7ff4bdffb640 1 -- 172.21.15.150:0/1116931881 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v37) ==== 138+0+0 (secure 0 0 0) 0x7ff4c0059090 con 0x7ff4c80cd720 2024-03-17T23:48:09.370 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_metadata' created 2024-03-17T23:48:09.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.372+0000 7ff4ce19d640 1 -- 172.21.15.150:0/1116931881 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff4ac06bbd0 msgr2=0x7ff4ac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:09.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.372+0000 7ff4ce19d640 1 --2- 172.21.15.150:0/1116931881 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff4ac06bbd0 0x7ff4ac06e090 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4036000 tx=0x7ff4c403b040 comp rx=0 tx=0).stop 2024-03-17T23:48:09.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.372+0000 7ff4ce19d640 1 -- 172.21.15.150:0/1116931881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4c80cd720 msgr2=0x7ff4c80c7f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:09.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.372+0000 7ff4ce19d640 1 --2- 172.21.15.150:0/1116931881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4c80cd720 0x7ff4c80c7f70 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000c7b0 tx=0x7ff4c000cc70 comp rx=0 tx=0).stop 2024-03-17T23:48:09.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.373+0000 7ff4ce19d640 1 -- 172.21.15.150:0/1116931881 shutdown_connections 2024-03-17T23:48:09.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.373+0000 7ff4ce19d640 1 --2- 172.21.15.150:0/1116931881 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4c80cd720 0x7ff4c80c7f70 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:09.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.373+0000 7ff4ce19d640 1 --2- 172.21.15.150:0/1116931881 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff4ac06bbd0 0x7ff4ac06e090 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:09.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.373+0000 7ff4ce19d640 1 --2- 172.21.15.150:0/1116931881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4c80cd2b0 0x7ff4c80c7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:09.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.373+0000 7ff4ce19d640 1 --2- 172.21.15.150:0/1116931881 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4c8099cd0 0x7ff4c8155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:09.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.373+0000 7ff4ce19d640 1 -- 172.21.15.150:0/1116931881 >> 172.21.15.150:0/1116931881 conn(0x7ff4c80b3560 msgr2=0x7ff4c81604b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:09.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.373+0000 7ff4ce19d640 1 -- 172.21.15.150:0/1116931881 shutdown_connections 2024-03-17T23:48:09.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.373+0000 7ff4ce19d640 1 -- 172.21.15.150:0/1116931881 wait complete. 2024-03-17T23:48:09.388 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-17T23:48:09.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.495+0000 7f0b4f0ea640 1 Processor -- start 2024-03-17T23:48:09.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.495+0000 7f0b4f0ea640 1 -- start start 2024-03-17T23:48:09.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.496+0000 7f0b4f0ea640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b480b4040 0x7f0b48072030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:09.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.496+0000 7f0b4f0ea640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0b48072570 0x7f0b4806e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:09.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.496+0000 7f0b4f0ea640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b4815be10 con 0x7f0b4815ba30 2024-03-17T23:48:09.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.496+0000 7f0b4f0ea640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f0b480729a0 con 0x7f0b480b4040 2024-03-17T23:48:09.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.496+0000 7f0b4f0ea640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f0b4815ec90 con 0x7f0b48072570 2024-03-17T23:48:09.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.496+0000 7f0b47fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b480b4040 0x7f0b48072030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:09.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.496+0000 7f0b47fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b480b4040 0x7f0b48072030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43562/0 (socket says 172.21.15.150:43562) 2024-03-17T23:48:09.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.496+0000 7f0b47fff640 1 -- 172.21.15.150:0/610226790 learned_addr learned my addr 172.21.15.150:0/610226790 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:09.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.496+0000 7f0b4d687640 1 --2- 172.21.15.150:0/610226790 >> v2:172.21.15.157:3300/0 conn(0x7f0b48072570 0x7f0b4806e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:09.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.497+0000 7f0b47fff640 1 -- 172.21.15.150:0/610226790 >> v2:172.21.15.157:3300/0 conn(0x7f0b48072570 msgr2=0x7f0b4806e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:09.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.497+0000 7f0b47fff640 1 --2- 172.21.15.150:0/610226790 >> v2:172.21.15.157:3300/0 conn(0x7f0b48072570 0x7f0b4806e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:09.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.497+0000 7f0b47fff640 1 -- 172.21.15.150:0/610226790 >> v1:172.21.15.157:6789/0 conn(0x7f0b4815ba30 legacy=0x7f0b4815e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:09.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.497+0000 7f0b47fff640 1 -- 172.21.15.150:0/610226790 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b4815ef70 con 0x7f0b480b4040 2024-03-17T23:48:09.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.497+0000 7f0b47fff640 1 --2- 172.21.15.150:0/610226790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b480b4040 0x7f0b48072030 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0b480cfdc0 tx=0x7f0b3c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=634e739dd9d8b222 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:09.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.497+0000 7f0b477fe640 1 -- 172.21.15.150:0/610226790 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c034070 con 0x7f0b480b4040 2024-03-17T23:48:09.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.497+0000 7f0b477fe640 1 -- 172.21.15.150:0/610226790 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b3c030860 con 0x7f0b480b4040 2024-03-17T23:48:09.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.497+0000 7f0b477fe640 1 -- 172.21.15.150:0/610226790 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c030c30 con 0x7f0b480b4040 2024-03-17T23:48:09.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.498+0000 7f0b4f0ea640 1 -- 172.21.15.150:0/610226790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b480b4040 msgr2=0x7f0b48072030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:09.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.498+0000 7f0b4f0ea640 1 --2- 172.21.15.150:0/610226790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b480b4040 0x7f0b48072030 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f0b480cfdc0 tx=0x7f0b3c02b9a0 comp rx=0 tx=0).stop 2024-03-17T23:48:09.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.498+0000 7f0b4f0ea640 1 -- 172.21.15.150:0/610226790 shutdown_connections 2024-03-17T23:48:09.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.498+0000 7f0b4f0ea640 1 --2- 172.21.15.150:0/610226790 >> v2:172.21.15.157:3300/0 conn(0x7f0b48072570 0x7f0b4806e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:09.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.498+0000 7f0b4f0ea640 1 --2- 172.21.15.150:0/610226790 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b480b4040 0x7f0b48072030 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:09.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.498+0000 7f0b4f0ea640 1 -- 172.21.15.150:0/610226790 >> 172.21.15.150:0/610226790 conn(0x7f0b480b1b40 msgr2=0x7f0b480b1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:09.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.498+0000 7f0b4f0ea640 1 -- 172.21.15.150:0/610226790 shutdown_connections 2024-03-17T23:48:09.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.498+0000 7f0b4f0ea640 1 -- 172.21.15.150:0/610226790 wait complete. 2024-03-17T23:48:09.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.499+0000 7f0b4f0ea640 1 Processor -- start 2024-03-17T23:48:09.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.499+0000 7f0b4f0ea640 1 -- start start 2024-03-17T23:48:09.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.500+0000 7f0b4f0ea640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b48072570 0x7f0b48148800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:09.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.500+0000 7f0b4f0ea640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b480b4040 0x7f0b4815f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:09.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.500+0000 7f0b4f0ea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4815ba30 0x7f0b4815f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:09.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.500+0000 7f0b4f0ea640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0b4806e370 con 0x7f0b4815ba30 2024-03-17T23:48:09.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.500+0000 7f0b4f0ea640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0b4806e090 con 0x7f0b48072570 2024-03-17T23:48:09.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.500+0000 7f0b4f0ea640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0b4806edd0 con 0x7f0b480b4040 2024-03-17T23:48:09.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.500+0000 7f0b4d687640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4815ba30 0x7f0b4815f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:09.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.500+0000 7f0b4d687640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4815ba30 0x7f0b4815f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43576/0 (socket says 172.21.15.150:43576) 2024-03-17T23:48:09.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.500+0000 7f0b4d687640 1 -- 172.21.15.150:0/3632503750 learned_addr learned my addr 172.21.15.150:0/3632503750 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:09.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.501+0000 7f0b47fff640 1 --2- 172.21.15.150:0/3632503750 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b480b4040 0x7f0b4815f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:09.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.501+0000 7f0b4ce86640 1 --2- 172.21.15.150:0/3632503750 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b48072570 0x7f0b48148800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:09.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.501+0000 7f0b4d687640 1 -- 172.21.15.150:0/3632503750 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b480b4040 msgr2=0x7f0b4815f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:09.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.501+0000 7f0b4d687640 1 --2- 172.21.15.150:0/3632503750 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b480b4040 0x7f0b4815f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:09.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.501+0000 7f0b4d687640 1 -- 172.21.15.150:0/3632503750 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b48072570 msgr2=0x7f0b48148800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:09.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.501+0000 7f0b4d687640 1 --2- 172.21.15.150:0/3632503750 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b48072570 0x7f0b48148800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:09.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.501+0000 7f0b4d687640 1 -- 172.21.15.150:0/3632503750 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b48148d40 con 0x7f0b4815ba30 2024-03-17T23:48:09.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.501+0000 7f0b4d687640 1 --2- 172.21.15.150:0/3632503750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4815ba30 0x7f0b4815f9d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0b40009dd0 tx=0x7f0b4000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:09.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.501+0000 7f0b457fa640 1 -- 172.21.15.150:0/3632503750 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b40018020 con 0x7f0b4815ba30 2024-03-17T23:48:09.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.501+0000 7f0b457fa640 1 -- 172.21.15.150:0/3632503750 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b40014480 con 0x7f0b4815ba30 2024-03-17T23:48:09.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.501+0000 7f0b4f0ea640 1 -- 172.21.15.150:0/3632503750 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b48158820 con 0x7f0b4815ba30 2024-03-17T23:48:09.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.501+0000 7f0b457fa640 1 -- 172.21.15.150:0/3632503750 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b40014870 con 0x7f0b4815ba30 2024-03-17T23:48:09.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.501+0000 7f0b4f0ea640 1 -- 172.21.15.150:0/3632503750 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b48153410 con 0x7f0b4815ba30 2024-03-17T23:48:09.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.502+0000 7f0b4f0ea640 1 -- 172.21.15.150:0/3632503750 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b480b6160 con 0x7f0b4815ba30 2024-03-17T23:48:09.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.502+0000 7f0b457fa640 1 -- 172.21.15.150:0/3632503750 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0b40005d10 con 0x7f0b4815ba30 2024-03-17T23:48:09.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.503+0000 7f0b457fa640 1 --2- 172.21.15.150:0/3632503750 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b3406bc00 0x7f0b3406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:09.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.503+0000 7f0b457fa640 1 -- 172.21.15.150:0/3632503750 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 5619+0+0 (secure 0 0 0) 0x7f0b4008b4d0 con 0x7f0b4815ba30 2024-03-17T23:48:09.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.503+0000 7f0b4ce86640 1 --2- 172.21.15.150:0/3632503750 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b3406bc00 0x7f0b3406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:09.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.503+0000 7f0b4ce86640 1 --2- 172.21.15.150:0/3632503750 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b3406bc00 0x7f0b3406e0c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0b38000f60 tx=0x7f0b38009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:09.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.507+0000 7f0b457fa640 1 -- 172.21.15.150:0/3632503750 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0b480b6160 con 0x7f0b4815ba30 2024-03-17T23:48:09.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:09.680+0000 7f0b4f0ea640 1 -- 172.21.15.150:0/3632503750 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f0b480b2170 con 0x7f0b4815ba30 2024-03-17T23:48:10.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.374+0000 7f0b457fa640 1 -- 172.21.15.150:0/3632503750 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v38) ==== 170+0+0 (secure 0 0 0) 0x7f0b40022070 con 0x7f0b4815ba30 2024-03-17T23:48:10.375 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_data' created 2024-03-17T23:48:10.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.377+0000 7f0b4f0ea640 1 -- 172.21.15.150:0/3632503750 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b3406bc00 msgr2=0x7f0b3406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:10.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.377+0000 7f0b4f0ea640 1 --2- 172.21.15.150:0/3632503750 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b3406bc00 0x7f0b3406e0c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0b38000f60 tx=0x7f0b38009040 comp rx=0 tx=0).stop 2024-03-17T23:48:10.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.377+0000 7f0b4f0ea640 1 -- 172.21.15.150:0/3632503750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4815ba30 msgr2=0x7f0b4815f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:10.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.377+0000 7f0b4f0ea640 1 --2- 172.21.15.150:0/3632503750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4815ba30 0x7f0b4815f9d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0b40009dd0 tx=0x7f0b4000c490 comp rx=0 tx=0).stop 2024-03-17T23:48:10.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.377+0000 7f0b4f0ea640 1 -- 172.21.15.150:0/3632503750 shutdown_connections 2024-03-17T23:48:10.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.377+0000 7f0b4f0ea640 1 --2- 172.21.15.150:0/3632503750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4815ba30 0x7f0b4815f9d0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:10.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.377+0000 7f0b4f0ea640 1 --2- 172.21.15.150:0/3632503750 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b480b4040 0x7f0b4815f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:10.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.377+0000 7f0b4f0ea640 1 --2- 172.21.15.150:0/3632503750 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b3406bc00 0x7f0b3406e0c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:10.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.377+0000 7f0b4f0ea640 1 --2- 172.21.15.150:0/3632503750 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b48072570 0x7f0b48148800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:10.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.377+0000 7f0b4f0ea640 1 -- 172.21.15.150:0/3632503750 >> 172.21.15.150:0/3632503750 conn(0x7f0b480b1b40 msgr2=0x7f0b4806f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:10.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.377+0000 7f0b4f0ea640 1 -- 172.21.15.150:0/3632503750 shutdown_connections 2024-03-17T23:48:10.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.377+0000 7f0b4f0ea640 1 -- 172.21.15.150:0/3632503750 wait complete. 2024-03-17T23:48:10.403 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-17T23:48:10.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.507+0000 7ff62e4f1640 1 Processor -- start 2024-03-17T23:48:10.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.507+0000 7ff62e4f1640 1 -- start start 2024-03-17T23:48:10.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.508+0000 7ff62e4f1640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff628155320 0x7ff62815bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:10.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.508+0000 7ff62e4f1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff62806dd30 0x7ff6280ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:10.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.508+0000 7ff62e4f1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff6280cd250 0x7ff62815e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:10.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.508+0000 7ff62e4f1640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7ff62815c550 con 0x7ff6280cd250 2024-03-17T23:48:10.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.508+0000 7ff62e4f1640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff6280cf3b0 con 0x7ff628155320 2024-03-17T23:48:10.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.508+0000 7ff62e4f1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff6280cf690 con 0x7ff62806dd30 2024-03-17T23:48:10.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.508+0000 7ff62ca8e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff6280cd250 0x7ff62815e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:10.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.508+0000 7ff62ca8e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff6280cd250 0x7ff62815e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43592/0 (socket says 172.21.15.150:43592) 2024-03-17T23:48:10.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.508+0000 7ff62ca8e640 1 -- 172.21.15.150:0/1101315595 learned_addr learned my addr 172.21.15.150:0/1101315595 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:10.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.508+0000 7ff62ca8e640 1 -- 172.21.15.150:0/1101315595 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff62806dd30 msgr2=0x7ff6280ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:10.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.508+0000 7ff6277fe640 1 --2- 172.21.15.150:0/1101315595 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff62806dd30 0x7ff6280ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:10.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.508+0000 7ff627fff640 1 --2- 172.21.15.150:0/1101315595 >> v2:172.21.15.157:3300/0 conn(0x7ff628155320 0x7ff62815bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:10.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.508+0000 7ff62ca8e640 1 --2- 172.21.15.150:0/1101315595 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff62806dd30 0x7ff6280ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:10.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.508+0000 7ff62ca8e640 1 -- 172.21.15.150:0/1101315595 >> v2:172.21.15.157:3300/0 conn(0x7ff628155320 msgr2=0x7ff62815bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:10.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.508+0000 7ff62ca8e640 1 --2- 172.21.15.150:0/1101315595 >> v2:172.21.15.157:3300/0 conn(0x7ff628155320 0x7ff62815bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:10.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.508+0000 7ff62ca8e640 1 -- 172.21.15.150:0/1101315595 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff62815c7a0 con 0x7ff6280cd250 2024-03-17T23:48:10.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.508+0000 7ff6277fe640 1 --2- 172.21.15.150:0/1101315595 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff62806dd30 0x7ff6280ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:10.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.508+0000 7ff627fff640 1 --2- 172.21.15.150:0/1101315595 >> v2:172.21.15.157:3300/0 conn(0x7ff628155320 0x7ff62815bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:10.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.509+0000 7ff62ca8e640 1 --2- 172.21.15.150:0/1101315595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6280cd250 0x7ff62815e980 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff62806d5a0 tx=0x7ff62002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b180ed45c8b22b80 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:10.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.509+0000 7ff626ffd640 1 -- 172.21.15.150:0/1101315595 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62002ca70 con 0x7ff6280cd250 2024-03-17T23:48:10.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.509+0000 7ff626ffd640 1 -- 172.21.15.150:0/1101315595 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff62002cc10 con 0x7ff6280cd250 2024-03-17T23:48:10.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.509+0000 7ff626ffd640 1 -- 172.21.15.150:0/1101315595 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6200343f0 con 0x7ff6280cd250 2024-03-17T23:48:10.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.510+0000 7ff62e4f1640 1 -- 172.21.15.150:0/1101315595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6280cd250 msgr2=0x7ff62815e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:10.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.510+0000 7ff62e4f1640 1 --2- 172.21.15.150:0/1101315595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6280cd250 0x7ff62815e980 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff62806d5a0 tx=0x7ff62002bed0 comp rx=0 tx=0).stop 2024-03-17T23:48:10.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.510+0000 7ff62e4f1640 1 -- 172.21.15.150:0/1101315595 shutdown_connections 2024-03-17T23:48:10.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.510+0000 7ff62e4f1640 1 --2- 172.21.15.150:0/1101315595 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff6280cd250 0x7ff62815e980 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:10.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.510+0000 7ff62e4f1640 1 --2- 172.21.15.150:0/1101315595 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff62806dd30 0x7ff6280ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:10.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.510+0000 7ff62e4f1640 1 --2- 172.21.15.150:0/1101315595 >> v2:172.21.15.157:3300/0 conn(0x7ff628155320 0x7ff62815bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:10.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.510+0000 7ff62e4f1640 1 -- 172.21.15.150:0/1101315595 >> 172.21.15.150:0/1101315595 conn(0x7ff62806b190 msgr2=0x7ff62806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:10.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.510+0000 7ff62e4f1640 1 -- 172.21.15.150:0/1101315595 shutdown_connections 2024-03-17T23:48:10.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.510+0000 7ff62e4f1640 1 -- 172.21.15.150:0/1101315595 wait complete. 2024-03-17T23:48:10.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.511+0000 7ff62e4f1640 1 Processor -- start 2024-03-17T23:48:10.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.511+0000 7ff62e4f1640 1 -- start start 2024-03-17T23:48:10.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.512+0000 7ff62e4f1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff62806dd30 0x7ff6280b40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:10.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.512+0000 7ff62e4f1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6280cd250 0x7ff628157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:10.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.512+0000 7ff62e4f1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff628155320 0x7ff628158180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:10.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.512+0000 7ff62e4f1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff628155700 con 0x7ff62806dd30 2024-03-17T23:48:10.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.512+0000 7ff62e4f1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff62806e190 con 0x7ff6280cd250 2024-03-17T23:48:10.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.512+0000 7ff62e4f1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff628155880 con 0x7ff628155320 2024-03-17T23:48:10.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.512+0000 7ff627fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff62806dd30 0x7ff6280b40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:10.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.512+0000 7ff627fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff62806dd30 0x7ff6280b40b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43598/0 (socket says 172.21.15.150:43598) 2024-03-17T23:48:10.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.512+0000 7ff627fff640 1 -- 172.21.15.150:0/3049244713 learned_addr learned my addr 172.21.15.150:0/3049244713 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:10.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.512+0000 7ff6277fe640 1 --2- 172.21.15.150:0/3049244713 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6280cd250 0x7ff628157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:10.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.512+0000 7ff62ca8e640 1 --2- 172.21.15.150:0/3049244713 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff628155320 0x7ff628158180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:10.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.512+0000 7ff627fff640 1 -- 172.21.15.150:0/3049244713 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff628155320 msgr2=0x7ff628158180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:10.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.512+0000 7ff627fff640 1 --2- 172.21.15.150:0/3049244713 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff628155320 0x7ff628158180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:10.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.512+0000 7ff627fff640 1 -- 172.21.15.150:0/3049244713 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6280cd250 msgr2=0x7ff628157c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:10.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.512+0000 7ff627fff640 1 --2- 172.21.15.150:0/3049244713 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6280cd250 0x7ff628157c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:10.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.512+0000 7ff627fff640 1 -- 172.21.15.150:0/3049244713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6280c5b50 con 0x7ff62806dd30 2024-03-17T23:48:10.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.512+0000 7ff627fff640 1 --2- 172.21.15.150:0/3049244713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff62806dd30 0x7ff6280b40b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7ff61800c820 tx=0x7ff61800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:10.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.512+0000 7ff624ff9640 1 -- 172.21.15.150:0/3049244713 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff618015070 con 0x7ff62806dd30 2024-03-17T23:48:10.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.512+0000 7ff624ff9640 1 -- 172.21.15.150:0/3049244713 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff618007a30 con 0x7ff62806dd30 2024-03-17T23:48:10.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.512+0000 7ff62e4f1640 1 -- 172.21.15.150:0/3049244713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6280c4fd0 con 0x7ff62806dd30 2024-03-17T23:48:10.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.512+0000 7ff624ff9640 1 -- 172.21.15.150:0/3049244713 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff618007e00 con 0x7ff62806dd30 2024-03-17T23:48:10.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.512+0000 7ff62e4f1640 1 -- 172.21.15.150:0/3049244713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6280c5590 con 0x7ff62806dd30 2024-03-17T23:48:10.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.513+0000 7ff62e4f1640 1 -- 172.21.15.150:0/3049244713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6280681a0 con 0x7ff62806dd30 2024-03-17T23:48:10.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.514+0000 7ff624ff9640 1 -- 172.21.15.150:0/3049244713 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff618018070 con 0x7ff62806dd30 2024-03-17T23:48:10.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.514+0000 7ff624ff9640 1 --2- 172.21.15.150:0/3049244713 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff61406bbd0 0x7ff61406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:10.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.514+0000 7ff624ff9640 1 -- 172.21.15.150:0/3049244713 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6001+0+0 (secure 0 0 0) 0x7ff61808aa10 con 0x7ff62806dd30 2024-03-17T23:48:10.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.514+0000 7ff6277fe640 1 --2- 172.21.15.150:0/3049244713 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff61406bbd0 0x7ff61406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:10.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.515+0000 7ff6277fe640 1 --2- 172.21.15.150:0/3049244713 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff61406bbd0 0x7ff61406e090 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff6280cbec0 tx=0x7ff61c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:10.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.519+0000 7ff624ff9640 1 -- 172.21.15.150:0/3049244713 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff618058e10 con 0x7ff62806dd30 2024-03-17T23:48:10.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:10.729+0000 7ff62e4f1640 1 -- 172.21.15.150:0/3049244713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7ff62806b8d0 con 0x7ff62806dd30 2024-03-17T23:48:11.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.376+0000 7ff624ff9640 1 -- 172.21.15.150:0/3049244713 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v13) ==== 172+0+0 (secure 0 0 0) 0x7ff61805ceb0 con 0x7ff62806dd30 2024-03-17T23:48:11.377 INFO:teuthology.orchestra.run.smithi150.stderr:new fs with metadata pool 5 and data pool 6 2024-03-17T23:48:11.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.381+0000 7ff62e4f1640 1 -- 172.21.15.150:0/3049244713 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff61406bbd0 msgr2=0x7ff61406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:11.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.381+0000 7ff62e4f1640 1 --2- 172.21.15.150:0/3049244713 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff61406bbd0 0x7ff61406e090 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff6280cbec0 tx=0x7ff61c008040 comp rx=0 tx=0).stop 2024-03-17T23:48:11.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.381+0000 7ff62e4f1640 1 -- 172.21.15.150:0/3049244713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff62806dd30 msgr2=0x7ff6280b40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:11.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.381+0000 7ff62e4f1640 1 --2- 172.21.15.150:0/3049244713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff62806dd30 0x7ff6280b40b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7ff61800c820 tx=0x7ff61800cc20 comp rx=0 tx=0).stop 2024-03-17T23:48:11.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.381+0000 7ff62e4f1640 1 -- 172.21.15.150:0/3049244713 shutdown_connections 2024-03-17T23:48:11.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.381+0000 7ff62e4f1640 1 --2- 172.21.15.150:0/3049244713 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff628155320 0x7ff628158180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:11.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.381+0000 7ff62e4f1640 1 --2- 172.21.15.150:0/3049244713 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff61406bbd0 0x7ff61406e090 secure :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff6280cbec0 tx=0x7ff61c008040 comp rx=0 tx=0).stop 2024-03-17T23:48:11.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.381+0000 7ff62e4f1640 1 --2- 172.21.15.150:0/3049244713 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff6280cd250 0x7ff628157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:11.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.381+0000 7ff62e4f1640 1 --2- 172.21.15.150:0/3049244713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff62806dd30 0x7ff6280b40b0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:11.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.381+0000 7ff62e4f1640 1 -- 172.21.15.150:0/3049244713 >> 172.21.15.150:0/3049244713 conn(0x7ff62806b190 msgr2=0x7ff628152db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:11.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.382+0000 7ff62e4f1640 1 -- 172.21.15.150:0/3049244713 shutdown_connections 2024-03-17T23:48:11.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.384+0000 7ff62e4f1640 1 -- 172.21.15.150:0/3049244713 wait complete. 2024-03-17T23:48:11.410 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:48:11.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.511+0000 7fbe1acbf640 1 Processor -- start 2024-03-17T23:48:11.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.511+0000 7fbe1acbf640 1 -- start start 2024-03-17T23:48:11.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.512+0000 7fbe1acbf640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbe1415de30 0x7fbe141619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:11.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.512+0000 7fbe1acbf640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fbe140cf120 0x7fbe140d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:11.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.512+0000 7fbe1acbf640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe140b9cf0 con 0x7fbe140b9860 2024-03-17T23:48:11.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.512+0000 7fbe1acbf640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fbe14162940 con 0x7fbe1415de30 2024-03-17T23:48:11.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.512+0000 7fbe1acbf640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fbe141620a0 con 0x7fbe140cf120 2024-03-17T23:48:11.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.512+0000 7fbe18a5b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbe1415de30 0x7fbe141619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:11.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.512+0000 7fbe13fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fbe140b9860 0x7fbe140cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37558/0 (socket says 172.21.15.150:37558) 2024-03-17T23:48:11.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.512+0000 7fbe13fff640 1 -- 172.21.15.150:0/3679037853 learned_addr learned my addr 172.21.15.150:0/3679037853 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:11.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.513+0000 7fbe18a5b640 1 -- 172.21.15.150:0/3679037853 >> v2:172.21.15.157:3300/0 conn(0x7fbe140cf120 msgr2=0x7fbe140d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:11.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.513+0000 7fbe1925c640 1 --2- 172.21.15.150:0/3679037853 >> v2:172.21.15.157:3300/0 conn(0x7fbe140cf120 0x7fbe140d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:11.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.513+0000 7fbe18a5b640 1 --2- 172.21.15.150:0/3679037853 >> v2:172.21.15.157:3300/0 conn(0x7fbe140cf120 0x7fbe140d2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:11.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.513+0000 7fbe18a5b640 1 -- 172.21.15.150:0/3679037853 >> v1:172.21.15.150:6789/0 conn(0x7fbe140b9860 legacy=0x7fbe140cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:11.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.513+0000 7fbe137fe640 1 -- 172.21.15.150:0/3679037853 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3941761525 0 0) 0x7fbe140b9cf0 con 0x7fbe140b9860 2024-03-17T23:48:11.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.513+0000 7fbe18a5b640 1 -- 172.21.15.150:0/3679037853 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe14162380 con 0x7fbe1415de30 2024-03-17T23:48:11.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.513+0000 7fbe1925c640 1 --2- 172.21.15.150:0/3679037853 >> v2:172.21.15.157:3300/0 conn(0x7fbe140cf120 0x7fbe140d2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:11.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.513+0000 7fbe18a5b640 1 --2- 172.21.15.150:0/3679037853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe1415de30 0x7fbe141619a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fbe141618d0 tx=0x7fbe0402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b810a8f9bea413 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:11.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.513+0000 7fbe137fe640 1 -- 172.21.15.150:0/3679037853 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe040315a0 con 0x7fbe1415de30 2024-03-17T23:48:11.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.514+0000 7fbe137fe640 1 -- 172.21.15.150:0/3679037853 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe04037070 con 0x7fbe1415de30 2024-03-17T23:48:11.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.514+0000 7fbe137fe640 1 -- 172.21.15.150:0/3679037853 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe04031960 con 0x7fbe1415de30 2024-03-17T23:48:11.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.514+0000 7fbe1acbf640 1 -- 172.21.15.150:0/3679037853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe1415de30 msgr2=0x7fbe141619a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:11.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.514+0000 7fbe1acbf640 1 --2- 172.21.15.150:0/3679037853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe1415de30 0x7fbe141619a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fbe141618d0 tx=0x7fbe0402b840 comp rx=0 tx=0).stop 2024-03-17T23:48:11.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.514+0000 7fbe1acbf640 1 -- 172.21.15.150:0/3679037853 shutdown_connections 2024-03-17T23:48:11.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.514+0000 7fbe1acbf640 1 --2- 172.21.15.150:0/3679037853 >> v2:172.21.15.157:3300/0 conn(0x7fbe140cf120 0x7fbe140d2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:11.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.514+0000 7fbe1acbf640 1 --2- 172.21.15.150:0/3679037853 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe1415de30 0x7fbe141619a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:11.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.514+0000 7fbe1acbf640 1 -- 172.21.15.150:0/3679037853 >> 172.21.15.150:0/3679037853 conn(0x7fbe140730b0 msgr2=0x7fbe140d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:11.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.514+0000 7fbe1acbf640 1 -- 172.21.15.150:0/3679037853 shutdown_connections 2024-03-17T23:48:11.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.514+0000 7fbe1acbf640 1 -- 172.21.15.150:0/3679037853 wait complete. 2024-03-17T23:48:11.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.515+0000 7fbe1acbf640 1 Processor -- start 2024-03-17T23:48:11.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.515+0000 7fbe1acbf640 1 -- start start 2024-03-17T23:48:11.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.516+0000 7fbe1acbf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbe140b9860 0x7fbe140cdab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:11.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.516+0000 7fbe1acbf640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe140cf120 0x7fbe140aa730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:11.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.516+0000 7fbe1acbf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe1415de30 0x7fbe141f7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:11.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.516+0000 7fbe1acbf640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1415e210 con 0x7fbe1415de30 2024-03-17T23:48:11.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.516+0000 7fbe1acbf640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbe140d2170 con 0x7fbe140cf120 2024-03-17T23:48:11.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.516+0000 7fbe1acbf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbe140d2bd0 con 0x7fbe140b9860 2024-03-17T23:48:11.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.516+0000 7fbe1925c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe1415de30 0x7fbe141f7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:11.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.516+0000 7fbe1925c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe1415de30 0x7fbe141f7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43606/0 (socket says 172.21.15.150:43606) 2024-03-17T23:48:11.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.516+0000 7fbe1925c640 1 -- 172.21.15.150:0/4186967043 learned_addr learned my addr 172.21.15.150:0/4186967043 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:11.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.516+0000 7fbe18a5b640 1 --2- 172.21.15.150:0/4186967043 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbe140b9860 0x7fbe140cdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:11.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.516+0000 7fbe13fff640 1 --2- 172.21.15.150:0/4186967043 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe140cf120 0x7fbe140aa730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:11.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.516+0000 7fbe1925c640 1 -- 172.21.15.150:0/4186967043 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbe140b9860 msgr2=0x7fbe140cdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:11.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.516+0000 7fbe1925c640 1 --2- 172.21.15.150:0/4186967043 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbe140b9860 0x7fbe140cdab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:11.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.517+0000 7fbe1925c640 1 -- 172.21.15.150:0/4186967043 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe140cf120 msgr2=0x7fbe140aa730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:11.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.517+0000 7fbe1925c640 1 --2- 172.21.15.150:0/4186967043 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe140cf120 0x7fbe140aa730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:11.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.517+0000 7fbe1925c640 1 -- 172.21.15.150:0/4186967043 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe1415fe90 con 0x7fbe1415de30 2024-03-17T23:48:11.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.517+0000 7fbe1925c640 1 --2- 172.21.15.150:0/4186967043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe1415de30 0x7fbe141f7200 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c00cef0 tx=0x7fbe0c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:11.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.517+0000 7fbe117fa640 1 -- 172.21.15.150:0/4186967043 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c01a070 con 0x7fbe1415de30 2024-03-17T23:48:11.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.517+0000 7fbe117fa640 1 -- 172.21.15.150:0/4186967043 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe0c00d860 con 0x7fbe1415de30 2024-03-17T23:48:11.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.517+0000 7fbe1acbf640 1 -- 172.21.15.150:0/4186967043 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe14160170 con 0x7fbe1415de30 2024-03-17T23:48:11.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.517+0000 7fbe117fa640 1 -- 172.21.15.150:0/4186967043 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c00dc50 con 0x7fbe1415de30 2024-03-17T23:48:11.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.517+0000 7fbe1acbf640 1 -- 172.21.15.150:0/4186967043 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe14160730 con 0x7fbe1415de30 2024-03-17T23:48:11.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.518+0000 7fbe1acbf640 1 -- 172.21.15.150:0/4186967043 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe140681a0 con 0x7fbe1415de30 2024-03-17T23:48:11.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.518+0000 7fbe117fa640 1 -- 172.21.15.150:0/4186967043 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbe0c01d070 con 0x7fbe1415de30 2024-03-17T23:48:11.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.519+0000 7fbe117fa640 1 --2- 172.21.15.150:0/4186967043 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbe0006bbd0 0x7fbe0006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:11.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.519+0000 7fbe18a5b640 1 --2- 172.21.15.150:0/4186967043 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbe0006bbd0 0x7fbe0006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:11.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.519+0000 7fbe117fa640 1 -- 172.21.15.150:0/4186967043 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fbe0c08b4b0 con 0x7fbe1415de30 2024-03-17T23:48:11.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.519+0000 7fbe18a5b640 1 --2- 172.21.15.150:0/4186967043 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbe0006bbd0 0x7fbe0006e090 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fbe04036000 tx=0x7fbe04008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:11.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.523+0000 7fbe117fa640 1 -- 172.21.15.150:0/4186967043 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbe0c059ed0 con 0x7fbe1415de30 2024-03-17T23:48:11.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.697+0000 7fbe1acbf640 1 -- 172.21.15.150:0/4186967043 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbe140b33c0 con 0x7fbe1415de30 2024-03-17T23:48:11.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.698+0000 7fbe117fa640 1 -- 172.21.15.150:0/4186967043 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16991 (secure 0 0 0) 0x7fbe0c088270 con 0x7fbe1415de30 2024-03-17T23:48:11.699 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:11.699 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":39,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:48:10.730661+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:48:08.957879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-17T23:48:09.698808+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:48:11.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.701+0000 7fbe1acbf640 1 -- 172.21.15.150:0/4186967043 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbe0006bbd0 msgr2=0x7fbe0006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:11.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.701+0000 7fbe1acbf640 1 --2- 172.21.15.150:0/4186967043 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbe0006bbd0 0x7fbe0006e090 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fbe04036000 tx=0x7fbe04008030 comp rx=0 tx=0).stop 2024-03-17T23:48:11.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.701+0000 7fbe1acbf640 1 -- 172.21.15.150:0/4186967043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe1415de30 msgr2=0x7fbe141f7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:11.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.701+0000 7fbe1acbf640 1 --2- 172.21.15.150:0/4186967043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe1415de30 0x7fbe141f7200 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c00cef0 tx=0x7fbe0c00a710 comp rx=0 tx=0).stop 2024-03-17T23:48:11.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.701+0000 7fbe1acbf640 1 -- 172.21.15.150:0/4186967043 shutdown_connections 2024-03-17T23:48:11.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.701+0000 7fbe1acbf640 1 --2- 172.21.15.150:0/4186967043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbe1415de30 0x7fbe141f7200 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:11.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.701+0000 7fbe1acbf640 1 --2- 172.21.15.150:0/4186967043 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbe0006bbd0 0x7fbe0006e090 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:11.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.701+0000 7fbe1acbf640 1 --2- 172.21.15.150:0/4186967043 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe140cf120 0x7fbe140aa730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:11.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.701+0000 7fbe1acbf640 1 --2- 172.21.15.150:0/4186967043 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbe140b9860 0x7fbe140cdab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:11.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.701+0000 7fbe1acbf640 1 -- 172.21.15.150:0/4186967043 >> 172.21.15.150:0/4186967043 conn(0x7fbe140730b0 msgr2=0x7fbe140b37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:11.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.701+0000 7fbe1acbf640 1 -- 172.21.15.150:0/4186967043 shutdown_connections 2024-03-17T23:48:11.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.701+0000 7fbe1acbf640 1 -- 172.21.15.150:0/4186967043 wait complete. 2024-03-17T23:48:11.713 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:48:11.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.813+0000 7fea0289a640 1 Processor -- start 2024-03-17T23:48:11.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.813+0000 7fea0289a640 1 -- start start 2024-03-17T23:48:11.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.814+0000 7fea0289a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe9fc15dfa0 0x7fe9fc06e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:11.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.814+0000 7fea0289a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe9fc072a20 0x7fe9fc06eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:11.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.814+0000 7fea0289a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fe9fc067700 0x7fe9fc06b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:11.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.814+0000 7fea0289a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fe9fc0d0030 con 0x7fe9fc15dfa0 2024-03-17T23:48:11.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.814+0000 7fea0289a640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fe9fc0d0310 con 0x7fe9fc067700 2024-03-17T23:48:11.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.814+0000 7fea0289a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe9fc0d05f0 con 0x7fe9fc072a20 2024-03-17T23:48:11.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.814+0000 7fe9fbfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe9fc15dfa0 0x7fe9fc06e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:11.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.814+0000 7fe9fbfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe9fc15dfa0 0x7fe9fc06e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43614/0 (socket says 172.21.15.150:43614) 2024-03-17T23:48:11.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.814+0000 7fe9fbfff640 1 -- 172.21.15.150:0/322506461 learned_addr learned my addr 172.21.15.150:0/322506461 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:11.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.814+0000 7fe9fb7fe640 1 --2- 172.21.15.150:0/322506461 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe9fc072a20 0x7fe9fc06eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:11.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.814+0000 7fea00e37640 1 --2- 172.21.15.150:0/322506461 >> v2:172.21.15.157:3300/0 conn(0x7fe9fc067700 0x7fe9fc06b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:11.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.814+0000 7fe9fbfff640 1 -- 172.21.15.150:0/322506461 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe9fc072a20 msgr2=0x7fe9fc06eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:11.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.814+0000 7fe9fbfff640 1 --2- 172.21.15.150:0/322506461 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe9fc072a20 0x7fe9fc06eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:11.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.814+0000 7fe9fbfff640 1 -- 172.21.15.150:0/322506461 >> v2:172.21.15.157:3300/0 conn(0x7fe9fc067700 msgr2=0x7fe9fc06b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:11.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.814+0000 7fe9fbfff640 1 --2- 172.21.15.150:0/322506461 >> v2:172.21.15.157:3300/0 conn(0x7fe9fc067700 0x7fe9fc06b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:11.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.814+0000 7fe9fbfff640 1 -- 172.21.15.150:0/322506461 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9fc0d08d0 con 0x7fe9fc15dfa0 2024-03-17T23:48:11.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.815+0000 7fe9fbfff640 1 --2- 172.21.15.150:0/322506461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9fc15dfa0 0x7fe9fc06e580 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc0d0520 tx=0x7fe9ec02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab29178600ed1b7e server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:11.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.815+0000 7fe9faffd640 1 -- 172.21.15.150:0/322506461 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec0317c0 con 0x7fe9fc15dfa0 2024-03-17T23:48:11.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.815+0000 7fe9faffd640 1 -- 172.21.15.150:0/322506461 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9ec037070 con 0x7fe9fc15dfa0 2024-03-17T23:48:11.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.815+0000 7fe9faffd640 1 -- 172.21.15.150:0/322506461 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec031b80 con 0x7fe9fc15dfa0 2024-03-17T23:48:11.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.815+0000 7fea0289a640 1 -- 172.21.15.150:0/322506461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9fc15dfa0 msgr2=0x7fe9fc06e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:11.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.815+0000 7fea0289a640 1 --2- 172.21.15.150:0/322506461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9fc15dfa0 0x7fe9fc06e580 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc0d0520 tx=0x7fe9ec02b860 comp rx=0 tx=0).stop 2024-03-17T23:48:11.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.815+0000 7fea0289a640 1 -- 172.21.15.150:0/322506461 shutdown_connections 2024-03-17T23:48:11.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.815+0000 7fea0289a640 1 --2- 172.21.15.150:0/322506461 >> v2:172.21.15.157:3300/0 conn(0x7fe9fc067700 0x7fe9fc06b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:11.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.816+0000 7fea0289a640 1 --2- 172.21.15.150:0/322506461 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe9fc072a20 0x7fe9fc06eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:11.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.816+0000 7fea0289a640 1 --2- 172.21.15.150:0/322506461 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9fc15dfa0 0x7fe9fc06e580 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:11.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.816+0000 7fea0289a640 1 -- 172.21.15.150:0/322506461 >> 172.21.15.150:0/322506461 conn(0x7fe9fc0b3da0 msgr2=0x7fe9fc0b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:11.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.816+0000 7fea0289a640 1 -- 172.21.15.150:0/322506461 shutdown_connections 2024-03-17T23:48:11.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.816+0000 7fea0289a640 1 -- 172.21.15.150:0/322506461 wait complete. 2024-03-17T23:48:11.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.817+0000 7fea0289a640 1 Processor -- start 2024-03-17T23:48:11.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.817+0000 7fea0289a640 1 -- start start 2024-03-17T23:48:11.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.818+0000 7fea0289a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9fc067700 0x7fe9fc09dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:11.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.818+0000 7fea0289a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe9fc072a20 0x7fe9fc0c9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:11.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.818+0000 7fea0289a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe9fc15dfa0 0x7fe9fc0c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:11.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.818+0000 7fea0289a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc15e500 con 0x7fe9fc067700 2024-03-17T23:48:11.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.818+0000 7fea0289a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc15e380 con 0x7fe9fc072a20 2024-03-17T23:48:11.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.818+0000 7fea0289a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe9fc06b6d0 con 0x7fe9fc15dfa0 2024-03-17T23:48:11.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.818+0000 7fe9fbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9fc067700 0x7fe9fc09dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:11.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.818+0000 7fe9fbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9fc067700 0x7fe9fc09dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43630/0 (socket says 172.21.15.150:43630) 2024-03-17T23:48:11.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.818+0000 7fe9fbfff640 1 -- 172.21.15.150:0/3427079619 learned_addr learned my addr 172.21.15.150:0/3427079619 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:11.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.818+0000 7fe9fb7fe640 1 --2- 172.21.15.150:0/3427079619 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe9fc072a20 0x7fe9fc0c9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:11.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.818+0000 7fea00e37640 1 --2- 172.21.15.150:0/3427079619 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe9fc15dfa0 0x7fe9fc0c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:11.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.818+0000 7fe9fbfff640 1 -- 172.21.15.150:0/3427079619 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe9fc15dfa0 msgr2=0x7fe9fc0c9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:11.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.818+0000 7fe9fbfff640 1 --2- 172.21.15.150:0/3427079619 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe9fc15dfa0 0x7fe9fc0c9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:11.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.818+0000 7fe9fbfff640 1 -- 172.21.15.150:0/3427079619 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe9fc072a20 msgr2=0x7fe9fc0c9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:11.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.818+0000 7fe9fbfff640 1 --2- 172.21.15.150:0/3427079619 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe9fc072a20 0x7fe9fc0c9520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:11.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.818+0000 7fe9fbfff640 1 -- 172.21.15.150:0/3427079619 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9fc14ab90 con 0x7fe9fc067700 2024-03-17T23:48:11.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.818+0000 7fe9fbfff640 1 --2- 172.21.15.150:0/3427079619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9fc067700 0x7fe9fc09dd80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec02b8a0 tx=0x7fe9ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:11.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.818+0000 7fe9f8ff9640 1 -- 172.21.15.150:0/3427079619 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec032860 con 0x7fe9fc067700 2024-03-17T23:48:11.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.818+0000 7fe9f8ff9640 1 -- 172.21.15.150:0/3427079619 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9ec032a20 con 0x7fe9fc067700 2024-03-17T23:48:11.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.818+0000 7fe9f8ff9640 1 -- 172.21.15.150:0/3427079619 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec032e10 con 0x7fe9fc067700 2024-03-17T23:48:11.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.818+0000 7fea0289a640 1 -- 172.21.15.150:0/3427079619 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9fc14ae70 con 0x7fe9fc067700 2024-03-17T23:48:11.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.818+0000 7fea0289a640 1 -- 172.21.15.150:0/3427079619 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9fc09c0a0 con 0x7fe9fc067700 2024-03-17T23:48:11.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.819+0000 7fea0289a640 1 -- 172.21.15.150:0/3427079619 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9fc15dae0 con 0x7fe9fc067700 2024-03-17T23:48:11.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.820+0000 7fe9f8ff9640 1 -- 172.21.15.150:0/3427079619 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe9ec04c060 con 0x7fe9fc067700 2024-03-17T23:48:11.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.820+0000 7fe9f8ff9640 1 --2- 172.21.15.150:0/3427079619 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe9e806bbd0 0x7fe9e806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:11.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.820+0000 7fe9f8ff9640 1 -- 172.21.15.150:0/3427079619 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fe9ec0ac610 con 0x7fe9fc067700 2024-03-17T23:48:11.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.820+0000 7fe9fb7fe640 1 --2- 172.21.15.150:0/3427079619 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe9e806bbd0 0x7fe9e806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:11.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.821+0000 7fe9fb7fe640 1 --2- 172.21.15.150:0/3427079619 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe9e806bbd0 0x7fe9e806e090 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc161100 tx=0x7fe9f0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:11.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.825+0000 7fe9f8ff9640 1 -- 172.21.15.150:0/3427079619 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe9ec040070 con 0x7fe9fc067700 2024-03-17T23:48:11.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.998+0000 7fea0289a640 1 -- 172.21.15.150:0/3427079619 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe9fc0681a0 con 0x7fe9fc067700 2024-03-17T23:48:12.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:11.999+0000 7fe9f8ff9640 1 -- 172.21.15.150:0/3427079619 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16991 (secure 0 0 0) 0x7fe9ec031e90 con 0x7fe9fc067700 2024-03-17T23:48:12.000 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:12.000 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":39,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:48:10.730661+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:48:08.957879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-17T23:48:09.698808+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:48:12.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.001+0000 7fea0289a640 1 -- 172.21.15.150:0/3427079619 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe9e806bbd0 msgr2=0x7fe9e806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:12.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.001+0000 7fea0289a640 1 --2- 172.21.15.150:0/3427079619 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe9e806bbd0 0x7fe9e806e090 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc161100 tx=0x7fe9f0008040 comp rx=0 tx=0).stop 2024-03-17T23:48:12.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.001+0000 7fea0289a640 1 -- 172.21.15.150:0/3427079619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9fc067700 msgr2=0x7fe9fc09dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:12.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.001+0000 7fea0289a640 1 --2- 172.21.15.150:0/3427079619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9fc067700 0x7fe9fc09dd80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec02b8a0 tx=0x7fe9ec0046c0 comp rx=0 tx=0).stop 2024-03-17T23:48:12.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.001+0000 7fea0289a640 1 -- 172.21.15.150:0/3427079619 shutdown_connections 2024-03-17T23:48:12.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.001+0000 7fea0289a640 1 --2- 172.21.15.150:0/3427079619 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe9fc15dfa0 0x7fe9fc0c9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:12.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.001+0000 7fea0289a640 1 --2- 172.21.15.150:0/3427079619 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe9e806bbd0 0x7fe9e806e090 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:12.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.001+0000 7fea0289a640 1 --2- 172.21.15.150:0/3427079619 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe9fc072a20 0x7fe9fc0c9520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:12.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.001+0000 7fea0289a640 1 --2- 172.21.15.150:0/3427079619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9fc067700 0x7fe9fc09dd80 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:12.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.001+0000 7fea0289a640 1 -- 172.21.15.150:0/3427079619 >> 172.21.15.150:0/3427079619 conn(0x7fe9fc0b3da0 msgr2=0x7fe9fc0afde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:12.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.001+0000 7fea0289a640 1 -- 172.21.15.150:0/3427079619 shutdown_connections 2024-03-17T23:48:12.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.002+0000 7fea0289a640 1 -- 172.21.15.150:0/3427079619 wait complete. 2024-03-17T23:48:12.013 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-17T23:48:12.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.116+0000 7f8a13e0b640 1 Processor -- start 2024-03-17T23:48:12.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.116+0000 7f8a13e0b640 1 -- start start 2024-03-17T23:48:12.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.117+0000 7f8a13e0b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a0c15bac0 0x7f8a0c067570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:12.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.117+0000 7f8a13e0b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8a0c15f210 0x7f8a0c06e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:12.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.117+0000 7f8a13e0b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a0c099c10 con 0x7f8a0c072a20 2024-03-17T23:48:12.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.117+0000 7f8a13e0b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f8a0c099ef0 con 0x7f8a0c15f210 2024-03-17T23:48:12.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.117+0000 7f8a13e0b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8a0c15e730 con 0x7f8a0c15bac0 2024-03-17T23:48:12.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.117+0000 7f8a113a6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8a0c15f210 0x7f8a0c06e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:12.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.117+0000 7f8a123a8640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f8a0c072a20 0x7f8a0c161b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37570/0 (socket says 172.21.15.150:37570) 2024-03-17T23:48:12.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.117+0000 7f8a123a8640 1 -- 172.21.15.150:0/2609357233 learned_addr learned my addr 172.21.15.150:0/2609357233 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:12.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.117+0000 7f8a11ba7640 1 --2- 172.21.15.150:0/2609357233 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a0c15bac0 0x7f8a0c067570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:12.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.117+0000 7f8a113a6640 1 -- 172.21.15.150:0/2609357233 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a0c15bac0 msgr2=0x7f8a0c067570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:12.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.117+0000 7f8a113a6640 1 --2- 172.21.15.150:0/2609357233 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a0c15bac0 0x7f8a0c067570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:12.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.117+0000 7f8a113a6640 1 -- 172.21.15.150:0/2609357233 >> v1:172.21.15.150:6789/0 conn(0x7f8a0c072a20 legacy=0x7f8a0c161b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:12.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.117+0000 7f8a113a6640 1 -- 172.21.15.150:0/2609357233 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a0c15ea10 con 0x7f8a0c15f210 2024-03-17T23:48:12.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.118+0000 7f8a113a6640 1 --2- 172.21.15.150:0/2609357233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a0c15f210 0x7f8a0c06e580 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f89fc0097a0 tx=0x7f89fc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ca777747a8a8ad0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:12.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.118+0000 7f8a10ba5640 1 -- 172.21.15.150:0/2609357233 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc0315a0 con 0x7f8a0c15f210 2024-03-17T23:48:12.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.118+0000 7f8a10ba5640 1 -- 172.21.15.150:0/2609357233 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89fc037070 con 0x7f8a0c15f210 2024-03-17T23:48:12.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.118+0000 7f8a10ba5640 1 -- 172.21.15.150:0/2609357233 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc031960 con 0x7f8a0c15f210 2024-03-17T23:48:12.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.119+0000 7f8a13e0b640 1 -- 172.21.15.150:0/2609357233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a0c15f210 msgr2=0x7f8a0c06e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:12.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.119+0000 7f8a13e0b640 1 --2- 172.21.15.150:0/2609357233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a0c15f210 0x7f8a0c06e580 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f89fc0097a0 tx=0x7f89fc02b840 comp rx=0 tx=0).stop 2024-03-17T23:48:12.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.119+0000 7f8a13e0b640 1 -- 172.21.15.150:0/2609357233 shutdown_connections 2024-03-17T23:48:12.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.119+0000 7f8a13e0b640 1 --2- 172.21.15.150:0/2609357233 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a0c15f210 0x7f8a0c06e580 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:12.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.119+0000 7f8a13e0b640 1 --2- 172.21.15.150:0/2609357233 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a0c15bac0 0x7f8a0c067570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:12.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.119+0000 7f8a13e0b640 1 -- 172.21.15.150:0/2609357233 >> 172.21.15.150:0/2609357233 conn(0x7f8a0c0b3da0 msgr2=0x7f8a0c0b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:12.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.119+0000 7f8a13e0b640 1 -- 172.21.15.150:0/2609357233 shutdown_connections 2024-03-17T23:48:12.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.119+0000 7f8a13e0b640 1 -- 172.21.15.150:0/2609357233 wait complete. 2024-03-17T23:48:12.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.120+0000 7f8a13e0b640 1 Processor -- start 2024-03-17T23:48:12.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.120+0000 7f8a13e0b640 1 -- start start 2024-03-17T23:48:12.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.121+0000 7f8a13e0b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a0c072a20 0x7f8a0c14b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:12.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.121+0000 7f8a13e0b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a0c15bac0 0x7f8a0c0675f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:12.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.121+0000 7f8a13e0b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a0c15f210 0x7f8a0c159670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:12.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.121+0000 7f8a13e0b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c06ec40 con 0x7f8a0c15bac0 2024-03-17T23:48:12.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.121+0000 7f8a13e0b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c06eac0 con 0x7f8a0c072a20 2024-03-17T23:48:12.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.121+0000 7f8a13e0b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8a0c06edc0 con 0x7f8a0c15f210 2024-03-17T23:48:12.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.121+0000 7f8a113a6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a0c15bac0 0x7f8a0c0675f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:12.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.121+0000 7f8a113a6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a0c15bac0 0x7f8a0c0675f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43658/0 (socket says 172.21.15.150:43658) 2024-03-17T23:48:12.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.121+0000 7f8a113a6640 1 -- 172.21.15.150:0/969045228 learned_addr learned my addr 172.21.15.150:0/969045228 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:12.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.121+0000 7f8a123a8640 1 --2- 172.21.15.150:0/969045228 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a0c15f210 0x7f8a0c159670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:12.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.121+0000 7f8a11ba7640 1 --2- 172.21.15.150:0/969045228 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a0c072a20 0x7f8a0c14b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:12.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.121+0000 7f8a113a6640 1 -- 172.21.15.150:0/969045228 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a0c15f210 msgr2=0x7f8a0c159670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:12.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.121+0000 7f8a113a6640 1 --2- 172.21.15.150:0/969045228 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a0c15f210 0x7f8a0c159670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:12.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.121+0000 7f8a113a6640 1 -- 172.21.15.150:0/969045228 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a0c072a20 msgr2=0x7f8a0c14b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:12.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.121+0000 7f8a113a6640 1 --2- 172.21.15.150:0/969045228 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a0c072a20 0x7f8a0c14b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:12.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.121+0000 7f8a113a6640 1 -- 172.21.15.150:0/969045228 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a0c0b0380 con 0x7f8a0c15bac0 2024-03-17T23:48:12.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.121+0000 7f8a113a6640 1 --2- 172.21.15.150:0/969045228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a0c15bac0 0x7f8a0c0675f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f89fc034040 tx=0x7f89fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:12.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.121+0000 7f8a027fc640 1 -- 172.21.15.150:0/969045228 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc032840 con 0x7f8a0c15bac0 2024-03-17T23:48:12.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.121+0000 7f8a027fc640 1 -- 172.21.15.150:0/969045228 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89fc032a00 con 0x7f8a0c15bac0 2024-03-17T23:48:12.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.121+0000 7f8a027fc640 1 -- 172.21.15.150:0/969045228 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc032df0 con 0x7f8a0c15bac0 2024-03-17T23:48:12.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.121+0000 7f8a13e0b640 1 -- 172.21.15.150:0/969045228 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a0c0b0660 con 0x7f8a0c15bac0 2024-03-17T23:48:12.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.121+0000 7f8a13e0b640 1 -- 172.21.15.150:0/969045228 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a0c0b0c20 con 0x7f8a0c15bac0 2024-03-17T23:48:12.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.122+0000 7f8a13e0b640 1 -- 172.21.15.150:0/969045228 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a0c0681a0 con 0x7f8a0c15bac0 2024-03-17T23:48:12.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.123+0000 7f8a027fc640 1 -- 172.21.15.150:0/969045228 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f89fc04e430 con 0x7f8a0c15bac0 2024-03-17T23:48:12.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.123+0000 7f8a027fc640 1 --2- 172.21.15.150:0/969045228 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f89f006bbd0 0x7f89f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:12.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.123+0000 7f8a11ba7640 1 --2- 172.21.15.150:0/969045228 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f89f006bbd0 0x7f89f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:12.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.123+0000 7f8a027fc640 1 -- 172.21.15.150:0/969045228 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f89fc02f030 con 0x7f8a0c15bac0 2024-03-17T23:48:12.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.124+0000 7f8a11ba7640 1 --2- 172.21.15.150:0/969045228 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f89f006bbd0 0x7f89f006e090 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8a08006fd0 tx=0x7f8a08008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:12.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.128+0000 7f8a027fc640 1 -- 172.21.15.150:0/969045228 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f89fc040070 con 0x7f8a0c15bac0 2024-03-17T23:48:12.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.339+0000 7f8a13e0b640 1 -- 172.21.15.150:0/969045228 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f8a0c06d4c0 con 0x7f8a0c15bac0 2024-03-17T23:48:12.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.393+0000 7f8a027fc640 1 -- 172.21.15.150:0/969045228 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v14) ==== 118+0+0 (secure 0 0 0) 0x7f89fc07acf0 con 0x7f8a0c15bac0 2024-03-17T23:48:12.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.395+0000 7f8a13e0b640 1 -- 172.21.15.150:0/969045228 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f89f006bbd0 msgr2=0x7f89f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:12.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.395+0000 7f8a13e0b640 1 --2- 172.21.15.150:0/969045228 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f89f006bbd0 0x7f89f006e090 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8a08006fd0 tx=0x7f8a08008040 comp rx=0 tx=0).stop 2024-03-17T23:48:12.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.395+0000 7f8a13e0b640 1 -- 172.21.15.150:0/969045228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a0c15bac0 msgr2=0x7f8a0c0675f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:12.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.395+0000 7f8a13e0b640 1 --2- 172.21.15.150:0/969045228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a0c15bac0 0x7f8a0c0675f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f89fc034040 tx=0x7f89fc0046c0 comp rx=0 tx=0).stop 2024-03-17T23:48:12.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.396+0000 7f8a13e0b640 1 -- 172.21.15.150:0/969045228 shutdown_connections 2024-03-17T23:48:12.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.396+0000 7f8a13e0b640 1 --2- 172.21.15.150:0/969045228 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a0c15f210 0x7f8a0c159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:12.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.396+0000 7f8a13e0b640 1 --2- 172.21.15.150:0/969045228 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a0c15bac0 0x7f8a0c0675f0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:12.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.396+0000 7f8a13e0b640 1 --2- 172.21.15.150:0/969045228 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f89f006bbd0 0x7f89f006e090 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:12.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.396+0000 7f8a13e0b640 1 --2- 172.21.15.150:0/969045228 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a0c072a20 0x7f8a0c14b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:12.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.396+0000 7f8a13e0b640 1 -- 172.21.15.150:0/969045228 >> 172.21.15.150:0/969045228 conn(0x7f8a0c0b3da0 msgr2=0x7f8a0c15acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:12.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.396+0000 7f8a13e0b640 1 -- 172.21.15.150:0/969045228 shutdown_connections 2024-03-17T23:48:12.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.396+0000 7f8a13e0b640 1 -- 172.21.15.150:0/969045228 wait complete. 2024-03-17T23:48:12.408 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-17T23:48:12.408 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-17T23:48:12.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.512+0000 7f336bf5b640 1 Processor -- start 2024-03-17T23:48:12.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.512+0000 7f336bf5b640 1 -- start start 2024-03-17T23:48:12.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.513+0000 7f336bf5b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f33640d0ce0 0x7f33640b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:12.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.513+0000 7f336bf5b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f33640b2bf0 0x7f33640d1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:12.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.513+0000 7f336bf5b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3364161d30 con 0x7f3364161520 2024-03-17T23:48:12.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.513+0000 7f336bf5b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f33640b9bf0 con 0x7f33640b2bf0 2024-03-17T23:48:12.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.513+0000 7f336bf5b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3364162940 con 0x7f33640d0ce0 2024-03-17T23:48:12.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.513+0000 7f336a4f8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f33640b2bf0 0x7f33640d1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:12.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.513+0000 7f3369cf7640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f3364161520 0x7f33640d05d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37586/0 (socket says 172.21.15.150:37586) 2024-03-17T23:48:12.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.513+0000 7f3369cf7640 1 -- 172.21.15.150:0/3124852713 learned_addr learned my addr 172.21.15.150:0/3124852713 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:12.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.513+0000 7f33694f6640 1 --2- 172.21.15.150:0/3124852713 >> v2:172.21.15.157:3300/0 conn(0x7f33640d0ce0 0x7f33640b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:12.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.513+0000 7f336a4f8640 1 -- 172.21.15.150:0/3124852713 >> v2:172.21.15.157:3300/0 conn(0x7f33640d0ce0 msgr2=0x7f33640b96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:12.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.513+0000 7f336a4f8640 1 --2- 172.21.15.150:0/3124852713 >> v2:172.21.15.157:3300/0 conn(0x7f33640d0ce0 0x7f33640b96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:12.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.514+0000 7f336a4f8640 1 -- 172.21.15.150:0/3124852713 >> v1:172.21.15.150:6789/0 conn(0x7f3364161520 legacy=0x7f33640d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:12.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.514+0000 7f336a4f8640 1 -- 172.21.15.150:0/3124852713 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3364162010 con 0x7f33640b2bf0 2024-03-17T23:48:12.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.514+0000 7f3368cf5640 1 -- 172.21.15.150:0/3124852713 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 265004675 0 0) 0x7f3364161d30 con 0x7f3364161520 2024-03-17T23:48:12.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.514+0000 7f336a4f8640 1 --2- 172.21.15.150:0/3124852713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33640b2bf0 0x7f33640d1fa0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f33640b6c90 tx=0x7f335c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed69c32c3f75b461 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:12.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.514+0000 7f3368cf5640 1 -- 172.21.15.150:0/3124852713 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335c039070 con 0x7f33640b2bf0 2024-03-17T23:48:12.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.514+0000 7f3368cf5640 1 -- 172.21.15.150:0/3124852713 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f335c02cb60 con 0x7f33640b2bf0 2024-03-17T23:48:12.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.514+0000 7f3368cf5640 1 -- 172.21.15.150:0/3124852713 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335c034440 con 0x7f33640b2bf0 2024-03-17T23:48:12.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.515+0000 7f336bf5b640 1 -- 172.21.15.150:0/3124852713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33640b2bf0 msgr2=0x7f33640d1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:12.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.515+0000 7f336bf5b640 1 --2- 172.21.15.150:0/3124852713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33640b2bf0 0x7f33640d1fa0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f33640b6c90 tx=0x7f335c02bdb0 comp rx=0 tx=0).stop 2024-03-17T23:48:12.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.515+0000 7f336bf5b640 1 -- 172.21.15.150:0/3124852713 shutdown_connections 2024-03-17T23:48:12.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.515+0000 7f336bf5b640 1 --2- 172.21.15.150:0/3124852713 >> v2:172.21.15.157:3300/0 conn(0x7f33640d0ce0 0x7f33640b96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:12.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.515+0000 7f336bf5b640 1 --2- 172.21.15.150:0/3124852713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33640b2bf0 0x7f33640d1fa0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:12.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.515+0000 7f336bf5b640 1 -- 172.21.15.150:0/3124852713 >> 172.21.15.150:0/3124852713 conn(0x7f336406ca00 msgr2=0x7f336406ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:12.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.515+0000 7f336bf5b640 1 -- 172.21.15.150:0/3124852713 shutdown_connections 2024-03-17T23:48:12.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.515+0000 7f336bf5b640 1 -- 172.21.15.150:0/3124852713 wait complete. 2024-03-17T23:48:12.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.516+0000 7f336bf5b640 1 Processor -- start 2024-03-17T23:48:12.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.516+0000 7f336bf5b640 1 -- start start 2024-03-17T23:48:12.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.517+0000 7f336bf5b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f33640b2bf0 0x7f33640b4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:12.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.517+0000 7f336bf5b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33640d0ce0 0x7f3364151f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:12.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.517+0000 7f336bf5b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3364161520 0x7f33640b5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:12.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.517+0000 7f336bf5b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3364161900 con 0x7f33640d0ce0 2024-03-17T23:48:12.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.517+0000 7f336bf5b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f33641625d0 con 0x7f3364161520 2024-03-17T23:48:12.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.517+0000 7f336bf5b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f33640d24e0 con 0x7f33640b2bf0 2024-03-17T23:48:12.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.517+0000 7f33694f6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33640d0ce0 0x7f3364151f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:12.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.517+0000 7f33694f6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33640d0ce0 0x7f3364151f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43674/0 (socket says 172.21.15.150:43674) 2024-03-17T23:48:12.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.517+0000 7f33694f6640 1 -- 172.21.15.150:0/1041007221 learned_addr learned my addr 172.21.15.150:0/1041007221 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:12.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.517+0000 7f3369cf7640 1 --2- 172.21.15.150:0/1041007221 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f33640b2bf0 0x7f33640b4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:12.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.517+0000 7f336a4f8640 1 --2- 172.21.15.150:0/1041007221 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3364161520 0x7f33640b5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:12.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.517+0000 7f33694f6640 1 -- 172.21.15.150:0/1041007221 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f33640b2bf0 msgr2=0x7f33640b4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:12.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.517+0000 7f33694f6640 1 --2- 172.21.15.150:0/1041007221 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f33640b2bf0 0x7f33640b4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:12.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.517+0000 7f33694f6640 1 -- 172.21.15.150:0/1041007221 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3364161520 msgr2=0x7f33640b5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:12.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.517+0000 7f33694f6640 1 --2- 172.21.15.150:0/1041007221 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3364161520 0x7f33640b5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:12.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.517+0000 7f33694f6640 1 -- 172.21.15.150:0/1041007221 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33640cc780 con 0x7f33640d0ce0 2024-03-17T23:48:12.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.517+0000 7f33694f6640 1 --2- 172.21.15.150:0/1041007221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33640d0ce0 0x7f3364151f70 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f335400e980 tx=0x7f335400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:12.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.518+0000 7f335a7fc640 1 -- 172.21.15.150:0/1041007221 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3354017070 con 0x7f33640d0ce0 2024-03-17T23:48:12.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.518+0000 7f335a7fc640 1 -- 172.21.15.150:0/1041007221 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f335400ca00 con 0x7f33640d0ce0 2024-03-17T23:48:12.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.518+0000 7f336bf5b640 1 -- 172.21.15.150:0/1041007221 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33640cca60 con 0x7f33640d0ce0 2024-03-17T23:48:12.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.518+0000 7f335a7fc640 1 -- 172.21.15.150:0/1041007221 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335400cdd0 con 0x7f33640d0ce0 2024-03-17T23:48:12.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.518+0000 7f336bf5b640 1 -- 172.21.15.150:0/1041007221 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33640cd020 con 0x7f33640d0ce0 2024-03-17T23:48:12.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.519+0000 7f336bf5b640 1 -- 172.21.15.150:0/1041007221 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33640681a0 con 0x7f33640d0ce0 2024-03-17T23:48:12.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.519+0000 7f335a7fc640 1 -- 172.21.15.150:0/1041007221 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f335401a070 con 0x7f33640d0ce0 2024-03-17T23:48:12.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.519+0000 7f335a7fc640 1 --2- 172.21.15.150:0/1041007221 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f334806bbd0 0x7f334806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:12.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.520+0000 7f335a7fc640 1 -- 172.21.15.150:0/1041007221 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7f335408af80 con 0x7f33640d0ce0 2024-03-17T23:48:12.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.520+0000 7f3369cf7640 1 --2- 172.21.15.150:0/1041007221 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f334806bbd0 0x7f334806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:12.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.520+0000 7f3369cf7640 1 --2- 172.21.15.150:0/1041007221 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f334806bbd0 0x7f334806e090 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3360000f60 tx=0x7f3360007c00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:12.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.528+0000 7f335a7fc640 1 -- 172.21.15.150:0/1041007221 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3354059970 con 0x7f33640d0ce0 2024-03-17T23:48:12.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:12.746+0000 7f336bf5b640 1 -- 172.21.15.150:0/1041007221 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f33640d19c0 con 0x7f33640d0ce0 2024-03-17T23:48:13.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.395+0000 7f335a7fc640 1 -- 172.21.15.150:0/1041007221 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v15) ==== 122+0+0 (secure 0 0 0) 0x7f3354059270 con 0x7f33640d0ce0 2024-03-17T23:48:13.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.397+0000 7f336bf5b640 1 -- 172.21.15.150:0/1041007221 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f334806bbd0 msgr2=0x7f334806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:13.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.397+0000 7f336bf5b640 1 --2- 172.21.15.150:0/1041007221 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f334806bbd0 0x7f334806e090 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3360000f60 tx=0x7f3360007c00 comp rx=0 tx=0).stop 2024-03-17T23:48:13.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.397+0000 7f336bf5b640 1 -- 172.21.15.150:0/1041007221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33640d0ce0 msgr2=0x7f3364151f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:13.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.397+0000 7f336bf5b640 1 --2- 172.21.15.150:0/1041007221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33640d0ce0 0x7f3364151f70 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f335400e980 tx=0x7f335400ee40 comp rx=0 tx=0).stop 2024-03-17T23:48:13.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.397+0000 7f336bf5b640 1 -- 172.21.15.150:0/1041007221 shutdown_connections 2024-03-17T23:48:13.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.397+0000 7f336bf5b640 1 --2- 172.21.15.150:0/1041007221 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f334806bbd0 0x7f334806e090 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:13.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.397+0000 7f336bf5b640 1 --2- 172.21.15.150:0/1041007221 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3364161520 0x7f33640b5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:13.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.397+0000 7f336bf5b640 1 --2- 172.21.15.150:0/1041007221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33640d0ce0 0x7f3364151f70 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:13.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.397+0000 7f336bf5b640 1 --2- 172.21.15.150:0/1041007221 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f33640b2bf0 0x7f33640b4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:13.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.397+0000 7f336bf5b640 1 -- 172.21.15.150:0/1041007221 >> 172.21.15.150:0/1041007221 conn(0x7f336406ca00 msgr2=0x7f33640b6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:13.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.397+0000 7f336bf5b640 1 -- 172.21.15.150:0/1041007221 shutdown_connections 2024-03-17T23:48:13.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.397+0000 7f336bf5b640 1 -- 172.21.15.150:0/1041007221 wait complete. 2024-03-17T23:48:13.409 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-17T23:48:13.409 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:48:13.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.511+0000 7ff4c04e8640 1 Processor -- start 2024-03-17T23:48:13.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.511+0000 7ff4c04e8640 1 -- start start 2024-03-17T23:48:13.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.512+0000 7ff4c04e8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff4b815de30 0x7ff4b81619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:13.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.512+0000 7ff4c04e8640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4b80b9cf0 con 0x7ff4b80b9860 2024-03-17T23:48:13.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.512+0000 7ff4c04e8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4b81620a0 con 0x7ff4b80cf120 2024-03-17T23:48:13.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.512+0000 7ff4c04e8640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7ff4b8162940 con 0x7ff4b815de30 2024-03-17T23:48:13.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.512+0000 7ff4be284640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff4b815de30 0x7ff4b81619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:13.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.512+0000 7ff4bda83640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7ff4b80b9860 0x7ff4b80cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37596/0 (socket says 172.21.15.150:37596) 2024-03-17T23:48:13.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.512+0000 7ff4bda83640 1 -- 172.21.15.150:0/4020923585 learned_addr learned my addr 172.21.15.150:0/4020923585 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:13.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.512+0000 7ff4be284640 1 -- 172.21.15.150:0/4020923585 >> v1:172.21.15.157:6789/0 conn(0x7ff4b80cf120 legacy=0x7ff4b80d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:13.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.512+0000 7ff4be284640 1 -- 172.21.15.150:0/4020923585 >> v1:172.21.15.150:6789/0 conn(0x7ff4b80b9860 legacy=0x7ff4b80cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:13.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.512+0000 7ff4bd282640 1 -- 172.21.15.150:0/4020923585 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 131894417 0 0) 0x7ff4b80b9cf0 con 0x7ff4b80b9860 2024-03-17T23:48:13.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.512+0000 7ff4be284640 1 -- 172.21.15.150:0/4020923585 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b8162380 con 0x7ff4b815de30 2024-03-17T23:48:13.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.513+0000 7ff4be284640 1 --2- 172.21.15.150:0/4020923585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4b815de30 0x7ff4b81619a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ff4b40097a0 tx=0x7ff4b402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbf726f80e659dce server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:13.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.513+0000 7ff4bd282640 1 -- 172.21.15.150:0/4020923585 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4035020 con 0x7ff4b815de30 2024-03-17T23:48:13.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.513+0000 7ff4bd282640 1 -- 172.21.15.150:0/4020923585 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b4031830 con 0x7ff4b815de30 2024-03-17T23:48:13.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.513+0000 7ff4bd282640 1 -- 172.21.15.150:0/4020923585 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4031c00 con 0x7ff4b815de30 2024-03-17T23:48:13.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.514+0000 7ff4c04e8640 1 -- 172.21.15.150:0/4020923585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4b815de30 msgr2=0x7ff4b81619a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:13.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.514+0000 7ff4c04e8640 1 --2- 172.21.15.150:0/4020923585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4b815de30 0x7ff4b81619a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ff4b40097a0 tx=0x7ff4b402b8d0 comp rx=0 tx=0).stop 2024-03-17T23:48:13.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.514+0000 7ff4c04e8640 1 -- 172.21.15.150:0/4020923585 shutdown_connections 2024-03-17T23:48:13.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.514+0000 7ff4c04e8640 1 --2- 172.21.15.150:0/4020923585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4b815de30 0x7ff4b81619a0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:13.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.514+0000 7ff4c04e8640 1 -- 172.21.15.150:0/4020923585 >> 172.21.15.150:0/4020923585 conn(0x7ff4b80730b0 msgr2=0x7ff4b80d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:13.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.514+0000 7ff4c04e8640 1 -- 172.21.15.150:0/4020923585 shutdown_connections 2024-03-17T23:48:13.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.514+0000 7ff4c04e8640 1 -- 172.21.15.150:0/4020923585 wait complete. 2024-03-17T23:48:13.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.515+0000 7ff4c04e8640 1 Processor -- start 2024-03-17T23:48:13.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.515+0000 7ff4c04e8640 1 -- start start 2024-03-17T23:48:13.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.516+0000 7ff4c04e8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4b80b9860 0x7ff4b80ad0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:13.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.516+0000 7ff4c04e8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4b80cf120 0x7ff4b815e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:13.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.516+0000 7ff4c04e8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4b815de30 0x7ff4b81f70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:13.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.516+0000 7ff4c04e8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b815e210 con 0x7ff4b80cf120 2024-03-17T23:48:13.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.516+0000 7ff4c04e8640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b80d2170 con 0x7ff4b80b9860 2024-03-17T23:48:13.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.516+0000 7ff4c04e8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff4b80d2da0 con 0x7ff4b815de30 2024-03-17T23:48:13.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.516+0000 7ff4bda83640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4b80cf120 0x7ff4b815e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:13.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.516+0000 7ff4bda83640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4b80cf120 0x7ff4b815e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43688/0 (socket says 172.21.15.150:43688) 2024-03-17T23:48:13.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.516+0000 7ff4bda83640 1 -- 172.21.15.150:0/3213741341 learned_addr learned my addr 172.21.15.150:0/3213741341 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:13.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.516+0000 7ff4be284640 1 --2- 172.21.15.150:0/3213741341 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4b80b9860 0x7ff4b80ad0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:13.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.516+0000 7ff4bda83640 1 -- 172.21.15.150:0/3213741341 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4b815de30 msgr2=0x7ff4b81f70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:13.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.516+0000 7ff4bea85640 1 --2- 172.21.15.150:0/3213741341 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4b815de30 0x7ff4b81f70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:13.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.516+0000 7ff4bda83640 1 --2- 172.21.15.150:0/3213741341 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4b815de30 0x7ff4b81f70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:13.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.516+0000 7ff4bda83640 1 -- 172.21.15.150:0/3213741341 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4b80b9860 msgr2=0x7ff4b80ad0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:13.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.516+0000 7ff4bda83640 1 --2- 172.21.15.150:0/3213741341 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4b80b9860 0x7ff4b80ad0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:13.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.516+0000 7ff4bda83640 1 -- 172.21.15.150:0/3213741341 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b815ede0 con 0x7ff4b80cf120 2024-03-17T23:48:13.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.516+0000 7ff4be284640 1 --2- 172.21.15.150:0/3213741341 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4b80b9860 0x7ff4b80ad0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:13.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.516+0000 7ff4bea85640 1 --2- 172.21.15.150:0/3213741341 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4b815de30 0x7ff4b81f70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:13.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.516+0000 7ff4bda83640 1 --2- 172.21.15.150:0/3213741341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4b80cf120 0x7ff4b815e390 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff4a800c4a0 tx=0x7ff4a800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:13.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.516+0000 7ff4aeffd640 1 -- 172.21.15.150:0/3213741341 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8018070 con 0x7ff4b80cf120 2024-03-17T23:48:13.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.516+0000 7ff4aeffd640 1 -- 172.21.15.150:0/3213741341 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4a80074e0 con 0x7ff4b80cf120 2024-03-17T23:48:13.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.516+0000 7ff4aeffd640 1 -- 172.21.15.150:0/3213741341 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8004a20 con 0x7ff4b80cf120 2024-03-17T23:48:13.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.516+0000 7ff4c04e8640 1 -- 172.21.15.150:0/3213741341 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4b815f0c0 con 0x7ff4b80cf120 2024-03-17T23:48:13.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.516+0000 7ff4c04e8640 1 -- 172.21.15.150:0/3213741341 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4b815f960 con 0x7ff4b80cf120 2024-03-17T23:48:13.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.517+0000 7ff4c04e8640 1 -- 172.21.15.150:0/3213741341 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4b815f290 con 0x7ff4b80cf120 2024-03-17T23:48:13.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.518+0000 7ff4aeffd640 1 -- 172.21.15.150:0/3213741341 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff4a801b070 con 0x7ff4b80cf120 2024-03-17T23:48:13.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.519+0000 7ff4aeffd640 1 --2- 172.21.15.150:0/3213741341 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff49c06bc00 0x7ff49c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:13.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.519+0000 7ff4aeffd640 1 -- 172.21.15.150:0/3213741341 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7ff4a8089f20 con 0x7ff4b80cf120 2024-03-17T23:48:13.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.519+0000 7ff4be284640 1 --2- 172.21.15.150:0/3213741341 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff49c06bc00 0x7ff49c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:13.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.519+0000 7ff4be284640 1 --2- 172.21.15.150:0/3213741341 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff49c06bc00 0x7ff49c06e0c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4036000 tx=0x7ff4b4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:13.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.527+0000 7ff4aeffd640 1 -- 172.21.15.150:0/3213741341 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff4a8058880 con 0x7ff4b80cf120 2024-03-17T23:48:13.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.744+0000 7ff4c04e8640 1 -- 172.21.15.150:0/3213741341 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff4b80681a0 con 0x7ff4b80cf120 2024-03-17T23:48:13.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.745+0000 7ff4aeffd640 1 -- 172.21.15.150:0/3213741341 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+5193 (secure 0 0 0) 0x7ff4a8058180 con 0x7ff4b80cf120 2024-03-17T23:48:13.746 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:13.746 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5177,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6834/3126446677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":3126446677},{"type":"v1","addr":"172.21.15.157:6834","nonce":3126446677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1992779366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1992779366},{"type":"v1","addr":"172.21.15.150:6836","nonce":1992779366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5487,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/3615467865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6833","nonce":3615467865},{"type":"v1","addr":"172.21.15.157:6835","nonce":3615467865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-17T23:48:10.730994+0000","modified":"2024-03-17T23:48:12.747344+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5515":{"gid":5515,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6837/1413722826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1413722826},{"type":"v1","addr":"172.21.15.150:6837","nonce":1413722826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5515,"qdb_cluster":[5515]},"id":2}]} 2024-03-17T23:48:13.746 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 15 2024-03-17T23:48:13.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.747+0000 7ff4c04e8640 1 -- 172.21.15.150:0/3213741341 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff49c06bc00 msgr2=0x7ff49c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:13.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.747+0000 7ff4c04e8640 1 --2- 172.21.15.150:0/3213741341 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff49c06bc00 0x7ff49c06e0c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4036000 tx=0x7ff4b4008030 comp rx=0 tx=0).stop 2024-03-17T23:48:13.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.747+0000 7ff4c04e8640 1 -- 172.21.15.150:0/3213741341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4b80cf120 msgr2=0x7ff4b815e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:13.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.747+0000 7ff4c04e8640 1 --2- 172.21.15.150:0/3213741341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4b80cf120 0x7ff4b815e390 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff4a800c4a0 tx=0x7ff4a800c960 comp rx=0 tx=0).stop 2024-03-17T23:48:13.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.747+0000 7ff4c04e8640 1 -- 172.21.15.150:0/3213741341 shutdown_connections 2024-03-17T23:48:13.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.747+0000 7ff4c04e8640 1 --2- 172.21.15.150:0/3213741341 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4b815de30 0x7ff4b81f70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:13.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.747+0000 7ff4c04e8640 1 --2- 172.21.15.150:0/3213741341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4b80cf120 0x7ff4b815e390 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:13.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.747+0000 7ff4c04e8640 1 --2- 172.21.15.150:0/3213741341 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff49c06bc00 0x7ff49c06e0c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:13.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.747+0000 7ff4c04e8640 1 --2- 172.21.15.150:0/3213741341 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4b80b9860 0x7ff4b80ad0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:13.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.747+0000 7ff4c04e8640 1 -- 172.21.15.150:0/3213741341 >> 172.21.15.150:0/3213741341 conn(0x7ff4b80730b0 msgr2=0x7ff4b80b87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:13.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.747+0000 7ff4c04e8640 1 -- 172.21.15.150:0/3213741341 shutdown_connections 2024-03-17T23:48:13.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.747+0000 7ff4c04e8640 1 -- 172.21.15.150:0/3213741341 wait complete. 2024-03-17T23:48:13.759 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:48:13.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.860+0000 7f3ce1e36640 1 Processor -- start 2024-03-17T23:48:13.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.860+0000 7f3ce1e36640 1 -- start start 2024-03-17T23:48:13.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.861+0000 7f3ce1e36640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3cdc15ab90 0x7f3cdc15bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:13.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.861+0000 7f3ce1e36640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cdc0ce790 con 0x7f3cdc1514f0 2024-03-17T23:48:13.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.861+0000 7f3ce1e36640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cdc15b050 con 0x7f3cdc06ecd0 2024-03-17T23:48:13.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.861+0000 7f3ce1e36640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3cdc15b330 con 0x7f3cdc15ab90 2024-03-17T23:48:13.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.861+0000 7f3cdb7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f3cdc1514f0 0x7f3cdc06e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37610/0 (socket says 172.21.15.150:37610) 2024-03-17T23:48:13.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.861+0000 7f3cdb7fe640 1 -- 172.21.15.150:0/2075778931 learned_addr learned my addr 172.21.15.150:0/2075778931 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:13.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.861+0000 7f3cd2ffd640 1 --2- 172.21.15.150:0/2075778931 >> v2:172.21.15.157:3300/0 conn(0x7f3cdc15ab90 0x7f3cdc15bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:13.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.861+0000 7f3cdaffd640 1 -- 172.21.15.150:0/2075778931 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3587372490 0 0) 0x7f3ccc001110 con 0x7f3cdc1514f0 2024-03-17T23:48:13.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.862+0000 7f3cdaffd640 1 -- 172.21.15.150:0/2075778931 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cc8003520 con 0x7f3cdc1514f0 2024-03-17T23:48:13.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.862+0000 7f3cdaffd640 1 -- 172.21.15.150:0/2075778931 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2805990158 0 0) 0x7f3cc8003520 con 0x7f3cdc1514f0 2024-03-17T23:48:13.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.862+0000 7f3cdaffd640 1 -- 172.21.15.150:0/2075778931 >> v2:172.21.15.157:3300/0 conn(0x7f3cdc15ab90 msgr2=0x7f3cdc15bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:13.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.862+0000 7f3cdaffd640 1 --2- 172.21.15.150:0/2075778931 >> v2:172.21.15.157:3300/0 conn(0x7f3cdc15ab90 0x7f3cdc15bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:13.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.862+0000 7f3cdaffd640 1 -- 172.21.15.150:0/2075778931 >> v1:172.21.15.157:6789/0 conn(0x7f3cdc06ecd0 legacy=0x7f3cdc0cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:13.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.862+0000 7f3cdaffd640 1 -- 172.21.15.150:0/2075778931 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cdc15b610 con 0x7f3cdc1514f0 2024-03-17T23:48:13.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.862+0000 7f3cdaffd640 1 -- 172.21.15.150:0/2075778931 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f3ccc001840 con 0x7f3cdc1514f0 2024-03-17T23:48:13.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.862+0000 7f3cdaffd640 1 -- 172.21.15.150:0/2075778931 >> v1:172.21.15.150:6789/0 conn(0x7f3cdc1514f0 legacy=0x7f3cdc06e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:13.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.862+0000 7f3cdaffd640 1 --2- 172.21.15.150:0/2075778931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cc80039b0 0x7f3cc8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:13.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.862+0000 7f3cdaffd640 1 --2- 172.21.15.150:0/2075778931 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3cc8004800 0x7f3cc8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:13.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.862+0000 7f3cdaffd640 1 --2- 172.21.15.150:0/2075778931 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3cc80051c0 0x7f3cc800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:13.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.862+0000 7f3cdaffd640 1 -- 172.21.15.150:0/2075778931 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3cdc15b330 con 0x7f3cc80039b0 2024-03-17T23:48:13.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.862+0000 7f3cdaffd640 1 -- 172.21.15.150:0/2075778931 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc800c060 con 0x7f3cc80051c0 2024-03-17T23:48:13.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.862+0000 7f3cdaffd640 1 -- 172.21.15.150:0/2075778931 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3cc800c260 con 0x7f3cc8004800 2024-03-17T23:48:13.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.862+0000 7f3cdb7fe640 1 --2- 172.21.15.150:0/2075778931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cc80039b0 0x7f3cc8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:13.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.862+0000 7f3cdbfff640 1 --2- 172.21.15.150:0/2075778931 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3cc80051c0 0x7f3cc800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:13.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.862+0000 7f3cd2ffd640 1 --2- 172.21.15.150:0/2075778931 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3cc8004800 0x7f3cc8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:13.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.862+0000 7f3cdb7fe640 1 -- 172.21.15.150:0/2075778931 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3cc8004800 msgr2=0x7f3cc8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:13.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.862+0000 7f3cdb7fe640 1 --2- 172.21.15.150:0/2075778931 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3cc8004800 0x7f3cc8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:13.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.862+0000 7f3cdb7fe640 1 -- 172.21.15.150:0/2075778931 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3cc80051c0 msgr2=0x7f3cc800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:13.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.862+0000 7f3cdb7fe640 1 --2- 172.21.15.150:0/2075778931 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3cc80051c0 0x7f3cc800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:13.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.862+0000 7f3cdb7fe640 1 -- 172.21.15.150:0/2075778931 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cc8003520 con 0x7f3cc80039b0 2024-03-17T23:48:13.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.862+0000 7f3cdbfff640 1 -- 172.21.15.150:0/2075778931 reap_dead start 2024-03-17T23:48:13.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.863+0000 7f3cdb7fe640 1 --2- 172.21.15.150:0/2075778931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cc80039b0 0x7f3cc8003dd0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc00c360 tx=0x7f3cc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b665e4445d45d196 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:13.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.863+0000 7f3cdaffd640 1 -- 172.21.15.150:0/2075778931 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc00c0e0 con 0x7f3cc80039b0 2024-03-17T23:48:13.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.863+0000 7f3cdaffd640 1 -- 172.21.15.150:0/2075778931 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ccc005af0 con 0x7f3cc80039b0 2024-03-17T23:48:13.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.863+0000 7f3cdaffd640 1 -- 172.21.15.150:0/2075778931 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc007070 con 0x7f3cc80039b0 2024-03-17T23:48:13.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.864+0000 7f3ce1e36640 1 -- 172.21.15.150:0/2075778931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cc80039b0 msgr2=0x7f3cc8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:13.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.864+0000 7f3ce1e36640 1 --2- 172.21.15.150:0/2075778931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cc80039b0 0x7f3cc8003dd0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc00c360 tx=0x7f3cc8005830 comp rx=0 tx=0).stop 2024-03-17T23:48:13.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.864+0000 7f3ce1e36640 1 -- 172.21.15.150:0/2075778931 shutdown_connections 2024-03-17T23:48:13.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.864+0000 7f3ce1e36640 1 --2- 172.21.15.150:0/2075778931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cc80039b0 0x7f3cc8003dd0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:13.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.864+0000 7f3ce1e36640 1 -- 172.21.15.150:0/2075778931 >> 172.21.15.150:0/2075778931 conn(0x7f3cdc06b190 msgr2=0x7f3cdc06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:13.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.864+0000 7f3ce1e36640 1 -- 172.21.15.150:0/2075778931 shutdown_connections 2024-03-17T23:48:13.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.864+0000 7f3ce1e36640 1 -- 172.21.15.150:0/2075778931 wait complete. 2024-03-17T23:48:13.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.865+0000 7f3ce1e36640 1 Processor -- start 2024-03-17T23:48:13.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.865+0000 7f3ce1e36640 1 -- start start 2024-03-17T23:48:13.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.866+0000 7f3ce1e36640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cc80039b0 0x7f3cdc13d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:13.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.866+0000 7f3ce1e36640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3cdc13dd00 0x7f3cdc149d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:13.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.866+0000 7f3ce1e36640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3cdc14a270 0x7f3cdc1ef030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:13.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.866+0000 7f3ce1e36640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3cdc1518d0 con 0x7f3cc80039b0 2024-03-17T23:48:13.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.866+0000 7f3ce1e36640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3cdc0af510 con 0x7f3cdc14a270 2024-03-17T23:48:13.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.866+0000 7f3ce1e36640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3cdc151a50 con 0x7f3cdc13dd00 2024-03-17T23:48:13.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.866+0000 7f3cdb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cc80039b0 0x7f3cdc13d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:13.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.866+0000 7f3cdb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cc80039b0 0x7f3cdc13d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43706/0 (socket says 172.21.15.150:43706) 2024-03-17T23:48:13.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.866+0000 7f3cdb7fe640 1 -- 172.21.15.150:0/3023363691 learned_addr learned my addr 172.21.15.150:0/3023363691 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:13.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.866+0000 7f3cd2ffd640 1 --2- 172.21.15.150:0/3023363691 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3cdc13dd00 0x7f3cdc149d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:13.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.866+0000 7f3cdbfff640 1 --2- 172.21.15.150:0/3023363691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3cdc14a270 0x7f3cdc1ef030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:13.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.866+0000 7f3cdb7fe640 1 -- 172.21.15.150:0/3023363691 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3cdc13dd00 msgr2=0x7f3cdc149d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:13.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.866+0000 7f3cdb7fe640 1 --2- 172.21.15.150:0/3023363691 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3cdc13dd00 0x7f3cdc149d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:13.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.866+0000 7f3cdb7fe640 1 -- 172.21.15.150:0/3023363691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3cdc14a270 msgr2=0x7f3cdc1ef030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:13.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.866+0000 7f3cdb7fe640 1 --2- 172.21.15.150:0/3023363691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3cdc14a270 0x7f3cdc1ef030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:13.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.866+0000 7f3cdb7fe640 1 -- 172.21.15.150:0/3023363691 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cdc14aa30 con 0x7f3cc80039b0 2024-03-17T23:48:13.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.866+0000 7f3cdbfff640 1 --2- 172.21.15.150:0/3023363691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3cdc14a270 0x7f3cdc1ef030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:13.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.867+0000 7f3cdb7fe640 1 --2- 172.21.15.150:0/3023363691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cc80039b0 0x7f3cdc13d7c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc00c480 tx=0x7f3cdc15b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:13.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.867+0000 7f3cd8ff9640 1 -- 172.21.15.150:0/3023363691 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc001600 con 0x7f3cc80039b0 2024-03-17T23:48:13.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.867+0000 7f3cd8ff9640 1 -- 172.21.15.150:0/3023363691 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ccc048070 con 0x7f3cc80039b0 2024-03-17T23:48:13.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.867+0000 7f3cd8ff9640 1 -- 172.21.15.150:0/3023363691 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc007070 con 0x7f3cc80039b0 2024-03-17T23:48:13.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.867+0000 7f3ce1e36640 1 -- 172.21.15.150:0/3023363691 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cdc14ad10 con 0x7f3cc80039b0 2024-03-17T23:48:13.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.867+0000 7f3ce1e36640 1 -- 172.21.15.150:0/3023363691 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cdc14b2d0 con 0x7f3cc80039b0 2024-03-17T23:48:13.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.868+0000 7f3ce1e36640 1 -- 172.21.15.150:0/3023363691 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cdc0681a0 con 0x7f3cc80039b0 2024-03-17T23:48:13.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.869+0000 7f3cd8ff9640 1 -- 172.21.15.150:0/3023363691 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3ccc007210 con 0x7f3cc80039b0 2024-03-17T23:48:13.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.869+0000 7f3cd8ff9640 1 --2- 172.21.15.150:0/3023363691 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3cbc06bbd0 0x7f3cbc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:13.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.869+0000 7f3cd2ffd640 1 --2- 172.21.15.150:0/3023363691 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3cbc06bbd0 0x7f3cbc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:13.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.869+0000 7f3cd8ff9640 1 -- 172.21.15.150:0/3023363691 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7f3ccc0b0910 con 0x7f3cc80039b0 2024-03-17T23:48:13.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.870+0000 7f3cd2ffd640 1 --2- 172.21.15.150:0/3023363691 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3cbc06bbd0 0x7f3cbc06e090 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc14bb60 tx=0x7f3cc4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:13.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:13.874+0000 7f3cd8ff9640 1 -- 172.21.15.150:0/3023363691 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3cdc0ce790 con 0x7f3cc80039b0 2024-03-17T23:48:14.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.053+0000 7f3ce1e36640 1 -- 172.21.15.150:0/3023363691 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3cdc15ed30 con 0x7f3cc80039b0 2024-03-17T23:48:14.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.054+0000 7f3cd8ff9640 1 -- 172.21.15.150:0/3023363691 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) ==== 74+0+16978 (secure 0 0 0) 0x7f3ccc005ac0 con 0x7f3cc80039b0 2024-03-17T23:48:14.055 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:14.055 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":40,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:48:12.387924+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:48:08.957879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-17T23:48:09.698808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:48:14.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.056+0000 7f3ce1e36640 1 -- 172.21.15.150:0/3023363691 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3cbc06bbd0 msgr2=0x7f3cbc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:14.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.056+0000 7f3ce1e36640 1 --2- 172.21.15.150:0/3023363691 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3cbc06bbd0 0x7f3cbc06e090 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc14bb60 tx=0x7f3cc4009190 comp rx=0 tx=0).stop 2024-03-17T23:48:14.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.056+0000 7f3ce1e36640 1 -- 172.21.15.150:0/3023363691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cc80039b0 msgr2=0x7f3cdc13d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:14.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.056+0000 7f3ce1e36640 1 --2- 172.21.15.150:0/3023363691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cc80039b0 0x7f3cdc13d7c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc00c480 tx=0x7f3cdc15b7e0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.056+0000 7f3ce1e36640 1 -- 172.21.15.150:0/3023363691 shutdown_connections 2024-03-17T23:48:14.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.056+0000 7f3ce1e36640 1 --2- 172.21.15.150:0/3023363691 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3cbc06bbd0 0x7f3cbc06e090 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.056+0000 7f3ce1e36640 1 --2- 172.21.15.150:0/3023363691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3cdc14a270 0x7f3cdc1ef030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.056+0000 7f3ce1e36640 1 --2- 172.21.15.150:0/3023363691 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3cdc13dd00 0x7f3cdc149d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.056+0000 7f3ce1e36640 1 --2- 172.21.15.150:0/3023363691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cc80039b0 0x7f3cdc13d7c0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.056+0000 7f3ce1e36640 1 -- 172.21.15.150:0/3023363691 >> 172.21.15.150:0/3023363691 conn(0x7f3cdc06b190 msgr2=0x7f3cdc15dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:14.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.056+0000 7f3ce1e36640 1 -- 172.21.15.150:0/3023363691 shutdown_connections 2024-03-17T23:48:14.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.056+0000 7f3ce1e36640 1 -- 172.21.15.150:0/3023363691 wait complete. 2024-03-17T23:48:14.068 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-17T23:48:14.069 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-17T23:48:14.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.170+0000 7fce62bda640 1 Processor -- start 2024-03-17T23:48:14.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.170+0000 7fce62bda640 1 -- start start 2024-03-17T23:48:14.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.171+0000 7fce62bda640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fce5c15de30 0x7fce5c1619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:14.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.171+0000 7fce62bda640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fce5c0cf120 0x7fce5c0d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:14.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.171+0000 7fce62bda640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce5c0b9cf0 con 0x7fce5c0b9860 2024-03-17T23:48:14.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.171+0000 7fce62bda640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fce5c162940 con 0x7fce5c0cf120 2024-03-17T23:48:14.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.171+0000 7fce62bda640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fce5c1620a0 con 0x7fce5c15de30 2024-03-17T23:48:14.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.171+0000 7fce61177640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fce5c0cf120 0x7fce5c0d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:14.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.171+0000 7fce5bfff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fce5c0b9860 0x7fce5c0cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37616/0 (socket says 172.21.15.150:37616) 2024-03-17T23:48:14.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.171+0000 7fce5bfff640 1 -- 172.21.15.150:0/2605397627 learned_addr learned my addr 172.21.15.150:0/2605397627 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:14.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.171+0000 7fce60976640 1 --2- 172.21.15.150:0/2605397627 >> v2:172.21.15.157:3300/0 conn(0x7fce5c15de30 0x7fce5c1619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:14.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.171+0000 7fce61177640 1 -- 172.21.15.150:0/2605397627 >> v2:172.21.15.157:3300/0 conn(0x7fce5c15de30 msgr2=0x7fce5c1619a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:14.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.171+0000 7fce61177640 1 --2- 172.21.15.150:0/2605397627 >> v2:172.21.15.157:3300/0 conn(0x7fce5c15de30 0x7fce5c1619a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.171+0000 7fce61177640 1 -- 172.21.15.150:0/2605397627 >> v1:172.21.15.150:6789/0 conn(0x7fce5c0b9860 legacy=0x7fce5c0cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:14.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.171+0000 7fce5b7fe640 1 -- 172.21.15.150:0/2605397627 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4032438651 0 0) 0x7fce5c0b9cf0 con 0x7fce5c0b9860 2024-03-17T23:48:14.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.171+0000 7fce61177640 1 -- 172.21.15.150:0/2605397627 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce5c162380 con 0x7fce5c0cf120 2024-03-17T23:48:14.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.172+0000 7fce61177640 1 --2- 172.21.15.150:0/2605397627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5c0cf120 0x7fce5c0d2690 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fce5c06d7c0 tx=0x7fce5402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de15bf0fd311deeb server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:14.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.172+0000 7fce5b7fe640 1 -- 172.21.15.150:0/2605397627 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce54039070 con 0x7fce5c0cf120 2024-03-17T23:48:14.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.172+0000 7fce5b7fe640 1 -- 172.21.15.150:0/2605397627 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce5402cbf0 con 0x7fce5c0cf120 2024-03-17T23:48:14.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.172+0000 7fce5b7fe640 1 -- 172.21.15.150:0/2605397627 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce54034440 con 0x7fce5c0cf120 2024-03-17T23:48:14.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.173+0000 7fce62bda640 1 -- 172.21.15.150:0/2605397627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5c0cf120 msgr2=0x7fce5c0d2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:14.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.173+0000 7fce62bda640 1 --2- 172.21.15.150:0/2605397627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5c0cf120 0x7fce5c0d2690 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fce5c06d7c0 tx=0x7fce5402be40 comp rx=0 tx=0).stop 2024-03-17T23:48:14.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.173+0000 7fce62bda640 1 -- 172.21.15.150:0/2605397627 shutdown_connections 2024-03-17T23:48:14.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.173+0000 7fce62bda640 1 --2- 172.21.15.150:0/2605397627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5c0cf120 0x7fce5c0d2690 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.173+0000 7fce62bda640 1 --2- 172.21.15.150:0/2605397627 >> v2:172.21.15.157:3300/0 conn(0x7fce5c15de30 0x7fce5c1619a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.173+0000 7fce62bda640 1 -- 172.21.15.150:0/2605397627 >> 172.21.15.150:0/2605397627 conn(0x7fce5c0730b0 msgr2=0x7fce5c0d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:14.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.173+0000 7fce62bda640 1 -- 172.21.15.150:0/2605397627 shutdown_connections 2024-03-17T23:48:14.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.173+0000 7fce62bda640 1 -- 172.21.15.150:0/2605397627 wait complete. 2024-03-17T23:48:14.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.174+0000 7fce62bda640 1 Processor -- start 2024-03-17T23:48:14.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.174+0000 7fce62bda640 1 -- start start 2024-03-17T23:48:14.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.175+0000 7fce62bda640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5c0b9860 0x7fce5c152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:14.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.175+0000 7fce62bda640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce5c0cf120 0x7fce5c15ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:14.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.175+0000 7fce62bda640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce5c15de30 0x7fce5c15e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:14.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.175+0000 7fce62bda640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fce5c15e210 con 0x7fce5c0b9860 2024-03-17T23:48:14.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.175+0000 7fce62bda640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fce5c0d2170 con 0x7fce5c0cf120 2024-03-17T23:48:14.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.175+0000 7fce62bda640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fce5c0d2bd0 con 0x7fce5c15de30 2024-03-17T23:48:14.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.175+0000 7fce60976640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5c0b9860 0x7fce5c152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:14.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.175+0000 7fce60976640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5c0b9860 0x7fce5c152680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43726/0 (socket says 172.21.15.150:43726) 2024-03-17T23:48:14.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.175+0000 7fce60976640 1 -- 172.21.15.150:0/1899018411 learned_addr learned my addr 172.21.15.150:0/1899018411 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:14.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.175+0000 7fce5bfff640 1 --2- 172.21.15.150:0/1899018411 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce5c0cf120 0x7fce5c15ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:14.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.175+0000 7fce61177640 1 --2- 172.21.15.150:0/1899018411 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce5c15de30 0x7fce5c15e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:14.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.175+0000 7fce60976640 1 -- 172.21.15.150:0/1899018411 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce5c15de30 msgr2=0x7fce5c15e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:14.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.175+0000 7fce60976640 1 --2- 172.21.15.150:0/1899018411 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce5c15de30 0x7fce5c15e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.175+0000 7fce60976640 1 -- 172.21.15.150:0/1899018411 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce5c0cf120 msgr2=0x7fce5c15ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:14.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.175+0000 7fce60976640 1 --2- 172.21.15.150:0/1899018411 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce5c0cf120 0x7fce5c15ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.175+0000 7fce60976640 1 -- 172.21.15.150:0/1899018411 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce5c0cd0a0 con 0x7fce5c0b9860 2024-03-17T23:48:14.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.175+0000 7fce60976640 1 --2- 172.21.15.150:0/1899018411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5c0b9860 0x7fce5c152680 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fce5000d620 tx=0x7fce5000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:14.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.176+0000 7fce597fa640 1 -- 172.21.15.150:0/1899018411 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce500125f0 con 0x7fce5c0b9860 2024-03-17T23:48:14.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.176+0000 7fce597fa640 1 -- 172.21.15.150:0/1899018411 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce50019070 con 0x7fce5c0b9860 2024-03-17T23:48:14.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.176+0000 7fce597fa640 1 -- 172.21.15.150:0/1899018411 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce500129d0 con 0x7fce5c0b9860 2024-03-17T23:48:14.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.176+0000 7fce62bda640 1 -- 172.21.15.150:0/1899018411 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce5c15eec0 con 0x7fce5c0b9860 2024-03-17T23:48:14.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.176+0000 7fce62bda640 1 -- 172.21.15.150:0/1899018411 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce5c15f480 con 0x7fce5c0b9860 2024-03-17T23:48:14.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.177+0000 7fce62bda640 1 -- 172.21.15.150:0/1899018411 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce5c0681a0 con 0x7fce5c0b9860 2024-03-17T23:48:14.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.177+0000 7fce597fa640 1 -- 172.21.15.150:0/1899018411 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fce50012b70 con 0x7fce5c0b9860 2024-03-17T23:48:14.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.178+0000 7fce597fa640 1 --2- 172.21.15.150:0/1899018411 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fce4006bbd0 0x7fce4006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:14.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.178+0000 7fce5bfff640 1 --2- 172.21.15.150:0/1899018411 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fce4006bbd0 0x7fce4006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:14.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.178+0000 7fce597fa640 1 -- 172.21.15.150:0/1899018411 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7fce5008aea0 con 0x7fce5c0b9860 2024-03-17T23:48:14.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.178+0000 7fce5bfff640 1 --2- 172.21.15.150:0/1899018411 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fce4006bbd0 0x7fce4006e090 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fce5c161730 tx=0x7fce48007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:14.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.182+0000 7fce597fa640 1 -- 172.21.15.150:0/1899018411 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fce500591b0 con 0x7fce5c0b9860 2024-03-17T23:48:14.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.428+0000 7fce62bda640 1 -- 172.21.15.150:0/1899018411 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fce5c0d1910 con 0x7fce4006bbd0 2024-03-17T23:48:14.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.434+0000 7fce597fa640 1 -- 172.21.15.150:0/1899018411 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246405 (secure 0 0 0) 0x7fce5c0d1910 con 0x7fce4006bbd0 2024-03-17T23:48:14.437 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:14.439 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-03-17T23:48:14.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.438+0000 7fce62bda640 1 -- 172.21.15.150:0/1899018411 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fce4006bbd0 msgr2=0x7fce4006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:14.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.438+0000 7fce62bda640 1 --2- 172.21.15.150:0/1899018411 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fce4006bbd0 0x7fce4006e090 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fce5c161730 tx=0x7fce48007d50 comp rx=0 tx=0).stop 2024-03-17T23:48:14.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.438+0000 7fce62bda640 1 -- 172.21.15.150:0/1899018411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5c0b9860 msgr2=0x7fce5c152680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:14.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.438+0000 7fce62bda640 1 --2- 172.21.15.150:0/1899018411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5c0b9860 0x7fce5c152680 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fce5000d620 tx=0x7fce5000dae0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.438+0000 7fce62bda640 1 -- 172.21.15.150:0/1899018411 shutdown_connections 2024-03-17T23:48:14.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.438+0000 7fce62bda640 1 --2- 172.21.15.150:0/1899018411 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce5c15de30 0x7fce5c15e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.438+0000 7fce62bda640 1 --2- 172.21.15.150:0/1899018411 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fce4006bbd0 0x7fce4006e090 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.438+0000 7fce62bda640 1 --2- 172.21.15.150:0/1899018411 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce5c0cf120 0x7fce5c15ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.438+0000 7fce62bda640 1 --2- 172.21.15.150:0/1899018411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5c0b9860 0x7fce5c152680 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.438+0000 7fce62bda640 1 -- 172.21.15.150:0/1899018411 >> 172.21.15.150:0/1899018411 conn(0x7fce5c0730b0 msgr2=0x7fce5c0ce6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:14.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.438+0000 7fce62bda640 1 -- 172.21.15.150:0/1899018411 shutdown_connections 2024-03-17T23:48:14.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.438+0000 7fce62bda640 1 -- 172.21.15.150:0/1899018411 wait complete. 2024-03-17T23:48:14.449 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2024-03-17T23:48:13.612979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591062,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":192,"num_write_kb":2419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":122,"acting":122,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":92,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":6,"kb":754974720,"kb_used":237408,"kb_used_data":4592,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737312,"statfs":{"total":773094113280,"available":772851007488,"internally_reserved":0,"allocated":4702208,"data_stored":2684865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":694,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.241889"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.400417+0000","last_change":"2024-03-17T23:48:11.399627+0000","last_active":"2024-03-17T23:48:11.400417+0000","last_peered":"2024-03-17T23:48:11.400417+0000","last_clean":"2024-03-17T23:48:11.400417+0000","last_became_active":"2024-03-17T23:48:11.399364+0000","last_became_peered":"2024-03-17T23:48:11.399364+0000","last_unstale":"2024-03-17T23:48:11.400417+0000","last_undegraded":"2024-03-17T23:48:11.400417+0000","last_fullsized":"2024-03-17T23:48:11.400417+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:44:42.530638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409171+0000","last_change":"2024-03-17T23:48:11.403822+0000","last_active":"2024-03-17T23:48:11.409171+0000","last_peered":"2024-03-17T23:48:11.409171+0000","last_clean":"2024-03-17T23:48:11.409171+0000","last_became_active":"2024-03-17T23:48:11.403572+0000","last_became_peered":"2024-03-17T23:48:11.403572+0000","last_unstale":"2024-03-17T23:48:11.409171+0000","last_undegraded":"2024-03-17T23:48:11.409171+0000","last_fullsized":"2024-03-17T23:48:11.409171+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:53:53.408503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.406658+0000","last_change":"2024-03-17T23:48:11.395918+0000","last_active":"2024-03-17T23:48:11.406658+0000","last_peered":"2024-03-17T23:48:11.406658+0000","last_clean":"2024-03-17T23:48:11.406658+0000","last_became_active":"2024-03-17T23:48:11.394963+0000","last_became_peered":"2024-03-17T23:48:11.394963+0000","last_unstale":"2024-03-17T23:48:11.406658+0000","last_undegraded":"2024-03-17T23:48:11.406658+0000","last_fullsized":"2024-03-17T23:48:11.406658+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:20:48.927064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.399665+0000","last_change":"2024-03-17T23:48:11.394537+0000","last_active":"2024-03-17T23:48:11.399665+0000","last_peered":"2024-03-17T23:48:11.399665+0000","last_clean":"2024-03-17T23:48:11.399665+0000","last_became_active":"2024-03-17T23:48:11.394147+0000","last_became_peered":"2024-03-17T23:48:11.394147+0000","last_unstale":"2024-03-17T23:48:11.399665+0000","last_undegraded":"2024-03-17T23:48:11.399665+0000","last_fullsized":"2024-03-17T23:48:11.399665+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:45.409873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.394641+0000","last_change":"2024-03-17T23:48:11.393682+0000","last_active":"2024-03-17T23:48:11.394641+0000","last_peered":"2024-03-17T23:48:11.394641+0000","last_clean":"2024-03-17T23:48:11.394641+0000","last_became_active":"2024-03-17T23:48:11.393410+0000","last_became_peered":"2024-03-17T23:48:11.393410+0000","last_unstale":"2024-03-17T23:48:11.394641+0000","last_undegraded":"2024-03-17T23:48:11.394641+0000","last_fullsized":"2024-03-17T23:48:11.394641+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:17.358341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397913+0000","last_change":"2024-03-17T23:48:11.405838+0000","last_active":"2024-03-17T23:48:12.397913+0000","last_peered":"2024-03-17T23:48:12.397913+0000","last_clean":"2024-03-17T23:48:12.397913+0000","last_became_active":"2024-03-17T23:48:11.405573+0000","last_became_peered":"2024-03-17T23:48:11.405573+0000","last_unstale":"2024-03-17T23:48:12.397913+0000","last_undegraded":"2024-03-17T23:48:12.397913+0000","last_fullsized":"2024-03-17T23:48:12.397913+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:33:46.207300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409150+0000","last_change":"2024-03-17T23:48:11.403246+0000","last_active":"2024-03-17T23:48:11.409150+0000","last_peered":"2024-03-17T23:48:11.409150+0000","last_clean":"2024-03-17T23:48:11.409150+0000","last_became_active":"2024-03-17T23:48:11.402986+0000","last_became_peered":"2024-03-17T23:48:11.402986+0000","last_unstale":"2024-03-17T23:48:11.409150+0000","last_undegraded":"2024-03-17T23:48:11.409150+0000","last_fullsized":"2024-03-17T23:48:11.409150+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:16:44.071095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.402167+0000","last_change":"2024-03-17T23:48:11.398200+0000","last_active":"2024-03-17T23:48:11.402167+0000","last_peered":"2024-03-17T23:48:11.402167+0000","last_clean":"2024-03-17T23:48:11.402167+0000","last_became_active":"2024-03-17T23:48:11.397937+0000","last_became_peered":"2024-03-17T23:48:11.397937+0000","last_unstale":"2024-03-17T23:48:11.402167+0000","last_undegraded":"2024-03-17T23:48:11.402167+0000","last_fullsized":"2024-03-17T23:48:11.402167+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:45.409873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409193+0000","last_change":"2024-03-17T23:48:11.408181+0000","last_active":"2024-03-17T23:48:11.409193+0000","last_peered":"2024-03-17T23:48:11.409193+0000","last_clean":"2024-03-17T23:48:11.409193+0000","last_became_active":"2024-03-17T23:48:11.407876+0000","last_became_peered":"2024-03-17T23:48:11.407876+0000","last_unstale":"2024-03-17T23:48:11.409193+0000","last_undegraded":"2024-03-17T23:48:11.409193+0000","last_fullsized":"2024-03-17T23:48:11.409193+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:05.029387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398135+0000","last_change":"2024-03-17T23:48:11.402141+0000","last_active":"2024-03-17T23:48:12.398135+0000","last_peered":"2024-03-17T23:48:12.398135+0000","last_clean":"2024-03-17T23:48:12.398135+0000","last_became_active":"2024-03-17T23:48:11.401870+0000","last_became_peered":"2024-03-17T23:48:11.401870+0000","last_unstale":"2024-03-17T23:48:12.398135+0000","last_undegraded":"2024-03-17T23:48:12.398135+0000","last_fullsized":"2024-03-17T23:48:12.398135+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:10.929912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398329+0000","last_change":"2024-03-17T23:48:11.401263+0000","last_active":"2024-03-17T23:48:12.398329+0000","last_peered":"2024-03-17T23:48:12.398329+0000","last_clean":"2024-03-17T23:48:12.398329+0000","last_became_active":"2024-03-17T23:48:11.400951+0000","last_became_peered":"2024-03-17T23:48:11.400951+0000","last_unstale":"2024-03-17T23:48:12.398329+0000","last_undegraded":"2024-03-17T23:48:12.398329+0000","last_fullsized":"2024-03-17T23:48:12.398329+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:42:43.776415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408759+0000","last_change":"2024-03-17T23:48:11.400746+0000","last_active":"2024-03-17T23:48:11.408759+0000","last_peered":"2024-03-17T23:48:11.408759+0000","last_clean":"2024-03-17T23:48:11.408759+0000","last_became_active":"2024-03-17T23:48:11.400494+0000","last_became_peered":"2024-03-17T23:48:11.400494+0000","last_unstale":"2024-03-17T23:48:11.408759+0000","last_undegraded":"2024-03-17T23:48:11.408759+0000","last_fullsized":"2024-03-17T23:48:11.408759+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:05.029387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398311+0000","last_change":"2024-03-17T23:48:11.403046+0000","last_active":"2024-03-17T23:48:12.398311+0000","last_peered":"2024-03-17T23:48:12.398311+0000","last_clean":"2024-03-17T23:48:12.398311+0000","last_became_active":"2024-03-17T23:48:11.402769+0000","last_became_peered":"2024-03-17T23:48:11.402769+0000","last_unstale":"2024-03-17T23:48:12.398311+0000","last_undegraded":"2024-03-17T23:48:12.398311+0000","last_fullsized":"2024-03-17T23:48:12.398311+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:16:09.342548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408364+0000","last_change":"2024-03-17T23:48:11.402963+0000","last_active":"2024-03-17T23:48:11.408364+0000","last_peered":"2024-03-17T23:48:11.408364+0000","last_clean":"2024-03-17T23:48:11.408364+0000","last_became_active":"2024-03-17T23:48:11.402669+0000","last_became_peered":"2024-03-17T23:48:11.402669+0000","last_unstale":"2024-03-17T23:48:11.408364+0000","last_undegraded":"2024-03-17T23:48:11.408364+0000","last_fullsized":"2024-03-17T23:48:11.408364+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:20:48.927064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398382+0000","last_change":"2024-03-17T23:48:11.392442+0000","last_active":"2024-03-17T23:48:12.398382+0000","last_peered":"2024-03-17T23:48:12.398382+0000","last_clean":"2024-03-17T23:48:12.398382+0000","last_became_active":"2024-03-17T23:48:11.392184+0000","last_became_peered":"2024-03-17T23:48:11.392184+0000","last_unstale":"2024-03-17T23:48:12.398382+0000","last_undegraded":"2024-03-17T23:48:12.398382+0000","last_fullsized":"2024-03-17T23:48:12.398382+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:31.101445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.399825+0000","last_change":"2024-03-17T23:48:11.398013+0000","last_active":"2024-03-17T23:48:11.399825+0000","last_peered":"2024-03-17T23:48:11.399825+0000","last_clean":"2024-03-17T23:48:11.399825+0000","last_became_active":"2024-03-17T23:48:11.397709+0000","last_became_peered":"2024-03-17T23:48:11.397709+0000","last_unstale":"2024-03-17T23:48:11.399825+0000","last_undegraded":"2024-03-17T23:48:11.399825+0000","last_fullsized":"2024-03-17T23:48:11.399825+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:05.029387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397892+0000","last_change":"2024-03-17T23:48:11.404602+0000","last_active":"2024-03-17T23:48:12.397892+0000","last_peered":"2024-03-17T23:48:12.397892+0000","last_clean":"2024-03-17T23:48:12.397892+0000","last_became_active":"2024-03-17T23:48:11.404351+0000","last_became_peered":"2024-03-17T23:48:11.404351+0000","last_unstale":"2024-03-17T23:48:12.397892+0000","last_undegraded":"2024-03-17T23:48:12.397892+0000","last_fullsized":"2024-03-17T23:48:12.397892+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:45.409873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409206+0000","last_change":"2024-03-17T23:48:11.403804+0000","last_active":"2024-03-17T23:48:11.409206+0000","last_peered":"2024-03-17T23:48:11.409206+0000","last_clean":"2024-03-17T23:48:11.409206+0000","last_became_active":"2024-03-17T23:48:11.403537+0000","last_became_peered":"2024-03-17T23:48:11.403537+0000","last_unstale":"2024-03-17T23:48:11.409206+0000","last_undegraded":"2024-03-17T23:48:11.409206+0000","last_fullsized":"2024-03-17T23:48:11.409206+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:32:24.643171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397774+0000","last_change":"2024-03-17T23:48:11.398674+0000","last_active":"2024-03-17T23:48:12.397774+0000","last_peered":"2024-03-17T23:48:12.397774+0000","last_clean":"2024-03-17T23:48:12.397774+0000","last_became_active":"2024-03-17T23:48:11.398399+0000","last_became_peered":"2024-03-17T23:48:11.398399+0000","last_unstale":"2024-03-17T23:48:12.397774+0000","last_undegraded":"2024-03-17T23:48:12.397774+0000","last_fullsized":"2024-03-17T23:48:12.397774+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:17.358341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398058+0000","last_change":"2024-03-17T23:48:11.393129+0000","last_active":"2024-03-17T23:48:12.398058+0000","last_peered":"2024-03-17T23:48:12.398058+0000","last_clean":"2024-03-17T23:48:12.398058+0000","last_became_active":"2024-03-17T23:48:11.392847+0000","last_became_peered":"2024-03-17T23:48:11.392847+0000","last_unstale":"2024-03-17T23:48:12.398058+0000","last_undegraded":"2024-03-17T23:48:12.398058+0000","last_fullsized":"2024-03-17T23:48:12.398058+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:20:48.927064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408386+0000","last_change":"2024-03-17T23:48:11.403541+0000","last_active":"2024-03-17T23:48:11.408386+0000","last_peered":"2024-03-17T23:48:11.408386+0000","last_clean":"2024-03-17T23:48:11.408386+0000","last_became_active":"2024-03-17T23:48:11.403273+0000","last_became_peered":"2024-03-17T23:48:11.403273+0000","last_unstale":"2024-03-17T23:48:11.408386+0000","last_undegraded":"2024-03-17T23:48:11.408386+0000","last_fullsized":"2024-03-17T23:48:11.408386+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:10.929912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.397462+0000","last_change":"2024-03-17T23:48:11.396799+0000","last_active":"2024-03-17T23:48:11.397462+0000","last_peered":"2024-03-17T23:48:11.397462+0000","last_clean":"2024-03-17T23:48:11.397462+0000","last_became_active":"2024-03-17T23:48:11.396343+0000","last_became_peered":"2024-03-17T23:48:11.396343+0000","last_unstale":"2024-03-17T23:48:11.397462+0000","last_undegraded":"2024-03-17T23:48:11.397462+0000","last_fullsized":"2024-03-17T23:48:11.397462+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:42:43.776415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.399567+0000","last_change":"2024-03-17T23:48:11.397507+0000","last_active":"2024-03-17T23:48:11.399567+0000","last_peered":"2024-03-17T23:48:11.399567+0000","last_clean":"2024-03-17T23:48:11.399567+0000","last_became_active":"2024-03-17T23:48:11.397238+0000","last_became_peered":"2024-03-17T23:48:11.397238+0000","last_unstale":"2024-03-17T23:48:11.399567+0000","last_undegraded":"2024-03-17T23:48:11.399567+0000","last_fullsized":"2024-03-17T23:48:11.399567+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:28:11.274300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"39'4","reported_seq":18,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398478+0000","last_change":"2024-03-17T23:48:10.382890+0000","last_active":"2024-03-17T23:48:12.398478+0000","last_peered":"2024-03-17T23:48:12.398478+0000","last_clean":"2024-03-17T23:48:12.398478+0000","last_became_active":"2024-03-17T23:48:10.382565+0000","last_became_peered":"2024-03-17T23:48:10.382565+0000","last_unstale":"2024-03-17T23:48:12.398478+0000","last_undegraded":"2024-03-17T23:48:12.398478+0000","last_fullsized":"2024-03-17T23:48:12.398478+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:54.441315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.401433+0000","last_change":"2024-03-17T23:48:11.397742+0000","last_active":"2024-03-17T23:48:11.401433+0000","last_peered":"2024-03-17T23:48:11.401433+0000","last_clean":"2024-03-17T23:48:11.401433+0000","last_became_active":"2024-03-17T23:48:11.397189+0000","last_became_peered":"2024-03-17T23:48:11.397189+0000","last_unstale":"2024-03-17T23:48:11.401433+0000","last_undegraded":"2024-03-17T23:48:11.401433+0000","last_fullsized":"2024-03-17T23:48:11.401433+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:59:29.151560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398181+0000","last_change":"2024-03-17T23:48:11.406337+0000","last_active":"2024-03-17T23:48:12.398181+0000","last_peered":"2024-03-17T23:48:12.398181+0000","last_clean":"2024-03-17T23:48:12.398181+0000","last_became_active":"2024-03-17T23:48:11.405094+0000","last_became_peered":"2024-03-17T23:48:11.405094+0000","last_unstale":"2024-03-17T23:48:12.398181+0000","last_undegraded":"2024-03-17T23:48:12.398181+0000","last_fullsized":"2024-03-17T23:48:12.398181+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:38:38.676853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"39'2","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.406939+0000","last_change":"2024-03-17T23:48:10.396210+0000","last_active":"2024-03-17T23:48:11.406939+0000","last_peered":"2024-03-17T23:48:11.406939+0000","last_clean":"2024-03-17T23:48:11.406939+0000","last_became_active":"2024-03-17T23:48:10.395955+0000","last_became_peered":"2024-03-17T23:48:10.395955+0000","last_unstale":"2024-03-17T23:48:11.406939+0000","last_undegraded":"2024-03-17T23:48:11.406939+0000","last_fullsized":"2024-03-17T23:48:11.406939+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:20:47.924674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"39'2","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398582+0000","last_change":"2024-03-17T23:48:10.391635+0000","last_active":"2024-03-17T23:48:12.398582+0000","last_peered":"2024-03-17T23:48:12.398582+0000","last_clean":"2024-03-17T23:48:12.398582+0000","last_became_active":"2024-03-17T23:48:10.391241+0000","last_became_peered":"2024-03-17T23:48:10.391241+0000","last_unstale":"2024-03-17T23:48:12.398582+0000","last_undegraded":"2024-03-17T23:48:12.398582+0000","last_fullsized":"2024-03-17T23:48:12.398582+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:30.099054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"39'1","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409546+0000","last_change":"2024-03-17T23:48:10.392858+0000","last_active":"2024-03-17T23:48:11.409546+0000","last_peered":"2024-03-17T23:48:11.409546+0000","last_clean":"2024-03-17T23:48:11.409546+0000","last_became_active":"2024-03-17T23:48:10.392599+0000","last_became_peered":"2024-03-17T23:48:10.392599+0000","last_unstale":"2024-03-17T23:48:11.409546+0000","last_undegraded":"2024-03-17T23:48:11.409546+0000","last_fullsized":"2024-03-17T23:48:11.409546+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:12:23.322115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408725+0000","last_change":"2024-03-17T23:48:11.400180+0000","last_active":"2024-03-17T23:48:11.408725+0000","last_peered":"2024-03-17T23:48:11.408725+0000","last_clean":"2024-03-17T23:48:11.408725+0000","last_became_active":"2024-03-17T23:48:11.399900+0000","last_became_peered":"2024-03-17T23:48:11.399900+0000","last_unstale":"2024-03-17T23:48:11.408725+0000","last_undegraded":"2024-03-17T23:48:11.408725+0000","last_fullsized":"2024-03-17T23:48:11.408725+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:31.101445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"39'1","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.407796+0000","last_change":"2024-03-17T23:48:10.396685+0000","last_active":"2024-03-17T23:48:11.407796+0000","last_peered":"2024-03-17T23:48:11.407796+0000","last_clean":"2024-03-17T23:48:11.407796+0000","last_became_active":"2024-03-17T23:48:10.396218+0000","last_became_peered":"2024-03-17T23:48:10.396218+0000","last_unstale":"2024-03-17T23:48:11.407796+0000","last_undegraded":"2024-03-17T23:48:11.407796+0000","last_fullsized":"2024-03-17T23:48:11.407796+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:07:26.890907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398506+0000","last_change":"2024-03-17T23:48:11.398107+0000","last_active":"2024-03-17T23:48:12.398506+0000","last_peered":"2024-03-17T23:48:12.398506+0000","last_clean":"2024-03-17T23:48:12.398506+0000","last_became_active":"2024-03-17T23:48:11.397847+0000","last_became_peered":"2024-03-17T23:48:11.397847+0000","last_unstale":"2024-03-17T23:48:12.398506+0000","last_undegraded":"2024-03-17T23:48:12.398506+0000","last_fullsized":"2024-03-17T23:48:12.398506+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:47:20.574371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.383921+0000","last_change":"2024-03-17T23:48:10.395355+0000","last_active":"2024-03-17T23:48:11.383921+0000","last_peered":"2024-03-17T23:48:11.383921+0000","last_clean":"2024-03-17T23:48:11.383921+0000","last_became_active":"2024-03-17T23:48:10.395051+0000","last_became_peered":"2024-03-17T23:48:10.395051+0000","last_unstale":"2024-03-17T23:48:11.383921+0000","last_undegraded":"2024-03-17T23:48:11.383921+0000","last_fullsized":"2024-03-17T23:48:11.383921+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:28:10.271909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397331+0000","last_change":"2024-03-17T23:48:10.387055+0000","last_active":"2024-03-17T23:48:12.397331+0000","last_peered":"2024-03-17T23:48:12.397331+0000","last_clean":"2024-03-17T23:48:12.397331+0000","last_became_active":"2024-03-17T23:48:10.386732+0000","last_became_peered":"2024-03-17T23:48:10.386732+0000","last_unstale":"2024-03-17T23:48:12.397331+0000","last_undegraded":"2024-03-17T23:48:12.397331+0000","last_fullsized":"2024-03-17T23:48:12.397331+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:59:28.149170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.398574+0000","last_change":"2024-03-17T23:48:11.394613+0000","last_active":"2024-03-17T23:48:11.398574+0000","last_peered":"2024-03-17T23:48:11.398574+0000","last_clean":"2024-03-17T23:48:11.398574+0000","last_became_active":"2024-03-17T23:48:11.394206+0000","last_became_peered":"2024-03-17T23:48:11.394206+0000","last_unstale":"2024-03-17T23:48:11.398574+0000","last_undegraded":"2024-03-17T23:48:11.398574+0000","last_fullsized":"2024-03-17T23:48:11.398574+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:40:54.501510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.389970+0000","last_change":"2024-03-17T23:48:10.394514+0000","last_active":"2024-03-17T23:48:11.389970+0000","last_peered":"2024-03-17T23:48:11.389970+0000","last_clean":"2024-03-17T23:48:11.389970+0000","last_became_active":"2024-03-17T23:48:10.394237+0000","last_became_peered":"2024-03-17T23:48:10.394237+0000","last_unstale":"2024-03-17T23:48:11.389970+0000","last_undegraded":"2024-03-17T23:48:11.389970+0000","last_fullsized":"2024-03-17T23:48:11.389970+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:40:53.499120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"39'3","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409672+0000","last_change":"2024-03-17T23:48:10.400644+0000","last_active":"2024-03-17T23:48:11.409672+0000","last_peered":"2024-03-17T23:48:11.409672+0000","last_clean":"2024-03-17T23:48:11.409672+0000","last_became_active":"2024-03-17T23:48:10.400361+0000","last_became_peered":"2024-03-17T23:48:10.400361+0000","last_unstale":"2024-03-17T23:48:11.409672+0000","last_undegraded":"2024-03-17T23:48:11.409672+0000","last_fullsized":"2024-03-17T23:48:11.409672+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:28:10.271909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398068+0000","last_change":"2024-03-17T23:48:11.404493+0000","last_active":"2024-03-17T23:48:12.398068+0000","last_peered":"2024-03-17T23:48:12.398068+0000","last_clean":"2024-03-17T23:48:12.398068+0000","last_became_active":"2024-03-17T23:48:11.401330+0000","last_became_peered":"2024-03-17T23:48:11.401330+0000","last_unstale":"2024-03-17T23:48:12.398068+0000","last_undegraded":"2024-03-17T23:48:12.398068+0000","last_fullsized":"2024-03-17T23:48:12.398068+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:37:11.855078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.385582+0000","last_change":"2024-03-17T23:48:10.394916+0000","last_active":"2024-03-17T23:48:11.385582+0000","last_peered":"2024-03-17T23:48:11.385582+0000","last_clean":"2024-03-17T23:48:11.385582+0000","last_became_active":"2024-03-17T23:48:10.391532+0000","last_became_peered":"2024-03-17T23:48:10.391532+0000","last_unstale":"2024-03-17T23:48:11.385582+0000","last_undegraded":"2024-03-17T23:48:11.385582+0000","last_fullsized":"2024-03-17T23:48:11.385582+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:44.407483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409135+0000","last_change":"2024-03-17T23:48:11.407145+0000","last_active":"2024-03-17T23:48:11.409135+0000","last_peered":"2024-03-17T23:48:11.409135+0000","last_clean":"2024-03-17T23:48:11.409135+0000","last_became_active":"2024-03-17T23:48:11.406851+0000","last_became_peered":"2024-03-17T23:48:11.406851+0000","last_unstale":"2024-03-17T23:48:11.409135+0000","last_undegraded":"2024-03-17T23:48:11.409135+0000","last_fullsized":"2024-03-17T23:48:11.409135+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:59:29.151560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397440+0000","last_change":"2024-03-17T23:48:10.388069+0000","last_active":"2024-03-17T23:48:12.397440+0000","last_peered":"2024-03-17T23:48:12.397440+0000","last_clean":"2024-03-17T23:48:12.397440+0000","last_became_active":"2024-03-17T23:48:10.387765+0000","last_became_peered":"2024-03-17T23:48:10.387765+0000","last_unstale":"2024-03-17T23:48:12.397440+0000","last_undegraded":"2024-03-17T23:48:12.397440+0000","last_fullsized":"2024-03-17T23:48:12.397440+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:15.116223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"39'8","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.400672+0000","last_change":"2024-03-17T23:48:10.397744+0000","last_active":"2024-03-17T23:48:11.400672+0000","last_peered":"2024-03-17T23:48:11.400672+0000","last_clean":"2024-03-17T23:48:11.400672+0000","last_became_active":"2024-03-17T23:48:10.397467+0000","last_became_peered":"2024-03-17T23:48:10.397467+0000","last_unstale":"2024-03-17T23:48:11.400672+0000","last_undegraded":"2024-03-17T23:48:11.400672+0000","last_fullsized":"2024-03-17T23:48:11.400672+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:12:23.322115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"39'1","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409062+0000","last_change":"2024-03-17T23:48:10.393694+0000","last_active":"2024-03-17T23:48:11.409062+0000","last_peered":"2024-03-17T23:48:11.409062+0000","last_clean":"2024-03-17T23:48:11.409062+0000","last_became_active":"2024-03-17T23:48:10.393418+0000","last_became_peered":"2024-03-17T23:48:10.393418+0000","last_unstale":"2024-03-17T23:48:11.409062+0000","last_undegraded":"2024-03-17T23:48:11.409062+0000","last_fullsized":"2024-03-17T23:48:11.409062+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:42:42.774025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408740+0000","last_change":"2024-03-17T23:48:11.400412+0000","last_active":"2024-03-17T23:48:11.408740+0000","last_peered":"2024-03-17T23:48:11.408740+0000","last_clean":"2024-03-17T23:48:11.408740+0000","last_became_active":"2024-03-17T23:48:11.400006+0000","last_became_peered":"2024-03-17T23:48:11.400006+0000","last_unstale":"2024-03-17T23:48:11.408740+0000","last_undegraded":"2024-03-17T23:48:11.408740+0000","last_fullsized":"2024-03-17T23:48:11.408740+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:59:29.151560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"39'2","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398100+0000","last_change":"2024-03-17T23:48:10.391802+0000","last_active":"2024-03-17T23:48:12.398100+0000","last_peered":"2024-03-17T23:48:12.398100+0000","last_clean":"2024-03-17T23:48:12.398100+0000","last_became_active":"2024-03-17T23:48:10.391464+0000","last_became_peered":"2024-03-17T23:48:10.391464+0000","last_unstale":"2024-03-17T23:48:12.398100+0000","last_undegraded":"2024-03-17T23:48:12.398100+0000","last_fullsized":"2024-03-17T23:48:12.398100+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:44:41.528247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397957+0000","last_change":"2024-03-17T23:48:11.399808+0000","last_active":"2024-03-17T23:48:12.397957+0000","last_peered":"2024-03-17T23:48:12.397957+0000","last_clean":"2024-03-17T23:48:12.397957+0000","last_became_active":"2024-03-17T23:48:11.399460+0000","last_became_peered":"2024-03-17T23:48:11.399460+0000","last_unstale":"2024-03-17T23:48:12.397957+0000","last_undegraded":"2024-03-17T23:48:12.397957+0000","last_fullsized":"2024-03-17T23:48:12.397957+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:16:59.276395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.384478+0000","last_change":"2024-03-17T23:48:10.396820+0000","last_active":"2024-03-17T23:48:11.384478+0000","last_peered":"2024-03-17T23:48:11.384478+0000","last_clean":"2024-03-17T23:48:11.384478+0000","last_became_active":"2024-03-17T23:48:10.396549+0000","last_became_peered":"2024-03-17T23:48:10.396549+0000","last_unstale":"2024-03-17T23:48:11.384478+0000","last_undegraded":"2024-03-17T23:48:11.384478+0000","last_fullsized":"2024-03-17T23:48:11.384478+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:16.355950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408966+0000","last_change":"2024-03-17T23:48:11.403660+0000","last_active":"2024-03-17T23:48:11.408966+0000","last_peered":"2024-03-17T23:48:11.408966+0000","last_clean":"2024-03-17T23:48:11.408966+0000","last_became_active":"2024-03-17T23:48:11.403385+0000","last_became_peered":"2024-03-17T23:48:11.403385+0000","last_unstale":"2024-03-17T23:48:11.408966+0000","last_undegraded":"2024-03-17T23:48:11.408966+0000","last_fullsized":"2024-03-17T23:48:11.408966+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:42:43.776415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409211+0000","last_change":"2024-03-17T23:48:11.404518+0000","last_active":"2024-03-17T23:48:11.409211+0000","last_peered":"2024-03-17T23:48:11.409211+0000","last_clean":"2024-03-17T23:48:11.409211+0000","last_became_active":"2024-03-17T23:48:11.404261+0000","last_became_peered":"2024-03-17T23:48:11.404261+0000","last_unstale":"2024-03-17T23:48:11.409211+0000","last_undegraded":"2024-03-17T23:48:11.409211+0000","last_fullsized":"2024-03-17T23:48:11.409211+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:17:53.610029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.385531+0000","last_change":"2024-03-17T23:48:10.397213+0000","last_active":"2024-03-17T23:48:11.385531+0000","last_peered":"2024-03-17T23:48:11.385531+0000","last_clean":"2024-03-17T23:48:11.385531+0000","last_became_active":"2024-03-17T23:48:10.396913+0000","last_became_peered":"2024-03-17T23:48:10.396913+0000","last_unstale":"2024-03-17T23:48:11.385531+0000","last_undegraded":"2024-03-17T23:48:11.385531+0000","last_fullsized":"2024-03-17T23:48:11.385531+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:31:12.048530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.397234+0000","last_change":"2024-03-17T23:48:11.396445+0000","last_active":"2024-03-17T23:48:11.397234+0000","last_peered":"2024-03-17T23:48:11.397234+0000","last_clean":"2024-03-17T23:48:11.397234+0000","last_became_active":"2024-03-17T23:48:11.396189+0000","last_became_peered":"2024-03-17T23:48:11.396189+0000","last_unstale":"2024-03-17T23:48:11.397234+0000","last_undegraded":"2024-03-17T23:48:11.397234+0000","last_fullsized":"2024-03-17T23:48:11.397234+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:10.929912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"39'2","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.406980+0000","last_change":"2024-03-17T23:48:10.400812+0000","last_active":"2024-03-17T23:48:11.406980+0000","last_peered":"2024-03-17T23:48:11.406980+0000","last_clean":"2024-03-17T23:48:11.406980+0000","last_became_active":"2024-03-17T23:48:10.400479+0000","last_became_peered":"2024-03-17T23:48:10.400479+0000","last_unstale":"2024-03-17T23:48:11.406980+0000","last_undegraded":"2024-03-17T23:48:11.406980+0000","last_fullsized":"2024-03-17T23:48:11.406980+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:09.927522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408459+0000","last_change":"2024-03-17T23:48:11.403785+0000","last_active":"2024-03-17T23:48:11.408459+0000","last_peered":"2024-03-17T23:48:11.408459+0000","last_clean":"2024-03-17T23:48:11.408459+0000","last_became_active":"2024-03-17T23:48:11.402301+0000","last_became_peered":"2024-03-17T23:48:11.402301+0000","last_unstale":"2024-03-17T23:48:11.408459+0000","last_undegraded":"2024-03-17T23:48:11.408459+0000","last_fullsized":"2024-03-17T23:48:11.408459+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:31.101445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.397727+0000","last_change":"2024-03-17T23:48:11.397059+0000","last_active":"2024-03-17T23:48:11.397727+0000","last_peered":"2024-03-17T23:48:11.397727+0000","last_clean":"2024-03-17T23:48:11.397727+0000","last_became_active":"2024-03-17T23:48:11.396724+0000","last_became_peered":"2024-03-17T23:48:11.396724+0000","last_unstale":"2024-03-17T23:48:11.397727+0000","last_undegraded":"2024-03-17T23:48:11.397727+0000","last_fullsized":"2024-03-17T23:48:11.397727+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:31.101445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398165+0000","last_change":"2024-03-17T23:48:10.384305+0000","last_active":"2024-03-17T23:48:12.398165+0000","last_peered":"2024-03-17T23:48:12.398165+0000","last_clean":"2024-03-17T23:48:12.398165+0000","last_became_active":"2024-03-17T23:48:10.384034+0000","last_became_peered":"2024-03-17T23:48:10.384034+0000","last_unstale":"2024-03-17T23:48:12.398165+0000","last_undegraded":"2024-03-17T23:48:12.398165+0000","last_fullsized":"2024-03-17T23:48:12.398165+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:32:37.500442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398088+0000","last_change":"2024-03-17T23:48:11.394167+0000","last_active":"2024-03-17T23:48:12.398088+0000","last_peered":"2024-03-17T23:48:12.398088+0000","last_clean":"2024-03-17T23:48:12.398088+0000","last_became_active":"2024-03-17T23:48:11.392337+0000","last_became_peered":"2024-03-17T23:48:11.392337+0000","last_unstale":"2024-03-17T23:48:12.398088+0000","last_undegraded":"2024-03-17T23:48:12.398088+0000","last_fullsized":"2024-03-17T23:48:12.398088+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:04:14.826887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397422+0000","last_change":"2024-03-17T23:48:10.391429+0000","last_active":"2024-03-17T23:48:12.397422+0000","last_peered":"2024-03-17T23:48:12.397422+0000","last_clean":"2024-03-17T23:48:12.397422+0000","last_became_active":"2024-03-17T23:48:10.390974+0000","last_became_peered":"2024-03-17T23:48:10.390974+0000","last_unstale":"2024-03-17T23:48:12.397422+0000","last_undegraded":"2024-03-17T23:48:12.397422+0000","last_fullsized":"2024-03-17T23:48:12.397422+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:04.026997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.399296+0000","last_change":"2024-03-17T23:48:11.397803+0000","last_active":"2024-03-17T23:48:11.399296+0000","last_peered":"2024-03-17T23:48:11.399296+0000","last_clean":"2024-03-17T23:48:11.399296+0000","last_became_active":"2024-03-17T23:48:11.397541+0000","last_became_peered":"2024-03-17T23:48:11.397541+0000","last_unstale":"2024-03-17T23:48:11.399296+0000","last_undegraded":"2024-03-17T23:48:11.399296+0000","last_fullsized":"2024-03-17T23:48:11.399296+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:17.358341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409115+0000","last_change":"2024-03-17T23:48:11.403123+0000","last_active":"2024-03-17T23:48:11.409115+0000","last_peered":"2024-03-17T23:48:11.409115+0000","last_clean":"2024-03-17T23:48:11.409115+0000","last_became_active":"2024-03-17T23:48:11.402851+0000","last_became_peered":"2024-03-17T23:48:11.402851+0000","last_unstale":"2024-03-17T23:48:11.409115+0000","last_undegraded":"2024-03-17T23:48:11.409115+0000","last_fullsized":"2024-03-17T23:48:11.409115+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:44:42.530638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.384095+0000","last_change":"2024-03-17T23:48:10.398461+0000","last_active":"2024-03-17T23:48:11.384095+0000","last_peered":"2024-03-17T23:48:11.384095+0000","last_clean":"2024-03-17T23:48:11.384095+0000","last_became_active":"2024-03-17T23:48:10.398188+0000","last_became_peered":"2024-03-17T23:48:10.398188+0000","last_unstale":"2024-03-17T23:48:11.384095+0000","last_undegraded":"2024-03-17T23:48:11.384095+0000","last_fullsized":"2024-03-17T23:48:11.384095+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:06:33.058080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":204,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.385340+0000","last_change":"2024-03-17T23:46:44.768668+0000","last_active":"2024-03-17T23:48:11.385340+0000","last_peered":"2024-03-17T23:48:11.385340+0000","last_clean":"2024-03-17T23:48:11.385340+0000","last_became_active":"2024-03-17T23:46:44.767826+0000","last_became_peered":"2024-03-17T23:46:44.767826+0000","last_unstale":"2024-03-17T23:48:11.385340+0000","last_undegraded":"2024-03-17T23:48:11.385340+0000","last_fullsized":"2024-03-17T23:48:11.385340+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_clean_scrub_stamp":"2024-03-17T23:46:43.744878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:54:22.286155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":80,"acting":80,"num_store_stats":5},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":694,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":40,"acting":40,"num_store_stats":5},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27424,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":409600,"data_stored":174269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27420,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":409600,"data_stored":174269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542165,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28008,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343832,"statfs":{"total":96636764160,"available":96608083968,"internally_reserved":0,"allocated":1003520,"data_stored":764637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32212,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339628,"statfs":{"total":96636764160,"available":96603779072,"internally_reserved":0,"allocated":1056768,"data_stored":786362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31636,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340204,"statfs":{"total":96636764160,"available":96604368896,"internally_reserved":0,"allocated":462848,"data_stored":201586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31624,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":454656,"data_stored":194424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31624,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":454656,"data_stored":194916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":450560,"data_stored":194402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-17T23:48:14.453 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-17T23:48:14.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.590+0000 7fc846ff3640 1 Processor -- start 2024-03-17T23:48:14.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.590+0000 7fc846ff3640 1 -- start start 2024-03-17T23:48:14.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.590+0000 7fc846ff3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8400b31c0 0x7fc8400b4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:14.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.590+0000 7fc846ff3640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc84009a7d0 0x7fc84006d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:14.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.590+0000 7fc846ff3640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8400cc220 con 0x7fc84006ccb0 2024-03-17T23:48:14.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.591+0000 7fc846ff3640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fc84006d650 con 0x7fc84009a7d0 2024-03-17T23:48:14.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.591+0000 7fc846ff3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc8400cc500 con 0x7fc8400b31c0 2024-03-17T23:48:14.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.591+0000 7fc844d8f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8400b31c0 0x7fc8400b4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:14.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.591+0000 7fc845590640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc84009a7d0 0x7fc84006d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:14.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.591+0000 7fc844d8f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8400b31c0 0x7fc8400b4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:46800/0 (socket says 172.21.15.150:46800) 2024-03-17T23:48:14.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.591+0000 7fc845590640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fc84009a7d0 0x7fc84006d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:38714/0 (socket says 172.21.15.150:38714) 2024-03-17T23:48:14.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.591+0000 7fc844d8f640 1 -- 172.21.15.150:0/4176775993 learned_addr learned my addr 172.21.15.150:0/4176775993 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:14.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.591+0000 7fc845590640 1 -- 172.21.15.150:0/4176775993 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8400b31c0 msgr2=0x7fc8400b4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:14.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.591+0000 7fc845590640 1 --2- 172.21.15.150:0/4176775993 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8400b31c0 0x7fc8400b4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.591+0000 7fc845590640 1 -- 172.21.15.150:0/4176775993 >> v1:172.21.15.157:6789/0 conn(0x7fc84006ccb0 legacy=0x7fc84009a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:14.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.591+0000 7fc845590640 1 -- 172.21.15.150:0/4176775993 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8400cc7e0 con 0x7fc84009a7d0 2024-03-17T23:48:14.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.591+0000 7fc844d8f640 1 --2- 172.21.15.150:0/4176775993 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8400b31c0 0x7fc8400b4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:48:14.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.592+0000 7fc845590640 1 --2- 172.21.15.150:0/4176775993 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc84009a7d0 0x7fc84006d110 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc838009d90 tx=0x7fc83802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7d16ce1ed45002 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:14.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.592+0000 7fc83f7fe640 1 -- 172.21.15.150:0/4176775993 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc838039070 con 0x7fc84009a7d0 2024-03-17T23:48:14.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.592+0000 7fc83f7fe640 1 -- 172.21.15.150:0/4176775993 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc83802cc50 con 0x7fc84009a7d0 2024-03-17T23:48:14.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.592+0000 7fc83f7fe640 1 -- 172.21.15.150:0/4176775993 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc838034440 con 0x7fc84009a7d0 2024-03-17T23:48:14.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.593+0000 7fc846ff3640 1 -- 172.21.15.150:0/4176775993 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc84009a7d0 msgr2=0x7fc84006d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:14.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.593+0000 7fc846ff3640 1 --2- 172.21.15.150:0/4176775993 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc84009a7d0 0x7fc84006d110 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc838009d90 tx=0x7fc83802bea0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.593+0000 7fc846ff3640 1 -- 172.21.15.150:0/4176775993 shutdown_connections 2024-03-17T23:48:14.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.593+0000 7fc846ff3640 1 --2- 172.21.15.150:0/4176775993 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc84009a7d0 0x7fc84006d110 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.593+0000 7fc846ff3640 1 --2- 172.21.15.150:0/4176775993 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc8400b31c0 0x7fc8400b4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.593+0000 7fc846ff3640 1 -- 172.21.15.150:0/4176775993 >> 172.21.15.150:0/4176775993 conn(0x7fc8400b3620 msgr2=0x7fc8400b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:14.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.593+0000 7fc846ff3640 1 -- 172.21.15.150:0/4176775993 shutdown_connections 2024-03-17T23:48:14.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.593+0000 7fc846ff3640 1 -- 172.21.15.150:0/4176775993 wait complete. 2024-03-17T23:48:14.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.594+0000 7fc846ff3640 1 Processor -- start 2024-03-17T23:48:14.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.594+0000 7fc846ff3640 1 -- start start 2024-03-17T23:48:14.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.595+0000 7fc846ff3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc84006ccb0 0x7fc84015b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:14.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.595+0000 7fc846ff3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc84009a7d0 0x7fc8400cae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:14.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.595+0000 7fc846ff3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8400b31c0 0x7fc8400c5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:14.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.595+0000 7fc846ff3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc84015d7a0 con 0x7fc8400b31c0 2024-03-17T23:48:14.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.595+0000 7fc846ff3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc8400ccda0 con 0x7fc84006ccb0 2024-03-17T23:48:14.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.595+0000 7fc846ff3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc84015d920 con 0x7fc84009a7d0 2024-03-17T23:48:14.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.595+0000 7fc845590640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8400b31c0 0x7fc8400c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:14.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.595+0000 7fc845590640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8400b31c0 0x7fc8400c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43742/0 (socket says 172.21.15.150:43742) 2024-03-17T23:48:14.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.595+0000 7fc845590640 1 -- 172.21.15.150:0/1034610676 learned_addr learned my addr 172.21.15.150:0/1034610676 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:14.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.595+0000 7fc83ffff640 1 --2- 172.21.15.150:0/1034610676 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc84009a7d0 0x7fc8400cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:14.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.595+0000 7fc844d8f640 1 --2- 172.21.15.150:0/1034610676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc84006ccb0 0x7fc84015b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:14.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.595+0000 7fc845590640 1 -- 172.21.15.150:0/1034610676 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc84009a7d0 msgr2=0x7fc8400cae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:14.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.595+0000 7fc845590640 1 --2- 172.21.15.150:0/1034610676 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc84009a7d0 0x7fc8400cae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.595+0000 7fc845590640 1 -- 172.21.15.150:0/1034610676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc84006ccb0 msgr2=0x7fc84015b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:14.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.595+0000 7fc845590640 1 --2- 172.21.15.150:0/1034610676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc84006ccb0 0x7fc84015b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.595+0000 7fc845590640 1 -- 172.21.15.150:0/1034610676 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8400c9df0 con 0x7fc8400b31c0 2024-03-17T23:48:14.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.596+0000 7fc845590640 1 --2- 172.21.15.150:0/1034610676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8400b31c0 0x7fc8400c5610 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fc838009eb0 tx=0x7fc838004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:14.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.596+0000 7fc83d7fa640 1 -- 172.21.15.150:0/1034610676 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc838039050 con 0x7fc8400b31c0 2024-03-17T23:48:14.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.596+0000 7fc83d7fa640 1 -- 172.21.15.150:0/1034610676 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc838045070 con 0x7fc8400b31c0 2024-03-17T23:48:14.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.596+0000 7fc846ff3640 1 -- 172.21.15.150:0/1034610676 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8400ca0d0 con 0x7fc8400b31c0 2024-03-17T23:48:14.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.596+0000 7fc83d7fa640 1 -- 172.21.15.150:0/1034610676 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc838009450 con 0x7fc8400b31c0 2024-03-17T23:48:14.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.596+0000 7fc846ff3640 1 -- 172.21.15.150:0/1034610676 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8400ca690 con 0x7fc8400b31c0 2024-03-17T23:48:14.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.597+0000 7fc846ff3640 1 -- 172.21.15.150:0/1034610676 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8400688b0 con 0x7fc8400b31c0 2024-03-17T23:48:14.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.598+0000 7fc83d7fa640 1 -- 172.21.15.150:0/1034610676 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc83803e610 con 0x7fc8400b31c0 2024-03-17T23:48:14.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.598+0000 7fc83d7fa640 1 --2- 172.21.15.150:0/1034610676 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc82406bbd0 0x7fc82406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:14.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.598+0000 7fc83d7fa640 1 -- 172.21.15.150:0/1034610676 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7fc8380ad650 con 0x7fc8400b31c0 2024-03-17T23:48:14.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.598+0000 7fc844d8f640 1 --2- 172.21.15.150:0/1034610676 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc82406bbd0 0x7fc82406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:14.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.599+0000 7fc844d8f640 1 --2- 172.21.15.150:0/1034610676 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc82406bbd0 0x7fc82406e090 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc834009ac0 tx=0x7fc834008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:14.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.603+0000 7fc83d7fa640 1 -- 172.21.15.150:0/1034610676 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc83807c0d0 con 0x7fc8400b31c0 2024-03-17T23:48:14.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.777+0000 7fc846ff3640 1 -- 172.21.15.150:0/1034610676 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc84006ca90 con 0x7fc82406bbd0 2024-03-17T23:48:14.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.782+0000 7fc83d7fa640 1 -- 172.21.15.150:0/1034610676 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246405 (secure 0 0 0) 0x7fc84006ca90 con 0x7fc82406bbd0 2024-03-17T23:48:14.785 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:14.786 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-03-17T23:48:14.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.785+0000 7fc846ff3640 1 -- 172.21.15.150:0/1034610676 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc82406bbd0 msgr2=0x7fc82406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:14.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.785+0000 7fc846ff3640 1 --2- 172.21.15.150:0/1034610676 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc82406bbd0 0x7fc82406e090 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc834009ac0 tx=0x7fc834008040 comp rx=0 tx=0).stop 2024-03-17T23:48:14.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.785+0000 7fc846ff3640 1 -- 172.21.15.150:0/1034610676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8400b31c0 msgr2=0x7fc8400c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:14.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.785+0000 7fc846ff3640 1 --2- 172.21.15.150:0/1034610676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8400b31c0 0x7fc8400c5610 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fc838009eb0 tx=0x7fc838004a40 comp rx=0 tx=0).stop 2024-03-17T23:48:14.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.785+0000 7fc846ff3640 1 -- 172.21.15.150:0/1034610676 shutdown_connections 2024-03-17T23:48:14.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.785+0000 7fc846ff3640 1 --2- 172.21.15.150:0/1034610676 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8400b31c0 0x7fc8400c5610 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.785+0000 7fc846ff3640 1 --2- 172.21.15.150:0/1034610676 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc84009a7d0 0x7fc8400cae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.785+0000 7fc846ff3640 1 --2- 172.21.15.150:0/1034610676 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc82406bbd0 0x7fc82406e090 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.785+0000 7fc846ff3640 1 --2- 172.21.15.150:0/1034610676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc84006ccb0 0x7fc84015b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.785+0000 7fc846ff3640 1 -- 172.21.15.150:0/1034610676 >> 172.21.15.150:0/1034610676 conn(0x7fc8400b3620 msgr2=0x7fc8400ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:14.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.785+0000 7fc846ff3640 1 -- 172.21.15.150:0/1034610676 shutdown_connections 2024-03-17T23:48:14.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.786+0000 7fc846ff3640 1 -- 172.21.15.150:0/1034610676 wait complete. 2024-03-17T23:48:14.797 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2024-03-17T23:48:13.612979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591062,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":192,"num_write_kb":2419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":122,"acting":122,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":92,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":6,"kb":754974720,"kb_used":237408,"kb_used_data":4592,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737312,"statfs":{"total":773094113280,"available":772851007488,"internally_reserved":0,"allocated":4702208,"data_stored":2684865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":694,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.241889"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.400417+0000","last_change":"2024-03-17T23:48:11.399627+0000","last_active":"2024-03-17T23:48:11.400417+0000","last_peered":"2024-03-17T23:48:11.400417+0000","last_clean":"2024-03-17T23:48:11.400417+0000","last_became_active":"2024-03-17T23:48:11.399364+0000","last_became_peered":"2024-03-17T23:48:11.399364+0000","last_unstale":"2024-03-17T23:48:11.400417+0000","last_undegraded":"2024-03-17T23:48:11.400417+0000","last_fullsized":"2024-03-17T23:48:11.400417+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:44:42.530638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409171+0000","last_change":"2024-03-17T23:48:11.403822+0000","last_active":"2024-03-17T23:48:11.409171+0000","last_peered":"2024-03-17T23:48:11.409171+0000","last_clean":"2024-03-17T23:48:11.409171+0000","last_became_active":"2024-03-17T23:48:11.403572+0000","last_became_peered":"2024-03-17T23:48:11.403572+0000","last_unstale":"2024-03-17T23:48:11.409171+0000","last_undegraded":"2024-03-17T23:48:11.409171+0000","last_fullsized":"2024-03-17T23:48:11.409171+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:53:53.408503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.406658+0000","last_change":"2024-03-17T23:48:11.395918+0000","last_active":"2024-03-17T23:48:11.406658+0000","last_peered":"2024-03-17T23:48:11.406658+0000","last_clean":"2024-03-17T23:48:11.406658+0000","last_became_active":"2024-03-17T23:48:11.394963+0000","last_became_peered":"2024-03-17T23:48:11.394963+0000","last_unstale":"2024-03-17T23:48:11.406658+0000","last_undegraded":"2024-03-17T23:48:11.406658+0000","last_fullsized":"2024-03-17T23:48:11.406658+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:20:48.927064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.399665+0000","last_change":"2024-03-17T23:48:11.394537+0000","last_active":"2024-03-17T23:48:11.399665+0000","last_peered":"2024-03-17T23:48:11.399665+0000","last_clean":"2024-03-17T23:48:11.399665+0000","last_became_active":"2024-03-17T23:48:11.394147+0000","last_became_peered":"2024-03-17T23:48:11.394147+0000","last_unstale":"2024-03-17T23:48:11.399665+0000","last_undegraded":"2024-03-17T23:48:11.399665+0000","last_fullsized":"2024-03-17T23:48:11.399665+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:45.409873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.394641+0000","last_change":"2024-03-17T23:48:11.393682+0000","last_active":"2024-03-17T23:48:11.394641+0000","last_peered":"2024-03-17T23:48:11.394641+0000","last_clean":"2024-03-17T23:48:11.394641+0000","last_became_active":"2024-03-17T23:48:11.393410+0000","last_became_peered":"2024-03-17T23:48:11.393410+0000","last_unstale":"2024-03-17T23:48:11.394641+0000","last_undegraded":"2024-03-17T23:48:11.394641+0000","last_fullsized":"2024-03-17T23:48:11.394641+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:17.358341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397913+0000","last_change":"2024-03-17T23:48:11.405838+0000","last_active":"2024-03-17T23:48:12.397913+0000","last_peered":"2024-03-17T23:48:12.397913+0000","last_clean":"2024-03-17T23:48:12.397913+0000","last_became_active":"2024-03-17T23:48:11.405573+0000","last_became_peered":"2024-03-17T23:48:11.405573+0000","last_unstale":"2024-03-17T23:48:12.397913+0000","last_undegraded":"2024-03-17T23:48:12.397913+0000","last_fullsized":"2024-03-17T23:48:12.397913+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:33:46.207300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409150+0000","last_change":"2024-03-17T23:48:11.403246+0000","last_active":"2024-03-17T23:48:11.409150+0000","last_peered":"2024-03-17T23:48:11.409150+0000","last_clean":"2024-03-17T23:48:11.409150+0000","last_became_active":"2024-03-17T23:48:11.402986+0000","last_became_peered":"2024-03-17T23:48:11.402986+0000","last_unstale":"2024-03-17T23:48:11.409150+0000","last_undegraded":"2024-03-17T23:48:11.409150+0000","last_fullsized":"2024-03-17T23:48:11.409150+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:16:44.071095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.402167+0000","last_change":"2024-03-17T23:48:11.398200+0000","last_active":"2024-03-17T23:48:11.402167+0000","last_peered":"2024-03-17T23:48:11.402167+0000","last_clean":"2024-03-17T23:48:11.402167+0000","last_became_active":"2024-03-17T23:48:11.397937+0000","last_became_peered":"2024-03-17T23:48:11.397937+0000","last_unstale":"2024-03-17T23:48:11.402167+0000","last_undegraded":"2024-03-17T23:48:11.402167+0000","last_fullsized":"2024-03-17T23:48:11.402167+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:45.409873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409193+0000","last_change":"2024-03-17T23:48:11.408181+0000","last_active":"2024-03-17T23:48:11.409193+0000","last_peered":"2024-03-17T23:48:11.409193+0000","last_clean":"2024-03-17T23:48:11.409193+0000","last_became_active":"2024-03-17T23:48:11.407876+0000","last_became_peered":"2024-03-17T23:48:11.407876+0000","last_unstale":"2024-03-17T23:48:11.409193+0000","last_undegraded":"2024-03-17T23:48:11.409193+0000","last_fullsized":"2024-03-17T23:48:11.409193+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:05.029387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398135+0000","last_change":"2024-03-17T23:48:11.402141+0000","last_active":"2024-03-17T23:48:12.398135+0000","last_peered":"2024-03-17T23:48:12.398135+0000","last_clean":"2024-03-17T23:48:12.398135+0000","last_became_active":"2024-03-17T23:48:11.401870+0000","last_became_peered":"2024-03-17T23:48:11.401870+0000","last_unstale":"2024-03-17T23:48:12.398135+0000","last_undegraded":"2024-03-17T23:48:12.398135+0000","last_fullsized":"2024-03-17T23:48:12.398135+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:10.929912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398329+0000","last_change":"2024-03-17T23:48:11.401263+0000","last_active":"2024-03-17T23:48:12.398329+0000","last_peered":"2024-03-17T23:48:12.398329+0000","last_clean":"2024-03-17T23:48:12.398329+0000","last_became_active":"2024-03-17T23:48:11.400951+0000","last_became_peered":"2024-03-17T23:48:11.400951+0000","last_unstale":"2024-03-17T23:48:12.398329+0000","last_undegraded":"2024-03-17T23:48:12.398329+0000","last_fullsized":"2024-03-17T23:48:12.398329+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:42:43.776415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408759+0000","last_change":"2024-03-17T23:48:11.400746+0000","last_active":"2024-03-17T23:48:11.408759+0000","last_peered":"2024-03-17T23:48:11.408759+0000","last_clean":"2024-03-17T23:48:11.408759+0000","last_became_active":"2024-03-17T23:48:11.400494+0000","last_became_peered":"2024-03-17T23:48:11.400494+0000","last_unstale":"2024-03-17T23:48:11.408759+0000","last_undegraded":"2024-03-17T23:48:11.408759+0000","last_fullsized":"2024-03-17T23:48:11.408759+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:05.029387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398311+0000","last_change":"2024-03-17T23:48:11.403046+0000","last_active":"2024-03-17T23:48:12.398311+0000","last_peered":"2024-03-17T23:48:12.398311+0000","last_clean":"2024-03-17T23:48:12.398311+0000","last_became_active":"2024-03-17T23:48:11.402769+0000","last_became_peered":"2024-03-17T23:48:11.402769+0000","last_unstale":"2024-03-17T23:48:12.398311+0000","last_undegraded":"2024-03-17T23:48:12.398311+0000","last_fullsized":"2024-03-17T23:48:12.398311+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:16:09.342548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408364+0000","last_change":"2024-03-17T23:48:11.402963+0000","last_active":"2024-03-17T23:48:11.408364+0000","last_peered":"2024-03-17T23:48:11.408364+0000","last_clean":"2024-03-17T23:48:11.408364+0000","last_became_active":"2024-03-17T23:48:11.402669+0000","last_became_peered":"2024-03-17T23:48:11.402669+0000","last_unstale":"2024-03-17T23:48:11.408364+0000","last_undegraded":"2024-03-17T23:48:11.408364+0000","last_fullsized":"2024-03-17T23:48:11.408364+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:20:48.927064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398382+0000","last_change":"2024-03-17T23:48:11.392442+0000","last_active":"2024-03-17T23:48:12.398382+0000","last_peered":"2024-03-17T23:48:12.398382+0000","last_clean":"2024-03-17T23:48:12.398382+0000","last_became_active":"2024-03-17T23:48:11.392184+0000","last_became_peered":"2024-03-17T23:48:11.392184+0000","last_unstale":"2024-03-17T23:48:12.398382+0000","last_undegraded":"2024-03-17T23:48:12.398382+0000","last_fullsized":"2024-03-17T23:48:12.398382+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:31.101445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.399825+0000","last_change":"2024-03-17T23:48:11.398013+0000","last_active":"2024-03-17T23:48:11.399825+0000","last_peered":"2024-03-17T23:48:11.399825+0000","last_clean":"2024-03-17T23:48:11.399825+0000","last_became_active":"2024-03-17T23:48:11.397709+0000","last_became_peered":"2024-03-17T23:48:11.397709+0000","last_unstale":"2024-03-17T23:48:11.399825+0000","last_undegraded":"2024-03-17T23:48:11.399825+0000","last_fullsized":"2024-03-17T23:48:11.399825+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:05.029387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397892+0000","last_change":"2024-03-17T23:48:11.404602+0000","last_active":"2024-03-17T23:48:12.397892+0000","last_peered":"2024-03-17T23:48:12.397892+0000","last_clean":"2024-03-17T23:48:12.397892+0000","last_became_active":"2024-03-17T23:48:11.404351+0000","last_became_peered":"2024-03-17T23:48:11.404351+0000","last_unstale":"2024-03-17T23:48:12.397892+0000","last_undegraded":"2024-03-17T23:48:12.397892+0000","last_fullsized":"2024-03-17T23:48:12.397892+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:45.409873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409206+0000","last_change":"2024-03-17T23:48:11.403804+0000","last_active":"2024-03-17T23:48:11.409206+0000","last_peered":"2024-03-17T23:48:11.409206+0000","last_clean":"2024-03-17T23:48:11.409206+0000","last_became_active":"2024-03-17T23:48:11.403537+0000","last_became_peered":"2024-03-17T23:48:11.403537+0000","last_unstale":"2024-03-17T23:48:11.409206+0000","last_undegraded":"2024-03-17T23:48:11.409206+0000","last_fullsized":"2024-03-17T23:48:11.409206+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:32:24.643171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397774+0000","last_change":"2024-03-17T23:48:11.398674+0000","last_active":"2024-03-17T23:48:12.397774+0000","last_peered":"2024-03-17T23:48:12.397774+0000","last_clean":"2024-03-17T23:48:12.397774+0000","last_became_active":"2024-03-17T23:48:11.398399+0000","last_became_peered":"2024-03-17T23:48:11.398399+0000","last_unstale":"2024-03-17T23:48:12.397774+0000","last_undegraded":"2024-03-17T23:48:12.397774+0000","last_fullsized":"2024-03-17T23:48:12.397774+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:17.358341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398058+0000","last_change":"2024-03-17T23:48:11.393129+0000","last_active":"2024-03-17T23:48:12.398058+0000","last_peered":"2024-03-17T23:48:12.398058+0000","last_clean":"2024-03-17T23:48:12.398058+0000","last_became_active":"2024-03-17T23:48:11.392847+0000","last_became_peered":"2024-03-17T23:48:11.392847+0000","last_unstale":"2024-03-17T23:48:12.398058+0000","last_undegraded":"2024-03-17T23:48:12.398058+0000","last_fullsized":"2024-03-17T23:48:12.398058+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:20:48.927064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408386+0000","last_change":"2024-03-17T23:48:11.403541+0000","last_active":"2024-03-17T23:48:11.408386+0000","last_peered":"2024-03-17T23:48:11.408386+0000","last_clean":"2024-03-17T23:48:11.408386+0000","last_became_active":"2024-03-17T23:48:11.403273+0000","last_became_peered":"2024-03-17T23:48:11.403273+0000","last_unstale":"2024-03-17T23:48:11.408386+0000","last_undegraded":"2024-03-17T23:48:11.408386+0000","last_fullsized":"2024-03-17T23:48:11.408386+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:10.929912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.397462+0000","last_change":"2024-03-17T23:48:11.396799+0000","last_active":"2024-03-17T23:48:11.397462+0000","last_peered":"2024-03-17T23:48:11.397462+0000","last_clean":"2024-03-17T23:48:11.397462+0000","last_became_active":"2024-03-17T23:48:11.396343+0000","last_became_peered":"2024-03-17T23:48:11.396343+0000","last_unstale":"2024-03-17T23:48:11.397462+0000","last_undegraded":"2024-03-17T23:48:11.397462+0000","last_fullsized":"2024-03-17T23:48:11.397462+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:42:43.776415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.399567+0000","last_change":"2024-03-17T23:48:11.397507+0000","last_active":"2024-03-17T23:48:11.399567+0000","last_peered":"2024-03-17T23:48:11.399567+0000","last_clean":"2024-03-17T23:48:11.399567+0000","last_became_active":"2024-03-17T23:48:11.397238+0000","last_became_peered":"2024-03-17T23:48:11.397238+0000","last_unstale":"2024-03-17T23:48:11.399567+0000","last_undegraded":"2024-03-17T23:48:11.399567+0000","last_fullsized":"2024-03-17T23:48:11.399567+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:28:11.274300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"39'4","reported_seq":18,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398478+0000","last_change":"2024-03-17T23:48:10.382890+0000","last_active":"2024-03-17T23:48:12.398478+0000","last_peered":"2024-03-17T23:48:12.398478+0000","last_clean":"2024-03-17T23:48:12.398478+0000","last_became_active":"2024-03-17T23:48:10.382565+0000","last_became_peered":"2024-03-17T23:48:10.382565+0000","last_unstale":"2024-03-17T23:48:12.398478+0000","last_undegraded":"2024-03-17T23:48:12.398478+0000","last_fullsized":"2024-03-17T23:48:12.398478+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:54.441315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.401433+0000","last_change":"2024-03-17T23:48:11.397742+0000","last_active":"2024-03-17T23:48:11.401433+0000","last_peered":"2024-03-17T23:48:11.401433+0000","last_clean":"2024-03-17T23:48:11.401433+0000","last_became_active":"2024-03-17T23:48:11.397189+0000","last_became_peered":"2024-03-17T23:48:11.397189+0000","last_unstale":"2024-03-17T23:48:11.401433+0000","last_undegraded":"2024-03-17T23:48:11.401433+0000","last_fullsized":"2024-03-17T23:48:11.401433+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:59:29.151560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398181+0000","last_change":"2024-03-17T23:48:11.406337+0000","last_active":"2024-03-17T23:48:12.398181+0000","last_peered":"2024-03-17T23:48:12.398181+0000","last_clean":"2024-03-17T23:48:12.398181+0000","last_became_active":"2024-03-17T23:48:11.405094+0000","last_became_peered":"2024-03-17T23:48:11.405094+0000","last_unstale":"2024-03-17T23:48:12.398181+0000","last_undegraded":"2024-03-17T23:48:12.398181+0000","last_fullsized":"2024-03-17T23:48:12.398181+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:38:38.676853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"39'2","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.406939+0000","last_change":"2024-03-17T23:48:10.396210+0000","last_active":"2024-03-17T23:48:11.406939+0000","last_peered":"2024-03-17T23:48:11.406939+0000","last_clean":"2024-03-17T23:48:11.406939+0000","last_became_active":"2024-03-17T23:48:10.395955+0000","last_became_peered":"2024-03-17T23:48:10.395955+0000","last_unstale":"2024-03-17T23:48:11.406939+0000","last_undegraded":"2024-03-17T23:48:11.406939+0000","last_fullsized":"2024-03-17T23:48:11.406939+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:20:47.924674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"39'2","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398582+0000","last_change":"2024-03-17T23:48:10.391635+0000","last_active":"2024-03-17T23:48:12.398582+0000","last_peered":"2024-03-17T23:48:12.398582+0000","last_clean":"2024-03-17T23:48:12.398582+0000","last_became_active":"2024-03-17T23:48:10.391241+0000","last_became_peered":"2024-03-17T23:48:10.391241+0000","last_unstale":"2024-03-17T23:48:12.398582+0000","last_undegraded":"2024-03-17T23:48:12.398582+0000","last_fullsized":"2024-03-17T23:48:12.398582+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:30.099054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"39'1","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409546+0000","last_change":"2024-03-17T23:48:10.392858+0000","last_active":"2024-03-17T23:48:11.409546+0000","last_peered":"2024-03-17T23:48:11.409546+0000","last_clean":"2024-03-17T23:48:11.409546+0000","last_became_active":"2024-03-17T23:48:10.392599+0000","last_became_peered":"2024-03-17T23:48:10.392599+0000","last_unstale":"2024-03-17T23:48:11.409546+0000","last_undegraded":"2024-03-17T23:48:11.409546+0000","last_fullsized":"2024-03-17T23:48:11.409546+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:12:23.322115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408725+0000","last_change":"2024-03-17T23:48:11.400180+0000","last_active":"2024-03-17T23:48:11.408725+0000","last_peered":"2024-03-17T23:48:11.408725+0000","last_clean":"2024-03-17T23:48:11.408725+0000","last_became_active":"2024-03-17T23:48:11.399900+0000","last_became_peered":"2024-03-17T23:48:11.399900+0000","last_unstale":"2024-03-17T23:48:11.408725+0000","last_undegraded":"2024-03-17T23:48:11.408725+0000","last_fullsized":"2024-03-17T23:48:11.408725+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:31.101445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"39'1","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.407796+0000","last_change":"2024-03-17T23:48:10.396685+0000","last_active":"2024-03-17T23:48:11.407796+0000","last_peered":"2024-03-17T23:48:11.407796+0000","last_clean":"2024-03-17T23:48:11.407796+0000","last_became_active":"2024-03-17T23:48:10.396218+0000","last_became_peered":"2024-03-17T23:48:10.396218+0000","last_unstale":"2024-03-17T23:48:11.407796+0000","last_undegraded":"2024-03-17T23:48:11.407796+0000","last_fullsized":"2024-03-17T23:48:11.407796+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:07:26.890907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398506+0000","last_change":"2024-03-17T23:48:11.398107+0000","last_active":"2024-03-17T23:48:12.398506+0000","last_peered":"2024-03-17T23:48:12.398506+0000","last_clean":"2024-03-17T23:48:12.398506+0000","last_became_active":"2024-03-17T23:48:11.397847+0000","last_became_peered":"2024-03-17T23:48:11.397847+0000","last_unstale":"2024-03-17T23:48:12.398506+0000","last_undegraded":"2024-03-17T23:48:12.398506+0000","last_fullsized":"2024-03-17T23:48:12.398506+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:47:20.574371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.383921+0000","last_change":"2024-03-17T23:48:10.395355+0000","last_active":"2024-03-17T23:48:11.383921+0000","last_peered":"2024-03-17T23:48:11.383921+0000","last_clean":"2024-03-17T23:48:11.383921+0000","last_became_active":"2024-03-17T23:48:10.395051+0000","last_became_peered":"2024-03-17T23:48:10.395051+0000","last_unstale":"2024-03-17T23:48:11.383921+0000","last_undegraded":"2024-03-17T23:48:11.383921+0000","last_fullsized":"2024-03-17T23:48:11.383921+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:28:10.271909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397331+0000","last_change":"2024-03-17T23:48:10.387055+0000","last_active":"2024-03-17T23:48:12.397331+0000","last_peered":"2024-03-17T23:48:12.397331+0000","last_clean":"2024-03-17T23:48:12.397331+0000","last_became_active":"2024-03-17T23:48:10.386732+0000","last_became_peered":"2024-03-17T23:48:10.386732+0000","last_unstale":"2024-03-17T23:48:12.397331+0000","last_undegraded":"2024-03-17T23:48:12.397331+0000","last_fullsized":"2024-03-17T23:48:12.397331+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:59:28.149170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.398574+0000","last_change":"2024-03-17T23:48:11.394613+0000","last_active":"2024-03-17T23:48:11.398574+0000","last_peered":"2024-03-17T23:48:11.398574+0000","last_clean":"2024-03-17T23:48:11.398574+0000","last_became_active":"2024-03-17T23:48:11.394206+0000","last_became_peered":"2024-03-17T23:48:11.394206+0000","last_unstale":"2024-03-17T23:48:11.398574+0000","last_undegraded":"2024-03-17T23:48:11.398574+0000","last_fullsized":"2024-03-17T23:48:11.398574+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:40:54.501510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.389970+0000","last_change":"2024-03-17T23:48:10.394514+0000","last_active":"2024-03-17T23:48:11.389970+0000","last_peered":"2024-03-17T23:48:11.389970+0000","last_clean":"2024-03-17T23:48:11.389970+0000","last_became_active":"2024-03-17T23:48:10.394237+0000","last_became_peered":"2024-03-17T23:48:10.394237+0000","last_unstale":"2024-03-17T23:48:11.389970+0000","last_undegraded":"2024-03-17T23:48:11.389970+0000","last_fullsized":"2024-03-17T23:48:11.389970+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:40:53.499120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"39'3","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409672+0000","last_change":"2024-03-17T23:48:10.400644+0000","last_active":"2024-03-17T23:48:11.409672+0000","last_peered":"2024-03-17T23:48:11.409672+0000","last_clean":"2024-03-17T23:48:11.409672+0000","last_became_active":"2024-03-17T23:48:10.400361+0000","last_became_peered":"2024-03-17T23:48:10.400361+0000","last_unstale":"2024-03-17T23:48:11.409672+0000","last_undegraded":"2024-03-17T23:48:11.409672+0000","last_fullsized":"2024-03-17T23:48:11.409672+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:28:10.271909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398068+0000","last_change":"2024-03-17T23:48:11.404493+0000","last_active":"2024-03-17T23:48:12.398068+0000","last_peered":"2024-03-17T23:48:12.398068+0000","last_clean":"2024-03-17T23:48:12.398068+0000","last_became_active":"2024-03-17T23:48:11.401330+0000","last_became_peered":"2024-03-17T23:48:11.401330+0000","last_unstale":"2024-03-17T23:48:12.398068+0000","last_undegraded":"2024-03-17T23:48:12.398068+0000","last_fullsized":"2024-03-17T23:48:12.398068+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:37:11.855078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.385582+0000","last_change":"2024-03-17T23:48:10.394916+0000","last_active":"2024-03-17T23:48:11.385582+0000","last_peered":"2024-03-17T23:48:11.385582+0000","last_clean":"2024-03-17T23:48:11.385582+0000","last_became_active":"2024-03-17T23:48:10.391532+0000","last_became_peered":"2024-03-17T23:48:10.391532+0000","last_unstale":"2024-03-17T23:48:11.385582+0000","last_undegraded":"2024-03-17T23:48:11.385582+0000","last_fullsized":"2024-03-17T23:48:11.385582+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:44.407483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409135+0000","last_change":"2024-03-17T23:48:11.407145+0000","last_active":"2024-03-17T23:48:11.409135+0000","last_peered":"2024-03-17T23:48:11.409135+0000","last_clean":"2024-03-17T23:48:11.409135+0000","last_became_active":"2024-03-17T23:48:11.406851+0000","last_became_peered":"2024-03-17T23:48:11.406851+0000","last_unstale":"2024-03-17T23:48:11.409135+0000","last_undegraded":"2024-03-17T23:48:11.409135+0000","last_fullsized":"2024-03-17T23:48:11.409135+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:59:29.151560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397440+0000","last_change":"2024-03-17T23:48:10.388069+0000","last_active":"2024-03-17T23:48:12.397440+0000","last_peered":"2024-03-17T23:48:12.397440+0000","last_clean":"2024-03-17T23:48:12.397440+0000","last_became_active":"2024-03-17T23:48:10.387765+0000","last_became_peered":"2024-03-17T23:48:10.387765+0000","last_unstale":"2024-03-17T23:48:12.397440+0000","last_undegraded":"2024-03-17T23:48:12.397440+0000","last_fullsized":"2024-03-17T23:48:12.397440+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:15.116223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"39'8","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.400672+0000","last_change":"2024-03-17T23:48:10.397744+0000","last_active":"2024-03-17T23:48:11.400672+0000","last_peered":"2024-03-17T23:48:11.400672+0000","last_clean":"2024-03-17T23:48:11.400672+0000","last_became_active":"2024-03-17T23:48:10.397467+0000","last_became_peered":"2024-03-17T23:48:10.397467+0000","last_unstale":"2024-03-17T23:48:11.400672+0000","last_undegraded":"2024-03-17T23:48:11.400672+0000","last_fullsized":"2024-03-17T23:48:11.400672+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:12:23.322115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"39'1","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409062+0000","last_change":"2024-03-17T23:48:10.393694+0000","last_active":"2024-03-17T23:48:11.409062+0000","last_peered":"2024-03-17T23:48:11.409062+0000","last_clean":"2024-03-17T23:48:11.409062+0000","last_became_active":"2024-03-17T23:48:10.393418+0000","last_became_peered":"2024-03-17T23:48:10.393418+0000","last_unstale":"2024-03-17T23:48:11.409062+0000","last_undegraded":"2024-03-17T23:48:11.409062+0000","last_fullsized":"2024-03-17T23:48:11.409062+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:42:42.774025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408740+0000","last_change":"2024-03-17T23:48:11.400412+0000","last_active":"2024-03-17T23:48:11.408740+0000","last_peered":"2024-03-17T23:48:11.408740+0000","last_clean":"2024-03-17T23:48:11.408740+0000","last_became_active":"2024-03-17T23:48:11.400006+0000","last_became_peered":"2024-03-17T23:48:11.400006+0000","last_unstale":"2024-03-17T23:48:11.408740+0000","last_undegraded":"2024-03-17T23:48:11.408740+0000","last_fullsized":"2024-03-17T23:48:11.408740+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:59:29.151560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"39'2","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398100+0000","last_change":"2024-03-17T23:48:10.391802+0000","last_active":"2024-03-17T23:48:12.398100+0000","last_peered":"2024-03-17T23:48:12.398100+0000","last_clean":"2024-03-17T23:48:12.398100+0000","last_became_active":"2024-03-17T23:48:10.391464+0000","last_became_peered":"2024-03-17T23:48:10.391464+0000","last_unstale":"2024-03-17T23:48:12.398100+0000","last_undegraded":"2024-03-17T23:48:12.398100+0000","last_fullsized":"2024-03-17T23:48:12.398100+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:44:41.528247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397957+0000","last_change":"2024-03-17T23:48:11.399808+0000","last_active":"2024-03-17T23:48:12.397957+0000","last_peered":"2024-03-17T23:48:12.397957+0000","last_clean":"2024-03-17T23:48:12.397957+0000","last_became_active":"2024-03-17T23:48:11.399460+0000","last_became_peered":"2024-03-17T23:48:11.399460+0000","last_unstale":"2024-03-17T23:48:12.397957+0000","last_undegraded":"2024-03-17T23:48:12.397957+0000","last_fullsized":"2024-03-17T23:48:12.397957+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:16:59.276395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.384478+0000","last_change":"2024-03-17T23:48:10.396820+0000","last_active":"2024-03-17T23:48:11.384478+0000","last_peered":"2024-03-17T23:48:11.384478+0000","last_clean":"2024-03-17T23:48:11.384478+0000","last_became_active":"2024-03-17T23:48:10.396549+0000","last_became_peered":"2024-03-17T23:48:10.396549+0000","last_unstale":"2024-03-17T23:48:11.384478+0000","last_undegraded":"2024-03-17T23:48:11.384478+0000","last_fullsized":"2024-03-17T23:48:11.384478+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:16.355950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408966+0000","last_change":"2024-03-17T23:48:11.403660+0000","last_active":"2024-03-17T23:48:11.408966+0000","last_peered":"2024-03-17T23:48:11.408966+0000","last_clean":"2024-03-17T23:48:11.408966+0000","last_became_active":"2024-03-17T23:48:11.403385+0000","last_became_peered":"2024-03-17T23:48:11.403385+0000","last_unstale":"2024-03-17T23:48:11.408966+0000","last_undegraded":"2024-03-17T23:48:11.408966+0000","last_fullsized":"2024-03-17T23:48:11.408966+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:42:43.776415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409211+0000","last_change":"2024-03-17T23:48:11.404518+0000","last_active":"2024-03-17T23:48:11.409211+0000","last_peered":"2024-03-17T23:48:11.409211+0000","last_clean":"2024-03-17T23:48:11.409211+0000","last_became_active":"2024-03-17T23:48:11.404261+0000","last_became_peered":"2024-03-17T23:48:11.404261+0000","last_unstale":"2024-03-17T23:48:11.409211+0000","last_undegraded":"2024-03-17T23:48:11.409211+0000","last_fullsized":"2024-03-17T23:48:11.409211+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:17:53.610029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.385531+0000","last_change":"2024-03-17T23:48:10.397213+0000","last_active":"2024-03-17T23:48:11.385531+0000","last_peered":"2024-03-17T23:48:11.385531+0000","last_clean":"2024-03-17T23:48:11.385531+0000","last_became_active":"2024-03-17T23:48:10.396913+0000","last_became_peered":"2024-03-17T23:48:10.396913+0000","last_unstale":"2024-03-17T23:48:11.385531+0000","last_undegraded":"2024-03-17T23:48:11.385531+0000","last_fullsized":"2024-03-17T23:48:11.385531+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:31:12.048530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.397234+0000","last_change":"2024-03-17T23:48:11.396445+0000","last_active":"2024-03-17T23:48:11.397234+0000","last_peered":"2024-03-17T23:48:11.397234+0000","last_clean":"2024-03-17T23:48:11.397234+0000","last_became_active":"2024-03-17T23:48:11.396189+0000","last_became_peered":"2024-03-17T23:48:11.396189+0000","last_unstale":"2024-03-17T23:48:11.397234+0000","last_undegraded":"2024-03-17T23:48:11.397234+0000","last_fullsized":"2024-03-17T23:48:11.397234+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:10.929912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"39'2","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.406980+0000","last_change":"2024-03-17T23:48:10.400812+0000","last_active":"2024-03-17T23:48:11.406980+0000","last_peered":"2024-03-17T23:48:11.406980+0000","last_clean":"2024-03-17T23:48:11.406980+0000","last_became_active":"2024-03-17T23:48:10.400479+0000","last_became_peered":"2024-03-17T23:48:10.400479+0000","last_unstale":"2024-03-17T23:48:11.406980+0000","last_undegraded":"2024-03-17T23:48:11.406980+0000","last_fullsized":"2024-03-17T23:48:11.406980+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:09.927522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408459+0000","last_change":"2024-03-17T23:48:11.403785+0000","last_active":"2024-03-17T23:48:11.408459+0000","last_peered":"2024-03-17T23:48:11.408459+0000","last_clean":"2024-03-17T23:48:11.408459+0000","last_became_active":"2024-03-17T23:48:11.402301+0000","last_became_peered":"2024-03-17T23:48:11.402301+0000","last_unstale":"2024-03-17T23:48:11.408459+0000","last_undegraded":"2024-03-17T23:48:11.408459+0000","last_fullsized":"2024-03-17T23:48:11.408459+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:31.101445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.397727+0000","last_change":"2024-03-17T23:48:11.397059+0000","last_active":"2024-03-17T23:48:11.397727+0000","last_peered":"2024-03-17T23:48:11.397727+0000","last_clean":"2024-03-17T23:48:11.397727+0000","last_became_active":"2024-03-17T23:48:11.396724+0000","last_became_peered":"2024-03-17T23:48:11.396724+0000","last_unstale":"2024-03-17T23:48:11.397727+0000","last_undegraded":"2024-03-17T23:48:11.397727+0000","last_fullsized":"2024-03-17T23:48:11.397727+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:31.101445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398165+0000","last_change":"2024-03-17T23:48:10.384305+0000","last_active":"2024-03-17T23:48:12.398165+0000","last_peered":"2024-03-17T23:48:12.398165+0000","last_clean":"2024-03-17T23:48:12.398165+0000","last_became_active":"2024-03-17T23:48:10.384034+0000","last_became_peered":"2024-03-17T23:48:10.384034+0000","last_unstale":"2024-03-17T23:48:12.398165+0000","last_undegraded":"2024-03-17T23:48:12.398165+0000","last_fullsized":"2024-03-17T23:48:12.398165+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:32:37.500442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398088+0000","last_change":"2024-03-17T23:48:11.394167+0000","last_active":"2024-03-17T23:48:12.398088+0000","last_peered":"2024-03-17T23:48:12.398088+0000","last_clean":"2024-03-17T23:48:12.398088+0000","last_became_active":"2024-03-17T23:48:11.392337+0000","last_became_peered":"2024-03-17T23:48:11.392337+0000","last_unstale":"2024-03-17T23:48:12.398088+0000","last_undegraded":"2024-03-17T23:48:12.398088+0000","last_fullsized":"2024-03-17T23:48:12.398088+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:04:14.826887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397422+0000","last_change":"2024-03-17T23:48:10.391429+0000","last_active":"2024-03-17T23:48:12.397422+0000","last_peered":"2024-03-17T23:48:12.397422+0000","last_clean":"2024-03-17T23:48:12.397422+0000","last_became_active":"2024-03-17T23:48:10.390974+0000","last_became_peered":"2024-03-17T23:48:10.390974+0000","last_unstale":"2024-03-17T23:48:12.397422+0000","last_undegraded":"2024-03-17T23:48:12.397422+0000","last_fullsized":"2024-03-17T23:48:12.397422+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:04.026997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.399296+0000","last_change":"2024-03-17T23:48:11.397803+0000","last_active":"2024-03-17T23:48:11.399296+0000","last_peered":"2024-03-17T23:48:11.399296+0000","last_clean":"2024-03-17T23:48:11.399296+0000","last_became_active":"2024-03-17T23:48:11.397541+0000","last_became_peered":"2024-03-17T23:48:11.397541+0000","last_unstale":"2024-03-17T23:48:11.399296+0000","last_undegraded":"2024-03-17T23:48:11.399296+0000","last_fullsized":"2024-03-17T23:48:11.399296+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:17.358341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409115+0000","last_change":"2024-03-17T23:48:11.403123+0000","last_active":"2024-03-17T23:48:11.409115+0000","last_peered":"2024-03-17T23:48:11.409115+0000","last_clean":"2024-03-17T23:48:11.409115+0000","last_became_active":"2024-03-17T23:48:11.402851+0000","last_became_peered":"2024-03-17T23:48:11.402851+0000","last_unstale":"2024-03-17T23:48:11.409115+0000","last_undegraded":"2024-03-17T23:48:11.409115+0000","last_fullsized":"2024-03-17T23:48:11.409115+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:44:42.530638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.384095+0000","last_change":"2024-03-17T23:48:10.398461+0000","last_active":"2024-03-17T23:48:11.384095+0000","last_peered":"2024-03-17T23:48:11.384095+0000","last_clean":"2024-03-17T23:48:11.384095+0000","last_became_active":"2024-03-17T23:48:10.398188+0000","last_became_peered":"2024-03-17T23:48:10.398188+0000","last_unstale":"2024-03-17T23:48:11.384095+0000","last_undegraded":"2024-03-17T23:48:11.384095+0000","last_fullsized":"2024-03-17T23:48:11.384095+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:06:33.058080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":204,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.385340+0000","last_change":"2024-03-17T23:46:44.768668+0000","last_active":"2024-03-17T23:48:11.385340+0000","last_peered":"2024-03-17T23:48:11.385340+0000","last_clean":"2024-03-17T23:48:11.385340+0000","last_became_active":"2024-03-17T23:46:44.767826+0000","last_became_peered":"2024-03-17T23:46:44.767826+0000","last_unstale":"2024-03-17T23:48:11.385340+0000","last_undegraded":"2024-03-17T23:48:11.385340+0000","last_fullsized":"2024-03-17T23:48:11.385340+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_clean_scrub_stamp":"2024-03-17T23:46:43.744878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:54:22.286155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":80,"acting":80,"num_store_stats":5},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":694,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":40,"acting":40,"num_store_stats":5},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27424,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":409600,"data_stored":174269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27420,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":409600,"data_stored":174269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542165,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28008,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343832,"statfs":{"total":96636764160,"available":96608083968,"internally_reserved":0,"allocated":1003520,"data_stored":764637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32212,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339628,"statfs":{"total":96636764160,"available":96603779072,"internally_reserved":0,"allocated":1056768,"data_stored":786362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31636,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340204,"statfs":{"total":96636764160,"available":96604368896,"internally_reserved":0,"allocated":462848,"data_stored":201586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31624,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":454656,"data_stored":194424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31624,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":454656,"data_stored":194916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":450560,"data_stored":194402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-17T23:48:14.801 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-17T23:48:14.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.974+0000 7f2fa9014640 1 Processor -- start 2024-03-17T23:48:14.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.974+0000 7f2fa9014640 1 -- start start 2024-03-17T23:48:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.975+0000 7f2fa9014640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2fa41514f0 0x7f2fa415bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.975+0000 7f2fa9014640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2fa406ecd0 0x7f2fa406f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.975+0000 7f2fa9014640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fa40ce790 con 0x7f2fa415ab90 2024-03-17T23:48:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.975+0000 7f2fa9014640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f2fa406fba0 con 0x7f2fa406ecd0 2024-03-17T23:48:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.975+0000 7f2fa9014640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f2fa415b050 con 0x7f2fa41514f0 2024-03-17T23:48:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.975+0000 7f2fa359e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2fa406ecd0 0x7f2fa406f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.975+0000 7f2f9bfff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f2fa415ab90 0x7f2fa406e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37624/0 (socket says 172.21.15.150:37624) 2024-03-17T23:48:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.975+0000 7f2f9bfff640 1 -- 172.21.15.150:0/521108731 learned_addr learned my addr 172.21.15.150:0/521108731 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.975+0000 7f2fa2d9d640 1 --2- 172.21.15.150:0/521108731 >> v2:172.21.15.157:3300/0 conn(0x7f2fa41514f0 0x7f2fa415bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.975+0000 7f2fa359e640 1 -- 172.21.15.150:0/521108731 >> v2:172.21.15.157:3300/0 conn(0x7f2fa41514f0 msgr2=0x7f2fa415bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:14.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.975+0000 7f2fa359e640 1 --2- 172.21.15.150:0/521108731 >> v2:172.21.15.157:3300/0 conn(0x7f2fa41514f0 0x7f2fa415bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.975+0000 7f2fa259c640 1 -- 172.21.15.150:0/521108731 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1920978899 0 0) 0x7f2f8c001110 con 0x7f2fa415ab90 2024-03-17T23:48:14.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.975+0000 7f2fa359e640 1 -- 172.21.15.150:0/521108731 >> v1:172.21.15.150:6789/0 conn(0x7f2fa415ab90 legacy=0x7f2fa406e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:14.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.975+0000 7f2fa359e640 1 -- 172.21.15.150:0/521108731 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa415b330 con 0x7f2fa406ecd0 2024-03-17T23:48:14.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.976+0000 7f2fa359e640 1 --2- 172.21.15.150:0/521108731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fa406ecd0 0x7f2fa406f660 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4072d90 tx=0x7f2f9c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dc4fda768be691b server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:14.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.976+0000 7f2fa259c640 1 -- 172.21.15.150:0/521108731 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c039070 con 0x7f2fa406ecd0 2024-03-17T23:48:14.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.976+0000 7f2fa259c640 1 -- 172.21.15.150:0/521108731 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f9c02cc80 con 0x7f2fa406ecd0 2024-03-17T23:48:14.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.976+0000 7f2fa259c640 1 -- 172.21.15.150:0/521108731 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c034660 con 0x7f2fa406ecd0 2024-03-17T23:48:14.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.977+0000 7f2fa9014640 1 -- 172.21.15.150:0/521108731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fa406ecd0 msgr2=0x7f2fa406f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:14.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.977+0000 7f2fa9014640 1 --2- 172.21.15.150:0/521108731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fa406ecd0 0x7f2fa406f660 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4072d90 tx=0x7f2f9c02bed0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.977+0000 7f2fa9014640 1 -- 172.21.15.150:0/521108731 shutdown_connections 2024-03-17T23:48:14.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.977+0000 7f2fa9014640 1 --2- 172.21.15.150:0/521108731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fa406ecd0 0x7f2fa406f660 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.977+0000 7f2fa9014640 1 --2- 172.21.15.150:0/521108731 >> v2:172.21.15.157:3300/0 conn(0x7f2fa41514f0 0x7f2fa415bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.977+0000 7f2fa9014640 1 -- 172.21.15.150:0/521108731 >> 172.21.15.150:0/521108731 conn(0x7f2fa406b190 msgr2=0x7f2fa406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:14.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.977+0000 7f2fa9014640 1 -- 172.21.15.150:0/521108731 shutdown_connections 2024-03-17T23:48:14.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.977+0000 7f2fa9014640 1 -- 172.21.15.150:0/521108731 wait complete. 2024-03-17T23:48:14.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.978+0000 7f2fa9014640 1 Processor -- start 2024-03-17T23:48:14.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.978+0000 7f2fa9014640 1 -- start start 2024-03-17T23:48:14.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.979+0000 7f2fa9014640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fa406ecd0 0x7f2fa414e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:14.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.979+0000 7f2fa9014640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fa41514f0 0x7f2fa4146410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:14.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.979+0000 7f2fa9014640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2fa415ab90 0x7f2fa41441a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:14.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.979+0000 7f2fa9014640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa40af510 con 0x7f2fa406ecd0 2024-03-17T23:48:14.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.979+0000 7f2fa9014640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa415b8f0 con 0x7f2fa415ab90 2024-03-17T23:48:14.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.979+0000 7f2fa9014640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2fa41518d0 con 0x7f2fa41514f0 2024-03-17T23:48:14.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.979+0000 7f2fa2d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fa406ecd0 0x7f2fa414e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:14.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.979+0000 7f2fa2d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fa406ecd0 0x7f2fa414e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43768/0 (socket says 172.21.15.150:43768) 2024-03-17T23:48:14.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.979+0000 7f2fa2d9d640 1 -- 172.21.15.150:0/2307877616 learned_addr learned my addr 172.21.15.150:0/2307877616 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:14.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.979+0000 7f2f9bfff640 1 --2- 172.21.15.150:0/2307877616 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fa41514f0 0x7f2fa4146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:14.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.979+0000 7f2fa2d9d640 1 -- 172.21.15.150:0/2307877616 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fa41514f0 msgr2=0x7f2fa4146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:14.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.979+0000 7f2fa2d9d640 1 --2- 172.21.15.150:0/2307877616 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fa41514f0 0x7f2fa4146410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.979+0000 7f2fa2d9d640 1 -- 172.21.15.150:0/2307877616 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2fa415ab90 msgr2=0x7f2fa41441a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:14.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.979+0000 7f2fa2d9d640 1 --2- 172.21.15.150:0/2307877616 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2fa415ab90 0x7f2fa41441a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:14.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.979+0000 7f2fa2d9d640 1 -- 172.21.15.150:0/2307877616 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa4153de0 con 0x7f2fa406ecd0 2024-03-17T23:48:14.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.980+0000 7f2fa2d9d640 1 --2- 172.21.15.150:0/2307877616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fa406ecd0 0x7f2fa414e2c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2f9400d830 tx=0x7f2f9400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:14.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.980+0000 7f2f9b7fe640 1 -- 172.21.15.150:0/2307877616 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9400b660 con 0x7f2fa406ecd0 2024-03-17T23:48:14.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.980+0000 7f2f9b7fe640 1 -- 172.21.15.150:0/2307877616 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f94019070 con 0x7f2fa406ecd0 2024-03-17T23:48:14.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.980+0000 7f2f9b7fe640 1 -- 172.21.15.150:0/2307877616 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9400ba20 con 0x7f2fa406ecd0 2024-03-17T23:48:14.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.980+0000 7f2fa9014640 1 -- 172.21.15.150:0/2307877616 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fa414c440 con 0x7f2fa406ecd0 2024-03-17T23:48:14.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.980+0000 7f2fa9014640 1 -- 172.21.15.150:0/2307877616 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fa414cc80 con 0x7f2fa406ecd0 2024-03-17T23:48:14.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.981+0000 7f2fa9014640 1 -- 172.21.15.150:0/2307877616 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fa40681a0 con 0x7f2fa406ecd0 2024-03-17T23:48:14.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.981+0000 7f2f9b7fe640 1 -- 172.21.15.150:0/2307877616 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2f9401e460 con 0x7f2fa406ecd0 2024-03-17T23:48:14.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.982+0000 7f2f9b7fe640 1 --2- 172.21.15.150:0/2307877616 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2f8406bbd0 0x7f2f8406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:14.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.982+0000 7f2f9bfff640 1 --2- 172.21.15.150:0/2307877616 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2f8406bbd0 0x7f2f8406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:14.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.982+0000 7f2f9b7fe640 1 -- 172.21.15.150:0/2307877616 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7f2f9408b260 con 0x7f2fa406ecd0 2024-03-17T23:48:14.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.982+0000 7f2f9bfff640 1 --2- 172.21.15.150:0/2307877616 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2f8406bbd0 0x7f2f8406e090 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4147110 tx=0x7f2f8c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:14.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:14.986+0000 7f2f9b7fe640 1 -- 172.21.15.150:0/2307877616 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2fa414c440 con 0x7f2fa406ecd0 2024-03-17T23:48:15.142 INFO:tasks.ceph.mds.c.smithi150.stderr:2024-03-17T23:48:15.142+0000 7f6c778f1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-17T23:48:15.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.234+0000 7f2fa9014640 1 -- 172.21.15.150:0/2307877616 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2fa406bf90 con 0x7f2f8406bbd0 2024-03-17T23:48:15.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.245+0000 7f2f9b7fe640 1 -- 172.21.15.150:0/2307877616 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246405 (secure 0 0 0) 0x7f2fa406bf90 con 0x7f2f8406bbd0 2024-03-17T23:48:15.249 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:15.250 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-03-17T23:48:15.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.248+0000 7f2fa9014640 1 -- 172.21.15.150:0/2307877616 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2f8406bbd0 msgr2=0x7f2f8406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:15.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.248+0000 7f2fa9014640 1 --2- 172.21.15.150:0/2307877616 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2f8406bbd0 0x7f2f8406e090 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4147110 tx=0x7f2f8c009040 comp rx=0 tx=0).stop 2024-03-17T23:48:15.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.248+0000 7f2fa9014640 1 -- 172.21.15.150:0/2307877616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fa406ecd0 msgr2=0x7f2fa414e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:15.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.248+0000 7f2fa9014640 1 --2- 172.21.15.150:0/2307877616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fa406ecd0 0x7f2fa414e2c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2f9400d830 tx=0x7f2f9400dcf0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.248+0000 7f2fa9014640 1 -- 172.21.15.150:0/2307877616 shutdown_connections 2024-03-17T23:48:15.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.248+0000 7f2fa9014640 1 --2- 172.21.15.150:0/2307877616 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2f8406bbd0 0x7f2f8406e090 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.249+0000 7f2fa9014640 1 --2- 172.21.15.150:0/2307877616 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2fa415ab90 0x7f2fa41441a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.249+0000 7f2fa9014640 1 --2- 172.21.15.150:0/2307877616 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2fa41514f0 0x7f2fa4146410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.249+0000 7f2fa9014640 1 --2- 172.21.15.150:0/2307877616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2fa406ecd0 0x7f2fa414e2c0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.249+0000 7f2fa9014640 1 -- 172.21.15.150:0/2307877616 >> 172.21.15.150:0/2307877616 conn(0x7f2fa406b190 msgr2=0x7f2fa4071cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:15.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.249+0000 7f2fa9014640 1 -- 172.21.15.150:0/2307877616 shutdown_connections 2024-03-17T23:48:15.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.249+0000 7f2fa9014640 1 -- 172.21.15.150:0/2307877616 wait complete. 2024-03-17T23:48:15.260 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2024-03-17T23:48:13.612979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591062,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":192,"num_write_kb":2419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":122,"acting":122,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":92,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":6,"kb":754974720,"kb_used":237408,"kb_used_data":4592,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737312,"statfs":{"total":773094113280,"available":772851007488,"internally_reserved":0,"allocated":4702208,"data_stored":2684865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":694,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.241889"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.400417+0000","last_change":"2024-03-17T23:48:11.399627+0000","last_active":"2024-03-17T23:48:11.400417+0000","last_peered":"2024-03-17T23:48:11.400417+0000","last_clean":"2024-03-17T23:48:11.400417+0000","last_became_active":"2024-03-17T23:48:11.399364+0000","last_became_peered":"2024-03-17T23:48:11.399364+0000","last_unstale":"2024-03-17T23:48:11.400417+0000","last_undegraded":"2024-03-17T23:48:11.400417+0000","last_fullsized":"2024-03-17T23:48:11.400417+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:44:42.530638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409171+0000","last_change":"2024-03-17T23:48:11.403822+0000","last_active":"2024-03-17T23:48:11.409171+0000","last_peered":"2024-03-17T23:48:11.409171+0000","last_clean":"2024-03-17T23:48:11.409171+0000","last_became_active":"2024-03-17T23:48:11.403572+0000","last_became_peered":"2024-03-17T23:48:11.403572+0000","last_unstale":"2024-03-17T23:48:11.409171+0000","last_undegraded":"2024-03-17T23:48:11.409171+0000","last_fullsized":"2024-03-17T23:48:11.409171+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:53:53.408503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.406658+0000","last_change":"2024-03-17T23:48:11.395918+0000","last_active":"2024-03-17T23:48:11.406658+0000","last_peered":"2024-03-17T23:48:11.406658+0000","last_clean":"2024-03-17T23:48:11.406658+0000","last_became_active":"2024-03-17T23:48:11.394963+0000","last_became_peered":"2024-03-17T23:48:11.394963+0000","last_unstale":"2024-03-17T23:48:11.406658+0000","last_undegraded":"2024-03-17T23:48:11.406658+0000","last_fullsized":"2024-03-17T23:48:11.406658+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:20:48.927064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.399665+0000","last_change":"2024-03-17T23:48:11.394537+0000","last_active":"2024-03-17T23:48:11.399665+0000","last_peered":"2024-03-17T23:48:11.399665+0000","last_clean":"2024-03-17T23:48:11.399665+0000","last_became_active":"2024-03-17T23:48:11.394147+0000","last_became_peered":"2024-03-17T23:48:11.394147+0000","last_unstale":"2024-03-17T23:48:11.399665+0000","last_undegraded":"2024-03-17T23:48:11.399665+0000","last_fullsized":"2024-03-17T23:48:11.399665+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:45.409873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.394641+0000","last_change":"2024-03-17T23:48:11.393682+0000","last_active":"2024-03-17T23:48:11.394641+0000","last_peered":"2024-03-17T23:48:11.394641+0000","last_clean":"2024-03-17T23:48:11.394641+0000","last_became_active":"2024-03-17T23:48:11.393410+0000","last_became_peered":"2024-03-17T23:48:11.393410+0000","last_unstale":"2024-03-17T23:48:11.394641+0000","last_undegraded":"2024-03-17T23:48:11.394641+0000","last_fullsized":"2024-03-17T23:48:11.394641+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:17.358341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397913+0000","last_change":"2024-03-17T23:48:11.405838+0000","last_active":"2024-03-17T23:48:12.397913+0000","last_peered":"2024-03-17T23:48:12.397913+0000","last_clean":"2024-03-17T23:48:12.397913+0000","last_became_active":"2024-03-17T23:48:11.405573+0000","last_became_peered":"2024-03-17T23:48:11.405573+0000","last_unstale":"2024-03-17T23:48:12.397913+0000","last_undegraded":"2024-03-17T23:48:12.397913+0000","last_fullsized":"2024-03-17T23:48:12.397913+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:33:46.207300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409150+0000","last_change":"2024-03-17T23:48:11.403246+0000","last_active":"2024-03-17T23:48:11.409150+0000","last_peered":"2024-03-17T23:48:11.409150+0000","last_clean":"2024-03-17T23:48:11.409150+0000","last_became_active":"2024-03-17T23:48:11.402986+0000","last_became_peered":"2024-03-17T23:48:11.402986+0000","last_unstale":"2024-03-17T23:48:11.409150+0000","last_undegraded":"2024-03-17T23:48:11.409150+0000","last_fullsized":"2024-03-17T23:48:11.409150+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:16:44.071095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.402167+0000","last_change":"2024-03-17T23:48:11.398200+0000","last_active":"2024-03-17T23:48:11.402167+0000","last_peered":"2024-03-17T23:48:11.402167+0000","last_clean":"2024-03-17T23:48:11.402167+0000","last_became_active":"2024-03-17T23:48:11.397937+0000","last_became_peered":"2024-03-17T23:48:11.397937+0000","last_unstale":"2024-03-17T23:48:11.402167+0000","last_undegraded":"2024-03-17T23:48:11.402167+0000","last_fullsized":"2024-03-17T23:48:11.402167+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:45.409873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409193+0000","last_change":"2024-03-17T23:48:11.408181+0000","last_active":"2024-03-17T23:48:11.409193+0000","last_peered":"2024-03-17T23:48:11.409193+0000","last_clean":"2024-03-17T23:48:11.409193+0000","last_became_active":"2024-03-17T23:48:11.407876+0000","last_became_peered":"2024-03-17T23:48:11.407876+0000","last_unstale":"2024-03-17T23:48:11.409193+0000","last_undegraded":"2024-03-17T23:48:11.409193+0000","last_fullsized":"2024-03-17T23:48:11.409193+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:05.029387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398135+0000","last_change":"2024-03-17T23:48:11.402141+0000","last_active":"2024-03-17T23:48:12.398135+0000","last_peered":"2024-03-17T23:48:12.398135+0000","last_clean":"2024-03-17T23:48:12.398135+0000","last_became_active":"2024-03-17T23:48:11.401870+0000","last_became_peered":"2024-03-17T23:48:11.401870+0000","last_unstale":"2024-03-17T23:48:12.398135+0000","last_undegraded":"2024-03-17T23:48:12.398135+0000","last_fullsized":"2024-03-17T23:48:12.398135+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:10.929912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398329+0000","last_change":"2024-03-17T23:48:11.401263+0000","last_active":"2024-03-17T23:48:12.398329+0000","last_peered":"2024-03-17T23:48:12.398329+0000","last_clean":"2024-03-17T23:48:12.398329+0000","last_became_active":"2024-03-17T23:48:11.400951+0000","last_became_peered":"2024-03-17T23:48:11.400951+0000","last_unstale":"2024-03-17T23:48:12.398329+0000","last_undegraded":"2024-03-17T23:48:12.398329+0000","last_fullsized":"2024-03-17T23:48:12.398329+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:42:43.776415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408759+0000","last_change":"2024-03-17T23:48:11.400746+0000","last_active":"2024-03-17T23:48:11.408759+0000","last_peered":"2024-03-17T23:48:11.408759+0000","last_clean":"2024-03-17T23:48:11.408759+0000","last_became_active":"2024-03-17T23:48:11.400494+0000","last_became_peered":"2024-03-17T23:48:11.400494+0000","last_unstale":"2024-03-17T23:48:11.408759+0000","last_undegraded":"2024-03-17T23:48:11.408759+0000","last_fullsized":"2024-03-17T23:48:11.408759+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:05.029387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398311+0000","last_change":"2024-03-17T23:48:11.403046+0000","last_active":"2024-03-17T23:48:12.398311+0000","last_peered":"2024-03-17T23:48:12.398311+0000","last_clean":"2024-03-17T23:48:12.398311+0000","last_became_active":"2024-03-17T23:48:11.402769+0000","last_became_peered":"2024-03-17T23:48:11.402769+0000","last_unstale":"2024-03-17T23:48:12.398311+0000","last_undegraded":"2024-03-17T23:48:12.398311+0000","last_fullsized":"2024-03-17T23:48:12.398311+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:16:09.342548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408364+0000","last_change":"2024-03-17T23:48:11.402963+0000","last_active":"2024-03-17T23:48:11.408364+0000","last_peered":"2024-03-17T23:48:11.408364+0000","last_clean":"2024-03-17T23:48:11.408364+0000","last_became_active":"2024-03-17T23:48:11.402669+0000","last_became_peered":"2024-03-17T23:48:11.402669+0000","last_unstale":"2024-03-17T23:48:11.408364+0000","last_undegraded":"2024-03-17T23:48:11.408364+0000","last_fullsized":"2024-03-17T23:48:11.408364+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:20:48.927064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398382+0000","last_change":"2024-03-17T23:48:11.392442+0000","last_active":"2024-03-17T23:48:12.398382+0000","last_peered":"2024-03-17T23:48:12.398382+0000","last_clean":"2024-03-17T23:48:12.398382+0000","last_became_active":"2024-03-17T23:48:11.392184+0000","last_became_peered":"2024-03-17T23:48:11.392184+0000","last_unstale":"2024-03-17T23:48:12.398382+0000","last_undegraded":"2024-03-17T23:48:12.398382+0000","last_fullsized":"2024-03-17T23:48:12.398382+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:31.101445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.399825+0000","last_change":"2024-03-17T23:48:11.398013+0000","last_active":"2024-03-17T23:48:11.399825+0000","last_peered":"2024-03-17T23:48:11.399825+0000","last_clean":"2024-03-17T23:48:11.399825+0000","last_became_active":"2024-03-17T23:48:11.397709+0000","last_became_peered":"2024-03-17T23:48:11.397709+0000","last_unstale":"2024-03-17T23:48:11.399825+0000","last_undegraded":"2024-03-17T23:48:11.399825+0000","last_fullsized":"2024-03-17T23:48:11.399825+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:05.029387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397892+0000","last_change":"2024-03-17T23:48:11.404602+0000","last_active":"2024-03-17T23:48:12.397892+0000","last_peered":"2024-03-17T23:48:12.397892+0000","last_clean":"2024-03-17T23:48:12.397892+0000","last_became_active":"2024-03-17T23:48:11.404351+0000","last_became_peered":"2024-03-17T23:48:11.404351+0000","last_unstale":"2024-03-17T23:48:12.397892+0000","last_undegraded":"2024-03-17T23:48:12.397892+0000","last_fullsized":"2024-03-17T23:48:12.397892+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:45.409873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409206+0000","last_change":"2024-03-17T23:48:11.403804+0000","last_active":"2024-03-17T23:48:11.409206+0000","last_peered":"2024-03-17T23:48:11.409206+0000","last_clean":"2024-03-17T23:48:11.409206+0000","last_became_active":"2024-03-17T23:48:11.403537+0000","last_became_peered":"2024-03-17T23:48:11.403537+0000","last_unstale":"2024-03-17T23:48:11.409206+0000","last_undegraded":"2024-03-17T23:48:11.409206+0000","last_fullsized":"2024-03-17T23:48:11.409206+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:32:24.643171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397774+0000","last_change":"2024-03-17T23:48:11.398674+0000","last_active":"2024-03-17T23:48:12.397774+0000","last_peered":"2024-03-17T23:48:12.397774+0000","last_clean":"2024-03-17T23:48:12.397774+0000","last_became_active":"2024-03-17T23:48:11.398399+0000","last_became_peered":"2024-03-17T23:48:11.398399+0000","last_unstale":"2024-03-17T23:48:12.397774+0000","last_undegraded":"2024-03-17T23:48:12.397774+0000","last_fullsized":"2024-03-17T23:48:12.397774+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:17.358341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398058+0000","last_change":"2024-03-17T23:48:11.393129+0000","last_active":"2024-03-17T23:48:12.398058+0000","last_peered":"2024-03-17T23:48:12.398058+0000","last_clean":"2024-03-17T23:48:12.398058+0000","last_became_active":"2024-03-17T23:48:11.392847+0000","last_became_peered":"2024-03-17T23:48:11.392847+0000","last_unstale":"2024-03-17T23:48:12.398058+0000","last_undegraded":"2024-03-17T23:48:12.398058+0000","last_fullsized":"2024-03-17T23:48:12.398058+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:20:48.927064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408386+0000","last_change":"2024-03-17T23:48:11.403541+0000","last_active":"2024-03-17T23:48:11.408386+0000","last_peered":"2024-03-17T23:48:11.408386+0000","last_clean":"2024-03-17T23:48:11.408386+0000","last_became_active":"2024-03-17T23:48:11.403273+0000","last_became_peered":"2024-03-17T23:48:11.403273+0000","last_unstale":"2024-03-17T23:48:11.408386+0000","last_undegraded":"2024-03-17T23:48:11.408386+0000","last_fullsized":"2024-03-17T23:48:11.408386+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:10.929912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.397462+0000","last_change":"2024-03-17T23:48:11.396799+0000","last_active":"2024-03-17T23:48:11.397462+0000","last_peered":"2024-03-17T23:48:11.397462+0000","last_clean":"2024-03-17T23:48:11.397462+0000","last_became_active":"2024-03-17T23:48:11.396343+0000","last_became_peered":"2024-03-17T23:48:11.396343+0000","last_unstale":"2024-03-17T23:48:11.397462+0000","last_undegraded":"2024-03-17T23:48:11.397462+0000","last_fullsized":"2024-03-17T23:48:11.397462+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:42:43.776415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.399567+0000","last_change":"2024-03-17T23:48:11.397507+0000","last_active":"2024-03-17T23:48:11.399567+0000","last_peered":"2024-03-17T23:48:11.399567+0000","last_clean":"2024-03-17T23:48:11.399567+0000","last_became_active":"2024-03-17T23:48:11.397238+0000","last_became_peered":"2024-03-17T23:48:11.397238+0000","last_unstale":"2024-03-17T23:48:11.399567+0000","last_undegraded":"2024-03-17T23:48:11.399567+0000","last_fullsized":"2024-03-17T23:48:11.399567+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:28:11.274300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"39'4","reported_seq":18,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398478+0000","last_change":"2024-03-17T23:48:10.382890+0000","last_active":"2024-03-17T23:48:12.398478+0000","last_peered":"2024-03-17T23:48:12.398478+0000","last_clean":"2024-03-17T23:48:12.398478+0000","last_became_active":"2024-03-17T23:48:10.382565+0000","last_became_peered":"2024-03-17T23:48:10.382565+0000","last_unstale":"2024-03-17T23:48:12.398478+0000","last_undegraded":"2024-03-17T23:48:12.398478+0000","last_fullsized":"2024-03-17T23:48:12.398478+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:54.441315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.401433+0000","last_change":"2024-03-17T23:48:11.397742+0000","last_active":"2024-03-17T23:48:11.401433+0000","last_peered":"2024-03-17T23:48:11.401433+0000","last_clean":"2024-03-17T23:48:11.401433+0000","last_became_active":"2024-03-17T23:48:11.397189+0000","last_became_peered":"2024-03-17T23:48:11.397189+0000","last_unstale":"2024-03-17T23:48:11.401433+0000","last_undegraded":"2024-03-17T23:48:11.401433+0000","last_fullsized":"2024-03-17T23:48:11.401433+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:59:29.151560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398181+0000","last_change":"2024-03-17T23:48:11.406337+0000","last_active":"2024-03-17T23:48:12.398181+0000","last_peered":"2024-03-17T23:48:12.398181+0000","last_clean":"2024-03-17T23:48:12.398181+0000","last_became_active":"2024-03-17T23:48:11.405094+0000","last_became_peered":"2024-03-17T23:48:11.405094+0000","last_unstale":"2024-03-17T23:48:12.398181+0000","last_undegraded":"2024-03-17T23:48:12.398181+0000","last_fullsized":"2024-03-17T23:48:12.398181+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:38:38.676853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"39'2","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.406939+0000","last_change":"2024-03-17T23:48:10.396210+0000","last_active":"2024-03-17T23:48:11.406939+0000","last_peered":"2024-03-17T23:48:11.406939+0000","last_clean":"2024-03-17T23:48:11.406939+0000","last_became_active":"2024-03-17T23:48:10.395955+0000","last_became_peered":"2024-03-17T23:48:10.395955+0000","last_unstale":"2024-03-17T23:48:11.406939+0000","last_undegraded":"2024-03-17T23:48:11.406939+0000","last_fullsized":"2024-03-17T23:48:11.406939+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:20:47.924674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"39'2","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398582+0000","last_change":"2024-03-17T23:48:10.391635+0000","last_active":"2024-03-17T23:48:12.398582+0000","last_peered":"2024-03-17T23:48:12.398582+0000","last_clean":"2024-03-17T23:48:12.398582+0000","last_became_active":"2024-03-17T23:48:10.391241+0000","last_became_peered":"2024-03-17T23:48:10.391241+0000","last_unstale":"2024-03-17T23:48:12.398582+0000","last_undegraded":"2024-03-17T23:48:12.398582+0000","last_fullsized":"2024-03-17T23:48:12.398582+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:30.099054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"39'1","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409546+0000","last_change":"2024-03-17T23:48:10.392858+0000","last_active":"2024-03-17T23:48:11.409546+0000","last_peered":"2024-03-17T23:48:11.409546+0000","last_clean":"2024-03-17T23:48:11.409546+0000","last_became_active":"2024-03-17T23:48:10.392599+0000","last_became_peered":"2024-03-17T23:48:10.392599+0000","last_unstale":"2024-03-17T23:48:11.409546+0000","last_undegraded":"2024-03-17T23:48:11.409546+0000","last_fullsized":"2024-03-17T23:48:11.409546+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:12:23.322115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408725+0000","last_change":"2024-03-17T23:48:11.400180+0000","last_active":"2024-03-17T23:48:11.408725+0000","last_peered":"2024-03-17T23:48:11.408725+0000","last_clean":"2024-03-17T23:48:11.408725+0000","last_became_active":"2024-03-17T23:48:11.399900+0000","last_became_peered":"2024-03-17T23:48:11.399900+0000","last_unstale":"2024-03-17T23:48:11.408725+0000","last_undegraded":"2024-03-17T23:48:11.408725+0000","last_fullsized":"2024-03-17T23:48:11.408725+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:31.101445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"39'1","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.407796+0000","last_change":"2024-03-17T23:48:10.396685+0000","last_active":"2024-03-17T23:48:11.407796+0000","last_peered":"2024-03-17T23:48:11.407796+0000","last_clean":"2024-03-17T23:48:11.407796+0000","last_became_active":"2024-03-17T23:48:10.396218+0000","last_became_peered":"2024-03-17T23:48:10.396218+0000","last_unstale":"2024-03-17T23:48:11.407796+0000","last_undegraded":"2024-03-17T23:48:11.407796+0000","last_fullsized":"2024-03-17T23:48:11.407796+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:07:26.890907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398506+0000","last_change":"2024-03-17T23:48:11.398107+0000","last_active":"2024-03-17T23:48:12.398506+0000","last_peered":"2024-03-17T23:48:12.398506+0000","last_clean":"2024-03-17T23:48:12.398506+0000","last_became_active":"2024-03-17T23:48:11.397847+0000","last_became_peered":"2024-03-17T23:48:11.397847+0000","last_unstale":"2024-03-17T23:48:12.398506+0000","last_undegraded":"2024-03-17T23:48:12.398506+0000","last_fullsized":"2024-03-17T23:48:12.398506+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:47:20.574371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.383921+0000","last_change":"2024-03-17T23:48:10.395355+0000","last_active":"2024-03-17T23:48:11.383921+0000","last_peered":"2024-03-17T23:48:11.383921+0000","last_clean":"2024-03-17T23:48:11.383921+0000","last_became_active":"2024-03-17T23:48:10.395051+0000","last_became_peered":"2024-03-17T23:48:10.395051+0000","last_unstale":"2024-03-17T23:48:11.383921+0000","last_undegraded":"2024-03-17T23:48:11.383921+0000","last_fullsized":"2024-03-17T23:48:11.383921+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:28:10.271909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397331+0000","last_change":"2024-03-17T23:48:10.387055+0000","last_active":"2024-03-17T23:48:12.397331+0000","last_peered":"2024-03-17T23:48:12.397331+0000","last_clean":"2024-03-17T23:48:12.397331+0000","last_became_active":"2024-03-17T23:48:10.386732+0000","last_became_peered":"2024-03-17T23:48:10.386732+0000","last_unstale":"2024-03-17T23:48:12.397331+0000","last_undegraded":"2024-03-17T23:48:12.397331+0000","last_fullsized":"2024-03-17T23:48:12.397331+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:59:28.149170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.398574+0000","last_change":"2024-03-17T23:48:11.394613+0000","last_active":"2024-03-17T23:48:11.398574+0000","last_peered":"2024-03-17T23:48:11.398574+0000","last_clean":"2024-03-17T23:48:11.398574+0000","last_became_active":"2024-03-17T23:48:11.394206+0000","last_became_peered":"2024-03-17T23:48:11.394206+0000","last_unstale":"2024-03-17T23:48:11.398574+0000","last_undegraded":"2024-03-17T23:48:11.398574+0000","last_fullsized":"2024-03-17T23:48:11.398574+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:40:54.501510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.389970+0000","last_change":"2024-03-17T23:48:10.394514+0000","last_active":"2024-03-17T23:48:11.389970+0000","last_peered":"2024-03-17T23:48:11.389970+0000","last_clean":"2024-03-17T23:48:11.389970+0000","last_became_active":"2024-03-17T23:48:10.394237+0000","last_became_peered":"2024-03-17T23:48:10.394237+0000","last_unstale":"2024-03-17T23:48:11.389970+0000","last_undegraded":"2024-03-17T23:48:11.389970+0000","last_fullsized":"2024-03-17T23:48:11.389970+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:40:53.499120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"39'3","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409672+0000","last_change":"2024-03-17T23:48:10.400644+0000","last_active":"2024-03-17T23:48:11.409672+0000","last_peered":"2024-03-17T23:48:11.409672+0000","last_clean":"2024-03-17T23:48:11.409672+0000","last_became_active":"2024-03-17T23:48:10.400361+0000","last_became_peered":"2024-03-17T23:48:10.400361+0000","last_unstale":"2024-03-17T23:48:11.409672+0000","last_undegraded":"2024-03-17T23:48:11.409672+0000","last_fullsized":"2024-03-17T23:48:11.409672+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:28:10.271909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398068+0000","last_change":"2024-03-17T23:48:11.404493+0000","last_active":"2024-03-17T23:48:12.398068+0000","last_peered":"2024-03-17T23:48:12.398068+0000","last_clean":"2024-03-17T23:48:12.398068+0000","last_became_active":"2024-03-17T23:48:11.401330+0000","last_became_peered":"2024-03-17T23:48:11.401330+0000","last_unstale":"2024-03-17T23:48:12.398068+0000","last_undegraded":"2024-03-17T23:48:12.398068+0000","last_fullsized":"2024-03-17T23:48:12.398068+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:37:11.855078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.385582+0000","last_change":"2024-03-17T23:48:10.394916+0000","last_active":"2024-03-17T23:48:11.385582+0000","last_peered":"2024-03-17T23:48:11.385582+0000","last_clean":"2024-03-17T23:48:11.385582+0000","last_became_active":"2024-03-17T23:48:10.391532+0000","last_became_peered":"2024-03-17T23:48:10.391532+0000","last_unstale":"2024-03-17T23:48:11.385582+0000","last_undegraded":"2024-03-17T23:48:11.385582+0000","last_fullsized":"2024-03-17T23:48:11.385582+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:44.407483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409135+0000","last_change":"2024-03-17T23:48:11.407145+0000","last_active":"2024-03-17T23:48:11.409135+0000","last_peered":"2024-03-17T23:48:11.409135+0000","last_clean":"2024-03-17T23:48:11.409135+0000","last_became_active":"2024-03-17T23:48:11.406851+0000","last_became_peered":"2024-03-17T23:48:11.406851+0000","last_unstale":"2024-03-17T23:48:11.409135+0000","last_undegraded":"2024-03-17T23:48:11.409135+0000","last_fullsized":"2024-03-17T23:48:11.409135+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:59:29.151560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397440+0000","last_change":"2024-03-17T23:48:10.388069+0000","last_active":"2024-03-17T23:48:12.397440+0000","last_peered":"2024-03-17T23:48:12.397440+0000","last_clean":"2024-03-17T23:48:12.397440+0000","last_became_active":"2024-03-17T23:48:10.387765+0000","last_became_peered":"2024-03-17T23:48:10.387765+0000","last_unstale":"2024-03-17T23:48:12.397440+0000","last_undegraded":"2024-03-17T23:48:12.397440+0000","last_fullsized":"2024-03-17T23:48:12.397440+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:15.116223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"39'8","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.400672+0000","last_change":"2024-03-17T23:48:10.397744+0000","last_active":"2024-03-17T23:48:11.400672+0000","last_peered":"2024-03-17T23:48:11.400672+0000","last_clean":"2024-03-17T23:48:11.400672+0000","last_became_active":"2024-03-17T23:48:10.397467+0000","last_became_peered":"2024-03-17T23:48:10.397467+0000","last_unstale":"2024-03-17T23:48:11.400672+0000","last_undegraded":"2024-03-17T23:48:11.400672+0000","last_fullsized":"2024-03-17T23:48:11.400672+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:12:23.322115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"39'1","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409062+0000","last_change":"2024-03-17T23:48:10.393694+0000","last_active":"2024-03-17T23:48:11.409062+0000","last_peered":"2024-03-17T23:48:11.409062+0000","last_clean":"2024-03-17T23:48:11.409062+0000","last_became_active":"2024-03-17T23:48:10.393418+0000","last_became_peered":"2024-03-17T23:48:10.393418+0000","last_unstale":"2024-03-17T23:48:11.409062+0000","last_undegraded":"2024-03-17T23:48:11.409062+0000","last_fullsized":"2024-03-17T23:48:11.409062+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:42:42.774025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408740+0000","last_change":"2024-03-17T23:48:11.400412+0000","last_active":"2024-03-17T23:48:11.408740+0000","last_peered":"2024-03-17T23:48:11.408740+0000","last_clean":"2024-03-17T23:48:11.408740+0000","last_became_active":"2024-03-17T23:48:11.400006+0000","last_became_peered":"2024-03-17T23:48:11.400006+0000","last_unstale":"2024-03-17T23:48:11.408740+0000","last_undegraded":"2024-03-17T23:48:11.408740+0000","last_fullsized":"2024-03-17T23:48:11.408740+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:59:29.151560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"39'2","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398100+0000","last_change":"2024-03-17T23:48:10.391802+0000","last_active":"2024-03-17T23:48:12.398100+0000","last_peered":"2024-03-17T23:48:12.398100+0000","last_clean":"2024-03-17T23:48:12.398100+0000","last_became_active":"2024-03-17T23:48:10.391464+0000","last_became_peered":"2024-03-17T23:48:10.391464+0000","last_unstale":"2024-03-17T23:48:12.398100+0000","last_undegraded":"2024-03-17T23:48:12.398100+0000","last_fullsized":"2024-03-17T23:48:12.398100+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:44:41.528247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397957+0000","last_change":"2024-03-17T23:48:11.399808+0000","last_active":"2024-03-17T23:48:12.397957+0000","last_peered":"2024-03-17T23:48:12.397957+0000","last_clean":"2024-03-17T23:48:12.397957+0000","last_became_active":"2024-03-17T23:48:11.399460+0000","last_became_peered":"2024-03-17T23:48:11.399460+0000","last_unstale":"2024-03-17T23:48:12.397957+0000","last_undegraded":"2024-03-17T23:48:12.397957+0000","last_fullsized":"2024-03-17T23:48:12.397957+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:16:59.276395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.384478+0000","last_change":"2024-03-17T23:48:10.396820+0000","last_active":"2024-03-17T23:48:11.384478+0000","last_peered":"2024-03-17T23:48:11.384478+0000","last_clean":"2024-03-17T23:48:11.384478+0000","last_became_active":"2024-03-17T23:48:10.396549+0000","last_became_peered":"2024-03-17T23:48:10.396549+0000","last_unstale":"2024-03-17T23:48:11.384478+0000","last_undegraded":"2024-03-17T23:48:11.384478+0000","last_fullsized":"2024-03-17T23:48:11.384478+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:16.355950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408966+0000","last_change":"2024-03-17T23:48:11.403660+0000","last_active":"2024-03-17T23:48:11.408966+0000","last_peered":"2024-03-17T23:48:11.408966+0000","last_clean":"2024-03-17T23:48:11.408966+0000","last_became_active":"2024-03-17T23:48:11.403385+0000","last_became_peered":"2024-03-17T23:48:11.403385+0000","last_unstale":"2024-03-17T23:48:11.408966+0000","last_undegraded":"2024-03-17T23:48:11.408966+0000","last_fullsized":"2024-03-17T23:48:11.408966+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:42:43.776415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409211+0000","last_change":"2024-03-17T23:48:11.404518+0000","last_active":"2024-03-17T23:48:11.409211+0000","last_peered":"2024-03-17T23:48:11.409211+0000","last_clean":"2024-03-17T23:48:11.409211+0000","last_became_active":"2024-03-17T23:48:11.404261+0000","last_became_peered":"2024-03-17T23:48:11.404261+0000","last_unstale":"2024-03-17T23:48:11.409211+0000","last_undegraded":"2024-03-17T23:48:11.409211+0000","last_fullsized":"2024-03-17T23:48:11.409211+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:17:53.610029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.385531+0000","last_change":"2024-03-17T23:48:10.397213+0000","last_active":"2024-03-17T23:48:11.385531+0000","last_peered":"2024-03-17T23:48:11.385531+0000","last_clean":"2024-03-17T23:48:11.385531+0000","last_became_active":"2024-03-17T23:48:10.396913+0000","last_became_peered":"2024-03-17T23:48:10.396913+0000","last_unstale":"2024-03-17T23:48:11.385531+0000","last_undegraded":"2024-03-17T23:48:11.385531+0000","last_fullsized":"2024-03-17T23:48:11.385531+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:31:12.048530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.397234+0000","last_change":"2024-03-17T23:48:11.396445+0000","last_active":"2024-03-17T23:48:11.397234+0000","last_peered":"2024-03-17T23:48:11.397234+0000","last_clean":"2024-03-17T23:48:11.397234+0000","last_became_active":"2024-03-17T23:48:11.396189+0000","last_became_peered":"2024-03-17T23:48:11.396189+0000","last_unstale":"2024-03-17T23:48:11.397234+0000","last_undegraded":"2024-03-17T23:48:11.397234+0000","last_fullsized":"2024-03-17T23:48:11.397234+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:10.929912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"39'2","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.406980+0000","last_change":"2024-03-17T23:48:10.400812+0000","last_active":"2024-03-17T23:48:11.406980+0000","last_peered":"2024-03-17T23:48:11.406980+0000","last_clean":"2024-03-17T23:48:11.406980+0000","last_became_active":"2024-03-17T23:48:10.400479+0000","last_became_peered":"2024-03-17T23:48:10.400479+0000","last_unstale":"2024-03-17T23:48:11.406980+0000","last_undegraded":"2024-03-17T23:48:11.406980+0000","last_fullsized":"2024-03-17T23:48:11.406980+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:09.927522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408459+0000","last_change":"2024-03-17T23:48:11.403785+0000","last_active":"2024-03-17T23:48:11.408459+0000","last_peered":"2024-03-17T23:48:11.408459+0000","last_clean":"2024-03-17T23:48:11.408459+0000","last_became_active":"2024-03-17T23:48:11.402301+0000","last_became_peered":"2024-03-17T23:48:11.402301+0000","last_unstale":"2024-03-17T23:48:11.408459+0000","last_undegraded":"2024-03-17T23:48:11.408459+0000","last_fullsized":"2024-03-17T23:48:11.408459+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:31.101445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:10.371109+0000","last_change":"2024-03-17T23:48:10.371109+0000","last_active":"2024-03-17T23:48:10.371109+0000","last_peered":"2024-03-17T23:48:10.371109+0000","last_clean":"2024-03-17T23:48:10.371109+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:10.371109+0000","last_undegraded":"2024-03-17T23:48:10.371109+0000","last_fullsized":"2024-03-17T23:48:10.371109+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.397727+0000","last_change":"2024-03-17T23:48:11.397059+0000","last_active":"2024-03-17T23:48:11.397727+0000","last_peered":"2024-03-17T23:48:11.397727+0000","last_clean":"2024-03-17T23:48:11.397727+0000","last_became_active":"2024-03-17T23:48:11.396724+0000","last_became_peered":"2024-03-17T23:48:11.396724+0000","last_unstale":"2024-03-17T23:48:11.397727+0000","last_undegraded":"2024-03-17T23:48:11.397727+0000","last_fullsized":"2024-03-17T23:48:11.397727+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:31.101445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398165+0000","last_change":"2024-03-17T23:48:10.384305+0000","last_active":"2024-03-17T23:48:12.398165+0000","last_peered":"2024-03-17T23:48:12.398165+0000","last_clean":"2024-03-17T23:48:12.398165+0000","last_became_active":"2024-03-17T23:48:10.384034+0000","last_became_peered":"2024-03-17T23:48:10.384034+0000","last_unstale":"2024-03-17T23:48:12.398165+0000","last_undegraded":"2024-03-17T23:48:12.398165+0000","last_fullsized":"2024-03-17T23:48:12.398165+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:32:37.500442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398088+0000","last_change":"2024-03-17T23:48:11.394167+0000","last_active":"2024-03-17T23:48:12.398088+0000","last_peered":"2024-03-17T23:48:12.398088+0000","last_clean":"2024-03-17T23:48:12.398088+0000","last_became_active":"2024-03-17T23:48:11.392337+0000","last_became_peered":"2024-03-17T23:48:11.392337+0000","last_unstale":"2024-03-17T23:48:12.398088+0000","last_undegraded":"2024-03-17T23:48:12.398088+0000","last_fullsized":"2024-03-17T23:48:12.398088+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:04:14.826887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397422+0000","last_change":"2024-03-17T23:48:10.391429+0000","last_active":"2024-03-17T23:48:12.397422+0000","last_peered":"2024-03-17T23:48:12.397422+0000","last_clean":"2024-03-17T23:48:12.397422+0000","last_became_active":"2024-03-17T23:48:10.390974+0000","last_became_peered":"2024-03-17T23:48:10.390974+0000","last_unstale":"2024-03-17T23:48:12.397422+0000","last_undegraded":"2024-03-17T23:48:12.397422+0000","last_fullsized":"2024-03-17T23:48:12.397422+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:04.026997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.399296+0000","last_change":"2024-03-17T23:48:11.397803+0000","last_active":"2024-03-17T23:48:11.399296+0000","last_peered":"2024-03-17T23:48:11.399296+0000","last_clean":"2024-03-17T23:48:11.399296+0000","last_became_active":"2024-03-17T23:48:11.397541+0000","last_became_peered":"2024-03-17T23:48:11.397541+0000","last_unstale":"2024-03-17T23:48:11.399296+0000","last_undegraded":"2024-03-17T23:48:11.399296+0000","last_fullsized":"2024-03-17T23:48:11.399296+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:17.358341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:48:09.368718+0000","last_change":"2024-03-17T23:48:09.368718+0000","last_active":"2024-03-17T23:48:09.368718+0000","last_peered":"2024-03-17T23:48:09.368718+0000","last_clean":"2024-03-17T23:48:09.368718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:48:09.368718+0000","last_undegraded":"2024-03-17T23:48:09.368718+0000","last_fullsized":"2024-03-17T23:48:09.368718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409115+0000","last_change":"2024-03-17T23:48:11.403123+0000","last_active":"2024-03-17T23:48:11.409115+0000","last_peered":"2024-03-17T23:48:11.409115+0000","last_clean":"2024-03-17T23:48:11.409115+0000","last_became_active":"2024-03-17T23:48:11.402851+0000","last_became_peered":"2024-03-17T23:48:11.402851+0000","last_unstale":"2024-03-17T23:48:11.409115+0000","last_undegraded":"2024-03-17T23:48:11.409115+0000","last_fullsized":"2024-03-17T23:48:11.409115+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:44:42.530638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.384095+0000","last_change":"2024-03-17T23:48:10.398461+0000","last_active":"2024-03-17T23:48:11.384095+0000","last_peered":"2024-03-17T23:48:11.384095+0000","last_clean":"2024-03-17T23:48:11.384095+0000","last_became_active":"2024-03-17T23:48:10.398188+0000","last_became_peered":"2024-03-17T23:48:10.398188+0000","last_unstale":"2024-03-17T23:48:11.384095+0000","last_undegraded":"2024-03-17T23:48:11.384095+0000","last_fullsized":"2024-03-17T23:48:11.384095+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:06:33.058080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":204,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.385340+0000","last_change":"2024-03-17T23:46:44.768668+0000","last_active":"2024-03-17T23:48:11.385340+0000","last_peered":"2024-03-17T23:48:11.385340+0000","last_clean":"2024-03-17T23:48:11.385340+0000","last_became_active":"2024-03-17T23:46:44.767826+0000","last_became_peered":"2024-03-17T23:46:44.767826+0000","last_unstale":"2024-03-17T23:48:11.385340+0000","last_undegraded":"2024-03-17T23:48:11.385340+0000","last_fullsized":"2024-03-17T23:48:11.385340+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_clean_scrub_stamp":"2024-03-17T23:46:43.744878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:54:22.286155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":80,"acting":80,"num_store_stats":5},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":694,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":40,"acting":40,"num_store_stats":5},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27424,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":409600,"data_stored":174269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27420,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":409600,"data_stored":174269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542165,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28008,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343832,"statfs":{"total":96636764160,"available":96608083968,"internally_reserved":0,"allocated":1003520,"data_stored":764637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32212,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339628,"statfs":{"total":96636764160,"available":96603779072,"internally_reserved":0,"allocated":1056768,"data_stored":786362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31636,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340204,"statfs":{"total":96636764160,"available":96604368896,"internally_reserved":0,"allocated":462848,"data_stored":201586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31624,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":454656,"data_stored":194424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31624,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":454656,"data_stored":194916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":450560,"data_stored":194402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-17T23:48:15.263 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3f is not active+clean 2024-03-17T23:48:15.264 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:10.371109+0000', 'last_change': '2024-03-17T23:48:10.371109+0000', 'last_active': '2024-03-17T23:48:10.371109+0000', 'last_peered': '2024-03-17T23:48:10.371109+0000', 'last_clean': '2024-03-17T23:48:10.371109+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:10.371109+0000', 'last_undegraded': '2024-03-17T23:48:10.371109+0000', 'last_fullsized': '2024-03-17T23:48:10.371109+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.264 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3e is not active+clean 2024-03-17T23:48:15.264 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:10.371109+0000', 'last_change': '2024-03-17T23:48:10.371109+0000', 'last_active': '2024-03-17T23:48:10.371109+0000', 'last_peered': '2024-03-17T23:48:10.371109+0000', 'last_clean': '2024-03-17T23:48:10.371109+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:10.371109+0000', 'last_undegraded': '2024-03-17T23:48:10.371109+0000', 'last_fullsized': '2024-03-17T23:48:10.371109+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.264 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3d is not active+clean 2024-03-17T23:48:15.264 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:10.371109+0000', 'last_change': '2024-03-17T23:48:10.371109+0000', 'last_active': '2024-03-17T23:48:10.371109+0000', 'last_peered': '2024-03-17T23:48:10.371109+0000', 'last_clean': '2024-03-17T23:48:10.371109+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:10.371109+0000', 'last_undegraded': '2024-03-17T23:48:10.371109+0000', 'last_fullsized': '2024-03-17T23:48:10.371109+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.264 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3c is not active+clean 2024-03-17T23:48:15.264 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:10.371109+0000', 'last_change': '2024-03-17T23:48:10.371109+0000', 'last_active': '2024-03-17T23:48:10.371109+0000', 'last_peered': '2024-03-17T23:48:10.371109+0000', 'last_clean': '2024-03-17T23:48:10.371109+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:10.371109+0000', 'last_undegraded': '2024-03-17T23:48:10.371109+0000', 'last_fullsized': '2024-03-17T23:48:10.371109+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.264 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3b is not active+clean 2024-03-17T23:48:15.264 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:10.371109+0000', 'last_change': '2024-03-17T23:48:10.371109+0000', 'last_active': '2024-03-17T23:48:10.371109+0000', 'last_peered': '2024-03-17T23:48:10.371109+0000', 'last_clean': '2024-03-17T23:48:10.371109+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:10.371109+0000', 'last_undegraded': '2024-03-17T23:48:10.371109+0000', 'last_fullsized': '2024-03-17T23:48:10.371109+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.264 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.38 is not active+clean 2024-03-17T23:48:15.265 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:10.371109+0000', 'last_change': '2024-03-17T23:48:10.371109+0000', 'last_active': '2024-03-17T23:48:10.371109+0000', 'last_peered': '2024-03-17T23:48:10.371109+0000', 'last_clean': '2024-03-17T23:48:10.371109+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:10.371109+0000', 'last_undegraded': '2024-03-17T23:48:10.371109+0000', 'last_fullsized': '2024-03-17T23:48:10.371109+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.266 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.37 is not active+clean 2024-03-17T23:48:15.266 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:10.371109+0000', 'last_change': '2024-03-17T23:48:10.371109+0000', 'last_active': '2024-03-17T23:48:10.371109+0000', 'last_peered': '2024-03-17T23:48:10.371109+0000', 'last_clean': '2024-03-17T23:48:10.371109+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:10.371109+0000', 'last_undegraded': '2024-03-17T23:48:10.371109+0000', 'last_fullsized': '2024-03-17T23:48:10.371109+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.266 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.34 is not active+clean 2024-03-17T23:48:15.267 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:10.371109+0000', 'last_change': '2024-03-17T23:48:10.371109+0000', 'last_active': '2024-03-17T23:48:10.371109+0000', 'last_peered': '2024-03-17T23:48:10.371109+0000', 'last_clean': '2024-03-17T23:48:10.371109+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:10.371109+0000', 'last_undegraded': '2024-03-17T23:48:10.371109+0000', 'last_fullsized': '2024-03-17T23:48:10.371109+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.267 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2d is not active+clean 2024-03-17T23:48:15.267 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:10.371109+0000', 'last_change': '2024-03-17T23:48:10.371109+0000', 'last_active': '2024-03-17T23:48:10.371109+0000', 'last_peered': '2024-03-17T23:48:10.371109+0000', 'last_clean': '2024-03-17T23:48:10.371109+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:10.371109+0000', 'last_undegraded': '2024-03-17T23:48:10.371109+0000', 'last_fullsized': '2024-03-17T23:48:10.371109+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.267 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.28 is not active+clean 2024-03-17T23:48:15.267 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:10.371109+0000', 'last_change': '2024-03-17T23:48:10.371109+0000', 'last_active': '2024-03-17T23:48:10.371109+0000', 'last_peered': '2024-03-17T23:48:10.371109+0000', 'last_clean': '2024-03-17T23:48:10.371109+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:10.371109+0000', 'last_undegraded': '2024-03-17T23:48:10.371109+0000', 'last_fullsized': '2024-03-17T23:48:10.371109+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.267 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1e is not active+clean 2024-03-17T23:48:15.267 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:09.368718+0000', 'last_change': '2024-03-17T23:48:09.368718+0000', 'last_active': '2024-03-17T23:48:09.368718+0000', 'last_peered': '2024-03-17T23:48:09.368718+0000', 'last_clean': '2024-03-17T23:48:09.368718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:09.368718+0000', 'last_undegraded': '2024-03-17T23:48:09.368718+0000', 'last_fullsized': '2024-03-17T23:48:09.368718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.267 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1f is not active+clean 2024-03-17T23:48:15.267 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:10.371109+0000', 'last_change': '2024-03-17T23:48:10.371109+0000', 'last_active': '2024-03-17T23:48:10.371109+0000', 'last_peered': '2024-03-17T23:48:10.371109+0000', 'last_clean': '2024-03-17T23:48:10.371109+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:10.371109+0000', 'last_undegraded': '2024-03-17T23:48:10.371109+0000', 'last_fullsized': '2024-03-17T23:48:10.371109+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.267 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.18 is not active+clean 2024-03-17T23:48:15.267 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:10.371109+0000', 'last_change': '2024-03-17T23:48:10.371109+0000', 'last_active': '2024-03-17T23:48:10.371109+0000', 'last_peered': '2024-03-17T23:48:10.371109+0000', 'last_clean': '2024-03-17T23:48:10.371109+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:10.371109+0000', 'last_undegraded': '2024-03-17T23:48:10.371109+0000', 'last_fullsized': '2024-03-17T23:48:10.371109+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.268 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1b is not active+clean 2024-03-17T23:48:15.268 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:10.371109+0000', 'last_change': '2024-03-17T23:48:10.371109+0000', 'last_active': '2024-03-17T23:48:10.371109+0000', 'last_peered': '2024-03-17T23:48:10.371109+0000', 'last_clean': '2024-03-17T23:48:10.371109+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:10.371109+0000', 'last_undegraded': '2024-03-17T23:48:10.371109+0000', 'last_fullsized': '2024-03-17T23:48:10.371109+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.268 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.14 is not active+clean 2024-03-17T23:48:15.268 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:10.371109+0000', 'last_change': '2024-03-17T23:48:10.371109+0000', 'last_active': '2024-03-17T23:48:10.371109+0000', 'last_peered': '2024-03-17T23:48:10.371109+0000', 'last_clean': '2024-03-17T23:48:10.371109+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:10.371109+0000', 'last_undegraded': '2024-03-17T23:48:10.371109+0000', 'last_fullsized': '2024-03-17T23:48:10.371109+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.268 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.17 is not active+clean 2024-03-17T23:48:15.268 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:09.368718+0000', 'last_change': '2024-03-17T23:48:09.368718+0000', 'last_active': '2024-03-17T23:48:09.368718+0000', 'last_peered': '2024-03-17T23:48:09.368718+0000', 'last_clean': '2024-03-17T23:48:09.368718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:09.368718+0000', 'last_undegraded': '2024-03-17T23:48:09.368718+0000', 'last_fullsized': '2024-03-17T23:48:09.368718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.268 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.16 is not active+clean 2024-03-17T23:48:15.268 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:10.371109+0000', 'last_change': '2024-03-17T23:48:10.371109+0000', 'last_active': '2024-03-17T23:48:10.371109+0000', 'last_peered': '2024-03-17T23:48:10.371109+0000', 'last_clean': '2024-03-17T23:48:10.371109+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:10.371109+0000', 'last_undegraded': '2024-03-17T23:48:10.371109+0000', 'last_fullsized': '2024-03-17T23:48:10.371109+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.268 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.14 is not active+clean 2024-03-17T23:48:15.268 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:09.368718+0000', 'last_change': '2024-03-17T23:48:09.368718+0000', 'last_active': '2024-03-17T23:48:09.368718+0000', 'last_peered': '2024-03-17T23:48:09.368718+0000', 'last_clean': '2024-03-17T23:48:09.368718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:09.368718+0000', 'last_undegraded': '2024-03-17T23:48:09.368718+0000', 'last_fullsized': '2024-03-17T23:48:09.368718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.268 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.10 is not active+clean 2024-03-17T23:48:15.268 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:10.371109+0000', 'last_change': '2024-03-17T23:48:10.371109+0000', 'last_active': '2024-03-17T23:48:10.371109+0000', 'last_peered': '2024-03-17T23:48:10.371109+0000', 'last_clean': '2024-03-17T23:48:10.371109+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:10.371109+0000', 'last_undegraded': '2024-03-17T23:48:10.371109+0000', 'last_fullsized': '2024-03-17T23:48:10.371109+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.269 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.13 is not active+clean 2024-03-17T23:48:15.269 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:09.368718+0000', 'last_change': '2024-03-17T23:48:09.368718+0000', 'last_active': '2024-03-17T23:48:09.368718+0000', 'last_peered': '2024-03-17T23:48:09.368718+0000', 'last_clean': '2024-03-17T23:48:09.368718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:09.368718+0000', 'last_undegraded': '2024-03-17T23:48:09.368718+0000', 'last_fullsized': '2024-03-17T23:48:09.368718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.269 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.11 is not active+clean 2024-03-17T23:48:15.269 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:10.371109+0000', 'last_change': '2024-03-17T23:48:10.371109+0000', 'last_active': '2024-03-17T23:48:10.371109+0000', 'last_peered': '2024-03-17T23:48:10.371109+0000', 'last_clean': '2024-03-17T23:48:10.371109+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:10.371109+0000', 'last_undegraded': '2024-03-17T23:48:10.371109+0000', 'last_fullsized': '2024-03-17T23:48:10.371109+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.269 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.13 is not active+clean 2024-03-17T23:48:15.269 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:10.371109+0000', 'last_change': '2024-03-17T23:48:10.371109+0000', 'last_active': '2024-03-17T23:48:10.371109+0000', 'last_peered': '2024-03-17T23:48:10.371109+0000', 'last_clean': '2024-03-17T23:48:10.371109+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:10.371109+0000', 'last_undegraded': '2024-03-17T23:48:10.371109+0000', 'last_fullsized': '2024-03-17T23:48:10.371109+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.269 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.c is not active+clean 2024-03-17T23:48:15.269 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:10.371109+0000', 'last_change': '2024-03-17T23:48:10.371109+0000', 'last_active': '2024-03-17T23:48:10.371109+0000', 'last_peered': '2024-03-17T23:48:10.371109+0000', 'last_clean': '2024-03-17T23:48:10.371109+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:10.371109+0000', 'last_undegraded': '2024-03-17T23:48:10.371109+0000', 'last_fullsized': '2024-03-17T23:48:10.371109+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.269 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.d is not active+clean 2024-03-17T23:48:15.269 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:09.368718+0000', 'last_change': '2024-03-17T23:48:09.368718+0000', 'last_active': '2024-03-17T23:48:09.368718+0000', 'last_peered': '2024-03-17T23:48:09.368718+0000', 'last_clean': '2024-03-17T23:48:09.368718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:09.368718+0000', 'last_undegraded': '2024-03-17T23:48:09.368718+0000', 'last_fullsized': '2024-03-17T23:48:09.368718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.270 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.f is not active+clean 2024-03-17T23:48:15.270 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:10.371109+0000', 'last_change': '2024-03-17T23:48:10.371109+0000', 'last_active': '2024-03-17T23:48:10.371109+0000', 'last_peered': '2024-03-17T23:48:10.371109+0000', 'last_clean': '2024-03-17T23:48:10.371109+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:10.371109+0000', 'last_undegraded': '2024-03-17T23:48:10.371109+0000', 'last_fullsized': '2024-03-17T23:48:10.371109+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.270 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.b is not active+clean 2024-03-17T23:48:15.270 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:09.368718+0000', 'last_change': '2024-03-17T23:48:09.368718+0000', 'last_active': '2024-03-17T23:48:09.368718+0000', 'last_peered': '2024-03-17T23:48:09.368718+0000', 'last_clean': '2024-03-17T23:48:09.368718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:09.368718+0000', 'last_undegraded': '2024-03-17T23:48:09.368718+0000', 'last_fullsized': '2024-03-17T23:48:09.368718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.270 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.9 is not active+clean 2024-03-17T23:48:15.270 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:10.371109+0000', 'last_change': '2024-03-17T23:48:10.371109+0000', 'last_active': '2024-03-17T23:48:10.371109+0000', 'last_peered': '2024-03-17T23:48:10.371109+0000', 'last_clean': '2024-03-17T23:48:10.371109+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:10.371109+0000', 'last_undegraded': '2024-03-17T23:48:10.371109+0000', 'last_fullsized': '2024-03-17T23:48:10.371109+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.270 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.a is not active+clean 2024-03-17T23:48:15.270 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:09.368718+0000', 'last_change': '2024-03-17T23:48:09.368718+0000', 'last_active': '2024-03-17T23:48:09.368718+0000', 'last_peered': '2024-03-17T23:48:09.368718+0000', 'last_clean': '2024-03-17T23:48:09.368718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:09.368718+0000', 'last_undegraded': '2024-03-17T23:48:09.368718+0000', 'last_fullsized': '2024-03-17T23:48:09.368718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.270 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.b is not active+clean 2024-03-17T23:48:15.270 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:10.371109+0000', 'last_change': '2024-03-17T23:48:10.371109+0000', 'last_active': '2024-03-17T23:48:10.371109+0000', 'last_peered': '2024-03-17T23:48:10.371109+0000', 'last_clean': '2024-03-17T23:48:10.371109+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:10.371109+0000', 'last_undegraded': '2024-03-17T23:48:10.371109+0000', 'last_fullsized': '2024-03-17T23:48:10.371109+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.270 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.8 is not active+clean 2024-03-17T23:48:15.271 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:09.368718+0000', 'last_change': '2024-03-17T23:48:09.368718+0000', 'last_active': '2024-03-17T23:48:09.368718+0000', 'last_peered': '2024-03-17T23:48:09.368718+0000', 'last_clean': '2024-03-17T23:48:09.368718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:09.368718+0000', 'last_undegraded': '2024-03-17T23:48:09.368718+0000', 'last_fullsized': '2024-03-17T23:48:09.368718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.271 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.6 is not active+clean 2024-03-17T23:48:15.271 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:09.368718+0000', 'last_change': '2024-03-17T23:48:09.368718+0000', 'last_active': '2024-03-17T23:48:09.368718+0000', 'last_peered': '2024-03-17T23:48:09.368718+0000', 'last_clean': '2024-03-17T23:48:09.368718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:09.368718+0000', 'last_undegraded': '2024-03-17T23:48:09.368718+0000', 'last_fullsized': '2024-03-17T23:48:09.368718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.271 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.6 is not active+clean 2024-03-17T23:48:15.271 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:10.371109+0000', 'last_change': '2024-03-17T23:48:10.371109+0000', 'last_active': '2024-03-17T23:48:10.371109+0000', 'last_peered': '2024-03-17T23:48:10.371109+0000', 'last_clean': '2024-03-17T23:48:10.371109+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:10.371109+0000', 'last_undegraded': '2024-03-17T23:48:10.371109+0000', 'last_fullsized': '2024-03-17T23:48:10.371109+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.271 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.5 is not active+clean 2024-03-17T23:48:15.271 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:09.368718+0000', 'last_change': '2024-03-17T23:48:09.368718+0000', 'last_active': '2024-03-17T23:48:09.368718+0000', 'last_peered': '2024-03-17T23:48:09.368718+0000', 'last_clean': '2024-03-17T23:48:09.368718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:09.368718+0000', 'last_undegraded': '2024-03-17T23:48:09.368718+0000', 'last_fullsized': '2024-03-17T23:48:09.368718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.271 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.0 is not active+clean 2024-03-17T23:48:15.271 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:10.371109+0000', 'last_change': '2024-03-17T23:48:10.371109+0000', 'last_active': '2024-03-17T23:48:10.371109+0000', 'last_peered': '2024-03-17T23:48:10.371109+0000', 'last_clean': '2024-03-17T23:48:10.371109+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:10.371109+0000', 'last_undegraded': '2024-03-17T23:48:10.371109+0000', 'last_fullsized': '2024-03-17T23:48:10.371109+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:10.371109+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.271 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3 is not active+clean 2024-03-17T23:48:15.271 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:09.368718+0000', 'last_change': '2024-03-17T23:48:09.368718+0000', 'last_active': '2024-03-17T23:48:09.368718+0000', 'last_peered': '2024-03-17T23:48:09.368718+0000', 'last_clean': '2024-03-17T23:48:09.368718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:09.368718+0000', 'last_undegraded': '2024-03-17T23:48:09.368718+0000', 'last_fullsized': '2024-03-17T23:48:09.368718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.272 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.0 is not active+clean 2024-03-17T23:48:15.272 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:48:09.368718+0000', 'last_change': '2024-03-17T23:48:09.368718+0000', 'last_active': '2024-03-17T23:48:09.368718+0000', 'last_peered': '2024-03-17T23:48:09.368718+0000', 'last_clean': '2024-03-17T23:48:09.368718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:48:09.368718+0000', 'last_undegraded': '2024-03-17T23:48:09.368718+0000', 'last_fullsized': '2024-03-17T23:48:09.368718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'last_clean_scrub_stamp': '2024-03-17T23:48:09.368718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:48:15.272 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-17T23:48:15.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.412+0000 7efd408ac640 1 Processor -- start 2024-03-17T23:48:15.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.412+0000 7efd408ac640 1 -- start start 2024-03-17T23:48:15.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.414+0000 7efd408ac640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7efd3c0b0310 0x7efd3c15e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:15.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.414+0000 7efd408ac640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd3c15ec30 con 0x7efd3c06c920 2024-03-17T23:48:15.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.414+0000 7efd408ac640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd3c0b3fc0 con 0x7efd3c15a4b0 2024-03-17T23:48:15.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.414+0000 7efd408ac640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7efd3c099e80 con 0x7efd3c0b0310 2024-03-17T23:48:15.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.414+0000 7efd3ad9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7efd3c06c920 0x7efd3c06b3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37626/0 (socket says 172.21.15.150:37626) 2024-03-17T23:48:15.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.414+0000 7efd3ad9d640 1 -- 172.21.15.150:0/4115340457 learned_addr learned my addr 172.21.15.150:0/4115340457 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:15.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.414+0000 7efd39d9b640 1 --2- 172.21.15.150:0/4115340457 >> v2:172.21.15.157:3300/0 conn(0x7efd3c0b0310 0x7efd3c15e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:15.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.414+0000 7efd3959a640 1 -- 172.21.15.150:0/4115340457 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4111216625 0 0) 0x7efd34001490 con 0x7efd3c06c920 2024-03-17T23:48:15.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.414+0000 7efd3959a640 1 -- 172.21.15.150:0/4115340457 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd24004630 con 0x7efd3c06c920 2024-03-17T23:48:15.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.414+0000 7efd3959a640 1 -- 172.21.15.150:0/4115340457 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1273938653 0 0) 0x7efd24004630 con 0x7efd3c06c920 2024-03-17T23:48:15.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.414+0000 7efd3959a640 1 -- 172.21.15.150:0/4115340457 >> v2:172.21.15.157:3300/0 conn(0x7efd3c0b0310 msgr2=0x7efd3c15e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:15.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.414+0000 7efd3959a640 1 --2- 172.21.15.150:0/4115340457 >> v2:172.21.15.157:3300/0 conn(0x7efd3c0b0310 0x7efd3c15e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.414+0000 7efd3959a640 1 -- 172.21.15.150:0/4115340457 >> v1:172.21.15.157:6789/0 conn(0x7efd3c15a4b0 legacy=0x7efd3c15dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:15.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.414+0000 7efd3959a640 1 -- 172.21.15.150:0/4115340457 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd3c15a890 con 0x7efd3c06c920 2024-03-17T23:48:15.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.414+0000 7efd3959a640 1 -- 172.21.15.150:0/4115340457 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7efd34001bc0 con 0x7efd3c06c920 2024-03-17T23:48:15.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.414+0000 7efd3959a640 1 -- 172.21.15.150:0/4115340457 >> v1:172.21.15.150:6789/0 conn(0x7efd3c06c920 legacy=0x7efd3c06b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:15.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.414+0000 7efd3959a640 1 --2- 172.21.15.150:0/4115340457 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd24004ac0 0x7efd24004ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:15.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.414+0000 7efd3959a640 1 --2- 172.21.15.150:0/4115340457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd24005910 0x7efd24005d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:15.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.414+0000 7efd3959a640 1 --2- 172.21.15.150:0/4115340457 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd240062d0 0x7efd2400cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:15.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.414+0000 7efd3959a640 1 -- 172.21.15.150:0/4115340457 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7efd3c099e80 con 0x7efd24005910 2024-03-17T23:48:15.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.414+0000 7efd3959a640 1 -- 172.21.15.150:0/4115340457 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7efd2400d170 con 0x7efd24004ac0 2024-03-17T23:48:15.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.414+0000 7efd3959a640 1 -- 172.21.15.150:0/4115340457 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efd2400d370 con 0x7efd240062d0 2024-03-17T23:48:15.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.415+0000 7efd39d9b640 1 --2- 172.21.15.150:0/4115340457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd24005910 0x7efd24005d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:15.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.415+0000 7efd3a59c640 1 --2- 172.21.15.150:0/4115340457 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd24004ac0 0x7efd24004ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:15.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.415+0000 7efd3ad9d640 1 --2- 172.21.15.150:0/4115340457 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd240062d0 0x7efd2400cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:15.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.415+0000 7efd39d9b640 1 -- 172.21.15.150:0/4115340457 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd240062d0 msgr2=0x7efd2400cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:15.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.415+0000 7efd39d9b640 1 --2- 172.21.15.150:0/4115340457 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd240062d0 0x7efd2400cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.415+0000 7efd39d9b640 1 -- 172.21.15.150:0/4115340457 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd24004ac0 msgr2=0x7efd24004ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:15.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.415+0000 7efd39d9b640 1 --2- 172.21.15.150:0/4115340457 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd24004ac0 0x7efd24004ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.415+0000 7efd39d9b640 1 -- 172.21.15.150:0/4115340457 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd24004630 con 0x7efd24005910 2024-03-17T23:48:15.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.415+0000 7efd3a59c640 1 --2- 172.21.15.150:0/4115340457 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd24004ac0 0x7efd24004ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:15.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.415+0000 7efd3ad9d640 1 --2- 172.21.15.150:0/4115340457 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd240062d0 0x7efd2400cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:15.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.415+0000 7efd3ad9d640 1 -- 172.21.15.150:0/4115340457 reap_dead start 2024-03-17T23:48:15.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.416+0000 7efd39d9b640 1 --2- 172.21.15.150:0/4115340457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd24005910 0x7efd24005d90 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7efd30009ac0 tx=0x7efd24006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd9169fb35482b6f server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:15.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.416+0000 7efd3959a640 1 -- 172.21.15.150:0/4115340457 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd30039070 con 0x7efd24005910 2024-03-17T23:48:15.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.416+0000 7efd3959a640 1 -- 172.21.15.150:0/4115340457 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd3002c6b0 con 0x7efd24005910 2024-03-17T23:48:15.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.416+0000 7efd3959a640 1 -- 172.21.15.150:0/4115340457 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd30009690 con 0x7efd24005910 2024-03-17T23:48:15.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.417+0000 7efd408ac640 1 -- 172.21.15.150:0/4115340457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd24005910 msgr2=0x7efd24005d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:15.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.417+0000 7efd408ac640 1 --2- 172.21.15.150:0/4115340457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd24005910 0x7efd24005d90 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7efd30009ac0 tx=0x7efd24006940 comp rx=0 tx=0).stop 2024-03-17T23:48:15.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.417+0000 7efd408ac640 1 -- 172.21.15.150:0/4115340457 shutdown_connections 2024-03-17T23:48:15.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.417+0000 7efd408ac640 1 --2- 172.21.15.150:0/4115340457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd24005910 0x7efd24005d90 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.417+0000 7efd408ac640 1 -- 172.21.15.150:0/4115340457 >> 172.21.15.150:0/4115340457 conn(0x7efd3c072a60 msgr2=0x7efd3c0b5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:15.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.417+0000 7efd408ac640 1 -- 172.21.15.150:0/4115340457 shutdown_connections 2024-03-17T23:48:15.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.417+0000 7efd408ac640 1 -- 172.21.15.150:0/4115340457 wait complete. 2024-03-17T23:48:15.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.418+0000 7efd408ac640 1 Processor -- start 2024-03-17T23:48:15.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.418+0000 7efd408ac640 1 -- start start 2024-03-17T23:48:15.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.419+0000 7efd408ac640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd24005910 0x7efd3c141e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:15.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.419+0000 7efd408ac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd3c14ecf0 0x7efd3c1f1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:15.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.419+0000 7efd408ac640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd3c14deb0 0x7efd3c14e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:15.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.419+0000 7efd408ac640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7efd3c0ad900 con 0x7efd3c14ecf0 2024-03-17T23:48:15.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.419+0000 7efd408ac640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7efd3c099b10 con 0x7efd3c14deb0 2024-03-17T23:48:15.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.419+0000 7efd408ac640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efd3c0ada80 con 0x7efd24005910 2024-03-17T23:48:15.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.419+0000 7efd39d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd3c14ecf0 0x7efd3c1f1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:15.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.419+0000 7efd39d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd3c14ecf0 0x7efd3c1f1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43780/0 (socket says 172.21.15.150:43780) 2024-03-17T23:48:15.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.419+0000 7efd39d9b640 1 -- 172.21.15.150:0/436657469 learned_addr learned my addr 172.21.15.150:0/436657469 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:15.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.419+0000 7efd3ad9d640 1 --2- 172.21.15.150:0/436657469 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd3c14deb0 0x7efd3c14e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:15.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.419+0000 7efd3a59c640 1 --2- 172.21.15.150:0/436657469 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd24005910 0x7efd3c141e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:15.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.419+0000 7efd39d9b640 1 -- 172.21.15.150:0/436657469 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd24005910 msgr2=0x7efd3c141e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:15.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.419+0000 7efd39d9b640 1 --2- 172.21.15.150:0/436657469 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd24005910 0x7efd3c141e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.419+0000 7efd39d9b640 1 -- 172.21.15.150:0/436657469 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd3c14deb0 msgr2=0x7efd3c14e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:15.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.419+0000 7efd39d9b640 1 --2- 172.21.15.150:0/436657469 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd3c14deb0 0x7efd3c14e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.419+0000 7efd39d9b640 1 -- 172.21.15.150:0/436657469 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd3c141b60 con 0x7efd3c14ecf0 2024-03-17T23:48:15.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.419+0000 7efd39d9b640 1 --2- 172.21.15.150:0/436657469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd3c14ecf0 0x7efd3c1f1680 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7efd30009be0 tx=0x7efd30004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:15.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.420+0000 7efd2affd640 1 -- 172.21.15.150:0/436657469 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd30042020 con 0x7efd3c14ecf0 2024-03-17T23:48:15.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.420+0000 7efd2affd640 1 -- 172.21.15.150:0/436657469 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd3003a040 con 0x7efd3c14ecf0 2024-03-17T23:48:15.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.420+0000 7efd408ac640 1 -- 172.21.15.150:0/436657469 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd3c14f380 con 0x7efd3c14ecf0 2024-03-17T23:48:15.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.420+0000 7efd2affd640 1 -- 172.21.15.150:0/436657469 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd30039070 con 0x7efd3c14ecf0 2024-03-17T23:48:15.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.420+0000 7efd408ac640 1 -- 172.21.15.150:0/436657469 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd3c14f940 con 0x7efd3c14ecf0 2024-03-17T23:48:15.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.421+0000 7efd408ac640 1 -- 172.21.15.150:0/436657469 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd3c0688b0 con 0x7efd3c14ecf0 2024-03-17T23:48:15.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.422+0000 7efd2affd640 1 -- 172.21.15.150:0/436657469 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7efd30045020 con 0x7efd3c14ecf0 2024-03-17T23:48:15.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.422+0000 7efd2affd640 1 --2- 172.21.15.150:0/436657469 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7efd1c06bc00 0x7efd1c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:15.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.422+0000 7efd2affd640 1 -- 172.21.15.150:0/436657469 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7efd300ac2a0 con 0x7efd3c14ecf0 2024-03-17T23:48:15.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.422+0000 7efd3a59c640 1 --2- 172.21.15.150:0/436657469 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7efd1c06bc00 0x7efd1c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:15.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.423+0000 7efd3a59c640 1 --2- 172.21.15.150:0/436657469 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7efd1c06bc00 0x7efd1c06e0c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7efd3c04fa60 tx=0x7efd2c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:15.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.430+0000 7efd2affd640 1 -- 172.21.15.150:0/436657469 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7efd3004c030 con 0x7efd3c14ecf0 2024-03-17T23:48:15.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.679+0000 7efd408ac640 1 -- 172.21.15.150:0/436657469 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7efd3c14b770 con 0x7efd3c14ecf0 2024-03-17T23:48:15.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.680+0000 7efd2affd640 1 -- 172.21.15.150:0/436657469 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1249 (secure 0 0 0) 0x7efd3c141b60 con 0x7efd3c14ecf0 2024-03-17T23:48:15.681 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:15.681 INFO:teuthology.orchestra.run.smithi150.stdout:{"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":104,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":40,"num_osds":8,"num_up_osds":8,"osd_up_since":1710719202,"num_in_osds":8,"osd_in_since":1710719193,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":61},{"state_name":"unknown","count":36}],"num_pgs":97,"num_pools":3,"num_objects":15,"data_bytes":591062,"bytes_used":243105792,"bytes_avail":772851007488,"bytes_total":773094113280,"unknown_pgs_ratio":0.37113401293754578,"write_bytes_sec":724,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":15,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"c","status":"up:active","gid":5515}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-03-17T23:47:51.608885+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-17T23:48:15.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.682+0000 7efd408ac640 1 -- 172.21.15.150:0/436657469 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7efd1c06bc00 msgr2=0x7efd1c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:15.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.682+0000 7efd408ac640 1 --2- 172.21.15.150:0/436657469 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7efd1c06bc00 0x7efd1c06e0c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7efd3c04fa60 tx=0x7efd2c009040 comp rx=0 tx=0).stop 2024-03-17T23:48:15.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.682+0000 7efd408ac640 1 -- 172.21.15.150:0/436657469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd3c14ecf0 msgr2=0x7efd3c1f1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:15.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.682+0000 7efd408ac640 1 --2- 172.21.15.150:0/436657469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd3c14ecf0 0x7efd3c1f1680 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7efd30009be0 tx=0x7efd30004730 comp rx=0 tx=0).stop 2024-03-17T23:48:15.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.682+0000 7efd408ac640 1 -- 172.21.15.150:0/436657469 shutdown_connections 2024-03-17T23:48:15.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.682+0000 7efd408ac640 1 --2- 172.21.15.150:0/436657469 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7efd1c06bc00 0x7efd1c06e0c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.682+0000 7efd408ac640 1 --2- 172.21.15.150:0/436657469 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd3c14deb0 0x7efd3c14e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.682+0000 7efd408ac640 1 --2- 172.21.15.150:0/436657469 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd3c14ecf0 0x7efd3c1f1680 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.682+0000 7efd408ac640 1 --2- 172.21.15.150:0/436657469 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd24005910 0x7efd3c141e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.682+0000 7efd408ac640 1 -- 172.21.15.150:0/436657469 >> 172.21.15.150:0/436657469 conn(0x7efd3c072a60 msgr2=0x7efd3c0add60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:15.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.682+0000 7efd408ac640 1 -- 172.21.15.150:0/436657469 shutdown_connections 2024-03-17T23:48:15.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.682+0000 7efd408ac640 1 -- 172.21.15.150:0/436657469 wait complete. 2024-03-17T23:48:15.695 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-17T23:48:15.695 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-17T23:48:15.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.796+0000 7fdb7d275640 1 Processor -- start 2024-03-17T23:48:15.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.796+0000 7fdb7d275640 1 -- start start 2024-03-17T23:48:15.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.797+0000 7fdb7d275640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdb781514f0 0x7fdb7815bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:15.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.797+0000 7fdb7d275640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fdb7815ab90 0x7fdb7806f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:15.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.797+0000 7fdb7d275640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb7806e5c0 0x7fdb7806ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:15.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.797+0000 7fdb7d275640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fdb7806fba0 con 0x7fdb781514f0 2024-03-17T23:48:15.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.797+0000 7fdb7d275640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fdb7806efb0 con 0x7fdb7815ab90 2024-03-17T23:48:15.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.797+0000 7fdb7d275640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdb780ce790 con 0x7fdb7806e5c0 2024-03-17T23:48:15.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.797+0000 7fdb76ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdb781514f0 0x7fdb7815bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:15.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.797+0000 7fdb76ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdb781514f0 0x7fdb7815bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36616/0 (socket says 172.21.15.150:36616) 2024-03-17T23:48:15.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.797+0000 7fdb76ffd640 1 -- 172.21.15.150:0/1869696691 learned_addr learned my addr 172.21.15.150:0/1869696691 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:15.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.797+0000 7fdb6ffff640 1 --2- 172.21.15.150:0/1869696691 >> v2:172.21.15.157:3300/0 conn(0x7fdb7815ab90 0x7fdb7806f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:15.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.797+0000 7fdb777fe640 1 --2- 172.21.15.150:0/1869696691 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb7806e5c0 0x7fdb7806ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:15.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.797+0000 7fdb76ffd640 1 -- 172.21.15.150:0/1869696691 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb7806e5c0 msgr2=0x7fdb7806ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:15.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.797+0000 7fdb76ffd640 1 --2- 172.21.15.150:0/1869696691 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb7806e5c0 0x7fdb7806ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.797+0000 7fdb76ffd640 1 -- 172.21.15.150:0/1869696691 >> v2:172.21.15.157:3300/0 conn(0x7fdb7815ab90 msgr2=0x7fdb7806f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:15.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.797+0000 7fdb76ffd640 1 --2- 172.21.15.150:0/1869696691 >> v2:172.21.15.157:3300/0 conn(0x7fdb7815ab90 0x7fdb7806f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.797+0000 7fdb76ffd640 1 -- 172.21.15.150:0/1869696691 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb7815b050 con 0x7fdb781514f0 2024-03-17T23:48:15.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.797+0000 7fdb6ffff640 1 --2- 172.21.15.150:0/1869696691 >> v2:172.21.15.157:3300/0 conn(0x7fdb7815ab90 0x7fdb7806f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:15.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.798+0000 7fdb76ffd640 1 --2- 172.21.15.150:0/1869696691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb781514f0 0x7fdb7815bd10 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fdb7806f2d0 tx=0x7fdb6802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50f07fdf76f5aeba server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:15.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.799+0000 7fdb767fc640 1 -- 172.21.15.150:0/1869696691 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68035020 con 0x7fdb781514f0 2024-03-17T23:48:15.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.799+0000 7fdb767fc640 1 -- 172.21.15.150:0/1869696691 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb68031870 con 0x7fdb781514f0 2024-03-17T23:48:15.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.799+0000 7fdb767fc640 1 -- 172.21.15.150:0/1869696691 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb68031c40 con 0x7fdb781514f0 2024-03-17T23:48:15.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.799+0000 7fdb7d275640 1 -- 172.21.15.150:0/1869696691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb781514f0 msgr2=0x7fdb7815bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:15.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.799+0000 7fdb7d275640 1 --2- 172.21.15.150:0/1869696691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb781514f0 0x7fdb7815bd10 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fdb7806f2d0 tx=0x7fdb6802b910 comp rx=0 tx=0).stop 2024-03-17T23:48:15.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.799+0000 7fdb7d275640 1 -- 172.21.15.150:0/1869696691 shutdown_connections 2024-03-17T23:48:15.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.799+0000 7fdb7d275640 1 --2- 172.21.15.150:0/1869696691 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb7806e5c0 0x7fdb7806ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.799+0000 7fdb7d275640 1 --2- 172.21.15.150:0/1869696691 >> v2:172.21.15.157:3300/0 conn(0x7fdb7815ab90 0x7fdb7806f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.799+0000 7fdb7d275640 1 --2- 172.21.15.150:0/1869696691 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb781514f0 0x7fdb7815bd10 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.799+0000 7fdb7d275640 1 -- 172.21.15.150:0/1869696691 >> 172.21.15.150:0/1869696691 conn(0x7fdb7806b190 msgr2=0x7fdb7806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:15.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.799+0000 7fdb7d275640 1 -- 172.21.15.150:0/1869696691 shutdown_connections 2024-03-17T23:48:15.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.799+0000 7fdb7d275640 1 -- 172.21.15.150:0/1869696691 wait complete. 2024-03-17T23:48:15.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.800+0000 7fdb7d275640 1 Processor -- start 2024-03-17T23:48:15.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.801+0000 7fdb7d275640 1 -- start start 2024-03-17T23:48:15.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.801+0000 7fdb7d275640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb7806e5c0 0x7fdb78145b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:15.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.801+0000 7fdb7d275640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb781514f0 0x7fdb781434c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:15.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.801+0000 7fdb7d275640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb7815ab90 0x7fdb78150640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:15.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.801+0000 7fdb7d275640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7815b8f0 con 0x7fdb781514f0 2024-03-17T23:48:15.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.801+0000 7fdb7d275640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7815b610 con 0x7fdb7815ab90 2024-03-17T23:48:15.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.801+0000 7fdb7d275640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdb780af510 con 0x7fdb7806e5c0 2024-03-17T23:48:15.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.801+0000 7fdb6ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb781514f0 0x7fdb781434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:15.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.801+0000 7fdb6ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb781514f0 0x7fdb781434c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36622/0 (socket says 172.21.15.150:36622) 2024-03-17T23:48:15.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.801+0000 7fdb6ffff640 1 -- 172.21.15.150:0/2215275363 learned_addr learned my addr 172.21.15.150:0/2215275363 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:15.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.802+0000 7fdb777fe640 1 --2- 172.21.15.150:0/2215275363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb7815ab90 0x7fdb78150640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:15.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.802+0000 7fdb76ffd640 1 --2- 172.21.15.150:0/2215275363 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb7806e5c0 0x7fdb78145b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:15.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.802+0000 7fdb6ffff640 1 -- 172.21.15.150:0/2215275363 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb7806e5c0 msgr2=0x7fdb78145b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:15.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.802+0000 7fdb6ffff640 1 --2- 172.21.15.150:0/2215275363 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb7806e5c0 0x7fdb78145b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.802+0000 7fdb6ffff640 1 -- 172.21.15.150:0/2215275363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb7815ab90 msgr2=0x7fdb78150640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:15.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.802+0000 7fdb6ffff640 1 --2- 172.21.15.150:0/2215275363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb7815ab90 0x7fdb78150640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.802+0000 7fdb6ffff640 1 -- 172.21.15.150:0/2215275363 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb78146180 con 0x7fdb781514f0 2024-03-17T23:48:15.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.802+0000 7fdb777fe640 1 --2- 172.21.15.150:0/2215275363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb7815ab90 0x7fdb78150640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:15.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.802+0000 7fdb6ffff640 1 --2- 172.21.15.150:0/2215275363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb781514f0 0x7fdb781434c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fdb6000c7f0 tx=0x7fdb6000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:15.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.802+0000 7fdb6f7fe640 1 -- 172.21.15.150:0/2215275363 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb60015070 con 0x7fdb781514f0 2024-03-17T23:48:15.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.802+0000 7fdb6f7fe640 1 -- 172.21.15.150:0/2215275363 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb60007820 con 0x7fdb781514f0 2024-03-17T23:48:15.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.802+0000 7fdb7d275640 1 -- 172.21.15.150:0/2215275363 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb78146460 con 0x7fdb781514f0 2024-03-17T23:48:15.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.802+0000 7fdb6f7fe640 1 -- 172.21.15.150:0/2215275363 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb60007bf0 con 0x7fdb781514f0 2024-03-17T23:48:15.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.802+0000 7fdb7d275640 1 -- 172.21.15.150:0/2215275363 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb78146a20 con 0x7fdb781514f0 2024-03-17T23:48:15.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.803+0000 7fdb7d275640 1 -- 172.21.15.150:0/2215275363 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb780681a0 con 0x7fdb781514f0 2024-03-17T23:48:15.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.803+0000 7fdb6f7fe640 1 -- 172.21.15.150:0/2215275363 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fdb60018070 con 0x7fdb781514f0 2024-03-17T23:48:15.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.804+0000 7fdb6f7fe640 1 --2- 172.21.15.150:0/2215275363 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fdb5806bbd0 0x7fdb5806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:15.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.804+0000 7fdb76ffd640 1 --2- 172.21.15.150:0/2215275363 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fdb5806bbd0 0x7fdb5806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:15.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.804+0000 7fdb6f7fe640 1 -- 172.21.15.150:0/2215275363 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7fdb6008ad40 con 0x7fdb781514f0 2024-03-17T23:48:15.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.804+0000 7fdb76ffd640 1 --2- 172.21.15.150:0/2215275363 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fdb5806bbd0 0x7fdb5806e090 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fdb68036000 tx=0x7fdb68008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:15.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.808+0000 7fdb6f7fe640 1 -- 172.21.15.150:0/2215275363 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fdb600596a0 con 0x7fdb781514f0 2024-03-17T23:48:15.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.983+0000 7fdb7d275640 1 -- 172.21.15.150:0/2215275363 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdb7806bf90 con 0x7fdb5806bbd0 2024-03-17T23:48:15.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.989+0000 7fdb6f7fe640 1 -- 172.21.15.150:0/2215275363 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251931 (secure 0 0 0) 0x7fdb7806bf90 con 0x7fdb5806bbd0 2024-03-17T23:48:15.990 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:15.994 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-03-17T23:48:15.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.992+0000 7fdb7d275640 1 -- 172.21.15.150:0/2215275363 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fdb5806bbd0 msgr2=0x7fdb5806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:15.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.992+0000 7fdb7d275640 1 --2- 172.21.15.150:0/2215275363 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fdb5806bbd0 0x7fdb5806e090 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fdb68036000 tx=0x7fdb68008030 comp rx=0 tx=0).stop 2024-03-17T23:48:15.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.992+0000 7fdb7d275640 1 -- 172.21.15.150:0/2215275363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb781514f0 msgr2=0x7fdb781434c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:15.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.992+0000 7fdb7d275640 1 --2- 172.21.15.150:0/2215275363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb781514f0 0x7fdb781434c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fdb6000c7f0 tx=0x7fdb6000ccb0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.992+0000 7fdb7d275640 1 -- 172.21.15.150:0/2215275363 shutdown_connections 2024-03-17T23:48:15.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.992+0000 7fdb7d275640 1 --2- 172.21.15.150:0/2215275363 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fdb5806bbd0 0x7fdb5806e090 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.992+0000 7fdb7d275640 1 --2- 172.21.15.150:0/2215275363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb7815ab90 0x7fdb78150640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.992+0000 7fdb7d275640 1 --2- 172.21.15.150:0/2215275363 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdb781514f0 0x7fdb781434c0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.992+0000 7fdb7d275640 1 --2- 172.21.15.150:0/2215275363 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdb7806e5c0 0x7fdb78145b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:15.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.992+0000 7fdb7d275640 1 -- 172.21.15.150:0/2215275363 >> 172.21.15.150:0/2215275363 conn(0x7fdb7806b190 msgr2=0x7fdb780aecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:15.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.992+0000 7fdb7d275640 1 -- 172.21.15.150:0/2215275363 shutdown_connections 2024-03-17T23:48:15.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:15.992+0000 7fdb7d275640 1 -- 172.21.15.150:0/2215275363 wait complete. 2024-03-17T23:48:16.004 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":80,"stamp":"2024-03-17T23:48:15.613323+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237560,"kb_used_data":4748,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737160,"statfs":{"total":773094113280,"available":772850851840,"internally_reserved":0,"allocated":4861952,"data_stored":2760128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000944"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.399005+0000","last_change":"2024-03-17T23:48:11.394564+0000","last_active":"2024-03-17T23:48:12.399005+0000","last_peered":"2024-03-17T23:48:12.399005+0000","last_clean":"2024-03-17T23:48:12.399005+0000","last_became_active":"2024-03-17T23:48:11.394313+0000","last_became_peered":"2024-03-17T23:48:11.394313+0000","last_unstale":"2024-03-17T23:48:12.399005+0000","last_undegraded":"2024-03-17T23:48:12.399005+0000","last_fullsized":"2024-03-17T23:48:12.399005+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:50:41.482483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398864+0000","last_change":"2024-03-17T23:48:11.390494+0000","last_active":"2024-03-17T23:48:12.398864+0000","last_peered":"2024-03-17T23:48:12.398864+0000","last_clean":"2024-03-17T23:48:12.398864+0000","last_became_active":"2024-03-17T23:48:11.390235+0000","last_became_peered":"2024-03-17T23:48:11.390235+0000","last_unstale":"2024-03-17T23:48:12.398864+0000","last_undegraded":"2024-03-17T23:48:12.398864+0000","last_fullsized":"2024-03-17T23:48:12.398864+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:20:58.243543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:13.023233+0000","last_change":"2024-03-17T23:48:11.396000+0000","last_active":"2024-03-17T23:48:13.023233+0000","last_peered":"2024-03-17T23:48:13.023233+0000","last_clean":"2024-03-17T23:48:13.023233+0000","last_became_active":"2024-03-17T23:48:11.395653+0000","last_became_peered":"2024-03-17T23:48:11.395653+0000","last_unstale":"2024-03-17T23:48:13.023233+0000","last_undegraded":"2024-03-17T23:48:13.023233+0000","last_fullsized":"2024-03-17T23:48:13.023233+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:56:06.470585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.823942+0000","last_change":"2024-03-17T23:48:11.396619+0000","last_active":"2024-03-17T23:48:12.823942+0000","last_peered":"2024-03-17T23:48:12.823942+0000","last_clean":"2024-03-17T23:48:12.823942+0000","last_became_active":"2024-03-17T23:48:11.396382+0000","last_became_peered":"2024-03-17T23:48:11.396382+0000","last_unstale":"2024-03-17T23:48:12.823942+0000","last_undegraded":"2024-03-17T23:48:12.823942+0000","last_fullsized":"2024-03-17T23:48:12.823942+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:06:34.060471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.823934+0000","last_change":"2024-03-17T23:48:11.402400+0000","last_active":"2024-03-17T23:48:12.823934+0000","last_peered":"2024-03-17T23:48:12.823934+0000","last_clean":"2024-03-17T23:48:12.823934+0000","last_became_active":"2024-03-17T23:48:11.402145+0000","last_became_peered":"2024-03-17T23:48:11.402145+0000","last_unstale":"2024-03-17T23:48:12.823934+0000","last_undegraded":"2024-03-17T23:48:12.823934+0000","last_fullsized":"2024-03-17T23:48:12.823934+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:32:38.502832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.400417+0000","last_change":"2024-03-17T23:48:11.399627+0000","last_active":"2024-03-17T23:48:11.400417+0000","last_peered":"2024-03-17T23:48:11.400417+0000","last_clean":"2024-03-17T23:48:11.400417+0000","last_became_active":"2024-03-17T23:48:11.399364+0000","last_became_peered":"2024-03-17T23:48:11.399364+0000","last_unstale":"2024-03-17T23:48:11.400417+0000","last_undegraded":"2024-03-17T23:48:11.400417+0000","last_fullsized":"2024-03-17T23:48:11.400417+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:44:42.530638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409171+0000","last_change":"2024-03-17T23:48:11.403822+0000","last_active":"2024-03-17T23:48:11.409171+0000","last_peered":"2024-03-17T23:48:11.409171+0000","last_clean":"2024-03-17T23:48:11.409171+0000","last_became_active":"2024-03-17T23:48:11.403572+0000","last_became_peered":"2024-03-17T23:48:11.403572+0000","last_unstale":"2024-03-17T23:48:11.409171+0000","last_undegraded":"2024-03-17T23:48:11.409171+0000","last_fullsized":"2024-03-17T23:48:11.409171+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:53:53.408503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398516+0000","last_change":"2024-03-17T23:48:11.397742+0000","last_active":"2024-03-17T23:48:12.398516+0000","last_peered":"2024-03-17T23:48:12.398516+0000","last_clean":"2024-03-17T23:48:12.398516+0000","last_became_active":"2024-03-17T23:48:11.397483+0000","last_became_peered":"2024-03-17T23:48:11.397483+0000","last_unstale":"2024-03-17T23:48:12.398516+0000","last_undegraded":"2024-03-17T23:48:12.398516+0000","last_fullsized":"2024-03-17T23:48:12.398516+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:41:05.763125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:13.023637+0000","last_change":"2024-03-17T23:48:11.407925+0000","last_active":"2024-03-17T23:48:13.023637+0000","last_peered":"2024-03-17T23:48:13.023637+0000","last_clean":"2024-03-17T23:48:13.023637+0000","last_became_active":"2024-03-17T23:48:11.407612+0000","last_became_peered":"2024-03-17T23:48:11.407612+0000","last_unstale":"2024-03-17T23:48:13.023637+0000","last_undegraded":"2024-03-17T23:48:13.023637+0000","last_fullsized":"2024-03-17T23:48:13.023637+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:37:11.855078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.406658+0000","last_change":"2024-03-17T23:48:11.395918+0000","last_active":"2024-03-17T23:48:11.406658+0000","last_peered":"2024-03-17T23:48:11.406658+0000","last_clean":"2024-03-17T23:48:11.406658+0000","last_became_active":"2024-03-17T23:48:11.394963+0000","last_became_peered":"2024-03-17T23:48:11.394963+0000","last_unstale":"2024-03-17T23:48:11.406658+0000","last_undegraded":"2024-03-17T23:48:11.406658+0000","last_fullsized":"2024-03-17T23:48:11.406658+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:20:48.927064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.399665+0000","last_change":"2024-03-17T23:48:11.394537+0000","last_active":"2024-03-17T23:48:11.399665+0000","last_peered":"2024-03-17T23:48:11.399665+0000","last_clean":"2024-03-17T23:48:11.399665+0000","last_became_active":"2024-03-17T23:48:11.394147+0000","last_became_peered":"2024-03-17T23:48:11.394147+0000","last_unstale":"2024-03-17T23:48:11.399665+0000","last_undegraded":"2024-03-17T23:48:11.399665+0000","last_fullsized":"2024-03-17T23:48:11.399665+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:45.409873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:13.023223+0000","last_change":"2024-03-17T23:48:11.402407+0000","last_active":"2024-03-17T23:48:13.023223+0000","last_peered":"2024-03-17T23:48:13.023223+0000","last_clean":"2024-03-17T23:48:13.023223+0000","last_became_active":"2024-03-17T23:48:11.402140+0000","last_became_peered":"2024-03-17T23:48:11.402140+0000","last_unstale":"2024-03-17T23:48:13.023223+0000","last_undegraded":"2024-03-17T23:48:13.023223+0000","last_fullsized":"2024-03-17T23:48:13.023223+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:03:52.711284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.394641+0000","last_change":"2024-03-17T23:48:11.393682+0000","last_active":"2024-03-17T23:48:11.394641+0000","last_peered":"2024-03-17T23:48:11.394641+0000","last_clean":"2024-03-17T23:48:11.394641+0000","last_became_active":"2024-03-17T23:48:11.393410+0000","last_became_peered":"2024-03-17T23:48:11.393410+0000","last_unstale":"2024-03-17T23:48:11.394641+0000","last_undegraded":"2024-03-17T23:48:11.394641+0000","last_fullsized":"2024-03-17T23:48:11.394641+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:17.358341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397913+0000","last_change":"2024-03-17T23:48:11.405838+0000","last_active":"2024-03-17T23:48:12.397913+0000","last_peered":"2024-03-17T23:48:12.397913+0000","last_clean":"2024-03-17T23:48:12.397913+0000","last_became_active":"2024-03-17T23:48:11.405573+0000","last_became_peered":"2024-03-17T23:48:11.405573+0000","last_unstale":"2024-03-17T23:48:12.397913+0000","last_undegraded":"2024-03-17T23:48:12.397913+0000","last_fullsized":"2024-03-17T23:48:12.397913+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:33:46.207300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409150+0000","last_change":"2024-03-17T23:48:11.403246+0000","last_active":"2024-03-17T23:48:11.409150+0000","last_peered":"2024-03-17T23:48:11.409150+0000","last_clean":"2024-03-17T23:48:11.409150+0000","last_became_active":"2024-03-17T23:48:11.402986+0000","last_became_peered":"2024-03-17T23:48:11.402986+0000","last_unstale":"2024-03-17T23:48:11.409150+0000","last_undegraded":"2024-03-17T23:48:11.409150+0000","last_fullsized":"2024-03-17T23:48:11.409150+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:16:44.071095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.402167+0000","last_change":"2024-03-17T23:48:11.398200+0000","last_active":"2024-03-17T23:48:11.402167+0000","last_peered":"2024-03-17T23:48:11.402167+0000","last_clean":"2024-03-17T23:48:11.402167+0000","last_became_active":"2024-03-17T23:48:11.397937+0000","last_became_peered":"2024-03-17T23:48:11.397937+0000","last_unstale":"2024-03-17T23:48:11.402167+0000","last_undegraded":"2024-03-17T23:48:11.402167+0000","last_fullsized":"2024-03-17T23:48:11.402167+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:45.409873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409193+0000","last_change":"2024-03-17T23:48:11.408181+0000","last_active":"2024-03-17T23:48:11.409193+0000","last_peered":"2024-03-17T23:48:11.409193+0000","last_clean":"2024-03-17T23:48:11.409193+0000","last_became_active":"2024-03-17T23:48:11.407876+0000","last_became_peered":"2024-03-17T23:48:11.407876+0000","last_unstale":"2024-03-17T23:48:11.409193+0000","last_undegraded":"2024-03-17T23:48:11.409193+0000","last_fullsized":"2024-03-17T23:48:11.409193+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:05.029387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398135+0000","last_change":"2024-03-17T23:48:11.402141+0000","last_active":"2024-03-17T23:48:12.398135+0000","last_peered":"2024-03-17T23:48:12.398135+0000","last_clean":"2024-03-17T23:48:12.398135+0000","last_became_active":"2024-03-17T23:48:11.401870+0000","last_became_peered":"2024-03-17T23:48:11.401870+0000","last_unstale":"2024-03-17T23:48:12.398135+0000","last_undegraded":"2024-03-17T23:48:12.398135+0000","last_fullsized":"2024-03-17T23:48:12.398135+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:10.929912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.823828+0000","last_change":"2024-03-17T23:48:11.401430+0000","last_active":"2024-03-17T23:48:12.823828+0000","last_peered":"2024-03-17T23:48:12.823828+0000","last_clean":"2024-03-17T23:48:12.823828+0000","last_became_active":"2024-03-17T23:48:11.401220+0000","last_became_peered":"2024-03-17T23:48:11.401220+0000","last_unstale":"2024-03-17T23:48:12.823828+0000","last_undegraded":"2024-03-17T23:48:12.823828+0000","last_fullsized":"2024-03-17T23:48:12.823828+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:55.443705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398329+0000","last_change":"2024-03-17T23:48:11.401263+0000","last_active":"2024-03-17T23:48:12.398329+0000","last_peered":"2024-03-17T23:48:12.398329+0000","last_clean":"2024-03-17T23:48:12.398329+0000","last_became_active":"2024-03-17T23:48:11.400951+0000","last_became_peered":"2024-03-17T23:48:11.400951+0000","last_unstale":"2024-03-17T23:48:12.398329+0000","last_undegraded":"2024-03-17T23:48:12.398329+0000","last_fullsized":"2024-03-17T23:48:12.398329+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:42:43.776415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408759+0000","last_change":"2024-03-17T23:48:11.400746+0000","last_active":"2024-03-17T23:48:11.408759+0000","last_peered":"2024-03-17T23:48:11.408759+0000","last_clean":"2024-03-17T23:48:11.408759+0000","last_became_active":"2024-03-17T23:48:11.400494+0000","last_became_peered":"2024-03-17T23:48:11.400494+0000","last_unstale":"2024-03-17T23:48:11.408759+0000","last_undegraded":"2024-03-17T23:48:11.408759+0000","last_fullsized":"2024-03-17T23:48:11.408759+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:05.029387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398311+0000","last_change":"2024-03-17T23:48:11.403046+0000","last_active":"2024-03-17T23:48:12.398311+0000","last_peered":"2024-03-17T23:48:12.398311+0000","last_clean":"2024-03-17T23:48:12.398311+0000","last_became_active":"2024-03-17T23:48:11.402769+0000","last_became_peered":"2024-03-17T23:48:11.402769+0000","last_unstale":"2024-03-17T23:48:12.398311+0000","last_undegraded":"2024-03-17T23:48:12.398311+0000","last_fullsized":"2024-03-17T23:48:12.398311+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:16:09.342548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408364+0000","last_change":"2024-03-17T23:48:11.402963+0000","last_active":"2024-03-17T23:48:11.408364+0000","last_peered":"2024-03-17T23:48:11.408364+0000","last_clean":"2024-03-17T23:48:11.408364+0000","last_became_active":"2024-03-17T23:48:11.402669+0000","last_became_peered":"2024-03-17T23:48:11.402669+0000","last_unstale":"2024-03-17T23:48:11.408364+0000","last_undegraded":"2024-03-17T23:48:11.408364+0000","last_fullsized":"2024-03-17T23:48:11.408364+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:20:48.927064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.823768+0000","last_change":"2024-03-17T23:48:11.392903+0000","last_active":"2024-03-17T23:48:12.823768+0000","last_peered":"2024-03-17T23:48:12.823768+0000","last_clean":"2024-03-17T23:48:12.823768+0000","last_became_active":"2024-03-17T23:48:11.392621+0000","last_became_peered":"2024-03-17T23:48:11.392621+0000","last_unstale":"2024-03-17T23:48:12.823768+0000","last_undegraded":"2024-03-17T23:48:12.823768+0000","last_fullsized":"2024-03-17T23:48:12.823768+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:16.118613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398382+0000","last_change":"2024-03-17T23:48:11.392442+0000","last_active":"2024-03-17T23:48:12.398382+0000","last_peered":"2024-03-17T23:48:12.398382+0000","last_clean":"2024-03-17T23:48:12.398382+0000","last_became_active":"2024-03-17T23:48:11.392184+0000","last_became_peered":"2024-03-17T23:48:11.392184+0000","last_unstale":"2024-03-17T23:48:12.398382+0000","last_undegraded":"2024-03-17T23:48:12.398382+0000","last_fullsized":"2024-03-17T23:48:12.398382+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:31.101445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.399825+0000","last_change":"2024-03-17T23:48:11.398013+0000","last_active":"2024-03-17T23:48:11.399825+0000","last_peered":"2024-03-17T23:48:11.399825+0000","last_clean":"2024-03-17T23:48:11.399825+0000","last_became_active":"2024-03-17T23:48:11.397709+0000","last_became_peered":"2024-03-17T23:48:11.397709+0000","last_unstale":"2024-03-17T23:48:11.399825+0000","last_undegraded":"2024-03-17T23:48:11.399825+0000","last_fullsized":"2024-03-17T23:48:11.399825+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:05.029387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397892+0000","last_change":"2024-03-17T23:48:11.404602+0000","last_active":"2024-03-17T23:48:12.397892+0000","last_peered":"2024-03-17T23:48:12.397892+0000","last_clean":"2024-03-17T23:48:12.397892+0000","last_became_active":"2024-03-17T23:48:11.404351+0000","last_became_peered":"2024-03-17T23:48:11.404351+0000","last_unstale":"2024-03-17T23:48:12.397892+0000","last_undegraded":"2024-03-17T23:48:12.397892+0000","last_fullsized":"2024-03-17T23:48:12.397892+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:45.409873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409206+0000","last_change":"2024-03-17T23:48:11.403804+0000","last_active":"2024-03-17T23:48:11.409206+0000","last_peered":"2024-03-17T23:48:11.409206+0000","last_clean":"2024-03-17T23:48:11.409206+0000","last_became_active":"2024-03-17T23:48:11.403537+0000","last_became_peered":"2024-03-17T23:48:11.403537+0000","last_unstale":"2024-03-17T23:48:11.409206+0000","last_undegraded":"2024-03-17T23:48:11.409206+0000","last_fullsized":"2024-03-17T23:48:11.409206+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:32:24.643171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397774+0000","last_change":"2024-03-17T23:48:11.398674+0000","last_active":"2024-03-17T23:48:12.397774+0000","last_peered":"2024-03-17T23:48:12.397774+0000","last_clean":"2024-03-17T23:48:12.397774+0000","last_became_active":"2024-03-17T23:48:11.398399+0000","last_became_peered":"2024-03-17T23:48:11.398399+0000","last_unstale":"2024-03-17T23:48:12.397774+0000","last_undegraded":"2024-03-17T23:48:12.397774+0000","last_fullsized":"2024-03-17T23:48:12.397774+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:17.358341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398058+0000","last_change":"2024-03-17T23:48:11.393129+0000","last_active":"2024-03-17T23:48:12.398058+0000","last_peered":"2024-03-17T23:48:12.398058+0000","last_clean":"2024-03-17T23:48:12.398058+0000","last_became_active":"2024-03-17T23:48:11.392847+0000","last_became_peered":"2024-03-17T23:48:11.392847+0000","last_unstale":"2024-03-17T23:48:12.398058+0000","last_undegraded":"2024-03-17T23:48:12.398058+0000","last_fullsized":"2024-03-17T23:48:12.398058+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:20:48.927064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408386+0000","last_change":"2024-03-17T23:48:11.403541+0000","last_active":"2024-03-17T23:48:11.408386+0000","last_peered":"2024-03-17T23:48:11.408386+0000","last_clean":"2024-03-17T23:48:11.408386+0000","last_became_active":"2024-03-17T23:48:11.403273+0000","last_became_peered":"2024-03-17T23:48:11.403273+0000","last_unstale":"2024-03-17T23:48:11.408386+0000","last_undegraded":"2024-03-17T23:48:11.408386+0000","last_fullsized":"2024-03-17T23:48:11.408386+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:10.929912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.397462+0000","last_change":"2024-03-17T23:48:11.396799+0000","last_active":"2024-03-17T23:48:11.397462+0000","last_peered":"2024-03-17T23:48:11.397462+0000","last_clean":"2024-03-17T23:48:11.397462+0000","last_became_active":"2024-03-17T23:48:11.396343+0000","last_became_peered":"2024-03-17T23:48:11.396343+0000","last_unstale":"2024-03-17T23:48:11.397462+0000","last_undegraded":"2024-03-17T23:48:11.397462+0000","last_fullsized":"2024-03-17T23:48:11.397462+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:42:43.776415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.399567+0000","last_change":"2024-03-17T23:48:11.397507+0000","last_active":"2024-03-17T23:48:11.399567+0000","last_peered":"2024-03-17T23:48:11.399567+0000","last_clean":"2024-03-17T23:48:11.399567+0000","last_became_active":"2024-03-17T23:48:11.397238+0000","last_became_peered":"2024-03-17T23:48:11.397238+0000","last_unstale":"2024-03-17T23:48:11.399567+0000","last_undegraded":"2024-03-17T23:48:11.399567+0000","last_fullsized":"2024-03-17T23:48:11.399567+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:28:11.274300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"39'4","reported_seq":18,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398478+0000","last_change":"2024-03-17T23:48:10.382890+0000","last_active":"2024-03-17T23:48:12.398478+0000","last_peered":"2024-03-17T23:48:12.398478+0000","last_clean":"2024-03-17T23:48:12.398478+0000","last_became_active":"2024-03-17T23:48:10.382565+0000","last_became_peered":"2024-03-17T23:48:10.382565+0000","last_unstale":"2024-03-17T23:48:12.398478+0000","last_undegraded":"2024-03-17T23:48:12.398478+0000","last_fullsized":"2024-03-17T23:48:12.398478+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:54.441315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.401433+0000","last_change":"2024-03-17T23:48:11.397742+0000","last_active":"2024-03-17T23:48:11.401433+0000","last_peered":"2024-03-17T23:48:11.401433+0000","last_clean":"2024-03-17T23:48:11.401433+0000","last_became_active":"2024-03-17T23:48:11.397189+0000","last_became_peered":"2024-03-17T23:48:11.397189+0000","last_unstale":"2024-03-17T23:48:11.401433+0000","last_undegraded":"2024-03-17T23:48:11.401433+0000","last_fullsized":"2024-03-17T23:48:11.401433+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:59:29.151560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"39'1","reported_seq":15,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:13.023457+0000","last_change":"2024-03-17T23:48:10.395772+0000","last_active":"2024-03-17T23:48:13.023457+0000","last_peered":"2024-03-17T23:48:13.023457+0000","last_clean":"2024-03-17T23:48:13.023457+0000","last_became_active":"2024-03-17T23:48:10.395501+0000","last_became_peered":"2024-03-17T23:48:10.395501+0000","last_unstale":"2024-03-17T23:48:13.023457+0000","last_undegraded":"2024-03-17T23:48:13.023457+0000","last_fullsized":"2024-03-17T23:48:13.023457+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:44.407483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398181+0000","last_change":"2024-03-17T23:48:11.406337+0000","last_active":"2024-03-17T23:48:12.398181+0000","last_peered":"2024-03-17T23:48:12.398181+0000","last_clean":"2024-03-17T23:48:12.398181+0000","last_became_active":"2024-03-17T23:48:11.405094+0000","last_became_peered":"2024-03-17T23:48:11.405094+0000","last_unstale":"2024-03-17T23:48:12.398181+0000","last_undegraded":"2024-03-17T23:48:12.398181+0000","last_fullsized":"2024-03-17T23:48:12.398181+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:38:38.676853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"39'2","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.406939+0000","last_change":"2024-03-17T23:48:10.396210+0000","last_active":"2024-03-17T23:48:11.406939+0000","last_peered":"2024-03-17T23:48:11.406939+0000","last_clean":"2024-03-17T23:48:11.406939+0000","last_became_active":"2024-03-17T23:48:10.395955+0000","last_became_peered":"2024-03-17T23:48:10.395955+0000","last_unstale":"2024-03-17T23:48:11.406939+0000","last_undegraded":"2024-03-17T23:48:11.406939+0000","last_fullsized":"2024-03-17T23:48:11.406939+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:20:47.924674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.399005+0000","last_change":"2024-03-17T23:48:11.392163+0000","last_active":"2024-03-17T23:48:12.399005+0000","last_peered":"2024-03-17T23:48:12.399005+0000","last_clean":"2024-03-17T23:48:12.399005+0000","last_became_active":"2024-03-17T23:48:11.391029+0000","last_became_peered":"2024-03-17T23:48:11.391029+0000","last_unstale":"2024-03-17T23:48:12.399005+0000","last_undegraded":"2024-03-17T23:48:12.399005+0000","last_fullsized":"2024-03-17T23:48:12.399005+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:05:55.540409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"39'2","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398582+0000","last_change":"2024-03-17T23:48:10.391635+0000","last_active":"2024-03-17T23:48:12.398582+0000","last_peered":"2024-03-17T23:48:12.398582+0000","last_clean":"2024-03-17T23:48:12.398582+0000","last_became_active":"2024-03-17T23:48:10.391241+0000","last_became_peered":"2024-03-17T23:48:10.391241+0000","last_unstale":"2024-03-17T23:48:12.398582+0000","last_undegraded":"2024-03-17T23:48:12.398582+0000","last_fullsized":"2024-03-17T23:48:12.398582+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:30.099054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:13.023079+0000","last_change":"2024-03-17T23:48:11.399236+0000","last_active":"2024-03-17T23:48:13.023079+0000","last_peered":"2024-03-17T23:48:13.023079+0000","last_clean":"2024-03-17T23:48:13.023079+0000","last_became_active":"2024-03-17T23:48:11.398922+0000","last_became_peered":"2024-03-17T23:48:11.398922+0000","last_unstale":"2024-03-17T23:48:13.023079+0000","last_undegraded":"2024-03-17T23:48:13.023079+0000","last_fullsized":"2024-03-17T23:48:13.023079+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:53:53.408503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"39'1","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409546+0000","last_change":"2024-03-17T23:48:10.392858+0000","last_active":"2024-03-17T23:48:11.409546+0000","last_peered":"2024-03-17T23:48:11.409546+0000","last_clean":"2024-03-17T23:48:11.409546+0000","last_became_active":"2024-03-17T23:48:10.392599+0000","last_became_peered":"2024-03-17T23:48:10.392599+0000","last_unstale":"2024-03-17T23:48:11.409546+0000","last_undegraded":"2024-03-17T23:48:11.409546+0000","last_fullsized":"2024-03-17T23:48:11.409546+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:12:23.322115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408725+0000","last_change":"2024-03-17T23:48:11.400180+0000","last_active":"2024-03-17T23:48:11.408725+0000","last_peered":"2024-03-17T23:48:11.408725+0000","last_clean":"2024-03-17T23:48:11.408725+0000","last_became_active":"2024-03-17T23:48:11.399900+0000","last_became_peered":"2024-03-17T23:48:11.399900+0000","last_unstale":"2024-03-17T23:48:11.408725+0000","last_undegraded":"2024-03-17T23:48:11.408725+0000","last_fullsized":"2024-03-17T23:48:11.408725+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:31.101445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"39'1","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.407796+0000","last_change":"2024-03-17T23:48:10.396685+0000","last_active":"2024-03-17T23:48:11.407796+0000","last_peered":"2024-03-17T23:48:11.407796+0000","last_clean":"2024-03-17T23:48:11.407796+0000","last_became_active":"2024-03-17T23:48:10.396218+0000","last_became_peered":"2024-03-17T23:48:10.396218+0000","last_unstale":"2024-03-17T23:48:11.407796+0000","last_undegraded":"2024-03-17T23:48:11.407796+0000","last_fullsized":"2024-03-17T23:48:11.407796+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:07:26.890907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398506+0000","last_change":"2024-03-17T23:48:11.398107+0000","last_active":"2024-03-17T23:48:12.398506+0000","last_peered":"2024-03-17T23:48:12.398506+0000","last_clean":"2024-03-17T23:48:12.398506+0000","last_became_active":"2024-03-17T23:48:11.397847+0000","last_became_peered":"2024-03-17T23:48:11.397847+0000","last_unstale":"2024-03-17T23:48:12.398506+0000","last_undegraded":"2024-03-17T23:48:12.398506+0000","last_fullsized":"2024-03-17T23:48:12.398506+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:47:20.574371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.383921+0000","last_change":"2024-03-17T23:48:10.395355+0000","last_active":"2024-03-17T23:48:11.383921+0000","last_peered":"2024-03-17T23:48:11.383921+0000","last_clean":"2024-03-17T23:48:11.383921+0000","last_became_active":"2024-03-17T23:48:10.395051+0000","last_became_peered":"2024-03-17T23:48:10.395051+0000","last_unstale":"2024-03-17T23:48:11.383921+0000","last_undegraded":"2024-03-17T23:48:11.383921+0000","last_fullsized":"2024-03-17T23:48:11.383921+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:28:10.271909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398591+0000","last_change":"2024-03-17T23:48:11.392405+0000","last_active":"2024-03-17T23:48:12.398591+0000","last_peered":"2024-03-17T23:48:12.398591+0000","last_clean":"2024-03-17T23:48:12.398591+0000","last_became_active":"2024-03-17T23:48:11.391112+0000","last_became_peered":"2024-03-17T23:48:11.391112+0000","last_unstale":"2024-03-17T23:48:12.398591+0000","last_undegraded":"2024-03-17T23:48:12.398591+0000","last_fullsized":"2024-03-17T23:48:12.398591+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:03:40.343506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397331+0000","last_change":"2024-03-17T23:48:10.387055+0000","last_active":"2024-03-17T23:48:12.397331+0000","last_peered":"2024-03-17T23:48:12.397331+0000","last_clean":"2024-03-17T23:48:12.397331+0000","last_became_active":"2024-03-17T23:48:10.386732+0000","last_became_peered":"2024-03-17T23:48:10.386732+0000","last_unstale":"2024-03-17T23:48:12.397331+0000","last_undegraded":"2024-03-17T23:48:12.397331+0000","last_fullsized":"2024-03-17T23:48:12.397331+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:59:28.149170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.824012+0000","last_change":"2024-03-17T23:48:11.396039+0000","last_active":"2024-03-17T23:48:12.824012+0000","last_peered":"2024-03-17T23:48:12.824012+0000","last_clean":"2024-03-17T23:48:12.824012+0000","last_became_active":"2024-03-17T23:48:11.395647+0000","last_became_peered":"2024-03-17T23:48:11.395647+0000","last_unstale":"2024-03-17T23:48:12.824012+0000","last_undegraded":"2024-03-17T23:48:12.824012+0000","last_fullsized":"2024-03-17T23:48:12.824012+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:07:27.893298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.399060+0000","last_change":"2024-03-17T23:48:10.389198+0000","last_active":"2024-03-17T23:48:12.399060+0000","last_peered":"2024-03-17T23:48:12.399060+0000","last_clean":"2024-03-17T23:48:12.399060+0000","last_became_active":"2024-03-17T23:48:10.388927+0000","last_became_peered":"2024-03-17T23:48:10.388927+0000","last_unstale":"2024-03-17T23:48:12.399060+0000","last_undegraded":"2024-03-17T23:48:12.399060+0000","last_fullsized":"2024-03-17T23:48:12.399060+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:47:19.571981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.398574+0000","last_change":"2024-03-17T23:48:11.394613+0000","last_active":"2024-03-17T23:48:11.398574+0000","last_peered":"2024-03-17T23:48:11.398574+0000","last_clean":"2024-03-17T23:48:11.398574+0000","last_became_active":"2024-03-17T23:48:11.394206+0000","last_became_peered":"2024-03-17T23:48:11.394206+0000","last_unstale":"2024-03-17T23:48:11.398574+0000","last_undegraded":"2024-03-17T23:48:11.398574+0000","last_fullsized":"2024-03-17T23:48:11.398574+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:40:54.501510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.389970+0000","last_change":"2024-03-17T23:48:10.394514+0000","last_active":"2024-03-17T23:48:11.389970+0000","last_peered":"2024-03-17T23:48:11.389970+0000","last_clean":"2024-03-17T23:48:11.389970+0000","last_became_active":"2024-03-17T23:48:10.394237+0000","last_became_peered":"2024-03-17T23:48:10.394237+0000","last_unstale":"2024-03-17T23:48:11.389970+0000","last_undegraded":"2024-03-17T23:48:11.389970+0000","last_fullsized":"2024-03-17T23:48:11.389970+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:40:53.499120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:13.023359+0000","last_change":"2024-03-17T23:48:11.394352+0000","last_active":"2024-03-17T23:48:13.023359+0000","last_peered":"2024-03-17T23:48:13.023359+0000","last_clean":"2024-03-17T23:48:13.023359+0000","last_became_active":"2024-03-17T23:48:11.393966+0000","last_became_peered":"2024-03-17T23:48:11.393966+0000","last_unstale":"2024-03-17T23:48:13.023359+0000","last_undegraded":"2024-03-17T23:48:13.023359+0000","last_fullsized":"2024-03-17T23:48:13.023359+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:33:46.207300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"39'3","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409672+0000","last_change":"2024-03-17T23:48:10.400644+0000","last_active":"2024-03-17T23:48:11.409672+0000","last_peered":"2024-03-17T23:48:11.409672+0000","last_clean":"2024-03-17T23:48:11.409672+0000","last_became_active":"2024-03-17T23:48:10.400361+0000","last_became_peered":"2024-03-17T23:48:10.400361+0000","last_unstale":"2024-03-17T23:48:11.409672+0000","last_undegraded":"2024-03-17T23:48:11.409672+0000","last_fullsized":"2024-03-17T23:48:11.409672+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:28:10.271909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398068+0000","last_change":"2024-03-17T23:48:11.404493+0000","last_active":"2024-03-17T23:48:12.398068+0000","last_peered":"2024-03-17T23:48:12.398068+0000","last_clean":"2024-03-17T23:48:12.398068+0000","last_became_active":"2024-03-17T23:48:11.401330+0000","last_became_peered":"2024-03-17T23:48:11.401330+0000","last_unstale":"2024-03-17T23:48:12.398068+0000","last_undegraded":"2024-03-17T23:48:12.398068+0000","last_fullsized":"2024-03-17T23:48:12.398068+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:37:11.855078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"39'6","reported_seq":20,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398838+0000","last_change":"2024-03-17T23:48:10.396077+0000","last_active":"2024-03-17T23:48:12.398838+0000","last_peered":"2024-03-17T23:48:12.398838+0000","last_clean":"2024-03-17T23:48:12.398838+0000","last_became_active":"2024-03-17T23:48:10.395491+0000","last_became_peered":"2024-03-17T23:48:10.395491+0000","last_unstale":"2024-03-17T23:48:12.398838+0000","last_undegraded":"2024-03-17T23:48:12.398838+0000","last_fullsized":"2024-03-17T23:48:12.398838+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:16:08.340158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:13.023082+0000","last_change":"2024-03-17T23:48:11.396412+0000","last_active":"2024-03-17T23:48:13.023082+0000","last_peered":"2024-03-17T23:48:13.023082+0000","last_clean":"2024-03-17T23:48:13.023082+0000","last_became_active":"2024-03-17T23:48:11.396052+0000","last_became_peered":"2024-03-17T23:48:11.396052+0000","last_unstale":"2024-03-17T23:48:13.023082+0000","last_undegraded":"2024-03-17T23:48:13.023082+0000","last_fullsized":"2024-03-17T23:48:13.023082+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:17:53.610029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"39'2","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398764+0000","last_change":"2024-03-17T23:48:10.392240+0000","last_active":"2024-03-17T23:48:12.398764+0000","last_peered":"2024-03-17T23:48:12.398764+0000","last_clean":"2024-03-17T23:48:12.398764+0000","last_became_active":"2024-03-17T23:48:10.391973+0000","last_became_peered":"2024-03-17T23:48:10.391973+0000","last_unstale":"2024-03-17T23:48:12.398764+0000","last_undegraded":"2024-03-17T23:48:12.398764+0000","last_fullsized":"2024-03-17T23:48:12.398764+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:12:22.747059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398590+0000","last_change":"2024-03-17T23:48:11.393482+0000","last_active":"2024-03-17T23:48:12.398590+0000","last_peered":"2024-03-17T23:48:12.398590+0000","last_clean":"2024-03-17T23:48:12.398590+0000","last_became_active":"2024-03-17T23:48:11.393222+0000","last_became_peered":"2024-03-17T23:48:11.393222+0000","last_unstale":"2024-03-17T23:48:12.398590+0000","last_undegraded":"2024-03-17T23:48:12.398590+0000","last_fullsized":"2024-03-17T23:48:12.398590+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:00:33.846091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.385582+0000","last_change":"2024-03-17T23:48:10.394916+0000","last_active":"2024-03-17T23:48:11.385582+0000","last_peered":"2024-03-17T23:48:11.385582+0000","last_clean":"2024-03-17T23:48:11.385582+0000","last_became_active":"2024-03-17T23:48:10.391532+0000","last_became_peered":"2024-03-17T23:48:10.391532+0000","last_unstale":"2024-03-17T23:48:11.385582+0000","last_undegraded":"2024-03-17T23:48:11.385582+0000","last_fullsized":"2024-03-17T23:48:11.385582+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:44.407483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409135+0000","last_change":"2024-03-17T23:48:11.407145+0000","last_active":"2024-03-17T23:48:11.409135+0000","last_peered":"2024-03-17T23:48:11.409135+0000","last_clean":"2024-03-17T23:48:11.409135+0000","last_became_active":"2024-03-17T23:48:11.406851+0000","last_became_peered":"2024-03-17T23:48:11.406851+0000","last_unstale":"2024-03-17T23:48:11.409135+0000","last_undegraded":"2024-03-17T23:48:11.409135+0000","last_fullsized":"2024-03-17T23:48:11.409135+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:59:29.151560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397440+0000","last_change":"2024-03-17T23:48:10.388069+0000","last_active":"2024-03-17T23:48:12.397440+0000","last_peered":"2024-03-17T23:48:12.397440+0000","last_clean":"2024-03-17T23:48:12.397440+0000","last_became_active":"2024-03-17T23:48:10.387765+0000","last_became_peered":"2024-03-17T23:48:10.387765+0000","last_unstale":"2024-03-17T23:48:12.397440+0000","last_undegraded":"2024-03-17T23:48:12.397440+0000","last_fullsized":"2024-03-17T23:48:12.397440+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:15.116223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398662+0000","last_change":"2024-03-17T23:48:11.396828+0000","last_active":"2024-03-17T23:48:12.398662+0000","last_peered":"2024-03-17T23:48:12.398662+0000","last_clean":"2024-03-17T23:48:12.398662+0000","last_became_active":"2024-03-17T23:48:11.396370+0000","last_became_peered":"2024-03-17T23:48:11.396370+0000","last_unstale":"2024-03-17T23:48:12.398662+0000","last_undegraded":"2024-03-17T23:48:12.398662+0000","last_fullsized":"2024-03-17T23:48:12.398662+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:07:28.504390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"39'8","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.400672+0000","last_change":"2024-03-17T23:48:10.397744+0000","last_active":"2024-03-17T23:48:11.400672+0000","last_peered":"2024-03-17T23:48:11.400672+0000","last_clean":"2024-03-17T23:48:11.400672+0000","last_became_active":"2024-03-17T23:48:10.397467+0000","last_became_peered":"2024-03-17T23:48:10.397467+0000","last_unstale":"2024-03-17T23:48:11.400672+0000","last_undegraded":"2024-03-17T23:48:11.400672+0000","last_fullsized":"2024-03-17T23:48:11.400672+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:12:23.322115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398645+0000","last_change":"2024-03-17T23:48:11.390497+0000","last_active":"2024-03-17T23:48:12.398645+0000","last_peered":"2024-03-17T23:48:12.398645+0000","last_clean":"2024-03-17T23:48:12.398645+0000","last_became_active":"2024-03-17T23:48:11.390222+0000","last_became_peered":"2024-03-17T23:48:11.390222+0000","last_unstale":"2024-03-17T23:48:12.398645+0000","last_undegraded":"2024-03-17T23:48:12.398645+0000","last_fullsized":"2024-03-17T23:48:12.398645+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:52:24.543577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"39'1","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409062+0000","last_change":"2024-03-17T23:48:10.393694+0000","last_active":"2024-03-17T23:48:11.409062+0000","last_peered":"2024-03-17T23:48:11.409062+0000","last_clean":"2024-03-17T23:48:11.409062+0000","last_became_active":"2024-03-17T23:48:10.393418+0000","last_became_peered":"2024-03-17T23:48:10.393418+0000","last_unstale":"2024-03-17T23:48:11.409062+0000","last_undegraded":"2024-03-17T23:48:11.409062+0000","last_fullsized":"2024-03-17T23:48:11.409062+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:42:42.774025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408740+0000","last_change":"2024-03-17T23:48:11.400412+0000","last_active":"2024-03-17T23:48:11.408740+0000","last_peered":"2024-03-17T23:48:11.408740+0000","last_clean":"2024-03-17T23:48:11.408740+0000","last_became_active":"2024-03-17T23:48:11.400006+0000","last_became_peered":"2024-03-17T23:48:11.400006+0000","last_unstale":"2024-03-17T23:48:11.408740+0000","last_undegraded":"2024-03-17T23:48:11.408740+0000","last_fullsized":"2024-03-17T23:48:11.408740+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:59:29.151560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"39'2","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398100+0000","last_change":"2024-03-17T23:48:10.391802+0000","last_active":"2024-03-17T23:48:12.398100+0000","last_peered":"2024-03-17T23:48:12.398100+0000","last_clean":"2024-03-17T23:48:12.398100+0000","last_became_active":"2024-03-17T23:48:10.391464+0000","last_became_peered":"2024-03-17T23:48:10.391464+0000","last_unstale":"2024-03-17T23:48:12.398100+0000","last_undegraded":"2024-03-17T23:48:12.398100+0000","last_fullsized":"2024-03-17T23:48:12.398100+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:44:41.528247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397957+0000","last_change":"2024-03-17T23:48:11.399808+0000","last_active":"2024-03-17T23:48:12.397957+0000","last_peered":"2024-03-17T23:48:12.397957+0000","last_clean":"2024-03-17T23:48:12.397957+0000","last_became_active":"2024-03-17T23:48:11.399460+0000","last_became_peered":"2024-03-17T23:48:11.399460+0000","last_unstale":"2024-03-17T23:48:12.397957+0000","last_undegraded":"2024-03-17T23:48:12.397957+0000","last_fullsized":"2024-03-17T23:48:12.397957+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:16:59.276395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"39'1","reported_seq":15,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.823897+0000","last_change":"2024-03-17T23:48:10.396952+0000","last_active":"2024-03-17T23:48:12.823897+0000","last_peered":"2024-03-17T23:48:12.823897+0000","last_clean":"2024-03-17T23:48:12.823897+0000","last_became_active":"2024-03-17T23:48:10.396664+0000","last_became_peered":"2024-03-17T23:48:10.396664+0000","last_unstale":"2024-03-17T23:48:12.823897+0000","last_undegraded":"2024-03-17T23:48:12.823897+0000","last_fullsized":"2024-03-17T23:48:12.823897+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:19:37.436831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.824069+0000","last_change":"2024-03-17T23:48:11.403492+0000","last_active":"2024-03-17T23:48:12.824069+0000","last_peered":"2024-03-17T23:48:12.824069+0000","last_clean":"2024-03-17T23:48:12.824069+0000","last_became_active":"2024-03-17T23:48:11.403203+0000","last_became_peered":"2024-03-17T23:48:11.403203+0000","last_unstale":"2024-03-17T23:48:12.824069+0000","last_undegraded":"2024-03-17T23:48:12.824069+0000","last_fullsized":"2024-03-17T23:48:12.824069+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:31:13.050920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.384478+0000","last_change":"2024-03-17T23:48:10.396820+0000","last_active":"2024-03-17T23:48:11.384478+0000","last_peered":"2024-03-17T23:48:11.384478+0000","last_clean":"2024-03-17T23:48:11.384478+0000","last_became_active":"2024-03-17T23:48:10.396549+0000","last_became_peered":"2024-03-17T23:48:10.396549+0000","last_unstale":"2024-03-17T23:48:11.384478+0000","last_undegraded":"2024-03-17T23:48:11.384478+0000","last_fullsized":"2024-03-17T23:48:11.384478+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:16.355950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408966+0000","last_change":"2024-03-17T23:48:11.403660+0000","last_active":"2024-03-17T23:48:11.408966+0000","last_peered":"2024-03-17T23:48:11.408966+0000","last_clean":"2024-03-17T23:48:11.408966+0000","last_became_active":"2024-03-17T23:48:11.403385+0000","last_became_peered":"2024-03-17T23:48:11.403385+0000","last_unstale":"2024-03-17T23:48:11.408966+0000","last_undegraded":"2024-03-17T23:48:11.408966+0000","last_fullsized":"2024-03-17T23:48:11.408966+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:42:43.776415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"39'2","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.823764+0000","last_change":"2024-03-17T23:48:10.387115+0000","last_active":"2024-03-17T23:48:12.823764+0000","last_peered":"2024-03-17T23:48:12.823764+0000","last_clean":"2024-03-17T23:48:12.823764+0000","last_became_active":"2024-03-17T23:48:10.386894+0000","last_became_peered":"2024-03-17T23:48:10.386894+0000","last_unstale":"2024-03-17T23:48:12.823764+0000","last_undegraded":"2024-03-17T23:48:12.823764+0000","last_fullsized":"2024-03-17T23:48:12.823764+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:25:28.982482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:13.023584+0000","last_change":"2024-03-17T23:48:11.395193+0000","last_active":"2024-03-17T23:48:13.023584+0000","last_peered":"2024-03-17T23:48:13.023584+0000","last_clean":"2024-03-17T23:48:13.023584+0000","last_became_active":"2024-03-17T23:48:11.393952+0000","last_became_peered":"2024-03-17T23:48:11.393952+0000","last_unstale":"2024-03-17T23:48:13.023584+0000","last_undegraded":"2024-03-17T23:48:13.023584+0000","last_fullsized":"2024-03-17T23:48:13.023584+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:16:59.276395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.823862+0000","last_change":"2024-03-17T23:48:10.391728+0000","last_active":"2024-03-17T23:48:12.823862+0000","last_peered":"2024-03-17T23:48:12.823862+0000","last_clean":"2024-03-17T23:48:12.823862+0000","last_became_active":"2024-03-17T23:48:10.391455+0000","last_became_peered":"2024-03-17T23:48:10.391455+0000","last_unstale":"2024-03-17T23:48:12.823862+0000","last_undegraded":"2024-03-17T23:48:12.823862+0000","last_fullsized":"2024-03-17T23:48:12.823862+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:57:20.246583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409211+0000","last_change":"2024-03-17T23:48:11.404518+0000","last_active":"2024-03-17T23:48:11.409211+0000","last_peered":"2024-03-17T23:48:11.409211+0000","last_clean":"2024-03-17T23:48:11.409211+0000","last_became_active":"2024-03-17T23:48:11.404261+0000","last_became_peered":"2024-03-17T23:48:11.404261+0000","last_unstale":"2024-03-17T23:48:11.409211+0000","last_undegraded":"2024-03-17T23:48:11.409211+0000","last_fullsized":"2024-03-17T23:48:11.409211+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:17:53.610029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.385531+0000","last_change":"2024-03-17T23:48:10.397213+0000","last_active":"2024-03-17T23:48:11.385531+0000","last_peered":"2024-03-17T23:48:11.385531+0000","last_clean":"2024-03-17T23:48:11.385531+0000","last_became_active":"2024-03-17T23:48:10.396913+0000","last_became_peered":"2024-03-17T23:48:10.396913+0000","last_unstale":"2024-03-17T23:48:11.385531+0000","last_undegraded":"2024-03-17T23:48:11.385531+0000","last_fullsized":"2024-03-17T23:48:11.385531+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:31:12.048530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398741+0000","last_change":"2024-03-17T23:48:11.393768+0000","last_active":"2024-03-17T23:48:12.398741+0000","last_peered":"2024-03-17T23:48:12.398741+0000","last_clean":"2024-03-17T23:48:12.398741+0000","last_became_active":"2024-03-17T23:48:11.392652+0000","last_became_peered":"2024-03-17T23:48:11.392652+0000","last_unstale":"2024-03-17T23:48:12.398741+0000","last_undegraded":"2024-03-17T23:48:12.398741+0000","last_fullsized":"2024-03-17T23:48:12.398741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:00:12.503035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.823906+0000","last_change":"2024-03-17T23:48:10.387114+0000","last_active":"2024-03-17T23:48:12.823906+0000","last_peered":"2024-03-17T23:48:12.823906+0000","last_clean":"2024-03-17T23:48:12.823906+0000","last_became_active":"2024-03-17T23:48:10.386860+0000","last_became_peered":"2024-03-17T23:48:10.386860+0000","last_unstale":"2024-03-17T23:48:12.823906+0000","last_undegraded":"2024-03-17T23:48:12.823906+0000","last_fullsized":"2024-03-17T23:48:12.823906+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:05:24.288253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.397234+0000","last_change":"2024-03-17T23:48:11.396445+0000","last_active":"2024-03-17T23:48:11.397234+0000","last_peered":"2024-03-17T23:48:11.397234+0000","last_clean":"2024-03-17T23:48:11.397234+0000","last_became_active":"2024-03-17T23:48:11.396189+0000","last_became_peered":"2024-03-17T23:48:11.396189+0000","last_unstale":"2024-03-17T23:48:11.397234+0000","last_undegraded":"2024-03-17T23:48:11.397234+0000","last_fullsized":"2024-03-17T23:48:11.397234+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:10.929912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"39'2","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.406980+0000","last_change":"2024-03-17T23:48:10.400812+0000","last_active":"2024-03-17T23:48:11.406980+0000","last_peered":"2024-03-17T23:48:11.406980+0000","last_clean":"2024-03-17T23:48:11.406980+0000","last_became_active":"2024-03-17T23:48:10.400479+0000","last_became_peered":"2024-03-17T23:48:10.400479+0000","last_unstale":"2024-03-17T23:48:11.406980+0000","last_undegraded":"2024-03-17T23:48:11.406980+0000","last_fullsized":"2024-03-17T23:48:11.406980+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:09.927522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408459+0000","last_change":"2024-03-17T23:48:11.403785+0000","last_active":"2024-03-17T23:48:11.408459+0000","last_peered":"2024-03-17T23:48:11.408459+0000","last_clean":"2024-03-17T23:48:11.408459+0000","last_became_active":"2024-03-17T23:48:11.402301+0000","last_became_peered":"2024-03-17T23:48:11.402301+0000","last_unstale":"2024-03-17T23:48:11.408459+0000","last_undegraded":"2024-03-17T23:48:11.408459+0000","last_fullsized":"2024-03-17T23:48:11.408459+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:31.101445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"40'1","reported_seq":15,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:13.383658+0000","last_change":"2024-03-17T23:48:10.389968+0000","last_active":"2024-03-17T23:48:13.383658+0000","last_peered":"2024-03-17T23:48:13.383658+0000","last_clean":"2024-03-17T23:48:13.383658+0000","last_became_active":"2024-03-17T23:48:10.389694+0000","last_became_peered":"2024-03-17T23:48:10.389694+0000","last_unstale":"2024-03-17T23:48:13.383658+0000","last_undegraded":"2024-03-17T23:48:13.383658+0000","last_fullsized":"2024-03-17T23:48:13.383658+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:49:18.907672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398896+0000","last_change":"2024-03-17T23:48:11.391249+0000","last_active":"2024-03-17T23:48:12.398896+0000","last_peered":"2024-03-17T23:48:12.398896+0000","last_clean":"2024-03-17T23:48:12.398896+0000","last_became_active":"2024-03-17T23:48:11.390836+0000","last_became_peered":"2024-03-17T23:48:11.390836+0000","last_unstale":"2024-03-17T23:48:12.398896+0000","last_undegraded":"2024-03-17T23:48:12.398896+0000","last_fullsized":"2024-03-17T23:48:12.398896+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:51:42.776453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:13.023326+0000","last_change":"2024-03-17T23:48:10.396185+0000","last_active":"2024-03-17T23:48:13.023326+0000","last_peered":"2024-03-17T23:48:13.023326+0000","last_clean":"2024-03-17T23:48:13.023326+0000","last_became_active":"2024-03-17T23:48:10.395932+0000","last_became_peered":"2024-03-17T23:48:10.395932+0000","last_unstale":"2024-03-17T23:48:13.023326+0000","last_undegraded":"2024-03-17T23:48:13.023326+0000","last_fullsized":"2024-03-17T23:48:13.023326+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:16.355950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:13.023230+0000","last_change":"2024-03-17T23:48:11.403894+0000","last_active":"2024-03-17T23:48:13.023230+0000","last_peered":"2024-03-17T23:48:13.023230+0000","last_clean":"2024-03-17T23:48:13.023230+0000","last_became_active":"2024-03-17T23:48:11.403646+0000","last_became_peered":"2024-03-17T23:48:11.403646+0000","last_unstale":"2024-03-17T23:48:13.023230+0000","last_undegraded":"2024-03-17T23:48:13.023230+0000","last_fullsized":"2024-03-17T23:48:13.023230+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:19:33.805891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:13.023386+0000","last_change":"2024-03-17T23:48:10.388194+0000","last_active":"2024-03-17T23:48:13.023386+0000","last_peered":"2024-03-17T23:48:13.023386+0000","last_clean":"2024-03-17T23:48:13.023386+0000","last_became_active":"2024-03-17T23:48:10.387920+0000","last_became_peered":"2024-03-17T23:48:10.387920+0000","last_unstale":"2024-03-17T23:48:13.023386+0000","last_undegraded":"2024-03-17T23:48:13.023386+0000","last_fullsized":"2024-03-17T23:48:13.023386+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:20:47.924674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.397727+0000","last_change":"2024-03-17T23:48:11.397059+0000","last_active":"2024-03-17T23:48:11.397727+0000","last_peered":"2024-03-17T23:48:11.397727+0000","last_clean":"2024-03-17T23:48:11.397727+0000","last_became_active":"2024-03-17T23:48:11.396724+0000","last_became_peered":"2024-03-17T23:48:11.396724+0000","last_unstale":"2024-03-17T23:48:11.397727+0000","last_undegraded":"2024-03-17T23:48:11.397727+0000","last_fullsized":"2024-03-17T23:48:11.397727+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:31.101445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398165+0000","last_change":"2024-03-17T23:48:10.384305+0000","last_active":"2024-03-17T23:48:12.398165+0000","last_peered":"2024-03-17T23:48:12.398165+0000","last_clean":"2024-03-17T23:48:12.398165+0000","last_became_active":"2024-03-17T23:48:10.384034+0000","last_became_peered":"2024-03-17T23:48:10.384034+0000","last_unstale":"2024-03-17T23:48:12.398165+0000","last_undegraded":"2024-03-17T23:48:12.398165+0000","last_fullsized":"2024-03-17T23:48:12.398165+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:32:37.500442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398088+0000","last_change":"2024-03-17T23:48:11.394167+0000","last_active":"2024-03-17T23:48:12.398088+0000","last_peered":"2024-03-17T23:48:12.398088+0000","last_clean":"2024-03-17T23:48:12.398088+0000","last_became_active":"2024-03-17T23:48:11.392337+0000","last_became_peered":"2024-03-17T23:48:11.392337+0000","last_unstale":"2024-03-17T23:48:12.398088+0000","last_undegraded":"2024-03-17T23:48:12.398088+0000","last_fullsized":"2024-03-17T23:48:12.398088+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:04:14.826887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397422+0000","last_change":"2024-03-17T23:48:10.391429+0000","last_active":"2024-03-17T23:48:12.397422+0000","last_peered":"2024-03-17T23:48:12.397422+0000","last_clean":"2024-03-17T23:48:12.397422+0000","last_became_active":"2024-03-17T23:48:10.390974+0000","last_became_peered":"2024-03-17T23:48:10.390974+0000","last_unstale":"2024-03-17T23:48:12.397422+0000","last_undegraded":"2024-03-17T23:48:12.397422+0000","last_fullsized":"2024-03-17T23:48:12.397422+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:04.026997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.399296+0000","last_change":"2024-03-17T23:48:11.397803+0000","last_active":"2024-03-17T23:48:11.399296+0000","last_peered":"2024-03-17T23:48:11.399296+0000","last_clean":"2024-03-17T23:48:11.399296+0000","last_became_active":"2024-03-17T23:48:11.397541+0000","last_became_peered":"2024-03-17T23:48:11.397541+0000","last_unstale":"2024-03-17T23:48:11.399296+0000","last_undegraded":"2024-03-17T23:48:11.399296+0000","last_fullsized":"2024-03-17T23:48:11.399296+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:17.358341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398558+0000","last_change":"2024-03-17T23:48:10.389196+0000","last_active":"2024-03-17T23:48:12.398558+0000","last_peered":"2024-03-17T23:48:12.398558+0000","last_clean":"2024-03-17T23:48:12.398558+0000","last_became_active":"2024-03-17T23:48:10.388918+0000","last_became_peered":"2024-03-17T23:48:10.388918+0000","last_unstale":"2024-03-17T23:48:12.398558+0000","last_undegraded":"2024-03-17T23:48:12.398558+0000","last_fullsized":"2024-03-17T23:48:12.398558+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:38:37.674462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409115+0000","last_change":"2024-03-17T23:48:11.403123+0000","last_active":"2024-03-17T23:48:11.409115+0000","last_peered":"2024-03-17T23:48:11.409115+0000","last_clean":"2024-03-17T23:48:11.409115+0000","last_became_active":"2024-03-17T23:48:11.402851+0000","last_became_peered":"2024-03-17T23:48:11.402851+0000","last_unstale":"2024-03-17T23:48:11.409115+0000","last_undegraded":"2024-03-17T23:48:11.409115+0000","last_fullsized":"2024-03-17T23:48:11.409115+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:44:42.530638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.384095+0000","last_change":"2024-03-17T23:48:10.398461+0000","last_active":"2024-03-17T23:48:11.384095+0000","last_peered":"2024-03-17T23:48:11.384095+0000","last_clean":"2024-03-17T23:48:11.384095+0000","last_became_active":"2024-03-17T23:48:10.398188+0000","last_became_peered":"2024-03-17T23:48:10.398188+0000","last_unstale":"2024-03-17T23:48:11.384095+0000","last_undegraded":"2024-03-17T23:48:11.384095+0000","last_fullsized":"2024-03-17T23:48:11.384095+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:06:33.058080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":204,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.385340+0000","last_change":"2024-03-17T23:46:44.768668+0000","last_active":"2024-03-17T23:48:11.385340+0000","last_peered":"2024-03-17T23:48:11.385340+0000","last_clean":"2024-03-17T23:48:11.385340+0000","last_became_active":"2024-03-17T23:46:44.767826+0000","last_became_peered":"2024-03-17T23:46:44.767826+0000","last_unstale":"2024-03-17T23:48:11.385340+0000","last_undegraded":"2024-03-17T23:48:11.385340+0000","last_fullsized":"2024-03-17T23:48:11.385340+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_clean_scrub_stamp":"2024-03-17T23:46:43.744878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:54:22.286155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":14,"seq":60129542166,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27488,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344352,"statfs":{"total":96636764160,"available":96608616448,"internally_reserved":0,"allocated":475136,"data_stored":202650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542166,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":450560,"data_stored":194382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542166,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343784,"statfs":{"total":96636764160,"available":96608034816,"internally_reserved":0,"allocated":1056768,"data_stored":791406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32212,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339628,"statfs":{"total":96636764160,"available":96603779072,"internally_reserved":0,"allocated":1056768,"data_stored":786362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31636,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340204,"statfs":{"total":96636764160,"available":96604368896,"internally_reserved":0,"allocated":462848,"data_stored":201586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31624,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":454656,"data_stored":194424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31624,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":454656,"data_stored":194916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":450560,"data_stored":194402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-17T23:48:19.010 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-17T23:48:19.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.113+0000 7f0513729640 1 Processor -- start 2024-03-17T23:48:19.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.113+0000 7f0513729640 1 -- start start 2024-03-17T23:48:19.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.114+0000 7f0513729640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f050c1514f0 0x7f050c15bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:19.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.114+0000 7f0513729640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f050c15ab90 0x7f050c06f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:19.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.114+0000 7f0513729640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f050c0ce790 con 0x7f050c06e5c0 2024-03-17T23:48:19.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.114+0000 7f0513729640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f050c06fba0 con 0x7f050c1514f0 2024-03-17T23:48:19.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.114+0000 7f0513729640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f050c15b050 con 0x7f050c15ab90 2024-03-17T23:48:19.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.114+0000 7f05114c5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f050c1514f0 0x7f050c15bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:19.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.114+0000 7f0511cc6640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f050c06e5c0 0x7f050c0cec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:48770/0 (socket says 172.21.15.150:48770) 2024-03-17T23:48:19.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.114+0000 7f0511cc6640 1 -- 172.21.15.150:0/3878359545 learned_addr learned my addr 172.21.15.150:0/3878359545 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:19.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.114+0000 7f0510cc4640 1 --2- 172.21.15.150:0/3878359545 >> v2:172.21.15.157:3300/0 conn(0x7f050c15ab90 0x7f050c06f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:19.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.114+0000 7f05114c5640 1 -- 172.21.15.150:0/3878359545 >> v2:172.21.15.157:3300/0 conn(0x7f050c15ab90 msgr2=0x7f050c06f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:19.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.114+0000 7f05114c5640 1 --2- 172.21.15.150:0/3878359545 >> v2:172.21.15.157:3300/0 conn(0x7f050c15ab90 0x7f050c06f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.114+0000 7f05114c5640 1 -- 172.21.15.150:0/3878359545 >> v1:172.21.15.150:6789/0 conn(0x7f050c06e5c0 legacy=0x7f050c0cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:19.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.114+0000 7f0503fff640 1 -- 172.21.15.150:0/3878359545 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4026260498 0 0) 0x7f0504001490 con 0x7f050c06e5c0 2024-03-17T23:48:19.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.114+0000 7f05114c5640 1 -- 172.21.15.150:0/3878359545 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f050c15b330 con 0x7f050c1514f0 2024-03-17T23:48:19.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.114+0000 7f0510cc4640 1 --2- 172.21.15.150:0/3878359545 >> v2:172.21.15.157:3300/0 conn(0x7f050c15ab90 0x7f050c06f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:19.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.115+0000 7f05114c5640 1 --2- 172.21.15.150:0/3878359545 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f050c1514f0 0x7f050c15bd10 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f050c06f2d0 tx=0x7f050802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f95fc83019243fce server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:19.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.115+0000 7f0503fff640 1 -- 172.21.15.150:0/3878359545 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0508035020 con 0x7f050c1514f0 2024-03-17T23:48:19.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.115+0000 7f0503fff640 1 -- 172.21.15.150:0/3878359545 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0508031830 con 0x7f050c1514f0 2024-03-17T23:48:19.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.115+0000 7f0503fff640 1 -- 172.21.15.150:0/3878359545 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0508031c00 con 0x7f050c1514f0 2024-03-17T23:48:19.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.116+0000 7f0513729640 1 -- 172.21.15.150:0/3878359545 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f050c1514f0 msgr2=0x7f050c15bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:19.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.116+0000 7f0513729640 1 --2- 172.21.15.150:0/3878359545 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f050c1514f0 0x7f050c15bd10 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f050c06f2d0 tx=0x7f050802b8d0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.116+0000 7f0513729640 1 -- 172.21.15.150:0/3878359545 shutdown_connections 2024-03-17T23:48:19.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.116+0000 7f0513729640 1 --2- 172.21.15.150:0/3878359545 >> v2:172.21.15.157:3300/0 conn(0x7f050c15ab90 0x7f050c06f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.116+0000 7f0513729640 1 --2- 172.21.15.150:0/3878359545 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f050c1514f0 0x7f050c15bd10 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.116+0000 7f0513729640 1 -- 172.21.15.150:0/3878359545 >> 172.21.15.150:0/3878359545 conn(0x7f050c06b190 msgr2=0x7f050c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:19.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.116+0000 7f0513729640 1 -- 172.21.15.150:0/3878359545 shutdown_connections 2024-03-17T23:48:19.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.116+0000 7f0513729640 1 -- 172.21.15.150:0/3878359545 wait complete. 2024-03-17T23:48:19.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.118+0000 7f0513729640 1 Processor -- start 2024-03-17T23:48:19.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.118+0000 7f0513729640 1 -- start start 2024-03-17T23:48:19.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.119+0000 7f0513729640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f050c06e5c0 0x7f050c138840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:19.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.119+0000 7f0513729640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f050c1514f0 0x7f050c138f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:19.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.119+0000 7f0513729640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f050c15ab90 0x7f050c149df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:19.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.119+0000 7f0513729640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f050c0af510 con 0x7f050c15ab90 2024-03-17T23:48:19.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.119+0000 7f0513729640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f050c15b8f0 con 0x7f050c06e5c0 2024-03-17T23:48:19.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.119+0000 7f0513729640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f050c1518d0 con 0x7f050c1514f0 2024-03-17T23:48:19.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.119+0000 7f0511cc6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f050c15ab90 0x7f050c149df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:19.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.119+0000 7f0511cc6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f050c15ab90 0x7f050c149df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36638/0 (socket says 172.21.15.150:36638) 2024-03-17T23:48:19.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.119+0000 7f0511cc6640 1 -- 172.21.15.150:0/2898455778 learned_addr learned my addr 172.21.15.150:0/2898455778 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:19.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.120+0000 7f05114c5640 1 --2- 172.21.15.150:0/2898455778 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f050c06e5c0 0x7f050c138840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:19.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.120+0000 7f0510cc4640 1 --2- 172.21.15.150:0/2898455778 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f050c1514f0 0x7f050c138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:19.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.120+0000 7f0511cc6640 1 -- 172.21.15.150:0/2898455778 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f050c1514f0 msgr2=0x7f050c138f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:19.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.120+0000 7f0511cc6640 1 --2- 172.21.15.150:0/2898455778 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f050c1514f0 0x7f050c138f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.120+0000 7f0511cc6640 1 -- 172.21.15.150:0/2898455778 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f050c06e5c0 msgr2=0x7f050c138840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:19.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.120+0000 7f0511cc6640 1 --2- 172.21.15.150:0/2898455778 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f050c06e5c0 0x7f050c138840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.120+0000 7f0511cc6640 1 -- 172.21.15.150:0/2898455778 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f050c149330 con 0x7f050c15ab90 2024-03-17T23:48:19.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.120+0000 7f0511cc6640 1 --2- 172.21.15.150:0/2898455778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f050c15ab90 0x7f050c149df0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f050400c700 tx=0x7f050400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:19.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.120+0000 7f0501ffb640 1 -- 172.21.15.150:0/2898455778 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0504018070 con 0x7f050c15ab90 2024-03-17T23:48:19.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.120+0000 7f0501ffb640 1 -- 172.21.15.150:0/2898455778 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f050400ce10 con 0x7f050c15ab90 2024-03-17T23:48:19.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.120+0000 7f0501ffb640 1 -- 172.21.15.150:0/2898455778 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f050400a800 con 0x7f050c15ab90 2024-03-17T23:48:19.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.120+0000 7f0513729640 1 -- 172.21.15.150:0/2898455778 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f050c13bb90 con 0x7f050c15ab90 2024-03-17T23:48:19.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.120+0000 7f0513729640 1 -- 172.21.15.150:0/2898455778 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f050c1394d0 con 0x7f050c15ab90 2024-03-17T23:48:19.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.121+0000 7f0513729640 1 -- 172.21.15.150:0/2898455778 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f050c0681a0 con 0x7f050c15ab90 2024-03-17T23:48:19.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.122+0000 7f0501ffb640 1 -- 172.21.15.150:0/2898455778 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f050401b070 con 0x7f050c15ab90 2024-03-17T23:48:19.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.122+0000 7f0501ffb640 1 --2- 172.21.15.150:0/2898455778 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f04f006bbd0 0x7f04f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:19.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.123+0000 7f0501ffb640 1 -- 172.21.15.150:0/2898455778 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7f0504089fe0 con 0x7f050c15ab90 2024-03-17T23:48:19.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.123+0000 7f05114c5640 1 --2- 172.21.15.150:0/2898455778 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f04f006bbd0 0x7f04f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:19.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.123+0000 7f05114c5640 1 --2- 172.21.15.150:0/2898455778 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f04f006bbd0 0x7f04f006e090 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0508036000 tx=0x7f0508008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:19.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.131+0000 7f0501ffb640 1 -- 172.21.15.150:0/2898455778 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f050c0ce790 con 0x7f050c15ab90 2024-03-17T23:48:19.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.306+0000 7f0513729640 1 -- 172.21.15.150:0/2898455778 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f050c13be70 con 0x7f04f006bbd0 2024-03-17T23:48:19.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.311+0000 7f0501ffb640 1 -- 172.21.15.150:0/2898455778 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251931 (secure 0 0 0) 0x7f0508087830 con 0x7f04f006bbd0 2024-03-17T23:48:19.314 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:19.316 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-03-17T23:48:19.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.314+0000 7f0513729640 1 -- 172.21.15.150:0/2898455778 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f04f006bbd0 msgr2=0x7f04f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:19.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.314+0000 7f0513729640 1 --2- 172.21.15.150:0/2898455778 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f04f006bbd0 0x7f04f006e090 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0508036000 tx=0x7f0508008030 comp rx=0 tx=0).stop 2024-03-17T23:48:19.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.314+0000 7f0513729640 1 -- 172.21.15.150:0/2898455778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f050c15ab90 msgr2=0x7f050c149df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:19.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.314+0000 7f0513729640 1 --2- 172.21.15.150:0/2898455778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f050c15ab90 0x7f050c149df0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f050400c700 tx=0x7f050400cbc0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.314+0000 7f0513729640 1 -- 172.21.15.150:0/2898455778 shutdown_connections 2024-03-17T23:48:19.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.314+0000 7f0513729640 1 --2- 172.21.15.150:0/2898455778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f050c15ab90 0x7f050c149df0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.314+0000 7f0513729640 1 --2- 172.21.15.150:0/2898455778 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f050c1514f0 0x7f050c138f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.314+0000 7f0513729640 1 --2- 172.21.15.150:0/2898455778 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f04f006bbd0 0x7f04f006e090 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.314+0000 7f0513729640 1 --2- 172.21.15.150:0/2898455778 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f050c06e5c0 0x7f050c138840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.314+0000 7f0513729640 1 -- 172.21.15.150:0/2898455778 >> 172.21.15.150:0/2898455778 conn(0x7f050c06b190 msgr2=0x7f050c071cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:19.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.315+0000 7f0513729640 1 -- 172.21.15.150:0/2898455778 shutdown_connections 2024-03-17T23:48:19.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.315+0000 7f0513729640 1 -- 172.21.15.150:0/2898455778 wait complete. 2024-03-17T23:48:19.326 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2024-03-17T23:48:17.613745+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237560,"kb_used_data":4748,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737160,"statfs":{"total":773094113280,"available":772850851840,"internally_reserved":0,"allocated":4861952,"data_stored":2760128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.237688"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.399005+0000","last_change":"2024-03-17T23:48:11.394564+0000","last_active":"2024-03-17T23:48:12.399005+0000","last_peered":"2024-03-17T23:48:12.399005+0000","last_clean":"2024-03-17T23:48:12.399005+0000","last_became_active":"2024-03-17T23:48:11.394313+0000","last_became_peered":"2024-03-17T23:48:11.394313+0000","last_unstale":"2024-03-17T23:48:12.399005+0000","last_undegraded":"2024-03-17T23:48:12.399005+0000","last_fullsized":"2024-03-17T23:48:12.399005+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:50:41.482483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398864+0000","last_change":"2024-03-17T23:48:11.390494+0000","last_active":"2024-03-17T23:48:12.398864+0000","last_peered":"2024-03-17T23:48:12.398864+0000","last_clean":"2024-03-17T23:48:12.398864+0000","last_became_active":"2024-03-17T23:48:11.390235+0000","last_became_peered":"2024-03-17T23:48:11.390235+0000","last_unstale":"2024-03-17T23:48:12.398864+0000","last_undegraded":"2024-03-17T23:48:12.398864+0000","last_fullsized":"2024-03-17T23:48:12.398864+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:20:58.243543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:13.023233+0000","last_change":"2024-03-17T23:48:11.396000+0000","last_active":"2024-03-17T23:48:13.023233+0000","last_peered":"2024-03-17T23:48:13.023233+0000","last_clean":"2024-03-17T23:48:13.023233+0000","last_became_active":"2024-03-17T23:48:11.395653+0000","last_became_peered":"2024-03-17T23:48:11.395653+0000","last_unstale":"2024-03-17T23:48:13.023233+0000","last_undegraded":"2024-03-17T23:48:13.023233+0000","last_fullsized":"2024-03-17T23:48:13.023233+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:56:06.470585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.823942+0000","last_change":"2024-03-17T23:48:11.396619+0000","last_active":"2024-03-17T23:48:12.823942+0000","last_peered":"2024-03-17T23:48:12.823942+0000","last_clean":"2024-03-17T23:48:12.823942+0000","last_became_active":"2024-03-17T23:48:11.396382+0000","last_became_peered":"2024-03-17T23:48:11.396382+0000","last_unstale":"2024-03-17T23:48:12.823942+0000","last_undegraded":"2024-03-17T23:48:12.823942+0000","last_fullsized":"2024-03-17T23:48:12.823942+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:06:34.060471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.823934+0000","last_change":"2024-03-17T23:48:11.402400+0000","last_active":"2024-03-17T23:48:12.823934+0000","last_peered":"2024-03-17T23:48:12.823934+0000","last_clean":"2024-03-17T23:48:12.823934+0000","last_became_active":"2024-03-17T23:48:11.402145+0000","last_became_peered":"2024-03-17T23:48:11.402145+0000","last_unstale":"2024-03-17T23:48:12.823934+0000","last_undegraded":"2024-03-17T23:48:12.823934+0000","last_fullsized":"2024-03-17T23:48:12.823934+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:32:38.502832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.400417+0000","last_change":"2024-03-17T23:48:11.399627+0000","last_active":"2024-03-17T23:48:11.400417+0000","last_peered":"2024-03-17T23:48:11.400417+0000","last_clean":"2024-03-17T23:48:11.400417+0000","last_became_active":"2024-03-17T23:48:11.399364+0000","last_became_peered":"2024-03-17T23:48:11.399364+0000","last_unstale":"2024-03-17T23:48:11.400417+0000","last_undegraded":"2024-03-17T23:48:11.400417+0000","last_fullsized":"2024-03-17T23:48:11.400417+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:44:42.530638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409171+0000","last_change":"2024-03-17T23:48:11.403822+0000","last_active":"2024-03-17T23:48:11.409171+0000","last_peered":"2024-03-17T23:48:11.409171+0000","last_clean":"2024-03-17T23:48:11.409171+0000","last_became_active":"2024-03-17T23:48:11.403572+0000","last_became_peered":"2024-03-17T23:48:11.403572+0000","last_unstale":"2024-03-17T23:48:11.409171+0000","last_undegraded":"2024-03-17T23:48:11.409171+0000","last_fullsized":"2024-03-17T23:48:11.409171+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:53:53.408503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398516+0000","last_change":"2024-03-17T23:48:11.397742+0000","last_active":"2024-03-17T23:48:12.398516+0000","last_peered":"2024-03-17T23:48:12.398516+0000","last_clean":"2024-03-17T23:48:12.398516+0000","last_became_active":"2024-03-17T23:48:11.397483+0000","last_became_peered":"2024-03-17T23:48:11.397483+0000","last_unstale":"2024-03-17T23:48:12.398516+0000","last_undegraded":"2024-03-17T23:48:12.398516+0000","last_fullsized":"2024-03-17T23:48:12.398516+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:41:05.763125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:13.023637+0000","last_change":"2024-03-17T23:48:11.407925+0000","last_active":"2024-03-17T23:48:13.023637+0000","last_peered":"2024-03-17T23:48:13.023637+0000","last_clean":"2024-03-17T23:48:13.023637+0000","last_became_active":"2024-03-17T23:48:11.407612+0000","last_became_peered":"2024-03-17T23:48:11.407612+0000","last_unstale":"2024-03-17T23:48:13.023637+0000","last_undegraded":"2024-03-17T23:48:13.023637+0000","last_fullsized":"2024-03-17T23:48:13.023637+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:37:11.855078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.406658+0000","last_change":"2024-03-17T23:48:11.395918+0000","last_active":"2024-03-17T23:48:11.406658+0000","last_peered":"2024-03-17T23:48:11.406658+0000","last_clean":"2024-03-17T23:48:11.406658+0000","last_became_active":"2024-03-17T23:48:11.394963+0000","last_became_peered":"2024-03-17T23:48:11.394963+0000","last_unstale":"2024-03-17T23:48:11.406658+0000","last_undegraded":"2024-03-17T23:48:11.406658+0000","last_fullsized":"2024-03-17T23:48:11.406658+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:20:48.927064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.399665+0000","last_change":"2024-03-17T23:48:11.394537+0000","last_active":"2024-03-17T23:48:11.399665+0000","last_peered":"2024-03-17T23:48:11.399665+0000","last_clean":"2024-03-17T23:48:11.399665+0000","last_became_active":"2024-03-17T23:48:11.394147+0000","last_became_peered":"2024-03-17T23:48:11.394147+0000","last_unstale":"2024-03-17T23:48:11.399665+0000","last_undegraded":"2024-03-17T23:48:11.399665+0000","last_fullsized":"2024-03-17T23:48:11.399665+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:45.409873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:13.023223+0000","last_change":"2024-03-17T23:48:11.402407+0000","last_active":"2024-03-17T23:48:13.023223+0000","last_peered":"2024-03-17T23:48:13.023223+0000","last_clean":"2024-03-17T23:48:13.023223+0000","last_became_active":"2024-03-17T23:48:11.402140+0000","last_became_peered":"2024-03-17T23:48:11.402140+0000","last_unstale":"2024-03-17T23:48:13.023223+0000","last_undegraded":"2024-03-17T23:48:13.023223+0000","last_fullsized":"2024-03-17T23:48:13.023223+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:03:52.711284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.394641+0000","last_change":"2024-03-17T23:48:11.393682+0000","last_active":"2024-03-17T23:48:11.394641+0000","last_peered":"2024-03-17T23:48:11.394641+0000","last_clean":"2024-03-17T23:48:11.394641+0000","last_became_active":"2024-03-17T23:48:11.393410+0000","last_became_peered":"2024-03-17T23:48:11.393410+0000","last_unstale":"2024-03-17T23:48:11.394641+0000","last_undegraded":"2024-03-17T23:48:11.394641+0000","last_fullsized":"2024-03-17T23:48:11.394641+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:17.358341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397913+0000","last_change":"2024-03-17T23:48:11.405838+0000","last_active":"2024-03-17T23:48:12.397913+0000","last_peered":"2024-03-17T23:48:12.397913+0000","last_clean":"2024-03-17T23:48:12.397913+0000","last_became_active":"2024-03-17T23:48:11.405573+0000","last_became_peered":"2024-03-17T23:48:11.405573+0000","last_unstale":"2024-03-17T23:48:12.397913+0000","last_undegraded":"2024-03-17T23:48:12.397913+0000","last_fullsized":"2024-03-17T23:48:12.397913+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:33:46.207300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409150+0000","last_change":"2024-03-17T23:48:11.403246+0000","last_active":"2024-03-17T23:48:11.409150+0000","last_peered":"2024-03-17T23:48:11.409150+0000","last_clean":"2024-03-17T23:48:11.409150+0000","last_became_active":"2024-03-17T23:48:11.402986+0000","last_became_peered":"2024-03-17T23:48:11.402986+0000","last_unstale":"2024-03-17T23:48:11.409150+0000","last_undegraded":"2024-03-17T23:48:11.409150+0000","last_fullsized":"2024-03-17T23:48:11.409150+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:16:44.071095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.402167+0000","last_change":"2024-03-17T23:48:11.398200+0000","last_active":"2024-03-17T23:48:11.402167+0000","last_peered":"2024-03-17T23:48:11.402167+0000","last_clean":"2024-03-17T23:48:11.402167+0000","last_became_active":"2024-03-17T23:48:11.397937+0000","last_became_peered":"2024-03-17T23:48:11.397937+0000","last_unstale":"2024-03-17T23:48:11.402167+0000","last_undegraded":"2024-03-17T23:48:11.402167+0000","last_fullsized":"2024-03-17T23:48:11.402167+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:45.409873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409193+0000","last_change":"2024-03-17T23:48:11.408181+0000","last_active":"2024-03-17T23:48:11.409193+0000","last_peered":"2024-03-17T23:48:11.409193+0000","last_clean":"2024-03-17T23:48:11.409193+0000","last_became_active":"2024-03-17T23:48:11.407876+0000","last_became_peered":"2024-03-17T23:48:11.407876+0000","last_unstale":"2024-03-17T23:48:11.409193+0000","last_undegraded":"2024-03-17T23:48:11.409193+0000","last_fullsized":"2024-03-17T23:48:11.409193+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:05.029387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398135+0000","last_change":"2024-03-17T23:48:11.402141+0000","last_active":"2024-03-17T23:48:12.398135+0000","last_peered":"2024-03-17T23:48:12.398135+0000","last_clean":"2024-03-17T23:48:12.398135+0000","last_became_active":"2024-03-17T23:48:11.401870+0000","last_became_peered":"2024-03-17T23:48:11.401870+0000","last_unstale":"2024-03-17T23:48:12.398135+0000","last_undegraded":"2024-03-17T23:48:12.398135+0000","last_fullsized":"2024-03-17T23:48:12.398135+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:10.929912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.823828+0000","last_change":"2024-03-17T23:48:11.401430+0000","last_active":"2024-03-17T23:48:12.823828+0000","last_peered":"2024-03-17T23:48:12.823828+0000","last_clean":"2024-03-17T23:48:12.823828+0000","last_became_active":"2024-03-17T23:48:11.401220+0000","last_became_peered":"2024-03-17T23:48:11.401220+0000","last_unstale":"2024-03-17T23:48:12.823828+0000","last_undegraded":"2024-03-17T23:48:12.823828+0000","last_fullsized":"2024-03-17T23:48:12.823828+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:55.443705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398329+0000","last_change":"2024-03-17T23:48:11.401263+0000","last_active":"2024-03-17T23:48:12.398329+0000","last_peered":"2024-03-17T23:48:12.398329+0000","last_clean":"2024-03-17T23:48:12.398329+0000","last_became_active":"2024-03-17T23:48:11.400951+0000","last_became_peered":"2024-03-17T23:48:11.400951+0000","last_unstale":"2024-03-17T23:48:12.398329+0000","last_undegraded":"2024-03-17T23:48:12.398329+0000","last_fullsized":"2024-03-17T23:48:12.398329+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:42:43.776415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408759+0000","last_change":"2024-03-17T23:48:11.400746+0000","last_active":"2024-03-17T23:48:11.408759+0000","last_peered":"2024-03-17T23:48:11.408759+0000","last_clean":"2024-03-17T23:48:11.408759+0000","last_became_active":"2024-03-17T23:48:11.400494+0000","last_became_peered":"2024-03-17T23:48:11.400494+0000","last_unstale":"2024-03-17T23:48:11.408759+0000","last_undegraded":"2024-03-17T23:48:11.408759+0000","last_fullsized":"2024-03-17T23:48:11.408759+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:05.029387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398311+0000","last_change":"2024-03-17T23:48:11.403046+0000","last_active":"2024-03-17T23:48:12.398311+0000","last_peered":"2024-03-17T23:48:12.398311+0000","last_clean":"2024-03-17T23:48:12.398311+0000","last_became_active":"2024-03-17T23:48:11.402769+0000","last_became_peered":"2024-03-17T23:48:11.402769+0000","last_unstale":"2024-03-17T23:48:12.398311+0000","last_undegraded":"2024-03-17T23:48:12.398311+0000","last_fullsized":"2024-03-17T23:48:12.398311+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:16:09.342548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408364+0000","last_change":"2024-03-17T23:48:11.402963+0000","last_active":"2024-03-17T23:48:11.408364+0000","last_peered":"2024-03-17T23:48:11.408364+0000","last_clean":"2024-03-17T23:48:11.408364+0000","last_became_active":"2024-03-17T23:48:11.402669+0000","last_became_peered":"2024-03-17T23:48:11.402669+0000","last_unstale":"2024-03-17T23:48:11.408364+0000","last_undegraded":"2024-03-17T23:48:11.408364+0000","last_fullsized":"2024-03-17T23:48:11.408364+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:20:48.927064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.823768+0000","last_change":"2024-03-17T23:48:11.392903+0000","last_active":"2024-03-17T23:48:12.823768+0000","last_peered":"2024-03-17T23:48:12.823768+0000","last_clean":"2024-03-17T23:48:12.823768+0000","last_became_active":"2024-03-17T23:48:11.392621+0000","last_became_peered":"2024-03-17T23:48:11.392621+0000","last_unstale":"2024-03-17T23:48:12.823768+0000","last_undegraded":"2024-03-17T23:48:12.823768+0000","last_fullsized":"2024-03-17T23:48:12.823768+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:16.118613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398382+0000","last_change":"2024-03-17T23:48:11.392442+0000","last_active":"2024-03-17T23:48:12.398382+0000","last_peered":"2024-03-17T23:48:12.398382+0000","last_clean":"2024-03-17T23:48:12.398382+0000","last_became_active":"2024-03-17T23:48:11.392184+0000","last_became_peered":"2024-03-17T23:48:11.392184+0000","last_unstale":"2024-03-17T23:48:12.398382+0000","last_undegraded":"2024-03-17T23:48:12.398382+0000","last_fullsized":"2024-03-17T23:48:12.398382+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:31.101445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.399825+0000","last_change":"2024-03-17T23:48:11.398013+0000","last_active":"2024-03-17T23:48:11.399825+0000","last_peered":"2024-03-17T23:48:11.399825+0000","last_clean":"2024-03-17T23:48:11.399825+0000","last_became_active":"2024-03-17T23:48:11.397709+0000","last_became_peered":"2024-03-17T23:48:11.397709+0000","last_unstale":"2024-03-17T23:48:11.399825+0000","last_undegraded":"2024-03-17T23:48:11.399825+0000","last_fullsized":"2024-03-17T23:48:11.399825+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:05.029387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397892+0000","last_change":"2024-03-17T23:48:11.404602+0000","last_active":"2024-03-17T23:48:12.397892+0000","last_peered":"2024-03-17T23:48:12.397892+0000","last_clean":"2024-03-17T23:48:12.397892+0000","last_became_active":"2024-03-17T23:48:11.404351+0000","last_became_peered":"2024-03-17T23:48:11.404351+0000","last_unstale":"2024-03-17T23:48:12.397892+0000","last_undegraded":"2024-03-17T23:48:12.397892+0000","last_fullsized":"2024-03-17T23:48:12.397892+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:45.409873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409206+0000","last_change":"2024-03-17T23:48:11.403804+0000","last_active":"2024-03-17T23:48:11.409206+0000","last_peered":"2024-03-17T23:48:11.409206+0000","last_clean":"2024-03-17T23:48:11.409206+0000","last_became_active":"2024-03-17T23:48:11.403537+0000","last_became_peered":"2024-03-17T23:48:11.403537+0000","last_unstale":"2024-03-17T23:48:11.409206+0000","last_undegraded":"2024-03-17T23:48:11.409206+0000","last_fullsized":"2024-03-17T23:48:11.409206+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:32:24.643171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397774+0000","last_change":"2024-03-17T23:48:11.398674+0000","last_active":"2024-03-17T23:48:12.397774+0000","last_peered":"2024-03-17T23:48:12.397774+0000","last_clean":"2024-03-17T23:48:12.397774+0000","last_became_active":"2024-03-17T23:48:11.398399+0000","last_became_peered":"2024-03-17T23:48:11.398399+0000","last_unstale":"2024-03-17T23:48:12.397774+0000","last_undegraded":"2024-03-17T23:48:12.397774+0000","last_fullsized":"2024-03-17T23:48:12.397774+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:17.358341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398058+0000","last_change":"2024-03-17T23:48:11.393129+0000","last_active":"2024-03-17T23:48:12.398058+0000","last_peered":"2024-03-17T23:48:12.398058+0000","last_clean":"2024-03-17T23:48:12.398058+0000","last_became_active":"2024-03-17T23:48:11.392847+0000","last_became_peered":"2024-03-17T23:48:11.392847+0000","last_unstale":"2024-03-17T23:48:12.398058+0000","last_undegraded":"2024-03-17T23:48:12.398058+0000","last_fullsized":"2024-03-17T23:48:12.398058+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:20:48.927064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408386+0000","last_change":"2024-03-17T23:48:11.403541+0000","last_active":"2024-03-17T23:48:11.408386+0000","last_peered":"2024-03-17T23:48:11.408386+0000","last_clean":"2024-03-17T23:48:11.408386+0000","last_became_active":"2024-03-17T23:48:11.403273+0000","last_became_peered":"2024-03-17T23:48:11.403273+0000","last_unstale":"2024-03-17T23:48:11.408386+0000","last_undegraded":"2024-03-17T23:48:11.408386+0000","last_fullsized":"2024-03-17T23:48:11.408386+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:10.929912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.397462+0000","last_change":"2024-03-17T23:48:11.396799+0000","last_active":"2024-03-17T23:48:11.397462+0000","last_peered":"2024-03-17T23:48:11.397462+0000","last_clean":"2024-03-17T23:48:11.397462+0000","last_became_active":"2024-03-17T23:48:11.396343+0000","last_became_peered":"2024-03-17T23:48:11.396343+0000","last_unstale":"2024-03-17T23:48:11.397462+0000","last_undegraded":"2024-03-17T23:48:11.397462+0000","last_fullsized":"2024-03-17T23:48:11.397462+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:42:43.776415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.399567+0000","last_change":"2024-03-17T23:48:11.397507+0000","last_active":"2024-03-17T23:48:11.399567+0000","last_peered":"2024-03-17T23:48:11.399567+0000","last_clean":"2024-03-17T23:48:11.399567+0000","last_became_active":"2024-03-17T23:48:11.397238+0000","last_became_peered":"2024-03-17T23:48:11.397238+0000","last_unstale":"2024-03-17T23:48:11.399567+0000","last_undegraded":"2024-03-17T23:48:11.399567+0000","last_fullsized":"2024-03-17T23:48:11.399567+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:28:11.274300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"39'4","reported_seq":18,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398478+0000","last_change":"2024-03-17T23:48:10.382890+0000","last_active":"2024-03-17T23:48:12.398478+0000","last_peered":"2024-03-17T23:48:12.398478+0000","last_clean":"2024-03-17T23:48:12.398478+0000","last_became_active":"2024-03-17T23:48:10.382565+0000","last_became_peered":"2024-03-17T23:48:10.382565+0000","last_unstale":"2024-03-17T23:48:12.398478+0000","last_undegraded":"2024-03-17T23:48:12.398478+0000","last_fullsized":"2024-03-17T23:48:12.398478+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:21:54.441315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.401433+0000","last_change":"2024-03-17T23:48:11.397742+0000","last_active":"2024-03-17T23:48:11.401433+0000","last_peered":"2024-03-17T23:48:11.401433+0000","last_clean":"2024-03-17T23:48:11.401433+0000","last_became_active":"2024-03-17T23:48:11.397189+0000","last_became_peered":"2024-03-17T23:48:11.397189+0000","last_unstale":"2024-03-17T23:48:11.401433+0000","last_undegraded":"2024-03-17T23:48:11.401433+0000","last_fullsized":"2024-03-17T23:48:11.401433+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:59:29.151560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"39'1","reported_seq":15,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:13.023457+0000","last_change":"2024-03-17T23:48:10.395772+0000","last_active":"2024-03-17T23:48:13.023457+0000","last_peered":"2024-03-17T23:48:13.023457+0000","last_clean":"2024-03-17T23:48:13.023457+0000","last_became_active":"2024-03-17T23:48:10.395501+0000","last_became_peered":"2024-03-17T23:48:10.395501+0000","last_unstale":"2024-03-17T23:48:13.023457+0000","last_undegraded":"2024-03-17T23:48:13.023457+0000","last_fullsized":"2024-03-17T23:48:13.023457+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:44.407483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398181+0000","last_change":"2024-03-17T23:48:11.406337+0000","last_active":"2024-03-17T23:48:12.398181+0000","last_peered":"2024-03-17T23:48:12.398181+0000","last_clean":"2024-03-17T23:48:12.398181+0000","last_became_active":"2024-03-17T23:48:11.405094+0000","last_became_peered":"2024-03-17T23:48:11.405094+0000","last_unstale":"2024-03-17T23:48:12.398181+0000","last_undegraded":"2024-03-17T23:48:12.398181+0000","last_fullsized":"2024-03-17T23:48:12.398181+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:38:38.676853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"39'2","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.406939+0000","last_change":"2024-03-17T23:48:10.396210+0000","last_active":"2024-03-17T23:48:11.406939+0000","last_peered":"2024-03-17T23:48:11.406939+0000","last_clean":"2024-03-17T23:48:11.406939+0000","last_became_active":"2024-03-17T23:48:10.395955+0000","last_became_peered":"2024-03-17T23:48:10.395955+0000","last_unstale":"2024-03-17T23:48:11.406939+0000","last_undegraded":"2024-03-17T23:48:11.406939+0000","last_fullsized":"2024-03-17T23:48:11.406939+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:20:47.924674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.399005+0000","last_change":"2024-03-17T23:48:11.392163+0000","last_active":"2024-03-17T23:48:12.399005+0000","last_peered":"2024-03-17T23:48:12.399005+0000","last_clean":"2024-03-17T23:48:12.399005+0000","last_became_active":"2024-03-17T23:48:11.391029+0000","last_became_peered":"2024-03-17T23:48:11.391029+0000","last_unstale":"2024-03-17T23:48:12.399005+0000","last_undegraded":"2024-03-17T23:48:12.399005+0000","last_fullsized":"2024-03-17T23:48:12.399005+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:05:55.540409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"39'2","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398582+0000","last_change":"2024-03-17T23:48:10.391635+0000","last_active":"2024-03-17T23:48:12.398582+0000","last_peered":"2024-03-17T23:48:12.398582+0000","last_clean":"2024-03-17T23:48:12.398582+0000","last_became_active":"2024-03-17T23:48:10.391241+0000","last_became_peered":"2024-03-17T23:48:10.391241+0000","last_unstale":"2024-03-17T23:48:12.398582+0000","last_undegraded":"2024-03-17T23:48:12.398582+0000","last_fullsized":"2024-03-17T23:48:12.398582+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:30.099054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:13.023079+0000","last_change":"2024-03-17T23:48:11.399236+0000","last_active":"2024-03-17T23:48:13.023079+0000","last_peered":"2024-03-17T23:48:13.023079+0000","last_clean":"2024-03-17T23:48:13.023079+0000","last_became_active":"2024-03-17T23:48:11.398922+0000","last_became_peered":"2024-03-17T23:48:11.398922+0000","last_unstale":"2024-03-17T23:48:13.023079+0000","last_undegraded":"2024-03-17T23:48:13.023079+0000","last_fullsized":"2024-03-17T23:48:13.023079+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:53:53.408503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"39'1","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409546+0000","last_change":"2024-03-17T23:48:10.392858+0000","last_active":"2024-03-17T23:48:11.409546+0000","last_peered":"2024-03-17T23:48:11.409546+0000","last_clean":"2024-03-17T23:48:11.409546+0000","last_became_active":"2024-03-17T23:48:10.392599+0000","last_became_peered":"2024-03-17T23:48:10.392599+0000","last_unstale":"2024-03-17T23:48:11.409546+0000","last_undegraded":"2024-03-17T23:48:11.409546+0000","last_fullsized":"2024-03-17T23:48:11.409546+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:12:23.322115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408725+0000","last_change":"2024-03-17T23:48:11.400180+0000","last_active":"2024-03-17T23:48:11.408725+0000","last_peered":"2024-03-17T23:48:11.408725+0000","last_clean":"2024-03-17T23:48:11.408725+0000","last_became_active":"2024-03-17T23:48:11.399900+0000","last_became_peered":"2024-03-17T23:48:11.399900+0000","last_unstale":"2024-03-17T23:48:11.408725+0000","last_undegraded":"2024-03-17T23:48:11.408725+0000","last_fullsized":"2024-03-17T23:48:11.408725+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:31.101445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"39'1","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.407796+0000","last_change":"2024-03-17T23:48:10.396685+0000","last_active":"2024-03-17T23:48:11.407796+0000","last_peered":"2024-03-17T23:48:11.407796+0000","last_clean":"2024-03-17T23:48:11.407796+0000","last_became_active":"2024-03-17T23:48:10.396218+0000","last_became_peered":"2024-03-17T23:48:10.396218+0000","last_unstale":"2024-03-17T23:48:11.407796+0000","last_undegraded":"2024-03-17T23:48:11.407796+0000","last_fullsized":"2024-03-17T23:48:11.407796+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:07:26.890907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398506+0000","last_change":"2024-03-17T23:48:11.398107+0000","last_active":"2024-03-17T23:48:12.398506+0000","last_peered":"2024-03-17T23:48:12.398506+0000","last_clean":"2024-03-17T23:48:12.398506+0000","last_became_active":"2024-03-17T23:48:11.397847+0000","last_became_peered":"2024-03-17T23:48:11.397847+0000","last_unstale":"2024-03-17T23:48:12.398506+0000","last_undegraded":"2024-03-17T23:48:12.398506+0000","last_fullsized":"2024-03-17T23:48:12.398506+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:47:20.574371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.383921+0000","last_change":"2024-03-17T23:48:10.395355+0000","last_active":"2024-03-17T23:48:11.383921+0000","last_peered":"2024-03-17T23:48:11.383921+0000","last_clean":"2024-03-17T23:48:11.383921+0000","last_became_active":"2024-03-17T23:48:10.395051+0000","last_became_peered":"2024-03-17T23:48:10.395051+0000","last_unstale":"2024-03-17T23:48:11.383921+0000","last_undegraded":"2024-03-17T23:48:11.383921+0000","last_fullsized":"2024-03-17T23:48:11.383921+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:28:10.271909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398591+0000","last_change":"2024-03-17T23:48:11.392405+0000","last_active":"2024-03-17T23:48:12.398591+0000","last_peered":"2024-03-17T23:48:12.398591+0000","last_clean":"2024-03-17T23:48:12.398591+0000","last_became_active":"2024-03-17T23:48:11.391112+0000","last_became_peered":"2024-03-17T23:48:11.391112+0000","last_unstale":"2024-03-17T23:48:12.398591+0000","last_undegraded":"2024-03-17T23:48:12.398591+0000","last_fullsized":"2024-03-17T23:48:12.398591+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:03:40.343506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397331+0000","last_change":"2024-03-17T23:48:10.387055+0000","last_active":"2024-03-17T23:48:12.397331+0000","last_peered":"2024-03-17T23:48:12.397331+0000","last_clean":"2024-03-17T23:48:12.397331+0000","last_became_active":"2024-03-17T23:48:10.386732+0000","last_became_peered":"2024-03-17T23:48:10.386732+0000","last_unstale":"2024-03-17T23:48:12.397331+0000","last_undegraded":"2024-03-17T23:48:12.397331+0000","last_fullsized":"2024-03-17T23:48:12.397331+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:59:28.149170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.824012+0000","last_change":"2024-03-17T23:48:11.396039+0000","last_active":"2024-03-17T23:48:12.824012+0000","last_peered":"2024-03-17T23:48:12.824012+0000","last_clean":"2024-03-17T23:48:12.824012+0000","last_became_active":"2024-03-17T23:48:11.395647+0000","last_became_peered":"2024-03-17T23:48:11.395647+0000","last_unstale":"2024-03-17T23:48:12.824012+0000","last_undegraded":"2024-03-17T23:48:12.824012+0000","last_fullsized":"2024-03-17T23:48:12.824012+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:07:27.893298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.399060+0000","last_change":"2024-03-17T23:48:10.389198+0000","last_active":"2024-03-17T23:48:12.399060+0000","last_peered":"2024-03-17T23:48:12.399060+0000","last_clean":"2024-03-17T23:48:12.399060+0000","last_became_active":"2024-03-17T23:48:10.388927+0000","last_became_peered":"2024-03-17T23:48:10.388927+0000","last_unstale":"2024-03-17T23:48:12.399060+0000","last_undegraded":"2024-03-17T23:48:12.399060+0000","last_fullsized":"2024-03-17T23:48:12.399060+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:47:19.571981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.398574+0000","last_change":"2024-03-17T23:48:11.394613+0000","last_active":"2024-03-17T23:48:11.398574+0000","last_peered":"2024-03-17T23:48:11.398574+0000","last_clean":"2024-03-17T23:48:11.398574+0000","last_became_active":"2024-03-17T23:48:11.394206+0000","last_became_peered":"2024-03-17T23:48:11.394206+0000","last_unstale":"2024-03-17T23:48:11.398574+0000","last_undegraded":"2024-03-17T23:48:11.398574+0000","last_fullsized":"2024-03-17T23:48:11.398574+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:40:54.501510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.389970+0000","last_change":"2024-03-17T23:48:10.394514+0000","last_active":"2024-03-17T23:48:11.389970+0000","last_peered":"2024-03-17T23:48:11.389970+0000","last_clean":"2024-03-17T23:48:11.389970+0000","last_became_active":"2024-03-17T23:48:10.394237+0000","last_became_peered":"2024-03-17T23:48:10.394237+0000","last_unstale":"2024-03-17T23:48:11.389970+0000","last_undegraded":"2024-03-17T23:48:11.389970+0000","last_fullsized":"2024-03-17T23:48:11.389970+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:40:53.499120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:13.023359+0000","last_change":"2024-03-17T23:48:11.394352+0000","last_active":"2024-03-17T23:48:13.023359+0000","last_peered":"2024-03-17T23:48:13.023359+0000","last_clean":"2024-03-17T23:48:13.023359+0000","last_became_active":"2024-03-17T23:48:11.393966+0000","last_became_peered":"2024-03-17T23:48:11.393966+0000","last_unstale":"2024-03-17T23:48:13.023359+0000","last_undegraded":"2024-03-17T23:48:13.023359+0000","last_fullsized":"2024-03-17T23:48:13.023359+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:33:46.207300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"39'3","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409672+0000","last_change":"2024-03-17T23:48:10.400644+0000","last_active":"2024-03-17T23:48:11.409672+0000","last_peered":"2024-03-17T23:48:11.409672+0000","last_clean":"2024-03-17T23:48:11.409672+0000","last_became_active":"2024-03-17T23:48:10.400361+0000","last_became_peered":"2024-03-17T23:48:10.400361+0000","last_unstale":"2024-03-17T23:48:11.409672+0000","last_undegraded":"2024-03-17T23:48:11.409672+0000","last_fullsized":"2024-03-17T23:48:11.409672+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:28:10.271909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398068+0000","last_change":"2024-03-17T23:48:11.404493+0000","last_active":"2024-03-17T23:48:12.398068+0000","last_peered":"2024-03-17T23:48:12.398068+0000","last_clean":"2024-03-17T23:48:12.398068+0000","last_became_active":"2024-03-17T23:48:11.401330+0000","last_became_peered":"2024-03-17T23:48:11.401330+0000","last_unstale":"2024-03-17T23:48:12.398068+0000","last_undegraded":"2024-03-17T23:48:12.398068+0000","last_fullsized":"2024-03-17T23:48:12.398068+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:37:11.855078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"39'6","reported_seq":20,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398838+0000","last_change":"2024-03-17T23:48:10.396077+0000","last_active":"2024-03-17T23:48:12.398838+0000","last_peered":"2024-03-17T23:48:12.398838+0000","last_clean":"2024-03-17T23:48:12.398838+0000","last_became_active":"2024-03-17T23:48:10.395491+0000","last_became_peered":"2024-03-17T23:48:10.395491+0000","last_unstale":"2024-03-17T23:48:12.398838+0000","last_undegraded":"2024-03-17T23:48:12.398838+0000","last_fullsized":"2024-03-17T23:48:12.398838+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:16:08.340158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:13.023082+0000","last_change":"2024-03-17T23:48:11.396412+0000","last_active":"2024-03-17T23:48:13.023082+0000","last_peered":"2024-03-17T23:48:13.023082+0000","last_clean":"2024-03-17T23:48:13.023082+0000","last_became_active":"2024-03-17T23:48:11.396052+0000","last_became_peered":"2024-03-17T23:48:11.396052+0000","last_unstale":"2024-03-17T23:48:13.023082+0000","last_undegraded":"2024-03-17T23:48:13.023082+0000","last_fullsized":"2024-03-17T23:48:13.023082+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:17:53.610029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"39'2","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398764+0000","last_change":"2024-03-17T23:48:10.392240+0000","last_active":"2024-03-17T23:48:12.398764+0000","last_peered":"2024-03-17T23:48:12.398764+0000","last_clean":"2024-03-17T23:48:12.398764+0000","last_became_active":"2024-03-17T23:48:10.391973+0000","last_became_peered":"2024-03-17T23:48:10.391973+0000","last_unstale":"2024-03-17T23:48:12.398764+0000","last_undegraded":"2024-03-17T23:48:12.398764+0000","last_fullsized":"2024-03-17T23:48:12.398764+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:12:22.747059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398590+0000","last_change":"2024-03-17T23:48:11.393482+0000","last_active":"2024-03-17T23:48:12.398590+0000","last_peered":"2024-03-17T23:48:12.398590+0000","last_clean":"2024-03-17T23:48:12.398590+0000","last_became_active":"2024-03-17T23:48:11.393222+0000","last_became_peered":"2024-03-17T23:48:11.393222+0000","last_unstale":"2024-03-17T23:48:12.398590+0000","last_undegraded":"2024-03-17T23:48:12.398590+0000","last_fullsized":"2024-03-17T23:48:12.398590+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:00:33.846091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.385582+0000","last_change":"2024-03-17T23:48:10.394916+0000","last_active":"2024-03-17T23:48:11.385582+0000","last_peered":"2024-03-17T23:48:11.385582+0000","last_clean":"2024-03-17T23:48:11.385582+0000","last_became_active":"2024-03-17T23:48:10.391532+0000","last_became_peered":"2024-03-17T23:48:10.391532+0000","last_unstale":"2024-03-17T23:48:11.385582+0000","last_undegraded":"2024-03-17T23:48:11.385582+0000","last_fullsized":"2024-03-17T23:48:11.385582+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:44.407483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409135+0000","last_change":"2024-03-17T23:48:11.407145+0000","last_active":"2024-03-17T23:48:11.409135+0000","last_peered":"2024-03-17T23:48:11.409135+0000","last_clean":"2024-03-17T23:48:11.409135+0000","last_became_active":"2024-03-17T23:48:11.406851+0000","last_became_peered":"2024-03-17T23:48:11.406851+0000","last_unstale":"2024-03-17T23:48:11.409135+0000","last_undegraded":"2024-03-17T23:48:11.409135+0000","last_fullsized":"2024-03-17T23:48:11.409135+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:59:29.151560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397440+0000","last_change":"2024-03-17T23:48:10.388069+0000","last_active":"2024-03-17T23:48:12.397440+0000","last_peered":"2024-03-17T23:48:12.397440+0000","last_clean":"2024-03-17T23:48:12.397440+0000","last_became_active":"2024-03-17T23:48:10.387765+0000","last_became_peered":"2024-03-17T23:48:10.387765+0000","last_unstale":"2024-03-17T23:48:12.397440+0000","last_undegraded":"2024-03-17T23:48:12.397440+0000","last_fullsized":"2024-03-17T23:48:12.397440+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:31:15.116223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398662+0000","last_change":"2024-03-17T23:48:11.396828+0000","last_active":"2024-03-17T23:48:12.398662+0000","last_peered":"2024-03-17T23:48:12.398662+0000","last_clean":"2024-03-17T23:48:12.398662+0000","last_became_active":"2024-03-17T23:48:11.396370+0000","last_became_peered":"2024-03-17T23:48:11.396370+0000","last_unstale":"2024-03-17T23:48:12.398662+0000","last_undegraded":"2024-03-17T23:48:12.398662+0000","last_fullsized":"2024-03-17T23:48:12.398662+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:07:28.504390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"39'8","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.400672+0000","last_change":"2024-03-17T23:48:10.397744+0000","last_active":"2024-03-17T23:48:11.400672+0000","last_peered":"2024-03-17T23:48:11.400672+0000","last_clean":"2024-03-17T23:48:11.400672+0000","last_became_active":"2024-03-17T23:48:10.397467+0000","last_became_peered":"2024-03-17T23:48:10.397467+0000","last_unstale":"2024-03-17T23:48:11.400672+0000","last_undegraded":"2024-03-17T23:48:11.400672+0000","last_fullsized":"2024-03-17T23:48:11.400672+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:12:23.322115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398645+0000","last_change":"2024-03-17T23:48:11.390497+0000","last_active":"2024-03-17T23:48:12.398645+0000","last_peered":"2024-03-17T23:48:12.398645+0000","last_clean":"2024-03-17T23:48:12.398645+0000","last_became_active":"2024-03-17T23:48:11.390222+0000","last_became_peered":"2024-03-17T23:48:11.390222+0000","last_unstale":"2024-03-17T23:48:12.398645+0000","last_undegraded":"2024-03-17T23:48:12.398645+0000","last_fullsized":"2024-03-17T23:48:12.398645+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:52:24.543577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"39'1","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409062+0000","last_change":"2024-03-17T23:48:10.393694+0000","last_active":"2024-03-17T23:48:11.409062+0000","last_peered":"2024-03-17T23:48:11.409062+0000","last_clean":"2024-03-17T23:48:11.409062+0000","last_became_active":"2024-03-17T23:48:10.393418+0000","last_became_peered":"2024-03-17T23:48:10.393418+0000","last_unstale":"2024-03-17T23:48:11.409062+0000","last_undegraded":"2024-03-17T23:48:11.409062+0000","last_fullsized":"2024-03-17T23:48:11.409062+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:42:42.774025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408740+0000","last_change":"2024-03-17T23:48:11.400412+0000","last_active":"2024-03-17T23:48:11.408740+0000","last_peered":"2024-03-17T23:48:11.408740+0000","last_clean":"2024-03-17T23:48:11.408740+0000","last_became_active":"2024-03-17T23:48:11.400006+0000","last_became_peered":"2024-03-17T23:48:11.400006+0000","last_unstale":"2024-03-17T23:48:11.408740+0000","last_undegraded":"2024-03-17T23:48:11.408740+0000","last_fullsized":"2024-03-17T23:48:11.408740+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:59:29.151560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"39'2","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398100+0000","last_change":"2024-03-17T23:48:10.391802+0000","last_active":"2024-03-17T23:48:12.398100+0000","last_peered":"2024-03-17T23:48:12.398100+0000","last_clean":"2024-03-17T23:48:12.398100+0000","last_became_active":"2024-03-17T23:48:10.391464+0000","last_became_peered":"2024-03-17T23:48:10.391464+0000","last_unstale":"2024-03-17T23:48:12.398100+0000","last_undegraded":"2024-03-17T23:48:12.398100+0000","last_fullsized":"2024-03-17T23:48:12.398100+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:44:41.528247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397957+0000","last_change":"2024-03-17T23:48:11.399808+0000","last_active":"2024-03-17T23:48:12.397957+0000","last_peered":"2024-03-17T23:48:12.397957+0000","last_clean":"2024-03-17T23:48:12.397957+0000","last_became_active":"2024-03-17T23:48:11.399460+0000","last_became_peered":"2024-03-17T23:48:11.399460+0000","last_unstale":"2024-03-17T23:48:12.397957+0000","last_undegraded":"2024-03-17T23:48:12.397957+0000","last_fullsized":"2024-03-17T23:48:12.397957+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:16:59.276395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"39'1","reported_seq":15,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.823897+0000","last_change":"2024-03-17T23:48:10.396952+0000","last_active":"2024-03-17T23:48:12.823897+0000","last_peered":"2024-03-17T23:48:12.823897+0000","last_clean":"2024-03-17T23:48:12.823897+0000","last_became_active":"2024-03-17T23:48:10.396664+0000","last_became_peered":"2024-03-17T23:48:10.396664+0000","last_unstale":"2024-03-17T23:48:12.823897+0000","last_undegraded":"2024-03-17T23:48:12.823897+0000","last_fullsized":"2024-03-17T23:48:12.823897+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:19:37.436831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.824069+0000","last_change":"2024-03-17T23:48:11.403492+0000","last_active":"2024-03-17T23:48:12.824069+0000","last_peered":"2024-03-17T23:48:12.824069+0000","last_clean":"2024-03-17T23:48:12.824069+0000","last_became_active":"2024-03-17T23:48:11.403203+0000","last_became_peered":"2024-03-17T23:48:11.403203+0000","last_unstale":"2024-03-17T23:48:12.824069+0000","last_undegraded":"2024-03-17T23:48:12.824069+0000","last_fullsized":"2024-03-17T23:48:12.824069+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:31:13.050920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.384478+0000","last_change":"2024-03-17T23:48:10.396820+0000","last_active":"2024-03-17T23:48:11.384478+0000","last_peered":"2024-03-17T23:48:11.384478+0000","last_clean":"2024-03-17T23:48:11.384478+0000","last_became_active":"2024-03-17T23:48:10.396549+0000","last_became_peered":"2024-03-17T23:48:10.396549+0000","last_unstale":"2024-03-17T23:48:11.384478+0000","last_undegraded":"2024-03-17T23:48:11.384478+0000","last_fullsized":"2024-03-17T23:48:11.384478+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:16.355950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408966+0000","last_change":"2024-03-17T23:48:11.403660+0000","last_active":"2024-03-17T23:48:11.408966+0000","last_peered":"2024-03-17T23:48:11.408966+0000","last_clean":"2024-03-17T23:48:11.408966+0000","last_became_active":"2024-03-17T23:48:11.403385+0000","last_became_peered":"2024-03-17T23:48:11.403385+0000","last_unstale":"2024-03-17T23:48:11.408966+0000","last_undegraded":"2024-03-17T23:48:11.408966+0000","last_fullsized":"2024-03-17T23:48:11.408966+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:42:43.776415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"39'2","reported_seq":16,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.823764+0000","last_change":"2024-03-17T23:48:10.387115+0000","last_active":"2024-03-17T23:48:12.823764+0000","last_peered":"2024-03-17T23:48:12.823764+0000","last_clean":"2024-03-17T23:48:12.823764+0000","last_became_active":"2024-03-17T23:48:10.386894+0000","last_became_peered":"2024-03-17T23:48:10.386894+0000","last_unstale":"2024-03-17T23:48:12.823764+0000","last_undegraded":"2024-03-17T23:48:12.823764+0000","last_fullsized":"2024-03-17T23:48:12.823764+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:25:28.982482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:13.023584+0000","last_change":"2024-03-17T23:48:11.395193+0000","last_active":"2024-03-17T23:48:13.023584+0000","last_peered":"2024-03-17T23:48:13.023584+0000","last_clean":"2024-03-17T23:48:13.023584+0000","last_became_active":"2024-03-17T23:48:11.393952+0000","last_became_peered":"2024-03-17T23:48:11.393952+0000","last_unstale":"2024-03-17T23:48:13.023584+0000","last_undegraded":"2024-03-17T23:48:13.023584+0000","last_fullsized":"2024-03-17T23:48:13.023584+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:16:59.276395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.823862+0000","last_change":"2024-03-17T23:48:10.391728+0000","last_active":"2024-03-17T23:48:12.823862+0000","last_peered":"2024-03-17T23:48:12.823862+0000","last_clean":"2024-03-17T23:48:12.823862+0000","last_became_active":"2024-03-17T23:48:10.391455+0000","last_became_peered":"2024-03-17T23:48:10.391455+0000","last_unstale":"2024-03-17T23:48:12.823862+0000","last_undegraded":"2024-03-17T23:48:12.823862+0000","last_fullsized":"2024-03-17T23:48:12.823862+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:57:20.246583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409211+0000","last_change":"2024-03-17T23:48:11.404518+0000","last_active":"2024-03-17T23:48:11.409211+0000","last_peered":"2024-03-17T23:48:11.409211+0000","last_clean":"2024-03-17T23:48:11.409211+0000","last_became_active":"2024-03-17T23:48:11.404261+0000","last_became_peered":"2024-03-17T23:48:11.404261+0000","last_unstale":"2024-03-17T23:48:11.409211+0000","last_undegraded":"2024-03-17T23:48:11.409211+0000","last_fullsized":"2024-03-17T23:48:11.409211+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:17:53.610029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.385531+0000","last_change":"2024-03-17T23:48:10.397213+0000","last_active":"2024-03-17T23:48:11.385531+0000","last_peered":"2024-03-17T23:48:11.385531+0000","last_clean":"2024-03-17T23:48:11.385531+0000","last_became_active":"2024-03-17T23:48:10.396913+0000","last_became_peered":"2024-03-17T23:48:10.396913+0000","last_unstale":"2024-03-17T23:48:11.385531+0000","last_undegraded":"2024-03-17T23:48:11.385531+0000","last_fullsized":"2024-03-17T23:48:11.385531+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:31:12.048530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398741+0000","last_change":"2024-03-17T23:48:11.393768+0000","last_active":"2024-03-17T23:48:12.398741+0000","last_peered":"2024-03-17T23:48:12.398741+0000","last_clean":"2024-03-17T23:48:12.398741+0000","last_became_active":"2024-03-17T23:48:11.392652+0000","last_became_peered":"2024-03-17T23:48:11.392652+0000","last_unstale":"2024-03-17T23:48:12.398741+0000","last_undegraded":"2024-03-17T23:48:12.398741+0000","last_fullsized":"2024-03-17T23:48:12.398741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:00:12.503035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.823906+0000","last_change":"2024-03-17T23:48:10.387114+0000","last_active":"2024-03-17T23:48:12.823906+0000","last_peered":"2024-03-17T23:48:12.823906+0000","last_clean":"2024-03-17T23:48:12.823906+0000","last_became_active":"2024-03-17T23:48:10.386860+0000","last_became_peered":"2024-03-17T23:48:10.386860+0000","last_unstale":"2024-03-17T23:48:12.823906+0000","last_undegraded":"2024-03-17T23:48:12.823906+0000","last_fullsized":"2024-03-17T23:48:12.823906+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:05:24.288253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.397234+0000","last_change":"2024-03-17T23:48:11.396445+0000","last_active":"2024-03-17T23:48:11.397234+0000","last_peered":"2024-03-17T23:48:11.397234+0000","last_clean":"2024-03-17T23:48:11.397234+0000","last_became_active":"2024-03-17T23:48:11.396189+0000","last_became_peered":"2024-03-17T23:48:11.396189+0000","last_unstale":"2024-03-17T23:48:11.397234+0000","last_undegraded":"2024-03-17T23:48:11.397234+0000","last_fullsized":"2024-03-17T23:48:11.397234+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:10.929912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"39'2","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.406980+0000","last_change":"2024-03-17T23:48:10.400812+0000","last_active":"2024-03-17T23:48:11.406980+0000","last_peered":"2024-03-17T23:48:11.406980+0000","last_clean":"2024-03-17T23:48:11.406980+0000","last_became_active":"2024-03-17T23:48:10.400479+0000","last_became_peered":"2024-03-17T23:48:10.400479+0000","last_unstale":"2024-03-17T23:48:11.406980+0000","last_undegraded":"2024-03-17T23:48:11.406980+0000","last_fullsized":"2024-03-17T23:48:11.406980+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:09.927522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.408459+0000","last_change":"2024-03-17T23:48:11.403785+0000","last_active":"2024-03-17T23:48:11.408459+0000","last_peered":"2024-03-17T23:48:11.408459+0000","last_clean":"2024-03-17T23:48:11.408459+0000","last_became_active":"2024-03-17T23:48:11.402301+0000","last_became_peered":"2024-03-17T23:48:11.402301+0000","last_unstale":"2024-03-17T23:48:11.408459+0000","last_undegraded":"2024-03-17T23:48:11.408459+0000","last_fullsized":"2024-03-17T23:48:11.408459+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:31.101445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"40'1","reported_seq":15,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:13.383658+0000","last_change":"2024-03-17T23:48:10.389968+0000","last_active":"2024-03-17T23:48:13.383658+0000","last_peered":"2024-03-17T23:48:13.383658+0000","last_clean":"2024-03-17T23:48:13.383658+0000","last_became_active":"2024-03-17T23:48:10.389694+0000","last_became_peered":"2024-03-17T23:48:10.389694+0000","last_unstale":"2024-03-17T23:48:13.383658+0000","last_undegraded":"2024-03-17T23:48:13.383658+0000","last_fullsized":"2024-03-17T23:48:13.383658+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:49:18.907672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398896+0000","last_change":"2024-03-17T23:48:11.391249+0000","last_active":"2024-03-17T23:48:12.398896+0000","last_peered":"2024-03-17T23:48:12.398896+0000","last_clean":"2024-03-17T23:48:12.398896+0000","last_became_active":"2024-03-17T23:48:11.390836+0000","last_became_peered":"2024-03-17T23:48:11.390836+0000","last_unstale":"2024-03-17T23:48:12.398896+0000","last_undegraded":"2024-03-17T23:48:12.398896+0000","last_fullsized":"2024-03-17T23:48:12.398896+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:51:42.776453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:13.023326+0000","last_change":"2024-03-17T23:48:10.396185+0000","last_active":"2024-03-17T23:48:13.023326+0000","last_peered":"2024-03-17T23:48:13.023326+0000","last_clean":"2024-03-17T23:48:13.023326+0000","last_became_active":"2024-03-17T23:48:10.395932+0000","last_became_peered":"2024-03-17T23:48:10.395932+0000","last_unstale":"2024-03-17T23:48:13.023326+0000","last_undegraded":"2024-03-17T23:48:13.023326+0000","last_fullsized":"2024-03-17T23:48:13.023326+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:16.355950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:13.023230+0000","last_change":"2024-03-17T23:48:11.403894+0000","last_active":"2024-03-17T23:48:13.023230+0000","last_peered":"2024-03-17T23:48:13.023230+0000","last_clean":"2024-03-17T23:48:13.023230+0000","last_became_active":"2024-03-17T23:48:11.403646+0000","last_became_peered":"2024-03-17T23:48:11.403646+0000","last_unstale":"2024-03-17T23:48:13.023230+0000","last_undegraded":"2024-03-17T23:48:13.023230+0000","last_fullsized":"2024-03-17T23:48:13.023230+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:19:33.805891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:13.023386+0000","last_change":"2024-03-17T23:48:10.388194+0000","last_active":"2024-03-17T23:48:13.023386+0000","last_peered":"2024-03-17T23:48:13.023386+0000","last_clean":"2024-03-17T23:48:13.023386+0000","last_became_active":"2024-03-17T23:48:10.387920+0000","last_became_peered":"2024-03-17T23:48:10.387920+0000","last_unstale":"2024-03-17T23:48:13.023386+0000","last_undegraded":"2024-03-17T23:48:13.023386+0000","last_fullsized":"2024-03-17T23:48:13.023386+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:20:47.924674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.397727+0000","last_change":"2024-03-17T23:48:11.397059+0000","last_active":"2024-03-17T23:48:11.397727+0000","last_peered":"2024-03-17T23:48:11.397727+0000","last_clean":"2024-03-17T23:48:11.397727+0000","last_became_active":"2024-03-17T23:48:11.396724+0000","last_became_peered":"2024-03-17T23:48:11.396724+0000","last_unstale":"2024-03-17T23:48:11.397727+0000","last_undegraded":"2024-03-17T23:48:11.397727+0000","last_fullsized":"2024-03-17T23:48:11.397727+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:31.101445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398165+0000","last_change":"2024-03-17T23:48:10.384305+0000","last_active":"2024-03-17T23:48:12.398165+0000","last_peered":"2024-03-17T23:48:12.398165+0000","last_clean":"2024-03-17T23:48:12.398165+0000","last_became_active":"2024-03-17T23:48:10.384034+0000","last_became_peered":"2024-03-17T23:48:10.384034+0000","last_unstale":"2024-03-17T23:48:12.398165+0000","last_undegraded":"2024-03-17T23:48:12.398165+0000","last_fullsized":"2024-03-17T23:48:12.398165+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:32:37.500442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398088+0000","last_change":"2024-03-17T23:48:11.394167+0000","last_active":"2024-03-17T23:48:12.398088+0000","last_peered":"2024-03-17T23:48:12.398088+0000","last_clean":"2024-03-17T23:48:12.398088+0000","last_became_active":"2024-03-17T23:48:11.392337+0000","last_became_peered":"2024-03-17T23:48:11.392337+0000","last_unstale":"2024-03-17T23:48:12.398088+0000","last_undegraded":"2024-03-17T23:48:12.398088+0000","last_fullsized":"2024-03-17T23:48:12.398088+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:04:14.826887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.397422+0000","last_change":"2024-03-17T23:48:10.391429+0000","last_active":"2024-03-17T23:48:12.397422+0000","last_peered":"2024-03-17T23:48:12.397422+0000","last_clean":"2024-03-17T23:48:12.397422+0000","last_became_active":"2024-03-17T23:48:10.390974+0000","last_became_peered":"2024-03-17T23:48:10.390974+0000","last_unstale":"2024-03-17T23:48:12.397422+0000","last_undegraded":"2024-03-17T23:48:12.397422+0000","last_fullsized":"2024-03-17T23:48:12.397422+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:55:04.026997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.399296+0000","last_change":"2024-03-17T23:48:11.397803+0000","last_active":"2024-03-17T23:48:11.399296+0000","last_peered":"2024-03-17T23:48:11.399296+0000","last_clean":"2024-03-17T23:48:11.399296+0000","last_became_active":"2024-03-17T23:48:11.397541+0000","last_became_peered":"2024-03-17T23:48:11.397541+0000","last_unstale":"2024-03-17T23:48:11.399296+0000","last_undegraded":"2024-03-17T23:48:11.399296+0000","last_fullsized":"2024-03-17T23:48:11.399296+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:17.358341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-17T23:48:12.398558+0000","last_change":"2024-03-17T23:48:10.389196+0000","last_active":"2024-03-17T23:48:12.398558+0000","last_peered":"2024-03-17T23:48:12.398558+0000","last_clean":"2024-03-17T23:48:12.398558+0000","last_became_active":"2024-03-17T23:48:10.388918+0000","last_became_peered":"2024-03-17T23:48:10.388918+0000","last_unstale":"2024-03-17T23:48:12.398558+0000","last_undegraded":"2024-03-17T23:48:12.398558+0000","last_fullsized":"2024-03-17T23:48:12.398558+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:38:37.674462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.409115+0000","last_change":"2024-03-17T23:48:11.403123+0000","last_active":"2024-03-17T23:48:11.409115+0000","last_peered":"2024-03-17T23:48:11.409115+0000","last_clean":"2024-03-17T23:48:11.409115+0000","last_became_active":"2024-03-17T23:48:11.402851+0000","last_became_peered":"2024-03-17T23:48:11.402851+0000","last_unstale":"2024-03-17T23:48:11.409115+0000","last_undegraded":"2024-03-17T23:48:11.409115+0000","last_fullsized":"2024-03-17T23:48:11.409115+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:10.371109+0000","last_clean_scrub_stamp":"2024-03-17T23:48:10.371109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:44:42.530638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.384095+0000","last_change":"2024-03-17T23:48:10.398461+0000","last_active":"2024-03-17T23:48:11.384095+0000","last_peered":"2024-03-17T23:48:11.384095+0000","last_clean":"2024-03-17T23:48:11.384095+0000","last_became_active":"2024-03-17T23:48:10.398188+0000","last_became_peered":"2024-03-17T23:48:10.398188+0000","last_unstale":"2024-03-17T23:48:11.384095+0000","last_undegraded":"2024-03-17T23:48:11.384095+0000","last_fullsized":"2024-03-17T23:48:11.384095+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:48:09.368718+0000","last_clean_scrub_stamp":"2024-03-17T23:48:09.368718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:06:33.058080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":204,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-03-17T23:48:11.385340+0000","last_change":"2024-03-17T23:46:44.768668+0000","last_active":"2024-03-17T23:48:11.385340+0000","last_peered":"2024-03-17T23:48:11.385340+0000","last_clean":"2024-03-17T23:48:11.385340+0000","last_became_active":"2024-03-17T23:46:44.767826+0000","last_became_peered":"2024-03-17T23:46:44.767826+0000","last_unstale":"2024-03-17T23:48:11.385340+0000","last_undegraded":"2024-03-17T23:48:11.385340+0000","last_fullsized":"2024-03-17T23:48:11.385340+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_clean_scrub_stamp":"2024-03-17T23:46:43.744878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:54:22.286155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":14,"seq":60129542166,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27488,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344352,"statfs":{"total":96636764160,"available":96608616448,"internally_reserved":0,"allocated":475136,"data_stored":202650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542166,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":450560,"data_stored":194382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542166,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343784,"statfs":{"total":96636764160,"available":96608034816,"internally_reserved":0,"allocated":1056768,"data_stored":791406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32212,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339628,"statfs":{"total":96636764160,"available":96603779072,"internally_reserved":0,"allocated":1056768,"data_stored":786362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31636,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340204,"statfs":{"total":96636764160,"available":96604368896,"internally_reserved":0,"allocated":462848,"data_stored":201586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31624,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":454656,"data_stored":194424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31624,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":454656,"data_stored":194916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":450560,"data_stored":194402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-17T23:48:19.330 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-17T23:48:19.331 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-17T23:48:19.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.479+0000 7fecefa36640 1 Processor -- start 2024-03-17T23:48:19.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.479+0000 7fecefa36640 1 -- start start 2024-03-17T23:48:19.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.480+0000 7fecefa36640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fece806b8a0 0x7fece815e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:19.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.480+0000 7fecefa36640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fece806ee10 0x7fece80afb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:19.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.480+0000 7fecefa36640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fece806f360 con 0x7fece815dce0 2024-03-17T23:48:19.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.480+0000 7fecefa36640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fece80cd480 con 0x7fece806ee10 2024-03-17T23:48:19.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.480+0000 7fecefa36640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fece806ff70 con 0x7fece806b8a0 2024-03-17T23:48:19.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.480+0000 7fecedfd3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fece806ee10 0x7fece80afb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:19.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.480+0000 7fecedfd3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fece806ee10 0x7fece80afb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36646/0 (socket says 172.21.15.150:36646) 2024-03-17T23:48:19.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.480+0000 7fecedfd3640 1 -- 172.21.15.150:0/1364393828 learned_addr learned my addr 172.21.15.150:0/1364393828 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:19.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.481+0000 7fececfd1640 1 --2- 172.21.15.150:0/1364393828 >> v2:172.21.15.157:3300/0 conn(0x7fece806b8a0 0x7fece815e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:19.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.481+0000 7fecedfd3640 1 -- 172.21.15.150:0/1364393828 >> v2:172.21.15.157:3300/0 conn(0x7fece806b8a0 msgr2=0x7fece815e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:19.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.481+0000 7fecedfd3640 1 --2- 172.21.15.150:0/1364393828 >> v2:172.21.15.157:3300/0 conn(0x7fece806b8a0 0x7fece815e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.481+0000 7fecedfd3640 1 -- 172.21.15.150:0/1364393828 >> v1:172.21.15.157:6789/0 conn(0x7fece815dce0 legacy=0x7fece806b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:19.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.481+0000 7fecedfd3640 1 -- 172.21.15.150:0/1364393828 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fece806f640 con 0x7fece806ee10 2024-03-17T23:48:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.482+0000 7fecedfd3640 1 --2- 172.21.15.150:0/1364393828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fece806ee10 0x7fece80afb10 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fece80b01a0 tx=0x7fecd802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2f7915bcaf48b4d server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.482+0000 7fecdffff640 1 -- 172.21.15.150:0/1364393828 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecd8034070 con 0x7fece806ee10 2024-03-17T23:48:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.482+0000 7fecdffff640 1 -- 172.21.15.150:0/1364393828 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecd8030860 con 0x7fece806ee10 2024-03-17T23:48:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.482+0000 7fecdffff640 1 -- 172.21.15.150:0/1364393828 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecd8030c30 con 0x7fece806ee10 2024-03-17T23:48:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.483+0000 7fecefa36640 1 -- 172.21.15.150:0/1364393828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fece806ee10 msgr2=0x7fece80afb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.483+0000 7fecefa36640 1 --2- 172.21.15.150:0/1364393828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fece806ee10 0x7fece80afb10 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fece80b01a0 tx=0x7fecd802b9a0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.483+0000 7fecefa36640 1 -- 172.21.15.150:0/1364393828 shutdown_connections 2024-03-17T23:48:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.483+0000 7fecefa36640 1 --2- 172.21.15.150:0/1364393828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fece806ee10 0x7fece80afb10 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.483+0000 7fecefa36640 1 --2- 172.21.15.150:0/1364393828 >> v2:172.21.15.157:3300/0 conn(0x7fece806b8a0 0x7fece815e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.483+0000 7fecefa36640 1 -- 172.21.15.150:0/1364393828 >> 172.21.15.150:0/1364393828 conn(0x7fece8099a70 msgr2=0x7fece8099e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.483+0000 7fecefa36640 1 -- 172.21.15.150:0/1364393828 shutdown_connections 2024-03-17T23:48:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.483+0000 7fecefa36640 1 -- 172.21.15.150:0/1364393828 wait complete. 2024-03-17T23:48:19.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.484+0000 7fecefa36640 1 Processor -- start 2024-03-17T23:48:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.484+0000 7fecefa36640 1 -- start start 2024-03-17T23:48:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.485+0000 7fecefa36640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fece806b8a0 0x7fece80b4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.485+0000 7fecefa36640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fece806ee10 0x7fece815cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.485+0000 7fecefa36640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fece815dce0 0x7fece815d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.485+0000 7fecefa36640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fece80cfbe0 con 0x7fece806b8a0 2024-03-17T23:48:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.485+0000 7fecefa36640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fece806fc00 con 0x7fece806ee10 2024-03-17T23:48:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.485+0000 7fecefa36640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fece80cfd60 con 0x7fece815dce0 2024-03-17T23:48:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.485+0000 7feced7d2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fece806b8a0 0x7fece80b4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.485+0000 7feced7d2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fece806b8a0 0x7fece80b4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36656/0 (socket says 172.21.15.150:36656) 2024-03-17T23:48:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.485+0000 7feced7d2640 1 -- 172.21.15.150:0/1541071844 learned_addr learned my addr 172.21.15.150:0/1541071844 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.485+0000 7fecedfd3640 1 --2- 172.21.15.150:0/1541071844 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fece815dce0 0x7fece815d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.485+0000 7fececfd1640 1 --2- 172.21.15.150:0/1541071844 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fece806ee10 0x7fece815cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.485+0000 7feced7d2640 1 -- 172.21.15.150:0/1541071844 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fece815dce0 msgr2=0x7fece815d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.485+0000 7feced7d2640 1 --2- 172.21.15.150:0/1541071844 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fece815dce0 0x7fece815d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.485+0000 7feced7d2640 1 -- 172.21.15.150:0/1541071844 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fece806ee10 msgr2=0x7fece815cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.485+0000 7feced7d2640 1 --2- 172.21.15.150:0/1541071844 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fece806ee10 0x7fece815cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.485+0000 7feced7d2640 1 -- 172.21.15.150:0/1541071844 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fece80b0040 con 0x7fece806b8a0 2024-03-17T23:48:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.485+0000 7fecedfd3640 1 --2- 172.21.15.150:0/1541071844 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fece815dce0 0x7fece815d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.485+0000 7fececfd1640 1 --2- 172.21.15.150:0/1541071844 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fece806ee10 0x7fece815cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.485+0000 7feced7d2640 1 --2- 172.21.15.150:0/1541071844 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fece806b8a0 0x7fece80b4f10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fece000c760 tx=0x7fece000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.485+0000 7fecddffb640 1 -- 172.21.15.150:0/1541071844 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece0016020 con 0x7fece806b8a0 2024-03-17T23:48:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.485+0000 7fecddffb640 1 -- 172.21.15.150:0/1541071844 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fece0006d50 con 0x7fece806b8a0 2024-03-17T23:48:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.485+0000 7fecefa36640 1 -- 172.21.15.150:0/1541071844 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fece809df00 con 0x7fece806b8a0 2024-03-17T23:48:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.486+0000 7fecddffb640 1 -- 172.21.15.150:0/1541071844 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece0003bb0 con 0x7fece806b8a0 2024-03-17T23:48:19.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.486+0000 7fecefa36640 1 -- 172.21.15.150:0/1541071844 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fece809e4c0 con 0x7fece806b8a0 2024-03-17T23:48:19.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.487+0000 7fecefa36640 1 -- 172.21.15.150:0/1541071844 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fece80688b0 con 0x7fece806b8a0 2024-03-17T23:48:19.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.487+0000 7fecddffb640 1 -- 172.21.15.150:0/1541071844 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fece000a850 con 0x7fece806b8a0 2024-03-17T23:48:19.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.488+0000 7fecddffb640 1 --2- 172.21.15.150:0/1541071844 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fecd406bbd0 0x7fecd406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:19.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.488+0000 7fecddffb640 1 -- 172.21.15.150:0/1541071844 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7fece008a810 con 0x7fece806b8a0 2024-03-17T23:48:19.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.488+0000 7fececfd1640 1 --2- 172.21.15.150:0/1541071844 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fecd406bbd0 0x7fecd406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:19.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.488+0000 7fececfd1640 1 --2- 172.21.15.150:0/1541071844 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fecd406bbd0 0x7fecd406e090 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fece80cd3c0 tx=0x7fece4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:19.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.496+0000 7fecddffb640 1 -- 172.21.15.150:0/1541071844 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fece0020020 con 0x7fece806b8a0 2024-03-17T23:48:19.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.739+0000 7fecefa36640 1 -- 172.21.15.150:0/1541071844 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fece80c7130 con 0x7fece806b8a0 2024-03-17T23:48:19.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.741+0000 7fecddffb640 1 -- 172.21.15.150:0/1541071844 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) ==== 190+0+147 (secure 0 0 0) 0x7fece0019310 con 0x7fece806b8a0 2024-03-17T23:48:19.742 INFO:teuthology.orchestra.run.smithi150.stdout:[client.0] 2024-03-17T23:48:19.742 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQC3gPdlvlPJNRAAkUfetPJK4hZQG5rNiSDozw== 2024-03-17T23:48:19.742 INFO:teuthology.orchestra.run.smithi150.stdout: caps mds = "allow" 2024-03-17T23:48:19.742 INFO:teuthology.orchestra.run.smithi150.stdout: caps mon = "allow r" 2024-03-17T23:48:19.742 INFO:teuthology.orchestra.run.smithi150.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-17T23:48:19.743 INFO:teuthology.orchestra.run.smithi150.stderr:updated caps for client.0 2024-03-17T23:48:19.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.744+0000 7fecefa36640 1 -- 172.21.15.150:0/1541071844 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fecd406bbd0 msgr2=0x7fecd406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:19.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.744+0000 7fecefa36640 1 --2- 172.21.15.150:0/1541071844 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fecd406bbd0 0x7fecd406e090 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fece80cd3c0 tx=0x7fece4008040 comp rx=0 tx=0).stop 2024-03-17T23:48:19.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.744+0000 7fecefa36640 1 -- 172.21.15.150:0/1541071844 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fece806b8a0 msgr2=0x7fece80b4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:19.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.744+0000 7fecefa36640 1 --2- 172.21.15.150:0/1541071844 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fece806b8a0 0x7fece80b4f10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fece000c760 tx=0x7fece000cc20 comp rx=0 tx=0).stop 2024-03-17T23:48:19.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.744+0000 7fecefa36640 1 -- 172.21.15.150:0/1541071844 shutdown_connections 2024-03-17T23:48:19.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.744+0000 7fecefa36640 1 --2- 172.21.15.150:0/1541071844 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fece815dce0 0x7fece815d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.744+0000 7fecefa36640 1 --2- 172.21.15.150:0/1541071844 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fecd406bbd0 0x7fecd406e090 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.744+0000 7fecefa36640 1 --2- 172.21.15.150:0/1541071844 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fece806ee10 0x7fece815cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.744+0000 7fecefa36640 1 --2- 172.21.15.150:0/1541071844 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fece806b8a0 0x7fece80b4f10 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.744+0000 7fecefa36640 1 -- 172.21.15.150:0/1541071844 >> 172.21.15.150:0/1541071844 conn(0x7fece8099a70 msgr2=0x7fece80d0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:19.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.744+0000 7fecefa36640 1 -- 172.21.15.150:0/1541071844 shutdown_connections 2024-03-17T23:48:19.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.744+0000 7fecefa36640 1 -- 172.21.15.150:0/1541071844 wait complete. 2024-03-17T23:48:19.756 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:48:19.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.857+0000 7f2168309640 1 Processor -- start 2024-03-17T23:48:19.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.857+0000 7f2168309640 1 -- start start 2024-03-17T23:48:19.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.858+0000 7f2168309640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21600b5c40 0x7f21600b7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:19.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.858+0000 7f2168309640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21600d2890 con 0x7f216015e060 2024-03-17T23:48:19.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.858+0000 7f2168309640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21600d2b70 con 0x7f21600b57d0 2024-03-17T23:48:19.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.858+0000 7f2168309640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f21600d2630 con 0x7f21600b5c40 2024-03-17T23:48:19.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.858+0000 7f21660a5640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f216015e060 0x7f21600d1f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:48772/0 (socket says 172.21.15.150:48772) 2024-03-17T23:48:19.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.858+0000 7f21660a5640 1 -- 172.21.15.150:0/2400235898 learned_addr learned my addr 172.21.15.150:0/2400235898 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:19.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.859+0000 7f21650a3640 1 -- 172.21.15.150:0/2400235898 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1043541071 0 0) 0x7f21600d2890 con 0x7f216015e060 2024-03-17T23:48:19.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.859+0000 7f21650a3640 1 -- 172.21.15.150:0/2400235898 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f214c003520 con 0x7f216015e060 2024-03-17T23:48:19.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.859+0000 7f21668a6640 1 --2- 172.21.15.150:0/2400235898 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21600b5c40 0x7f21600b7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:19.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.859+0000 7f21650a3640 1 -- 172.21.15.150:0/2400235898 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2105117430 0 0) 0x7f214c003520 con 0x7f216015e060 2024-03-17T23:48:19.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.859+0000 7f21650a3640 1 -- 172.21.15.150:0/2400235898 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21600b5c40 msgr2=0x7f21600b7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:19.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.859+0000 7f21650a3640 1 --2- 172.21.15.150:0/2400235898 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21600b5c40 0x7f21600b7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.859+0000 7f21650a3640 1 -- 172.21.15.150:0/2400235898 >> v1:172.21.15.157:6789/0 conn(0x7f21600b57d0 legacy=0x7f21600b6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:19.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.859+0000 7f21650a3640 1 -- 172.21.15.150:0/2400235898 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21600d2e50 con 0x7f216015e060 2024-03-17T23:48:19.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.859+0000 7f21650a3640 1 -- 172.21.15.150:0/2400235898 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f215c001670 con 0x7f216015e060 2024-03-17T23:48:19.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.859+0000 7f21650a3640 1 -- 172.21.15.150:0/2400235898 >> v1:172.21.15.150:6789/0 conn(0x7f216015e060 legacy=0x7f21600d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:19.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.859+0000 7f21650a3640 1 --2- 172.21.15.150:0/2400235898 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f214c0039e0 0x7f214c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:19.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.859+0000 7f21650a3640 1 --2- 172.21.15.150:0/2400235898 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21600b5c40 0x7f214c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:19.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.859+0000 7f21650a3640 1 --2- 172.21.15.150:0/2400235898 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f214c004d40 0x7f214c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:19.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.859+0000 7f21650a3640 1 -- 172.21.15.150:0/2400235898 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f21600d2630 con 0x7f214c0039e0 2024-03-17T23:48:19.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.859+0000 7f21650a3640 1 -- 172.21.15.150:0/2400235898 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f214c00baf0 con 0x7f214c004d40 2024-03-17T23:48:19.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.859+0000 7f21650a3640 1 -- 172.21.15.150:0/2400235898 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f214c00bcf0 con 0x7f21600b5c40 2024-03-17T23:48:19.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.859+0000 7f21660a5640 1 --2- 172.21.15.150:0/2400235898 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f214c0039e0 0x7f214c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:19.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.859+0000 7f21658a4640 1 --2- 172.21.15.150:0/2400235898 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21600b5c40 0x7f214c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:19.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.859+0000 7f21668a6640 1 --2- 172.21.15.150:0/2400235898 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f214c004d40 0x7f214c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:19.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.859+0000 7f21660a5640 1 -- 172.21.15.150:0/2400235898 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21600b5c40 msgr2=0x7f214c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:19.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.859+0000 7f21660a5640 1 --2- 172.21.15.150:0/2400235898 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21600b5c40 0x7f214c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.859+0000 7f21660a5640 1 -- 172.21.15.150:0/2400235898 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f214c004d40 msgr2=0x7f214c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:19.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.859+0000 7f21660a5640 1 --2- 172.21.15.150:0/2400235898 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f214c004d40 0x7f214c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.859+0000 7f21660a5640 1 -- 172.21.15.150:0/2400235898 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f214c003520 con 0x7f214c0039e0 2024-03-17T23:48:19.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.859+0000 7f21658a4640 1 --2- 172.21.15.150:0/2400235898 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21600b5c40 0x7f214c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:19.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.860+0000 7f21660a5640 1 --2- 172.21.15.150:0/2400235898 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f214c0039e0 0x7f214c003e00 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f215c00c240 tx=0x7f214c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=640318427e52438 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:19.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.860+0000 7f21650a3640 1 -- 172.21.15.150:0/2400235898 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215c007cf0 con 0x7f214c0039e0 2024-03-17T23:48:19.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.860+0000 7f21650a3640 1 -- 172.21.15.150:0/2400235898 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f215c030070 con 0x7f214c0039e0 2024-03-17T23:48:19.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.860+0000 7f21650a3640 1 -- 172.21.15.150:0/2400235898 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215c005d30 con 0x7f214c0039e0 2024-03-17T23:48:19.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.861+0000 7f2168309640 1 -- 172.21.15.150:0/2400235898 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f214c0039e0 msgr2=0x7f214c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:19.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.861+0000 7f2168309640 1 --2- 172.21.15.150:0/2400235898 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f214c0039e0 0x7f214c003e00 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f215c00c240 tx=0x7f214c005830 comp rx=0 tx=0).stop 2024-03-17T23:48:19.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.861+0000 7f21668a6640 1 -- 172.21.15.150:0/2400235898 reap_dead start 2024-03-17T23:48:19.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.861+0000 7f2168309640 1 -- 172.21.15.150:0/2400235898 shutdown_connections 2024-03-17T23:48:19.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.861+0000 7f2168309640 1 -- 172.21.15.150:0/2400235898 >> 172.21.15.150:0/2400235898 conn(0x7f2160073200 msgr2=0x7f21600d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:19.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.861+0000 7f2168309640 1 -- 172.21.15.150:0/2400235898 shutdown_connections 2024-03-17T23:48:19.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.861+0000 7f2168309640 1 -- 172.21.15.150:0/2400235898 wait complete. 2024-03-17T23:48:19.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.862+0000 7f2168309640 1 Processor -- start 2024-03-17T23:48:19.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.862+0000 7f2168309640 1 -- start start 2024-03-17T23:48:19.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.863+0000 7f2168309640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f214c0039e0 0x7f21600cb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:19.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.863+0000 7f2168309640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f216015c2c0 0x7f21600aa820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:19.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.863+0000 7f2168309640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21600cc690 0x7f21600ccb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:19.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.863+0000 7f2168309640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f21600b6080 con 0x7f214c0039e0 2024-03-17T23:48:19.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.863+0000 7f2168309640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f21600d3410 con 0x7f216015c2c0 2024-03-17T23:48:19.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.863+0000 7f2168309640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f216015e440 con 0x7f21600cc690 2024-03-17T23:48:19.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.863+0000 7f21660a5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f214c0039e0 0x7f21600cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:19.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.863+0000 7f21660a5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f214c0039e0 0x7f21600cb010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36664/0 (socket says 172.21.15.150:36664) 2024-03-17T23:48:19.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.863+0000 7f21660a5640 1 -- 172.21.15.150:0/3261475720 learned_addr learned my addr 172.21.15.150:0/3261475720 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:19.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.863+0000 7f21658a4640 1 --2- 172.21.15.150:0/3261475720 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f216015c2c0 0x7f21600aa820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:19.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.863+0000 7f21668a6640 1 --2- 172.21.15.150:0/3261475720 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21600cc690 0x7f21600ccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:19.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.863+0000 7f21660a5640 1 -- 172.21.15.150:0/3261475720 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21600cc690 msgr2=0x7f21600ccb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:19.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.863+0000 7f21660a5640 1 --2- 172.21.15.150:0/3261475720 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21600cc690 0x7f21600ccb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.863+0000 7f21660a5640 1 -- 172.21.15.150:0/3261475720 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f216015c2c0 msgr2=0x7f21600aa820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:19.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.863+0000 7f21660a5640 1 --2- 172.21.15.150:0/3261475720 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f216015c2c0 0x7f21600aa820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:19.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.863+0000 7f21660a5640 1 -- 172.21.15.150:0/3261475720 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21600cb5f0 con 0x7f214c0039e0 2024-03-17T23:48:19.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.863+0000 7f21658a4640 1 --2- 172.21.15.150:0/3261475720 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f216015c2c0 0x7f21600aa820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:19.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.863+0000 7f21668a6640 1 --2- 172.21.15.150:0/3261475720 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21600cc690 0x7f21600ccb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:19.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.863+0000 7f21660a5640 1 --2- 172.21.15.150:0/3261475720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f214c0039e0 0x7f21600cb010 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f215c00c2a0 tx=0x7f21600d3020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:19.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.863+0000 7f2156ffd640 1 -- 172.21.15.150:0/3261475720 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215c045070 con 0x7f214c0039e0 2024-03-17T23:48:19.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.863+0000 7f2156ffd640 1 -- 172.21.15.150:0/3261475720 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f215c000ed0 con 0x7f214c0039e0 2024-03-17T23:48:19.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.863+0000 7f2156ffd640 1 -- 172.21.15.150:0/3261475720 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215c001c30 con 0x7f214c0039e0 2024-03-17T23:48:19.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.863+0000 7f2168309640 1 -- 172.21.15.150:0/3261475720 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21600cb8d0 con 0x7f214c0039e0 2024-03-17T23:48:19.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.864+0000 7f2168309640 1 -- 172.21.15.150:0/3261475720 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21600cc170 con 0x7f214c0039e0 2024-03-17T23:48:19.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.864+0000 7f2168309640 1 -- 172.21.15.150:0/3261475720 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21600688b0 con 0x7f214c0039e0 2024-03-17T23:48:19.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.865+0000 7f2156ffd640 1 -- 172.21.15.150:0/3261475720 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f215c007070 con 0x7f214c0039e0 2024-03-17T23:48:19.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.865+0000 7f2156ffd640 1 --2- 172.21.15.150:0/3261475720 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f214406bc00 0x7f214406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:19.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.865+0000 7f2156ffd640 1 -- 172.21.15.150:0/3261475720 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7f215c0abb40 con 0x7f214c0039e0 2024-03-17T23:48:19.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.865+0000 7f21658a4640 1 --2- 172.21.15.150:0/3261475720 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f214406bc00 0x7f214406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:19.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.866+0000 7f21658a4640 1 --2- 172.21.15.150:0/3261475720 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f214406bc00 0x7f214406e0c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2150007fd0 tx=0x7f2150009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:19.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:19.870+0000 7f2156ffd640 1 -- 172.21.15.150:0/3261475720 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f215c079f30 con 0x7f214c0039e0 2024-03-17T23:48:20.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.092+0000 7f2168309640 1 -- 172.21.15.150:0/3261475720 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2160161730 con 0x7f214c0039e0 2024-03-17T23:48:20.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.092+0000 7f2156ffd640 1 -- 172.21.15.150:0/3261475720 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+5193 (secure 0 0 0) 0x7f215c07dfd0 con 0x7f214c0039e0 2024-03-17T23:48:20.093 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:20.093 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5177,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6834/3126446677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":3126446677},{"type":"v1","addr":"172.21.15.157:6834","nonce":3126446677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1992779366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1992779366},{"type":"v1","addr":"172.21.15.150:6836","nonce":1992779366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5487,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/3615467865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6833","nonce":3615467865},{"type":"v1","addr":"172.21.15.157:6835","nonce":3615467865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-17T23:48:10.730994+0000","modified":"2024-03-17T23:48:12.747344+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5515":{"gid":5515,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6837/1413722826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1413722826},{"type":"v1","addr":"172.21.15.150:6837","nonce":1413722826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5515,"qdb_cluster":[5515]},"id":2}]} 2024-03-17T23:48:20.094 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 15 2024-03-17T23:48:20.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.094+0000 7f2168309640 1 -- 172.21.15.150:0/3261475720 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f214406bc00 msgr2=0x7f214406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:20.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.094+0000 7f2168309640 1 --2- 172.21.15.150:0/3261475720 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f214406bc00 0x7f214406e0c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2150007fd0 tx=0x7f2150009040 comp rx=0 tx=0).stop 2024-03-17T23:48:20.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.094+0000 7f2168309640 1 -- 172.21.15.150:0/3261475720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f214c0039e0 msgr2=0x7f21600cb010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:20.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.094+0000 7f2168309640 1 --2- 172.21.15.150:0/3261475720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f214c0039e0 0x7f21600cb010 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f215c00c2a0 tx=0x7f21600d3020 comp rx=0 tx=0).stop 2024-03-17T23:48:20.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.094+0000 7f2168309640 1 -- 172.21.15.150:0/3261475720 shutdown_connections 2024-03-17T23:48:20.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.094+0000 7f2168309640 1 --2- 172.21.15.150:0/3261475720 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f21600cc690 0x7f21600ccb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.094+0000 7f2168309640 1 --2- 172.21.15.150:0/3261475720 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f214406bc00 0x7f214406e0c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.094+0000 7f2168309640 1 --2- 172.21.15.150:0/3261475720 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f216015c2c0 0x7f21600aa820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.094+0000 7f2168309640 1 --2- 172.21.15.150:0/3261475720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f214c0039e0 0x7f21600cb010 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.094+0000 7f2168309640 1 -- 172.21.15.150:0/3261475720 >> 172.21.15.150:0/3261475720 conn(0x7f2160073200 msgr2=0x7f21600b7400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:20.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.094+0000 7f2168309640 1 -- 172.21.15.150:0/3261475720 shutdown_connections 2024-03-17T23:48:20.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.094+0000 7f2168309640 1 -- 172.21.15.150:0/3261475720 wait complete. 2024-03-17T23:48:20.513 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-17T23:48:10.730994+0000', 'modified': '2024-03-17T23:48:12.747344+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5515}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5515': {'gid': 5515, 'name': 'c', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6837/1413722826', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6835', 'nonce': 1413722826}, {'type': 'v1', 'addr': '172.21.15.150:6837', 'nonce': 1413722826}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5515, 'qdb_cluster': [5515]} 2024-03-17T23:48:20.514 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-17T23:48:20.514 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-17T23:48:20.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.616+0000 7fa473e4d640 1 Processor -- start 2024-03-17T23:48:20.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.616+0000 7fa473e4d640 1 -- start start 2024-03-17T23:48:20.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.617+0000 7fa473e4d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa46c09a830 0x7fa46c15e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:20.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.617+0000 7fa473e4d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa46c0d04c0 0x7fa46c0d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:20.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.617+0000 7fa473e4d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa46c06c750 con 0x7fa46c06ccb0 2024-03-17T23:48:20.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.617+0000 7fa473e4d640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fa46c09ac90 con 0x7fa46c09a830 2024-03-17T23:48:20.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.617+0000 7fa473e4d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa46c0d0030 con 0x7fa46c0d04c0 2024-03-17T23:48:20.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.617+0000 7fa4723ea640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa46c0d04c0 0x7fa46c0d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:20.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.618+0000 7fa4713e8640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa46c09a830 0x7fa46c15e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:20.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.618+0000 7fa4723ea640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa46c0d04c0 0x7fa46c0d0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:60852/0 (socket says 172.21.15.150:60852) 2024-03-17T23:48:20.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.618+0000 7fa4723ea640 1 -- 172.21.15.150:0/2945689440 learned_addr learned my addr 172.21.15.150:0/2945689440 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:20.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.618+0000 7fa471be9640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fa46c06ccb0 0x7fa46c06d090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:53934/0 (socket says 172.21.15.150:53934) 2024-03-17T23:48:20.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.618+0000 7fa4713e8640 1 -- 172.21.15.150:0/2945689440 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa46c0d04c0 msgr2=0x7fa46c0d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:20.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.618+0000 7fa4713e8640 1 --2- 172.21.15.150:0/2945689440 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa46c0d04c0 0x7fa46c0d0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.618+0000 7fa4713e8640 1 -- 172.21.15.150:0/2945689440 >> v1:172.21.15.157:6789/0 conn(0x7fa46c06ccb0 legacy=0x7fa46c06d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:20.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.618+0000 7fa4713e8640 1 -- 172.21.15.150:0/2945689440 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa46c15edf0 con 0x7fa46c09a830 2024-03-17T23:48:20.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.618+0000 7fa4723ea640 1 --2- 172.21.15.150:0/2945689440 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa46c0d04c0 0x7fa46c0d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:48:20.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.619+0000 7fa4713e8640 1 --2- 172.21.15.150:0/2945689440 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa46c09a830 0x7fa46c15e4d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa4680097a0 tx=0x7fa46802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=222a892aa47d0c21 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:20.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.619+0000 7fa470be7640 1 -- 172.21.15.150:0/2945689440 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa468035020 con 0x7fa46c09a830 2024-03-17T23:48:20.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.619+0000 7fa470be7640 1 -- 172.21.15.150:0/2945689440 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa468031830 con 0x7fa46c09a830 2024-03-17T23:48:20.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.619+0000 7fa470be7640 1 -- 172.21.15.150:0/2945689440 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa468031c00 con 0x7fa46c09a830 2024-03-17T23:48:20.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.619+0000 7fa473e4d640 1 -- 172.21.15.150:0/2945689440 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa46c09a830 msgr2=0x7fa46c15e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:20.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.619+0000 7fa473e4d640 1 --2- 172.21.15.150:0/2945689440 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa46c09a830 0x7fa46c15e4d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa4680097a0 tx=0x7fa46802b8d0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.619+0000 7fa473e4d640 1 -- 172.21.15.150:0/2945689440 shutdown_connections 2024-03-17T23:48:20.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.619+0000 7fa473e4d640 1 --2- 172.21.15.150:0/2945689440 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa46c0d04c0 0x7fa46c0d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.619+0000 7fa473e4d640 1 --2- 172.21.15.150:0/2945689440 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa46c09a830 0x7fa46c15e4d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.619+0000 7fa473e4d640 1 -- 172.21.15.150:0/2945689440 >> 172.21.15.150:0/2945689440 conn(0x7fa46c0b36b0 msgr2=0x7fa46c0b3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:20.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.620+0000 7fa473e4d640 1 -- 172.21.15.150:0/2945689440 shutdown_connections 2024-03-17T23:48:20.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.620+0000 7fa473e4d640 1 -- 172.21.15.150:0/2945689440 wait complete. 2024-03-17T23:48:20.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.621+0000 7fa473e4d640 1 Processor -- start 2024-03-17T23:48:20.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.621+0000 7fa473e4d640 1 -- start start 2024-03-17T23:48:20.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.622+0000 7fa473e4d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa46c06ccb0 0x7fa46c1592f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:20.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.622+0000 7fa473e4d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa46c09a830 0x7fa46c0d0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:20.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.622+0000 7fa473e4d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa46c0d04c0 0x7fa46c0d0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:20.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.622+0000 7fa473e4d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c15f690 con 0x7fa46c0d04c0 2024-03-17T23:48:20.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.622+0000 7fa473e4d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c15f3b0 con 0x7fa46c09a830 2024-03-17T23:48:20.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.622+0000 7fa473e4d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa46c15ea10 con 0x7fa46c06ccb0 2024-03-17T23:48:20.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.622+0000 7fa4723ea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa46c0d04c0 0x7fa46c0d0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:20.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.622+0000 7fa4723ea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa46c0d04c0 0x7fa46c0d0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36678/0 (socket says 172.21.15.150:36678) 2024-03-17T23:48:20.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.622+0000 7fa471be9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa46c06ccb0 0x7fa46c1592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:20.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.622+0000 7fa4723ea640 1 -- 172.21.15.150:0/4003070725 learned_addr learned my addr 172.21.15.150:0/4003070725 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:20.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.622+0000 7fa4713e8640 1 --2- 172.21.15.150:0/4003070725 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa46c09a830 0x7fa46c0d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:20.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.622+0000 7fa4723ea640 1 -- 172.21.15.150:0/4003070725 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa46c06ccb0 msgr2=0x7fa46c1592f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:20.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.622+0000 7fa4723ea640 1 --2- 172.21.15.150:0/4003070725 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa46c06ccb0 0x7fa46c1592f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.622+0000 7fa4723ea640 1 -- 172.21.15.150:0/4003070725 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa46c09a830 msgr2=0x7fa46c0d0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:20.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.622+0000 7fa4723ea640 1 --2- 172.21.15.150:0/4003070725 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa46c09a830 0x7fa46c0d0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.622+0000 7fa4723ea640 1 -- 172.21.15.150:0/4003070725 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa46c1f8710 con 0x7fa46c0d04c0 2024-03-17T23:48:20.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.622+0000 7fa471be9640 1 --2- 172.21.15.150:0/4003070725 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa46c06ccb0 0x7fa46c1592f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:48:20.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.622+0000 7fa4723ea640 1 --2- 172.21.15.150:0/4003070725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa46c0d04c0 0x7fa46c0d0e70 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fa46400efc0 tx=0x7fa46400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:20.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.622+0000 7fa4627fc640 1 -- 172.21.15.150:0/4003070725 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46400cbe0 con 0x7fa46c0d04c0 2024-03-17T23:48:20.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.622+0000 7fa4627fc640 1 -- 172.21.15.150:0/4003070725 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa46400cd80 con 0x7fa46c0d04c0 2024-03-17T23:48:20.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.622+0000 7fa473e4d640 1 -- 172.21.15.150:0/4003070725 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa46c1f7950 con 0x7fa46c0d04c0 2024-03-17T23:48:20.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.623+0000 7fa4627fc640 1 -- 172.21.15.150:0/4003070725 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa464003bb0 con 0x7fa46c0d04c0 2024-03-17T23:48:20.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.623+0000 7fa473e4d640 1 -- 172.21.15.150:0/4003070725 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa46c1f7f10 con 0x7fa46c0d04c0 2024-03-17T23:48:20.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.624+0000 7fa4627fc640 1 -- 172.21.15.150:0/4003070725 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa46401b030 con 0x7fa46c0d04c0 2024-03-17T23:48:20.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.624+0000 7fa4627fc640 1 --2- 172.21.15.150:0/4003070725 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa45006bbd0 0x7fa45006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:20.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.625+0000 7fa471be9640 1 --2- 172.21.15.150:0/4003070725 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa45006bbd0 0x7fa45006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:20.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.625+0000 7fa4627fc640 1 -- 172.21.15.150:0/4003070725 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7fa46408aed0 con 0x7fa46c0d04c0 2024-03-17T23:48:20.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.625+0000 7fa471be9640 1 --2- 172.21.15.150:0/4003070725 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa45006bbd0 0x7fa45006e090 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa45c007fd0 tx=0x7fa45c007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:20.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.626+0000 7fa476a30740 1 Processor -- start 2024-03-17T23:48:20.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.626+0000 7fa476a30740 1 -- start start 2024-03-17T23:48:20.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.626+0000 7fa476a30740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af83cc80 0x5601af80fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:20.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.626+0000 7fa476a30740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5601af83d060 0x5601af83fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:20.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.626+0000 7fa476a30740 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5601af810160 0x5601af83ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:20.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.626+0000 7fa476a30740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5601af778c80 con 0x5601af83cc80 2024-03-17T23:48:20.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.626+0000 7fa476a30740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5601af721aa0 con 0x5601af83d060 2024-03-17T23:48:20.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.626+0000 7fa476a30740 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x5601af774d20 con 0x5601af810160 2024-03-17T23:48:20.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.626+0000 7fa471be9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af83cc80 0x5601af80fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:20.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.626+0000 7fa471be9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af83cc80 0x5601af80fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36694/0 (socket says 172.21.15.150:36694) 2024-03-17T23:48:20.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.626+0000 7fa471be9640 1 -- 172.21.15.150:0/1235452178 learned_addr learned my addr 172.21.15.150:0/1235452178 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:20.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.627+0000 7fa471be9640 1 -- 172.21.15.150:0/1235452178 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5601af810160 msgr2=0x5601af83ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:20.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.627+0000 7fa471be9640 1 --2- 172.21.15.150:0/1235452178 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5601af810160 0x5601af83ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.627+0000 7fa471be9640 1 -- 172.21.15.150:0/1235452178 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5601af83d060 msgr2=0x5601af83fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:20.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.627+0000 7fa4713e8640 1 --2- 172.21.15.150:0/1235452178 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5601af83d060 0x5601af83fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:20.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.627+0000 7fa471be9640 1 --2- 172.21.15.150:0/1235452178 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5601af83d060 0x5601af83fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.627+0000 7fa471be9640 1 -- 172.21.15.150:0/1235452178 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601af743ff0 con 0x5601af83cc80 2024-03-17T23:48:20.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.627+0000 7fa471be9640 1 --2- 172.21.15.150:0/1235452178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af83cc80 0x5601af80fc20 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fa45c007520 tx=0x7fa45c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35695c435aabf019 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:20.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.627+0000 7fa44f7fe640 1 -- 172.21.15.150:0/1235452178 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c026070 con 0x5601af83cc80 2024-03-17T23:48:20.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.627+0000 7fa44f7fe640 1 -- 172.21.15.150:0/1235452178 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa45c013e00 con 0x5601af83cc80 2024-03-17T23:48:20.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.627+0000 7fa44f7fe640 1 -- 172.21.15.150:0/1235452178 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c012550 con 0x5601af83cc80 2024-03-17T23:48:20.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.628+0000 7fa476a30740 1 -- 172.21.15.150:0/1235452178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af83cc80 msgr2=0x5601af80fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:20.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.628+0000 7fa476a30740 1 --2- 172.21.15.150:0/1235452178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af83cc80 0x5601af80fc20 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fa45c007520 tx=0x7fa45c013000 comp rx=0 tx=0).stop 2024-03-17T23:48:20.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.628+0000 7fa476a30740 1 -- 172.21.15.150:0/1235452178 shutdown_connections 2024-03-17T23:48:20.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.628+0000 7fa476a30740 1 --2- 172.21.15.150:0/1235452178 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5601af810160 0x5601af83ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.628+0000 7fa476a30740 1 --2- 172.21.15.150:0/1235452178 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5601af83d060 0x5601af83fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.628+0000 7fa476a30740 1 --2- 172.21.15.150:0/1235452178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af83cc80 0x5601af80fc20 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.628+0000 7fa476a30740 1 -- 172.21.15.150:0/1235452178 >> 172.21.15.150:0/1235452178 conn(0x5601af811e90 msgr2=0x5601af85aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:20.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.628+0000 7fa476a30740 1 -- 172.21.15.150:0/1235452178 shutdown_connections 2024-03-17T23:48:20.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.628+0000 7fa476a30740 1 -- 172.21.15.150:0/1235452178 wait complete. 2024-03-17T23:48:20.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.628+0000 7fa476a30740 1 -- start start 2024-03-17T23:48:20.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.628+0000 7fa476a30740 1 Processor -- start 2024-03-17T23:48:20.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.629+0000 7fa476a30740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5601af83d060 0x5601af85b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:20.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.629+0000 7fa476a30740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af840c20 0x5601af86e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:20.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.629+0000 7fa476a30740 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5601af841000 0x5601af86e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:20.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.629+0000 7fa476a30740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5601af814530 con 0x5601af840c20 2024-03-17T23:48:20.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.629+0000 7fa476a30740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5601af734ef0 con 0x5601af83d060 2024-03-17T23:48:20.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.629+0000 7fa476a30740 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x5601af816d70 con 0x5601af841000 2024-03-17T23:48:20.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.629+0000 7fa476a30740 10 client.0 _ll_register_callbacks cb 0x5601af72fdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:48:20.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.629+0000 7fa4713e8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af840c20 0x5601af86e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:20.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.629+0000 7fa4713e8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af840c20 0x5601af86e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36706/0 (socket says 172.21.15.150:36706) 2024-03-17T23:48:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.629+0000 7fa4713e8640 1 -- 172.21.15.150:0/3131444166 learned_addr learned my addr 172.21.15.150:0/3131444166 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.630+0000 7fa4723ea640 1 --2- 172.21.15.150:0/3131444166 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5601af841000 0x5601af86e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.630+0000 7fa471be9640 1 --2- 172.21.15.150:0/3131444166 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5601af83d060 0x5601af85b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.630+0000 7fa4713e8640 1 -- 172.21.15.150:0/3131444166 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5601af841000 msgr2=0x5601af86e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.630+0000 7fa4713e8640 1 --2- 172.21.15.150:0/3131444166 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5601af841000 0x5601af86e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.630+0000 7fa4713e8640 1 -- 172.21.15.150:0/3131444166 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5601af83d060 msgr2=0x5601af85b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.630+0000 7fa4713e8640 1 --2- 172.21.15.150:0/3131444166 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5601af83d060 0x5601af85b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.630+0000 7fa4713e8640 1 -- 172.21.15.150:0/3131444166 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601af741a30 con 0x5601af840c20 2024-03-17T23:48:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.630+0000 7fa4713e8640 1 --2- 172.21.15.150:0/3131444166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af840c20 0x5601af86e3b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa46802bee0 tx=0x7fa46802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2119a43c3074d298 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.630+0000 7fa44cff9640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-17T23:48:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.630+0000 7fa44cff9640 1 -- 172.21.15.150:0/3131444166 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa468049060 con 0x5601af840c20 2024-03-17T23:48:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.630+0000 7fa476a30740 1 -- 172.21.15.150:0/3131444166 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa45c013e00 con 0x5601af840c20 2024-03-17T23:48:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.630+0000 7fa44cff9640 1 -- 172.21.15.150:0/3131444166 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7fa468031df0 con 0x5601af840c20 2024-03-17T23:48:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.630+0000 7fa44cff9640 1 -- 172.21.15.150:0/3131444166 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7fa46804f050 con 0x5601af840c20 2024-03-17T23:48:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.630+0000 7fa476a30740 10 client.5691 fetch_fsmap learned FSMap version 15 2024-03-17T23:48:20.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.630+0000 7fa476a30740 1 -- 172.21.15.150:0/3131444166 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x5601af48d2d0 con 0x5601af840c20 2024-03-17T23:48:20.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.630+0000 7fa44cff9640 1 -- 172.21.15.150:0/3131444166 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa45c013e00 con 0x5601af840c20 2024-03-17T23:48:20.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.631+0000 7fa44cff9640 1 -- 172.21.15.150:0/3131444166 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 15) ==== 3494+0+0 (secure 0 0 0) 0x7fa468052020 con 0x5601af840c20 2024-03-17T23:48:20.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.631+0000 7fa476a30740 10 client.5691 fetch_fsmap finished waiting for FSMap version 15 2024-03-17T23:48:20.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.631+0000 7fa476a30740 10 client.5691 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-17T23:48:20.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.631+0000 7fa476a30740 20 client.5691 populate_metadata read hostname 'smithi150' 2024-03-17T23:48:20.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.631+0000 7fa476a30740 1 --2- 172.21.15.150:0/3131444166 >> [v2:172.21.15.150:6835/1413722826,v1:172.21.15.150:6837/1413722826] conn(0x5601af859ab0 0x5601af83e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:20.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.631+0000 7fa476a30740 4 client.5691 mds_command: new command op to 5515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-17T23:48:20.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.631+0000 7fa476a30740 1 -- 172.21.15.150:0/3131444166 --> [v2:172.21.15.150:6835/1413722826,v1:172.21.15.150:6837/1413722826] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5601af7861c0 con 0x5601af859ab0 2024-03-17T23:48:20.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.631+0000 7fa471be9640 1 --2- 172.21.15.150:0/3131444166 >> [v2:172.21.15.150:6835/1413722826,v1:172.21.15.150:6837/1413722826] conn(0x5601af859ab0 0x5601af83e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:20.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.631+0000 7fa44cff9640 10 client.5691 ms_handle_connect on v2:172.21.15.150:6835/1413722826 2024-03-17T23:48:20.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.631+0000 7fa471be9640 1 --2- 172.21.15.150:0/3131444166 >> [v2:172.21.15.150:6835/1413722826,v1:172.21.15.150:6837/1413722826] conn(0x5601af859ab0 0x5601af83e970 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e62f52e7411dea05 server_cookie=3716a4e282ec4cc7 in_seq=0 out_seq=0 2024-03-17T23:48:20.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.632+0000 7fa44cff9640 1 -- 172.21.15.150:0/3131444166 <== mds.0 v2:172.21.15.150:6835/1413722826 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5601af7861c0 con 0x5601af859ab0 2024-03-17T23:48:20.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.632+0000 7fa44cff9640 10 client.5691 handle_command_reply: tid=0 2024-03-17T23:48:20.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.632+0000 7fa476a30740 1 client.5691 shutdown 2024-03-17T23:48:20.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.632+0000 7fa476a30740 1 -- 172.21.15.150:0/3131444166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af840c20 msgr2=0x5601af86e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:20.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.632+0000 7fa476a30740 1 --2- 172.21.15.150:0/3131444166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af840c20 0x5601af86e3b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa46802bee0 tx=0x7fa46802e000 comp rx=0 tx=0).stop 2024-03-17T23:48:20.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.632+0000 7fa476a30740 1 -- 172.21.15.150:0/3131444166 shutdown_connections 2024-03-17T23:48:20.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.632+0000 7fa476a30740 1 --2- 172.21.15.150:0/3131444166 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5601af841000 0x5601af86e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.632+0000 7fa476a30740 1 --2- 172.21.15.150:0/3131444166 >> [v2:172.21.15.150:6835/1413722826,v1:172.21.15.150:6837/1413722826] conn(0x5601af859ab0 0x5601af83e970 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.632+0000 7fa476a30740 1 --2- 172.21.15.150:0/3131444166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af840c20 0x5601af86e3b0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.632+0000 7fa476a30740 1 --2- 172.21.15.150:0/3131444166 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5601af83d060 0x5601af85b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.632+0000 7fa44cff9640 0 client.5691 ms_handle_reset on v2:172.21.15.150:6835/1413722826 2024-03-17T23:48:20.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.632+0000 7fa476a30740 1 -- 172.21.15.150:0/3131444166 >> 172.21.15.150:0/3131444166 conn(0x5601af810160 msgr2=0x5601af85d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:20.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.632+0000 7fa476a30740 1 -- 172.21.15.150:0/3131444166 shutdown_connections 2024-03-17T23:48:20.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.632+0000 7fa476a30740 1 -- 172.21.15.150:0/3131444166 wait complete. 2024-03-17T23:48:20.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.632+0000 7fa476a30740 20 client.5691 trim_cache size 0 max 16384 2024-03-17T23:48:20.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.650+0000 7fa476a30740 1 Processor -- start 2024-03-17T23:48:20.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.650+0000 7fa476a30740 1 -- start start 2024-03-17T23:48:20.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.651+0000 7fa476a30740 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5601af83d060 0x5601af86e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:20.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.651+0000 7fa476a30740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af840c20 0x5601af86e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:20.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.651+0000 7fa476a30740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5601af841000 0x5601af814b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:20.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.651+0000 7fa476a30740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1ffc50 con 0x5601af840c20 2024-03-17T23:48:20.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.651+0000 7fa476a30740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c0c6b80 con 0x5601af841000 2024-03-17T23:48:20.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.651+0000 7fa476a30740 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa46c0c71d0 con 0x5601af83d060 2024-03-17T23:48:20.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.651+0000 7fa4713e8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af840c20 0x5601af86e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:20.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.651+0000 7fa4713e8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af840c20 0x5601af86e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36708/0 (socket says 172.21.15.150:36708) 2024-03-17T23:48:20.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.651+0000 7fa4713e8640 1 -- 172.21.15.150:0/2649287912 learned_addr learned my addr 172.21.15.150:0/2649287912 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:20.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.651+0000 7fa4713e8640 1 -- 172.21.15.150:0/2649287912 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5601af83d060 msgr2=0x5601af86e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:20.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.651+0000 7fa4713e8640 1 --2- 172.21.15.150:0/2649287912 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5601af83d060 0x5601af86e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.651+0000 7fa4713e8640 1 -- 172.21.15.150:0/2649287912 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5601af841000 msgr2=0x5601af814b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:20.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.651+0000 7fa4713e8640 1 --2- 172.21.15.150:0/2649287912 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5601af841000 0x5601af814b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.651+0000 7fa4713e8640 1 -- 172.21.15.150:0/2649287912 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601af5576b0 con 0x5601af840c20 2024-03-17T23:48:20.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.652+0000 7fa4713e8640 1 --2- 172.21.15.150:0/2649287912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af840c20 0x5601af86e880 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa468044440 tx=0x7fa468031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e91a3589cb18ccd server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:20.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.652+0000 7fa44effd640 1 -- 172.21.15.150:0/2649287912 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa468035040 con 0x5601af840c20 2024-03-17T23:48:20.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.652+0000 7fa44effd640 1 -- 172.21.15.150:0/2649287912 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa468004230 con 0x5601af840c20 2024-03-17T23:48:20.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.652+0000 7fa44effd640 1 -- 172.21.15.150:0/2649287912 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa468044820 con 0x5601af840c20 2024-03-17T23:48:20.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.652+0000 7fa476a30740 1 -- 172.21.15.150:0/2649287912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af840c20 msgr2=0x5601af86e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:20.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.652+0000 7fa476a30740 1 --2- 172.21.15.150:0/2649287912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af840c20 0x5601af86e880 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa468044440 tx=0x7fa468031000 comp rx=0 tx=0).stop 2024-03-17T23:48:20.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.652+0000 7fa476a30740 1 -- 172.21.15.150:0/2649287912 shutdown_connections 2024-03-17T23:48:20.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.652+0000 7fa476a30740 1 --2- 172.21.15.150:0/2649287912 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5601af841000 0x5601af814b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.652+0000 7fa476a30740 1 --2- 172.21.15.150:0/2649287912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af840c20 0x5601af86e880 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.652+0000 7fa476a30740 1 --2- 172.21.15.150:0/2649287912 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5601af83d060 0x5601af86e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.652+0000 7fa476a30740 1 -- 172.21.15.150:0/2649287912 >> 172.21.15.150:0/2649287912 conn(0x5601af810160 msgr2=0x5601af83e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:20.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.652+0000 7fa476a30740 1 -- 172.21.15.150:0/2649287912 shutdown_connections 2024-03-17T23:48:20.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.652+0000 7fa476a30740 1 -- 172.21.15.150:0/2649287912 wait complete. 2024-03-17T23:48:20.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.653+0000 7fa476a30740 1 -- start start 2024-03-17T23:48:20.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.653+0000 7fa476a30740 1 Processor -- start 2024-03-17T23:48:20.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.654+0000 7fa476a30740 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5601af83d060 0x5601af85b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:20.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.654+0000 7fa476a30740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5601af840c20 0x5601af8d5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:20.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.654+0000 7fa476a30740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af841000 0x5601af80eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:20.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.654+0000 7fa476a30740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5601af859930 con 0x5601af841000 2024-03-17T23:48:20.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.654+0000 7fa476a30740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5601af83fa50 con 0x5601af840c20 2024-03-17T23:48:20.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.654+0000 7fa476a30740 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x5601af859f50 con 0x5601af83d060 2024-03-17T23:48:20.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.654+0000 7fa476a30740 10 client.0 _ll_register_callbacks cb 0x7fa468009620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:48:20.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.654+0000 7fa4723ea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af841000 0x5601af80eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:20.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.654+0000 7fa4723ea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af841000 0x5601af80eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36712/0 (socket says 172.21.15.150:36712) 2024-03-17T23:48:20.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.654+0000 7fa4723ea640 1 -- 172.21.15.150:0/2292118547 learned_addr learned my addr 172.21.15.150:0/2292118547 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:20.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.654+0000 7fa471be9640 1 --2- 172.21.15.150:0/2292118547 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5601af83d060 0x5601af85b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:20.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.654+0000 7fa4713e8640 1 --2- 172.21.15.150:0/2292118547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5601af840c20 0x5601af8d5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:20.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.654+0000 7fa4723ea640 1 -- 172.21.15.150:0/2292118547 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5601af83d060 msgr2=0x5601af85b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:20.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.654+0000 7fa4723ea640 1 --2- 172.21.15.150:0/2292118547 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5601af83d060 0x5601af85b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.654+0000 7fa4723ea640 1 -- 172.21.15.150:0/2292118547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5601af840c20 msgr2=0x5601af8d5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:20.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.654+0000 7fa4723ea640 1 --2- 172.21.15.150:0/2292118547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5601af840c20 0x5601af8d5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.654+0000 7fa4723ea640 1 -- 172.21.15.150:0/2292118547 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601af86a2b0 con 0x5601af841000 2024-03-17T23:48:20.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.655+0000 7fa4723ea640 1 --2- 172.21.15.150:0/2292118547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af841000 0x5601af80eaf0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa4640556d0 tx=0x7fa464058ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fec525548734844 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:20.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.655+0000 7fa44cff9640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-17T23:48:20.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.655+0000 7fa44cff9640 1 -- 172.21.15.150:0/2292118547 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4640588c0 con 0x5601af841000 2024-03-17T23:48:20.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.655+0000 7fa44cff9640 1 -- 172.21.15.150:0/2292118547 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7fa46405f2a0 con 0x5601af841000 2024-03-17T23:48:20.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.655+0000 7fa476a30740 1 -- 172.21.15.150:0/2292118547 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa468004230 con 0x5601af841000 2024-03-17T23:48:20.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.655+0000 7fa44cff9640 1 -- 172.21.15.150:0/2292118547 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7fa464077050 con 0x5601af841000 2024-03-17T23:48:20.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.655+0000 7fa476a30740 10 client.5697 fetch_fsmap learned FSMap version 15 2024-03-17T23:48:20.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.655+0000 7fa476a30740 1 -- 172.21.15.150:0/2292118547 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x5601af80f550 con 0x5601af841000 2024-03-17T23:48:20.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.655+0000 7fa44cff9640 1 -- 172.21.15.150:0/2292118547 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa468004230 con 0x5601af841000 2024-03-17T23:48:20.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.655+0000 7fa44cff9640 1 -- 172.21.15.150:0/2292118547 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 15) ==== 3494+0+0 (secure 0 0 0) 0x7fa464054fc0 con 0x5601af841000 2024-03-17T23:48:20.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.655+0000 7fa476a30740 10 client.5697 fetch_fsmap finished waiting for FSMap version 15 2024-03-17T23:48:20.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.655+0000 7fa476a30740 10 client.5697 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-17T23:48:20.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.655+0000 7fa476a30740 20 client.5697 populate_metadata read hostname 'smithi150' 2024-03-17T23:48:20.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.655+0000 7fa476a30740 1 --2- 172.21.15.150:0/2292118547 >> [v2:172.21.15.150:6835/1413722826,v1:172.21.15.150:6837/1413722826] conn(0x5601af837560 0x5601af837940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:20.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.655+0000 7fa476a30740 4 client.5697 mds_command: new command op to 5515 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-17T23:48:20.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.655+0000 7fa476a30740 1 -- 172.21.15.150:0/2292118547 --> [v2:172.21.15.150:6835/1413722826,v1:172.21.15.150:6837/1413722826] -- command(tid 0: {"prefix": "status"}) -- 0x5601af842e30 con 0x5601af837560 2024-03-17T23:48:20.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.655+0000 7fa471be9640 1 --2- 172.21.15.150:0/2292118547 >> [v2:172.21.15.150:6835/1413722826,v1:172.21.15.150:6837/1413722826] conn(0x5601af837560 0x5601af837940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:20.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.656+0000 7fa44cff9640 10 client.5697 ms_handle_connect on v2:172.21.15.150:6835/1413722826 2024-03-17T23:48:20.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.656+0000 7fa471be9640 1 --2- 172.21.15.150:0/2292118547 >> [v2:172.21.15.150:6835/1413722826,v1:172.21.15.150:6837/1413722826] conn(0x5601af837560 0x5601af837940 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ed17d24fb9a79bc server_cookie=413b2743a0322d85 in_seq=0 out_seq=0 2024-03-17T23:48:20.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.656+0000 7fa44cff9640 1 -- 172.21.15.150:0/2292118547 <== mds.0 v2:172.21.15.150:6835/1413722826 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x5601af842e30 con 0x5601af837560 2024-03-17T23:48:20.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.656+0000 7fa44cff9640 10 client.5697 handle_command_reply: tid=0 2024-03-17T23:48:20.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.656+0000 7fa476a30740 1 client.5697 shutdown 2024-03-17T23:48:20.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.656+0000 7fa476a30740 1 -- 172.21.15.150:0/2292118547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af841000 msgr2=0x5601af80eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:20.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.656+0000 7fa476a30740 1 --2- 172.21.15.150:0/2292118547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af841000 0x5601af80eaf0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa4640556d0 tx=0x7fa464058ed0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.656+0000 7fa476a30740 1 -- 172.21.15.150:0/2292118547 shutdown_connections 2024-03-17T23:48:20.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.656+0000 7fa476a30740 1 --2- 172.21.15.150:0/2292118547 >> [v2:172.21.15.150:6835/1413722826,v1:172.21.15.150:6837/1413722826] conn(0x5601af837560 0x5601af837940 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.656+0000 7fa476a30740 1 --2- 172.21.15.150:0/2292118547 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5601af841000 0x5601af80eaf0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.656+0000 7fa44cff9640 0 client.5697 ms_handle_reset on v2:172.21.15.150:6835/1413722826 2024-03-17T23:48:20.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.656+0000 7fa476a30740 1 --2- 172.21.15.150:0/2292118547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5601af840c20 0x5601af8d5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.656+0000 7fa476a30740 1 --2- 172.21.15.150:0/2292118547 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x5601af83d060 0x5601af85b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.656+0000 7fa476a30740 1 -- 172.21.15.150:0/2292118547 >> 172.21.15.150:0/2292118547 conn(0x5601af810160 msgr2=0x5601af86e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:20.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.656+0000 7fa476a30740 1 -- 172.21.15.150:0/2292118547 shutdown_connections 2024-03-17T23:48:20.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.656+0000 7fa476a30740 1 -- 172.21.15.150:0/2292118547 wait complete. 2024-03-17T23:48:20.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.656+0000 7fa476a30740 20 client.5697 trim_cache size 0 max 16384 2024-03-17T23:48:20.663 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-17T23:48:20.663 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_fsid": "a1c5acd3-e97f-470b-8226-335736a7795b", 2024-03-17T23:48:20.663 INFO:teuthology.orchestra.run.smithi150.stdout: "whoami": 0, 2024-03-17T23:48:20.663 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 5515, 2024-03-17T23:48:20.663 INFO:teuthology.orchestra.run.smithi150.stdout: "want_state": "up:active", 2024-03-17T23:48:20.663 INFO:teuthology.orchestra.run.smithi150.stdout: "state": "up:active", 2024-03-17T23:48:20.663 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "cephfs", 2024-03-17T23:48:20.663 INFO:teuthology.orchestra.run.smithi150.stdout: "rank_uptime": 9.2788059230000002, 2024-03-17T23:48:20.663 INFO:teuthology.orchestra.run.smithi150.stdout: "mdsmap_epoch": 15, 2024-03-17T23:48:20.663 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch": 40, 2024-03-17T23:48:20.663 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch_barrier": 40, 2024-03-17T23:48:20.663 INFO:teuthology.orchestra.run.smithi150.stdout: "uptime": 16.374101598999999 2024-03-17T23:48:20.663 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-17T23:48:20.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.657+0000 7fa473e4d640 1 -- 172.21.15.150:0/4003070725 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa45006bbd0 msgr2=0x7fa45006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:20.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.657+0000 7fa473e4d640 1 --2- 172.21.15.150:0/4003070725 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa45006bbd0 0x7fa45006e090 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa45c007fd0 tx=0x7fa45c007e80 comp rx=0 tx=0).stop 2024-03-17T23:48:20.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.657+0000 7fa473e4d640 1 -- 172.21.15.150:0/4003070725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa46c0d04c0 msgr2=0x7fa46c0d0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:20.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.657+0000 7fa473e4d640 1 --2- 172.21.15.150:0/4003070725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa46c0d04c0 0x7fa46c0d0e70 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fa46400efc0 tx=0x7fa46400c490 comp rx=0 tx=0).stop 2024-03-17T23:48:20.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.657+0000 7fa473e4d640 1 -- 172.21.15.150:0/4003070725 shutdown_connections 2024-03-17T23:48:20.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.657+0000 7fa473e4d640 1 --2- 172.21.15.150:0/4003070725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa46c0d04c0 0x7fa46c0d0e70 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.657+0000 7fa473e4d640 1 --2- 172.21.15.150:0/4003070725 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa45006bbd0 0x7fa45006e090 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.657+0000 7fa473e4d640 1 --2- 172.21.15.150:0/4003070725 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa46c09a830 0x7fa46c0d0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.657+0000 7fa473e4d640 1 --2- 172.21.15.150:0/4003070725 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa46c06ccb0 0x7fa46c1592f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.657+0000 7fa473e4d640 1 -- 172.21.15.150:0/4003070725 >> 172.21.15.150:0/4003070725 conn(0x7fa46c0b36b0 msgr2=0x7fa46c073b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:20.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.657+0000 7fa473e4d640 1 -- 172.21.15.150:0/4003070725 shutdown_connections 2024-03-17T23:48:20.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.657+0000 7fa473e4d640 1 -- 172.21.15.150:0/4003070725 wait complete. 2024-03-17T23:48:20.668 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-17T23:48:20.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.768+0000 7feb29afe640 1 Processor -- start 2024-03-17T23:48:20.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.768+0000 7feb29afe640 1 -- start start 2024-03-17T23:48:20.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.769+0000 7feb29afe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7feb241619a0 0x7feb240b9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:20.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.769+0000 7feb29afe640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb24161dd0 con 0x7feb240cf120 2024-03-17T23:48:20.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.769+0000 7feb29afe640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb241620a0 con 0x7feb2415de30 2024-03-17T23:48:20.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.769+0000 7feb29afe640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7feb24162940 con 0x7feb241619a0 2024-03-17T23:48:20.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.769+0000 7feb22ffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7feb240cf120 0x7feb240d2690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:48776/0 (socket says 172.21.15.150:48776) 2024-03-17T23:48:20.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.769+0000 7feb22ffd640 1 -- 172.21.15.150:0/3028363780 learned_addr learned my addr 172.21.15.150:0/3028363780 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:20.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.769+0000 7feb23fff640 1 --2- 172.21.15.150:0/3028363780 >> v2:172.21.15.150:3300/0 conn(0x7feb241619a0 0x7feb240b9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:20.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.769+0000 7feb23fff640 1 -- 172.21.15.150:0/3028363780 >> v1:172.21.15.157:6789/0 conn(0x7feb2415de30 legacy=0x7feb240cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:20.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.769+0000 7feb23fff640 1 -- 172.21.15.150:0/3028363780 >> v1:172.21.15.150:6789/0 conn(0x7feb240cf120 legacy=0x7feb240d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:20.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.769+0000 7feb23fff640 1 -- 172.21.15.150:0/3028363780 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb24162380 con 0x7feb241619a0 2024-03-17T23:48:20.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.769+0000 7feb227fc640 1 -- 172.21.15.150:0/3028363780 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 112663813 0 0) 0x7feb24161dd0 con 0x7feb240cf120 2024-03-17T23:48:20.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.770+0000 7feb23fff640 1 --2- 172.21.15.150:0/3028363780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb241619a0 0x7feb240b9860 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7feb1c009d90 tx=0x7feb1c02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b24ff38974be8657 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:20.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.770+0000 7feb227fc640 1 -- 172.21.15.150:0/3028363780 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb1c039070 con 0x7feb241619a0 2024-03-17T23:48:20.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.770+0000 7feb227fc640 1 -- 172.21.15.150:0/3028363780 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb1c02cbd0 con 0x7feb241619a0 2024-03-17T23:48:20.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.770+0000 7feb227fc640 1 -- 172.21.15.150:0/3028363780 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb1c034440 con 0x7feb241619a0 2024-03-17T23:48:20.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.771+0000 7feb29afe640 1 -- 172.21.15.150:0/3028363780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb241619a0 msgr2=0x7feb240b9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:20.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.771+0000 7feb29afe640 1 --2- 172.21.15.150:0/3028363780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb241619a0 0x7feb240b9860 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7feb1c009d90 tx=0x7feb1c02be20 comp rx=0 tx=0).stop 2024-03-17T23:48:20.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.771+0000 7feb29afe640 1 -- 172.21.15.150:0/3028363780 shutdown_connections 2024-03-17T23:48:20.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.771+0000 7feb29afe640 1 --2- 172.21.15.150:0/3028363780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb241619a0 0x7feb240b9860 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.771+0000 7feb29afe640 1 -- 172.21.15.150:0/3028363780 >> 172.21.15.150:0/3028363780 conn(0x7feb240730b0 msgr2=0x7feb240d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:20.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.771+0000 7feb29afe640 1 -- 172.21.15.150:0/3028363780 shutdown_connections 2024-03-17T23:48:20.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.771+0000 7feb29afe640 1 -- 172.21.15.150:0/3028363780 wait complete. 2024-03-17T23:48:20.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.772+0000 7feb29afe640 1 Processor -- start 2024-03-17T23:48:20.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.772+0000 7feb29afe640 1 -- start start 2024-03-17T23:48:20.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.773+0000 7feb29afe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb240cf120 0x7feb241579c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:20.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.773+0000 7feb29afe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feb2415de30 0x7feb240c95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:20.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.773+0000 7feb29afe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feb241619a0 0x7feb24148c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:20.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.773+0000 7feb29afe640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7feb2415e210 con 0x7feb240cf120 2024-03-17T23:48:20.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.773+0000 7feb29afe640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7feb240d2170 con 0x7feb241619a0 2024-03-17T23:48:20.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.773+0000 7feb29afe640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7feb240d2da0 con 0x7feb2415de30 2024-03-17T23:48:20.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.773+0000 7feb237fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb240cf120 0x7feb241579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:20.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.773+0000 7feb237fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb240cf120 0x7feb241579c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36732/0 (socket says 172.21.15.150:36732) 2024-03-17T23:48:20.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.773+0000 7feb237fe640 1 -- 172.21.15.150:0/3483222398 learned_addr learned my addr 172.21.15.150:0/3483222398 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:20.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.773+0000 7feb23fff640 1 --2- 172.21.15.150:0/3483222398 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feb241619a0 0x7feb24148c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:20.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.773+0000 7feb22ffd640 1 --2- 172.21.15.150:0/3483222398 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feb2415de30 0x7feb240c95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:20.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.773+0000 7feb237fe640 1 -- 172.21.15.150:0/3483222398 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feb2415de30 msgr2=0x7feb240c95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:20.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.773+0000 7feb237fe640 1 --2- 172.21.15.150:0/3483222398 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feb2415de30 0x7feb240c95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.773+0000 7feb237fe640 1 -- 172.21.15.150:0/3483222398 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feb241619a0 msgr2=0x7feb24148c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:20.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.773+0000 7feb237fe640 1 --2- 172.21.15.150:0/3483222398 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feb241619a0 0x7feb24148c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:20.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.773+0000 7feb237fe640 1 -- 172.21.15.150:0/3483222398 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb240cdfc0 con 0x7feb240cf120 2024-03-17T23:48:20.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.773+0000 7feb23fff640 1 --2- 172.21.15.150:0/3483222398 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feb241619a0 0x7feb24148c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:20.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.773+0000 7feb22ffd640 1 --2- 172.21.15.150:0/3483222398 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feb2415de30 0x7feb240c95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:20.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.773+0000 7feb237fe640 1 --2- 172.21.15.150:0/3483222398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb240cf120 0x7feb241579c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7feb1400c430 tx=0x7feb1400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:20.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.774+0000 7feb1bfff640 1 -- 172.21.15.150:0/3483222398 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb14011790 con 0x7feb240cf120 2024-03-17T23:48:20.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.774+0000 7feb1bfff640 1 -- 172.21.15.150:0/3483222398 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb14017070 con 0x7feb240cf120 2024-03-17T23:48:20.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.774+0000 7feb29afe640 1 -- 172.21.15.150:0/3483222398 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb2415a060 con 0x7feb240cf120 2024-03-17T23:48:20.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.774+0000 7feb1bfff640 1 -- 172.21.15.150:0/3483222398 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb14011bb0 con 0x7feb240cf120 2024-03-17T23:48:20.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.774+0000 7feb29afe640 1 -- 172.21.15.150:0/3483222398 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb2415a620 con 0x7feb240cf120 2024-03-17T23:48:20.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.775+0000 7feb29afe640 1 -- 172.21.15.150:0/3483222398 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb2415a820 con 0x7feb240cf120 2024-03-17T23:48:20.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.775+0000 7feb1bfff640 1 -- 172.21.15.150:0/3483222398 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7feb1401c460 con 0x7feb240cf120 2024-03-17T23:48:20.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.776+0000 7feb1bfff640 1 --2- 172.21.15.150:0/3483222398 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7feb0406bbd0 0x7feb0406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:20.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.776+0000 7feb1bfff640 1 -- 172.21.15.150:0/3483222398 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7feb1408aa90 con 0x7feb240cf120 2024-03-17T23:48:20.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.776+0000 7feb22ffd640 1 --2- 172.21.15.150:0/3483222398 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7feb0406bbd0 0x7feb0406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:20.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.776+0000 7feb22ffd640 1 --2- 172.21.15.150:0/3483222398 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7feb0406bbd0 0x7feb0406e090 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7feb24157970 tx=0x7feb0c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:20.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:20.784+0000 7feb1bfff640 1 -- 172.21.15.150:0/3483222398 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7feb2415a820 con 0x7feb240cf120 2024-03-17T23:48:21.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:21.000+0000 7feb29afe640 1 -- 172.21.15.150:0/3483222398 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7feb240d1910 con 0x7feb240cf120 2024-03-17T23:48:21.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:21.001+0000 7feb1bfff640 1 -- 172.21.15.150:0/3483222398 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) ==== 53+0+73 (secure 0 0 0) 0x7feb14059480 con 0x7feb240cf120 2024-03-17T23:48:21.001 INFO:teuthology.orchestra.run.smithi150.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-17T23:48:21.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:21.003+0000 7feb29afe640 1 -- 172.21.15.150:0/3483222398 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7feb0406bbd0 msgr2=0x7feb0406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:21.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:21.003+0000 7feb29afe640 1 --2- 172.21.15.150:0/3483222398 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7feb0406bbd0 0x7feb0406e090 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7feb24157970 tx=0x7feb0c009040 comp rx=0 tx=0).stop 2024-03-17T23:48:21.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:21.003+0000 7feb29afe640 1 -- 172.21.15.150:0/3483222398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb240cf120 msgr2=0x7feb241579c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:21.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:21.003+0000 7feb29afe640 1 --2- 172.21.15.150:0/3483222398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb240cf120 0x7feb241579c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7feb1400c430 tx=0x7feb1400c8f0 comp rx=0 tx=0).stop 2024-03-17T23:48:21.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:21.003+0000 7feb29afe640 1 -- 172.21.15.150:0/3483222398 shutdown_connections 2024-03-17T23:48:21.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:21.003+0000 7feb29afe640 1 --2- 172.21.15.150:0/3483222398 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7feb0406bbd0 0x7feb0406e090 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:21.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:21.003+0000 7feb29afe640 1 --2- 172.21.15.150:0/3483222398 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feb241619a0 0x7feb24148c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:21.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:21.003+0000 7feb29afe640 1 --2- 172.21.15.150:0/3483222398 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feb2415de30 0x7feb240c95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:21.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:21.003+0000 7feb29afe640 1 --2- 172.21.15.150:0/3483222398 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feb240cf120 0x7feb241579c0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:21.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:21.003+0000 7feb29afe640 1 -- 172.21.15.150:0/3483222398 >> 172.21.15.150:0/3483222398 conn(0x7feb240730b0 msgr2=0x7feb240b83a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:21.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:21.003+0000 7feb29afe640 1 -- 172.21.15.150:0/3483222398 shutdown_connections 2024-03-17T23:48:21.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:21.003+0000 7feb29afe640 1 -- 172.21.15.150:0/3483222398 wait complete. 2024-03-17T23:48:21.014 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-17T23:48:21.014 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-17T23:48:21.014 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi150.front.sepia.ceph.com 2024-03-17T23:48:21.014 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-17T23:48:21.014 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-17T23:48:21.015 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-17T23:48:21.015 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-17T23:48:21.015 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-17T23:48:21.015 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-17T23:48:21.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:21.015 DEBUG:teuthology.orchestra.run.smithi150:> ip addr 2024-03-17T23:48:21.031 INFO:teuthology.orchestra.run.smithi150.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-17T23:48:21.032 INFO:teuthology.orchestra.run.smithi150.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-17T23:48:21.032 INFO:teuthology.orchestra.run.smithi150.stdout: inet 127.0.0.1/8 scope host lo 2024-03-17T23:48:21.032 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:48:21.032 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 ::1/128 scope host 2024-03-17T23:48:21.032 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:48:21.032 INFO:teuthology.orchestra.run.smithi150.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:48:21.033 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bc brd ff:ff:ff:ff:ff:ff 2024-03-17T23:48:21.033 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f0 2024-03-17T23:48:21.033 INFO:teuthology.orchestra.run.smithi150.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:48:21.033 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bd brd ff:ff:ff:ff:ff:ff 2024-03-17T23:48:21.033 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f1 2024-03-17T23:48:21.033 INFO:teuthology.orchestra.run.smithi150.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:48:21.033 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:34 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:48:21.034 INFO:teuthology.orchestra.run.smithi150.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-17T23:48:21.034 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:35 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:48:21.034 INFO:teuthology.orchestra.run.smithi150.stdout: inet 172.21.15.150/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-17T23:48:21.034 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft 42354sec preferred_lft 42354sec 2024-03-17T23:48:21.034 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::ec4:7aff:fe88:6e35/64 scope link 2024-03-17T23:48:21.034 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:48:21.035 INFO:teuthology.orchestra.run.smithi150.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-17T23:48:21.035 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether f6:2d:aa:e3:d6:a3 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:48:21.035 INFO:teuthology.orchestra.run.smithi150.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-17T23:48:21.035 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:48:21.035 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::bc2f:65ff:fe6d:abc3/64 scope link 2024-03-17T23:48:21.035 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:48:21.036 INFO:teuthology.orchestra.run.smithi150.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-17T23:48:21.036 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether f6:2d:aa:e3:d6:a3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:48:21.036 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::f42d:aaff:fee3:d6a3/64 scope link 2024-03-17T23:48:21.036 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:48:21.036 INFO:teuthology.orchestra.run.smithi150.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-17T23:48:21.036 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether fa:95:93:4e:f1:dd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:48:21.037 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::f895:93ff:fe4e:f1dd/64 scope link 2024-03-17T23:48:21.037 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:48:21.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:21.038 DEBUG:teuthology.orchestra.run.smithi150:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-17T23:48:21.097 INFO:teuthology.orchestra.run.smithi150.stdout:1 2024-03-17T23:48:21.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:21.098 DEBUG:teuthology.orchestra.run.smithi150:> route 2024-03-17T23:48:21.155 INFO:teuthology.orchestra.run.smithi150.stdout:Kernel IP routing table 2024-03-17T23:48:21.155 INFO:teuthology.orchestra.run.smithi150.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-17T23:48:21.155 INFO:teuthology.orchestra.run.smithi150.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-17T23:48:21.155 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-17T23:48:21.155 INFO:teuthology.orchestra.run.smithi150.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-17T23:48:21.155 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:48:21.156 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-03-17T23:48:21.156 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-17T23:48:21.156 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-17T23:48:21.156 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-17T23:48:21.156 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-03-17T23:48:21.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:48:21.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:48:21.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:21.294 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-03-17T23:48:21.309 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-17T23:48:21.309 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-17T23:48:21.309 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-17T23:48:21.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:21.309 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set brx.0 up 2024-03-17T23:48:21.377 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-17T23:48:21.377 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-17T23:48:21.377 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-17T23:48:21.432 INFO:teuthology.orchestra.run.smithi150.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-17T23:48:21.432 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-17T23:48:21.432 DEBUG:teuthology.orchestra.run.smithi150:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-17T23:48:21.487 DEBUG:teuthology.orchestra.run.smithi150:> sudo modprobe fuse 2024-03-17T23:48:21.554 DEBUG:teuthology.orchestra.run.smithi150:> cat /proc/self/mounts | awk '{print $2}' 2024-03-17T23:48:21.609 INFO:teuthology.orchestra.run.smithi150.stdout:/proc 2024-03-17T23:48:21.610 INFO:teuthology.orchestra.run.smithi150.stdout:/sys 2024-03-17T23:48:21.610 INFO:teuthology.orchestra.run.smithi150.stdout:/dev 2024-03-17T23:48:21.610 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/security 2024-03-17T23:48:21.610 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/shm 2024-03-17T23:48:21.610 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/pts 2024-03-17T23:48:21.610 INFO:teuthology.orchestra.run.smithi150.stdout:/run 2024-03-17T23:48:21.610 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/cgroup 2024-03-17T23:48:21.610 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/pstore 2024-03-17T23:48:21.610 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/bpf 2024-03-17T23:48:21.610 INFO:teuthology.orchestra.run.smithi150.stdout:/ 2024-03-17T23:48:21.610 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/selinux 2024-03-17T23:48:21.610 INFO:teuthology.orchestra.run.smithi150.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:48:21.611 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/hugepages 2024-03-17T23:48:21.611 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/mqueue 2024-03-17T23:48:21.611 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/debug 2024-03-17T23:48:21.611 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/tracing 2024-03-17T23:48:21.611 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-17T23:48:21.611 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/fuse/connections 2024-03-17T23:48:21.611 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/config 2024-03-17T23:48:21.611 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-sysctl.service 2024-03-17T23:48:21.611 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-17T23:48:21.611 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/nfs/rpc_pipefs 2024-03-17T23:48:21.611 INFO:teuthology.orchestra.run.smithi150.stdout:/run/user/1000 2024-03-17T23:48:21.611 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph 2024-03-17T23:48:21.611 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-0 2024-03-17T23:48:21.612 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-1 2024-03-17T23:48:21.612 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-2 2024-03-17T23:48:21.612 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-3 2024-03-17T23:48:21.612 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns 2024-03-17T23:48:21.612 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:48:21.612 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:48:21.612 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:48:21.612 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:48:21.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:21.612 DEBUG:teuthology.orchestra.run.smithi150:> ls /sys/fs/fuse/connections 2024-03-17T23:48:21.664 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-17T23:48:21.665 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-17T23:48:21.707 DEBUG:teuthology.orchestra.run.smithi150:> sudo modprobe fuse 2024-03-17T23:48:21.735 DEBUG:teuthology.orchestra.run.smithi150:> cat /proc/self/mounts | awk '{print $2}' 2024-03-17T23:48:21.791 INFO:teuthology.orchestra.run.smithi150.stdout:/proc 2024-03-17T23:48:21.791 INFO:teuthology.orchestra.run.smithi150.stdout:/sys 2024-03-17T23:48:21.791 INFO:teuthology.orchestra.run.smithi150.stdout:/dev 2024-03-17T23:48:21.791 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/security 2024-03-17T23:48:21.791 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/shm 2024-03-17T23:48:21.791 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/pts 2024-03-17T23:48:21.792 INFO:teuthology.orchestra.run.smithi150.stdout:/run 2024-03-17T23:48:21.792 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/cgroup 2024-03-17T23:48:21.792 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/pstore 2024-03-17T23:48:21.792 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/bpf 2024-03-17T23:48:21.792 INFO:teuthology.orchestra.run.smithi150.stdout:/ 2024-03-17T23:48:21.792 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/selinux 2024-03-17T23:48:21.792 INFO:teuthology.orchestra.run.smithi150.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:48:21.792 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/hugepages 2024-03-17T23:48:21.792 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/mqueue 2024-03-17T23:48:21.792 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/debug 2024-03-17T23:48:21.792 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/tracing 2024-03-17T23:48:21.792 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-17T23:48:21.792 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/fuse/connections 2024-03-17T23:48:21.792 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/config 2024-03-17T23:48:21.793 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-sysctl.service 2024-03-17T23:48:21.793 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-17T23:48:21.793 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/nfs/rpc_pipefs 2024-03-17T23:48:21.793 INFO:teuthology.orchestra.run.smithi150.stdout:/run/user/1000 2024-03-17T23:48:21.793 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph 2024-03-17T23:48:21.793 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-0 2024-03-17T23:48:21.793 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-1 2024-03-17T23:48:21.793 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-2 2024-03-17T23:48:21.793 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-3 2024-03-17T23:48:21.793 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns 2024-03-17T23:48:21.793 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:48:21.794 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:48:21.794 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:48:21.794 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:48:21.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:21.795 DEBUG:teuthology.orchestra.run.smithi150:> ls /sys/fs/fuse/connections 2024-03-17T23:48:21.795 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:2024-03-17T23:48:21.794+0000 7f771503a480 -1 init, newargv = 0x55d78e977510 newargc=15 2024-03-17T23:48:21.795 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:ceph-fuse[54700]: starting ceph client 2024-03-17T23:48:21.799 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:ceph-fuse[54700]: starting fuse 2024-03-17T23:48:21.815 INFO:teuthology.orchestra.run.smithi150.stdout:43 2024-03-17T23:48:21.816 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-17T23:48:21.816 DEBUG:teuthology.orchestra.run.smithi150:> sudo stdin-killer -- python3 -c ' 2024-03-17T23:48:21.816 DEBUG:teuthology.orchestra.run.smithi150:> import glob 2024-03-17T23:48:21.816 DEBUG:teuthology.orchestra.run.smithi150:> import re 2024-03-17T23:48:21.817 DEBUG:teuthology.orchestra.run.smithi150:> import os 2024-03-17T23:48:21.817 DEBUG:teuthology.orchestra.run.smithi150:> import subprocess 2024-03-17T23:48:21.817 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:48:21.817 DEBUG:teuthology.orchestra.run.smithi150:> def _find_admin_socket(client_name): 2024-03-17T23:48:21.817 DEBUG:teuthology.orchestra.run.smithi150:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-17T23:48:21.818 DEBUG:teuthology.orchestra.run.smithi150:> files = glob.glob(asok_path) 2024-03-17T23:48:21.818 DEBUG:teuthology.orchestra.run.smithi150:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-17T23:48:21.818 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:48:21.818 DEBUG:teuthology.orchestra.run.smithi150:> # Given a non-glob path, it better be there 2024-03-17T23:48:21.818 DEBUG:teuthology.orchestra.run.smithi150:> if "*" not in asok_path: 2024-03-17T23:48:21.818 DEBUG:teuthology.orchestra.run.smithi150:> assert(len(files) == 1) 2024-03-17T23:48:21.819 DEBUG:teuthology.orchestra.run.smithi150:> return files[0] 2024-03-17T23:48:21.819 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:48:21.819 DEBUG:teuthology.orchestra.run.smithi150:> for f in files: 2024-03-17T23:48:21.819 DEBUG:teuthology.orchestra.run.smithi150:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-17T23:48:21.819 DEBUG:teuthology.orchestra.run.smithi150:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-17T23:48:21.819 DEBUG:teuthology.orchestra.run.smithi150:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-17T23:48:21.819 DEBUG:teuthology.orchestra.run.smithi150:> contents = proc_f.read() 2024-03-17T23:48:21.820 DEBUG:teuthology.orchestra.run.smithi150:> if mountpoint in contents: 2024-03-17T23:48:21.820 DEBUG:teuthology.orchestra.run.smithi150:> return f 2024-03-17T23:48:21.820 DEBUG:teuthology.orchestra.run.smithi150:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-17T23:48:21.820 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:48:21.820 DEBUG:teuthology.orchestra.run.smithi150:> print(_find_admin_socket("client.0")) 2024-03-17T23:48:21.820 DEBUG:teuthology.orchestra.run.smithi150:> ' 2024-03-17T23:48:21.936 INFO:teuthology.orchestra.run.smithi150.stdout:/var/run/ceph/ceph-client.0.54700.asok 2024-03-17T23:48:21.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:48:21.946 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.54700.asok 2024-03-17T23:48:21.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:21.946 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.54700.asok status 2024-03-17T23:48:22.083 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-17T23:48:22.083 INFO:teuthology.orchestra.run.smithi150.stdout: "metadata": { 2024-03-17T23:48:22.084 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-17T23:48:22.084 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-17T23:48:22.084 INFO:teuthology.orchestra.run.smithi150.stdout: "entity_id": "0", 2024-03-17T23:48:22.084 INFO:teuthology.orchestra.run.smithi150.stdout: "hostname": "smithi150", 2024-03-17T23:48:22.084 INFO:teuthology.orchestra.run.smithi150.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-17T23:48:22.084 INFO:teuthology.orchestra.run.smithi150.stdout: "pid": "54700", 2024-03-17T23:48:22.084 INFO:teuthology.orchestra.run.smithi150.stdout: "root": "/" 2024-03-17T23:48:22.084 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:22.084 INFO:teuthology.orchestra.run.smithi150.stdout: "dentry_count": 0, 2024-03-17T23:48:22.084 INFO:teuthology.orchestra.run.smithi150.stdout: "dentry_pinned_count": 0, 2024-03-17T23:48:22.084 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 5712, 2024-03-17T23:48:22.084 INFO:teuthology.orchestra.run.smithi150.stdout: "inst": { 2024-03-17T23:48:22.084 INFO:teuthology.orchestra.run.smithi150.stdout: "name": { 2024-03-17T23:48:22.084 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "client", 2024-03-17T23:48:22.085 INFO:teuthology.orchestra.run.smithi150.stdout: "num": 5712 2024-03-17T23:48:22.085 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:22.085 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": { 2024-03-17T23:48:22.085 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v1", 2024-03-17T23:48:22.085 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "192.168.0.1:0", 2024-03-17T23:48:22.085 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 221336382 2024-03-17T23:48:22.085 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:22.085 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:22.085 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": { 2024-03-17T23:48:22.085 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v1", 2024-03-17T23:48:22.085 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "192.168.0.1:0", 2024-03-17T23:48:22.085 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 221336382 2024-03-17T23:48:22.085 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:22.085 INFO:teuthology.orchestra.run.smithi150.stdout: "inst_str": "client.5712 192.168.0.1:0/221336382", 2024-03-17T23:48:22.086 INFO:teuthology.orchestra.run.smithi150.stdout: "addr_str": "192.168.0.1:0/221336382", 2024-03-17T23:48:22.086 INFO:teuthology.orchestra.run.smithi150.stdout: "inode_count": 1, 2024-03-17T23:48:22.086 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_epoch": 15, 2024-03-17T23:48:22.086 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_epoch": 40, 2024-03-17T23:48:22.086 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_epoch_barrier": 0, 2024-03-17T23:48:22.086 INFO:teuthology.orchestra.run.smithi150.stdout: "blocklisted": false, 2024-03-17T23:48:22.086 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "cephfs" 2024-03-17T23:48:22.086 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-17T23:48:22.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:22.092 DEBUG:teuthology.orchestra.run.smithi150:> stat --file-system '--printf=%T 2024-03-17T23:48:22.092 DEBUG:teuthology.orchestra.run.smithi150:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-17T23:48:22.108 INFO:teuthology.orchestra.run.smithi150.stdout:fuseblk 2024-03-17T23:48:22.108 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-17T23:48:22.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:22.109 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-17T23:48:22.176 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create data_new --pg_num_min 64 2024-03-17T23:48:22.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.323+0000 7f0b8f481640 1 Processor -- start 2024-03-17T23:48:22.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.323+0000 7f0b8f481640 1 -- start start 2024-03-17T23:48:22.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.324+0000 7f0b8f481640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0b8815e060 0x7f0b88161ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:22.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.324+0000 7f0b8f481640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b880b9930 0x7f0b880ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:22.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.324+0000 7f0b8f481640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b88162310 con 0x7f0b880ceff0 2024-03-17T23:48:22.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.324+0000 7f0b8f481640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f0b8815e440 con 0x7f0b880b9930 2024-03-17T23:48:22.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.324+0000 7f0b8f481640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f0b880d2420 con 0x7f0b8815e060 2024-03-17T23:48:22.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.324+0000 7f0b8ca1c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b880b9930 0x7f0b880ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:22.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.324+0000 7f0b8ca1c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b880b9930 0x7f0b880ceab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36736/0 (socket says 172.21.15.150:36736) 2024-03-17T23:48:22.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.324+0000 7f0b8ca1c640 1 -- 172.21.15.150:0/2487599019 learned_addr learned my addr 172.21.15.150:0/2487599019 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:22.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.324+0000 7f0b8ca1c640 1 -- 172.21.15.150:0/2487599019 >> v2:172.21.15.157:3300/0 conn(0x7f0b8815e060 msgr2=0x7f0b88161ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:22.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.324+0000 7f0b8ca1c640 1 --2- 172.21.15.150:0/2487599019 >> v2:172.21.15.157:3300/0 conn(0x7f0b8815e060 0x7f0b88161ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:22.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.324+0000 7f0b8ca1c640 1 -- 172.21.15.150:0/2487599019 >> v1:172.21.15.157:6789/0 conn(0x7f0b880ceff0 legacy=0x7f0b880b5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:22.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.324+0000 7f0b8ca1c640 1 -- 172.21.15.150:0/2487599019 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b881625f0 con 0x7f0b880b9930 2024-03-17T23:48:22.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.325+0000 7f0b8ca1c640 1 --2- 172.21.15.150:0/2487599019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b880b9930 0x7f0b880ceab0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f0b880d4cf0 tx=0x7f0b7802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c9eeaf1d82a816f server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:22.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.325+0000 7f0b7ffff640 1 -- 172.21.15.150:0/2487599019 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b78035020 con 0x7f0b880b9930 2024-03-17T23:48:22.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.325+0000 7f0b7ffff640 1 -- 172.21.15.150:0/2487599019 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b78031870 con 0x7f0b880b9930 2024-03-17T23:48:22.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.325+0000 7f0b7ffff640 1 -- 172.21.15.150:0/2487599019 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b78031c40 con 0x7f0b880b9930 2024-03-17T23:48:22.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.325+0000 7f0b8f481640 1 -- 172.21.15.150:0/2487599019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b880b9930 msgr2=0x7f0b880ceab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:22.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.325+0000 7f0b8f481640 1 --2- 172.21.15.150:0/2487599019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b880b9930 0x7f0b880ceab0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f0b880d4cf0 tx=0x7f0b7802b910 comp rx=0 tx=0).stop 2024-03-17T23:48:22.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.326+0000 7f0b8f481640 1 -- 172.21.15.150:0/2487599019 shutdown_connections 2024-03-17T23:48:22.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.326+0000 7f0b8f481640 1 --2- 172.21.15.150:0/2487599019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b880b9930 0x7f0b880ceab0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:22.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.326+0000 7f0b8f481640 1 --2- 172.21.15.150:0/2487599019 >> v2:172.21.15.157:3300/0 conn(0x7f0b8815e060 0x7f0b88161ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:22.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.326+0000 7f0b8f481640 1 -- 172.21.15.150:0/2487599019 >> 172.21.15.150:0/2487599019 conn(0x7f0b8806c9e0 msgr2=0x7f0b8806cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:22.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.326+0000 7f0b8f481640 1 -- 172.21.15.150:0/2487599019 shutdown_connections 2024-03-17T23:48:22.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.326+0000 7f0b8f481640 1 -- 172.21.15.150:0/2487599019 wait complete. 2024-03-17T23:48:22.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.327+0000 7f0b8f481640 1 Processor -- start 2024-03-17T23:48:22.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.327+0000 7f0b8f481640 1 -- start start 2024-03-17T23:48:22.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.329+0000 7f0b8f481640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b880b9930 0x7f0b880ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:22.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.329+0000 7f0b8f481640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b880ceff0 0x7f0b880b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:22.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.329+0000 7f0b8f481640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b8815e060 0x7f0b880c9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:22.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.329+0000 7f0b8f481640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0b88162000 con 0x7f0b880b9930 2024-03-17T23:48:22.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.329+0000 7f0b8f481640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0b880d20b0 con 0x7f0b8815e060 2024-03-17T23:48:22.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.329+0000 7f0b8f481640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0b880b6e10 con 0x7f0b880ceff0 2024-03-17T23:48:22.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.329+0000 7f0b8d21d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b880b9930 0x7f0b880ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:22.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.329+0000 7f0b8d21d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b880b9930 0x7f0b880ccbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36752/0 (socket says 172.21.15.150:36752) 2024-03-17T23:48:22.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.329+0000 7f0b8d21d640 1 -- 172.21.15.150:0/3314749556 learned_addr learned my addr 172.21.15.150:0/3314749556 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:22.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.329+0000 7f0b8da1e640 1 --2- 172.21.15.150:0/3314749556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b8815e060 0x7f0b880c9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:22.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.329+0000 7f0b8ca1c640 1 --2- 172.21.15.150:0/3314749556 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b880ceff0 0x7f0b880b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:22.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.329+0000 7f0b8d21d640 1 -- 172.21.15.150:0/3314749556 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b880ceff0 msgr2=0x7f0b880b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:22.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.329+0000 7f0b8d21d640 1 --2- 172.21.15.150:0/3314749556 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b880ceff0 0x7f0b880b40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:22.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.329+0000 7f0b8d21d640 1 -- 172.21.15.150:0/3314749556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b8815e060 msgr2=0x7f0b880c9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:22.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.329+0000 7f0b8d21d640 1 --2- 172.21.15.150:0/3314749556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b8815e060 0x7f0b880c9da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:22.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.329+0000 7f0b8d21d640 1 -- 172.21.15.150:0/3314749556 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b880c8d00 con 0x7f0b880b9930 2024-03-17T23:48:22.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.329+0000 7f0b8d21d640 1 --2- 172.21.15.150:0/3314749556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b880b9930 0x7f0b880ccbc0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f0b8400b410 tx=0x7f0b8400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:22.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.329+0000 7f0b7dffb640 1 -- 172.21.15.150:0/3314749556 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b84015020 con 0x7f0b880b9930 2024-03-17T23:48:22.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.329+0000 7f0b7dffb640 1 -- 172.21.15.150:0/3314749556 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b84011850 con 0x7f0b880b9930 2024-03-17T23:48:22.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.329+0000 7f0b8f481640 1 -- 172.21.15.150:0/3314749556 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b880c8fe0 con 0x7f0b880b9930 2024-03-17T23:48:22.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.329+0000 7f0b7dffb640 1 -- 172.21.15.150:0/3314749556 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b84011c40 con 0x7f0b880b9930 2024-03-17T23:48:22.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.330+0000 7f0b8f481640 1 -- 172.21.15.150:0/3314749556 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b880c95a0 con 0x7f0b880b9930 2024-03-17T23:48:22.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.331+0000 7f0b7dffb640 1 -- 172.21.15.150:0/3314749556 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0b84018020 con 0x7f0b880b9930 2024-03-17T23:48:22.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.331+0000 7f0b8f481640 1 -- 172.21.15.150:0/3314749556 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b880681a0 con 0x7f0b880b9930 2024-03-17T23:48:22.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.331+0000 7f0b7dffb640 1 --2- 172.21.15.150:0/3314749556 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b6c06bbd0 0x7f0b6c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:22.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.331+0000 7f0b7dffb640 1 -- 172.21.15.150:0/3314749556 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6101+0+0 (secure 0 0 0) 0x7f0b8408abb0 con 0x7f0b880b9930 2024-03-17T23:48:22.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.331+0000 7f0b8ca1c640 1 --2- 172.21.15.150:0/3314749556 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b6c06bbd0 0x7f0b6c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:22.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.332+0000 7f0b8ca1c640 1 --2- 172.21.15.150:0/3314749556 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b6c06bbd0 0x7f0b6c06e090 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0b78036000 tx=0x7f0b7803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:22.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.336+0000 7f0b7dffb640 1 -- 172.21.15.150:0/3314749556 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0b8401f020 con 0x7f0b880b9930 2024-03-17T23:48:22.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:22.543+0000 7f0b8f481640 1 -- 172.21.15.150:0/3314749556 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "data_new", "pg_num_min": 64} v 0) -- 0x7f0b880d1980 con 0x7f0b880b9930 2024-03-17T23:48:23.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.425+0000 7f0b7dffb640 1 -- 172.21.15.150:0/3314749556 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "data_new", "pg_num_min": 64}]=0 pool 'data_new' created v41) ==== 124+0+0 (secure 0 0 0) 0x7f0b840172c0 con 0x7f0b880b9930 2024-03-17T23:48:23.426 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'data_new' created 2024-03-17T23:48:23.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.428+0000 7f0b8f481640 1 -- 172.21.15.150:0/3314749556 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b6c06bbd0 msgr2=0x7f0b6c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:23.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.428+0000 7f0b8f481640 1 --2- 172.21.15.150:0/3314749556 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b6c06bbd0 0x7f0b6c06e090 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0b78036000 tx=0x7f0b7803c040 comp rx=0 tx=0).stop 2024-03-17T23:48:23.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.428+0000 7f0b8f481640 1 -- 172.21.15.150:0/3314749556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b880b9930 msgr2=0x7f0b880ccbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:23.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.428+0000 7f0b8f481640 1 --2- 172.21.15.150:0/3314749556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b880b9930 0x7f0b880ccbc0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f0b8400b410 tx=0x7f0b8400b8d0 comp rx=0 tx=0).stop 2024-03-17T23:48:23.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.428+0000 7f0b8f481640 1 -- 172.21.15.150:0/3314749556 shutdown_connections 2024-03-17T23:48:23.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.428+0000 7f0b8f481640 1 --2- 172.21.15.150:0/3314749556 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0b6c06bbd0 0x7f0b6c06e090 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:23.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.428+0000 7f0b8f481640 1 --2- 172.21.15.150:0/3314749556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b8815e060 0x7f0b880c9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:23.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.428+0000 7f0b8f481640 1 --2- 172.21.15.150:0/3314749556 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b880ceff0 0x7f0b880b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:23.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.428+0000 7f0b8f481640 1 --2- 172.21.15.150:0/3314749556 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b880b9930 0x7f0b880ccbc0 secure :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f0b8400b410 tx=0x7f0b8400b8d0 comp rx=0 tx=0).stop 2024-03-17T23:48:23.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.428+0000 7f0b8f481640 1 -- 172.21.15.150:0/3314749556 >> 172.21.15.150:0/3314749556 conn(0x7f0b8806c9e0 msgr2=0x7f0b88160fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:23.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.428+0000 7f0b8f481640 1 -- 172.21.15.150:0/3314749556 shutdown_connections 2024-03-17T23:48:23.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.428+0000 7f0b8f481640 1 -- 172.21.15.150:0/3314749556 wait complete. 2024-03-17T23:48:23.445 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs data_new 2024-03-17T23:48:23.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.548+0000 7f192c275640 1 Processor -- start 2024-03-17T23:48:23.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.548+0000 7f192c275640 1 -- start start 2024-03-17T23:48:23.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.549+0000 7f192c275640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1924144bb0 0x7f192415ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:23.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.549+0000 7f192c275640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f192415d390 0x7f19240cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:23.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.549+0000 7f192c275640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1924145170 con 0x7f19240ce4d0 2024-03-17T23:48:23.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.549+0000 7f192c275640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f19240ce920 con 0x7f192415d390 2024-03-17T23:48:23.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.549+0000 7f192c275640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1924145450 con 0x7f1924144bb0 2024-03-17T23:48:23.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.549+0000 7f1929810640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f192415d390 0x7f19240cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:23.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.549+0000 7f1929810640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f192415d390 0x7f19240cdf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36764/0 (socket says 172.21.15.150:36764) 2024-03-17T23:48:23.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.549+0000 7f1929810640 1 -- 172.21.15.150:0/2507695832 learned_addr learned my addr 172.21.15.150:0/2507695832 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:23.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.550+0000 7f192a011640 1 --2- 172.21.15.150:0/2507695832 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1924144bb0 0x7f192415ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:23.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.550+0000 7f1929810640 1 -- 172.21.15.150:0/2507695832 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1924144bb0 msgr2=0x7f192415ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:23.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.550+0000 7f1929810640 1 --2- 172.21.15.150:0/2507695832 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1924144bb0 0x7f192415ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:23.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.550+0000 7f1929810640 1 -- 172.21.15.150:0/2507695832 >> v1:172.21.15.157:6789/0 conn(0x7f19240ce4d0 legacy=0x7f192406e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:23.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.550+0000 7f1929810640 1 -- 172.21.15.150:0/2507695832 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1924145730 con 0x7f192415d390 2024-03-17T23:48:23.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.551+0000 7f1929810640 1 --2- 172.21.15.150:0/2507695832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f192415d390 0x7f19240cdf90 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f1920009870 tx=0x7f192002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92fd6052a58a71aa server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:23.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.551+0000 7f192900f640 1 -- 172.21.15.150:0/2507695832 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1920035020 con 0x7f192415d390 2024-03-17T23:48:23.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.551+0000 7f192900f640 1 -- 172.21.15.150:0/2507695832 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1920031640 con 0x7f192415d390 2024-03-17T23:48:23.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.551+0000 7f192900f640 1 -- 172.21.15.150:0/2507695832 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192002d040 con 0x7f192415d390 2024-03-17T23:48:23.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.551+0000 7f192c275640 1 -- 172.21.15.150:0/2507695832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f192415d390 msgr2=0x7f19240cdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:23.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.551+0000 7f192c275640 1 --2- 172.21.15.150:0/2507695832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f192415d390 0x7f19240cdf90 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f1920009870 tx=0x7f192002b8f0 comp rx=0 tx=0).stop 2024-03-17T23:48:23.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.552+0000 7f192c275640 1 -- 172.21.15.150:0/2507695832 shutdown_connections 2024-03-17T23:48:23.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.552+0000 7f192c275640 1 --2- 172.21.15.150:0/2507695832 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f192415d390 0x7f19240cdf90 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:23.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.552+0000 7f192c275640 1 --2- 172.21.15.150:0/2507695832 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1924144bb0 0x7f192415ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:23.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.552+0000 7f192c275640 1 -- 172.21.15.150:0/2507695832 >> 172.21.15.150:0/2507695832 conn(0x7f192406b190 msgr2=0x7f192406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:23.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.552+0000 7f192c275640 1 -- 172.21.15.150:0/2507695832 shutdown_connections 2024-03-17T23:48:23.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.552+0000 7f192c275640 1 -- 172.21.15.150:0/2507695832 wait complete. 2024-03-17T23:48:23.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.553+0000 7f192c275640 1 Processor -- start 2024-03-17T23:48:23.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.554+0000 7f192c275640 1 -- start start 2024-03-17T23:48:23.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.555+0000 7f192c275640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19240ce4d0 0x7f192413aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:23.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.555+0000 7f192c275640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1924144bb0 0x7f19241574b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:23.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.555+0000 7f192c275640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f192415d390 0x7f19241579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:23.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.555+0000 7f192c275640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f192406ec80 con 0x7f1924144bb0 2024-03-17T23:48:23.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.555+0000 7f192c275640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f192406eb00 con 0x7f19240ce4d0 2024-03-17T23:48:23.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.555+0000 7f192c275640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f192406ee00 con 0x7f192415d390 2024-03-17T23:48:23.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.555+0000 7f1929810640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1924144bb0 0x7f19241574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:23.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.555+0000 7f1929810640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1924144bb0 0x7f19241574b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36778/0 (socket says 172.21.15.150:36778) 2024-03-17T23:48:23.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.555+0000 7f1929810640 1 -- 172.21.15.150:0/3195558324 learned_addr learned my addr 172.21.15.150:0/3195558324 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:23.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.555+0000 7f192a011640 1 --2- 172.21.15.150:0/3195558324 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19240ce4d0 0x7f192413aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:23.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.555+0000 7f192a812640 1 --2- 172.21.15.150:0/3195558324 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f192415d390 0x7f19241579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:23.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.555+0000 7f1929810640 1 -- 172.21.15.150:0/3195558324 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f192415d390 msgr2=0x7f19241579f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:23.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.555+0000 7f1929810640 1 --2- 172.21.15.150:0/3195558324 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f192415d390 0x7f19241579f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:23.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.555+0000 7f1929810640 1 -- 172.21.15.150:0/3195558324 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19240ce4d0 msgr2=0x7f192413aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:23.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.555+0000 7f1929810640 1 --2- 172.21.15.150:0/3195558324 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19240ce4d0 0x7f192413aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:23.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.555+0000 7f1929810640 1 -- 172.21.15.150:0/3195558324 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1924138160 con 0x7f1924144bb0 2024-03-17T23:48:23.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.555+0000 7f1929810640 1 --2- 172.21.15.150:0/3195558324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1924144bb0 0x7f19241574b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1920000c00 tx=0x7f19200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:23.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.555+0000 7f191affd640 1 -- 172.21.15.150:0/3195558324 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1920035040 con 0x7f1924144bb0 2024-03-17T23:48:23.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.556+0000 7f191affd640 1 -- 172.21.15.150:0/3195558324 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1920039bf0 con 0x7f1924144bb0 2024-03-17T23:48:23.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.556+0000 7f191affd640 1 -- 172.21.15.150:0/3195558324 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1920045480 con 0x7f1924144bb0 2024-03-17T23:48:23.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.556+0000 7f192c275640 1 -- 172.21.15.150:0/3195558324 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1924138440 con 0x7f1924144bb0 2024-03-17T23:48:23.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.556+0000 7f192c275640 1 -- 172.21.15.150:0/3195558324 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1924138a00 con 0x7f1924144bb0 2024-03-17T23:48:23.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.556+0000 7f192c275640 1 -- 172.21.15.150:0/3195558324 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19240681a0 con 0x7f1924144bb0 2024-03-17T23:48:23.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.557+0000 7f191affd640 1 -- 172.21.15.150:0/3195558324 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1920038020 con 0x7f1924144bb0 2024-03-17T23:48:23.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.557+0000 7f191affd640 1 --2- 172.21.15.150:0/3195558324 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f190806bbd0 0x7f190806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:23.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.558+0000 7f191affd640 1 -- 172.21.15.150:0/3195558324 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6464+0+0 (secure 0 0 0) 0x7f19200ac980 con 0x7f1924144bb0 2024-03-17T23:48:23.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.558+0000 7f192a011640 1 --2- 172.21.15.150:0/3195558324 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f190806bbd0 0x7f190806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:23.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.558+0000 7f192a011640 1 --2- 172.21.15.150:0/3195558324 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f190806bbd0 0x7f190806e090 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1914006fd0 tx=0x7f1914008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:23.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.566+0000 7f191affd640 1 -- 172.21.15.150:0/3195558324 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f192007abc0 con 0x7f1924144bb0 2024-03-17T23:48:23.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:23.826+0000 7f192c275640 1 -- 172.21.15.150:0/3195558324 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "data_new"} v 0) -- 0x7f192406e490 con 0x7f1924144bb0 2024-03-17T23:48:24.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.436+0000 7f191affd640 1 -- 172.21.15.150:0/3195558324 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "data_new"}]=0 added data pool 7 to fsmap v16) ==== 131+0+0 (secure 0 0 0) 0x7f192007ec60 con 0x7f1924144bb0 2024-03-17T23:48:24.437 INFO:teuthology.orchestra.run.smithi150.stderr:added data pool 7 to fsmap 2024-03-17T23:48:24.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.440+0000 7f192c275640 1 -- 172.21.15.150:0/3195558324 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f190806bbd0 msgr2=0x7f190806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:24.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.440+0000 7f192c275640 1 --2- 172.21.15.150:0/3195558324 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f190806bbd0 0x7f190806e090 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1914006fd0 tx=0x7f1914008040 comp rx=0 tx=0).stop 2024-03-17T23:48:24.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.440+0000 7f192c275640 1 -- 172.21.15.150:0/3195558324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1924144bb0 msgr2=0x7f19241574b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:24.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.440+0000 7f192c275640 1 --2- 172.21.15.150:0/3195558324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1924144bb0 0x7f19241574b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1920000c00 tx=0x7f19200046c0 comp rx=0 tx=0).stop 2024-03-17T23:48:24.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.441+0000 7f192c275640 1 -- 172.21.15.150:0/3195558324 shutdown_connections 2024-03-17T23:48:24.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.441+0000 7f192c275640 1 --2- 172.21.15.150:0/3195558324 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f192415d390 0x7f19241579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:24.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.441+0000 7f192c275640 1 --2- 172.21.15.150:0/3195558324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1924144bb0 0x7f19241574b0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:24.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.441+0000 7f192c275640 1 --2- 172.21.15.150:0/3195558324 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f190806bbd0 0x7f190806e090 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:24.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.441+0000 7f192c275640 1 --2- 172.21.15.150:0/3195558324 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19240ce4d0 0x7f192413aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:24.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.441+0000 7f192c275640 1 -- 172.21.15.150:0/3195558324 >> 172.21.15.150:0/3195558324 conn(0x7f192406b190 msgr2=0x7f19240b1060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:24.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.441+0000 7f192c275640 1 -- 172.21.15.150:0/3195558324 shutdown_connections 2024-03-17T23:48:24.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.441+0000 7f192c275640 1 -- 172.21.15.150:0/3195558324 wait complete. 2024-03-17T23:48:24.457 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:48:24.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.557+0000 7f81bdc12640 1 Processor -- start 2024-03-17T23:48:24.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.557+0000 7f81bdc12640 1 -- start start 2024-03-17T23:48:24.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.557+0000 7f81bdc12640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81b80b5c40 0x7f81b80b7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:24.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.557+0000 7f81bdc12640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81b80d2890 con 0x7f81b80b57d0 2024-03-17T23:48:24.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.557+0000 7f81bdc12640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81b80d2b70 con 0x7f81b815e060 2024-03-17T23:48:24.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.557+0000 7f81bdc12640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f81b80d2630 con 0x7f81b80b5c40 2024-03-17T23:48:24.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.558+0000 7f81b6ffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f81b80b57d0 0x7f81b80b6ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:48788/0 (socket says 172.21.15.150:48788) 2024-03-17T23:48:24.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.558+0000 7f81b6ffd640 1 -- 172.21.15.150:0/1159191602 learned_addr learned my addr 172.21.15.150:0/1159191602 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:24.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.558+0000 7f81b7fff640 1 --2- 172.21.15.150:0/1159191602 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81b80b5c40 0x7f81b80b7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:24.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.558+0000 7f81b67fc640 1 -- 172.21.15.150:0/1159191602 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2760474643 0 0) 0x7f81b80d2890 con 0x7f81b80b57d0 2024-03-17T23:48:24.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.558+0000 7f81b67fc640 1 -- 172.21.15.150:0/1159191602 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81a0003520 con 0x7f81b80b57d0 2024-03-17T23:48:24.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.558+0000 7f81b67fc640 1 -- 172.21.15.150:0/1159191602 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3046245832 0 0) 0x7f81b80d2b70 con 0x7f81b815e060 2024-03-17T23:48:24.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.558+0000 7f81b67fc640 1 -- 172.21.15.150:0/1159191602 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81b80d2890 con 0x7f81b815e060 2024-03-17T23:48:24.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.558+0000 7f81b67fc640 1 -- 172.21.15.150:0/1159191602 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 234835385 0 0) 0x7f81a0003520 con 0x7f81b80b57d0 2024-03-17T23:48:24.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.558+0000 7f81b7fff640 1 -- 172.21.15.150:0/1159191602 >> v1:172.21.15.157:6789/0 conn(0x7f81b815e060 legacy=0x7f81b80d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:24.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.558+0000 7f81b7fff640 1 -- 172.21.15.150:0/1159191602 >> v1:172.21.15.150:6789/0 conn(0x7f81b80b57d0 legacy=0x7f81b80b6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:24.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.558+0000 7f81b7fff640 1 -- 172.21.15.150:0/1159191602 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81b80d2e50 con 0x7f81b80b5c40 2024-03-17T23:48:24.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.559+0000 7f81b7fff640 1 --2- 172.21.15.150:0/1159191602 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81b80b5c40 0x7f81b80b7600 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f81b0009dc0 tx=0x7f81b002bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24b797740f92cc7e server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:24.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.559+0000 7f81b67fc640 1 -- 172.21.15.150:0/1159191602 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b0039070 con 0x7f81b80b5c40 2024-03-17T23:48:24.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.559+0000 7f81b67fc640 1 -- 172.21.15.150:0/1159191602 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81b002c880 con 0x7f81b80b5c40 2024-03-17T23:48:24.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.559+0000 7f81b67fc640 1 -- 172.21.15.150:0/1159191602 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81ac001670 con 0x7f81b80b5c40 2024-03-17T23:48:24.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.560+0000 7f81bdc12640 1 -- 172.21.15.150:0/1159191602 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81b80b5c40 msgr2=0x7f81b80b7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:24.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.560+0000 7f81bdc12640 1 --2- 172.21.15.150:0/1159191602 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81b80b5c40 0x7f81b80b7600 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f81b0009dc0 tx=0x7f81b002bd50 comp rx=0 tx=0).stop 2024-03-17T23:48:24.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.560+0000 7f81bdc12640 1 -- 172.21.15.150:0/1159191602 shutdown_connections 2024-03-17T23:48:24.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.560+0000 7f81bdc12640 1 --2- 172.21.15.150:0/1159191602 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81b80b5c40 0x7f81b80b7600 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:24.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.560+0000 7f81bdc12640 1 -- 172.21.15.150:0/1159191602 >> 172.21.15.150:0/1159191602 conn(0x7f81b8073200 msgr2=0x7f81b80d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:24.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.560+0000 7f81bdc12640 1 -- 172.21.15.150:0/1159191602 shutdown_connections 2024-03-17T23:48:24.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.560+0000 7f81bdc12640 1 -- 172.21.15.150:0/1159191602 wait complete. 2024-03-17T23:48:24.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.561+0000 7f81bdc12640 1 Processor -- start 2024-03-17T23:48:24.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.561+0000 7f81bdc12640 1 -- start start 2024-03-17T23:48:24.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.562+0000 7f81bdc12640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81b80b57d0 0x7f81b815a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:24.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.562+0000 7f81bdc12640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81b80b5c40 0x7f81b81f4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:24.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.562+0000 7f81bdc12640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81b815e060 0x7f81b80aafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:24.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.562+0000 7f81bdc12640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f81b80b6080 con 0x7f81b80b57d0 2024-03-17T23:48:24.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.562+0000 7f81bdc12640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f81b80d3410 con 0x7f81b815e060 2024-03-17T23:48:24.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.562+0000 7f81bdc12640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f81b815e440 con 0x7f81b80b5c40 2024-03-17T23:48:24.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.562+0000 7f81b77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81b80b57d0 0x7f81b815a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:24.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.562+0000 7f81b77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81b80b57d0 0x7f81b815a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36782/0 (socket says 172.21.15.150:36782) 2024-03-17T23:48:24.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.562+0000 7f81b77fe640 1 -- 172.21.15.150:0/3227566320 learned_addr learned my addr 172.21.15.150:0/3227566320 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:24.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.562+0000 7f81b6ffd640 1 --2- 172.21.15.150:0/3227566320 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81b80b5c40 0x7f81b81f4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:24.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.562+0000 7f81b7fff640 1 --2- 172.21.15.150:0/3227566320 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81b815e060 0x7f81b80aafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:24.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.562+0000 7f81b77fe640 1 -- 172.21.15.150:0/3227566320 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81b80b5c40 msgr2=0x7f81b81f4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:24.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.562+0000 7f81b77fe640 1 --2- 172.21.15.150:0/3227566320 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81b80b5c40 0x7f81b81f4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:24.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.562+0000 7f81b77fe640 1 -- 172.21.15.150:0/3227566320 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81b815e060 msgr2=0x7f81b80aafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:24.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.562+0000 7f81b77fe640 1 --2- 172.21.15.150:0/3227566320 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81b815e060 0x7f81b80aafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:24.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.562+0000 7f81b77fe640 1 -- 172.21.15.150:0/3227566320 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81b80cc100 con 0x7f81b80b57d0 2024-03-17T23:48:24.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.562+0000 7f81b77fe640 1 --2- 172.21.15.150:0/3227566320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81b80b57d0 0x7f81b815a600 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f81a800c430 tx=0x7f81a800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:24.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.562+0000 7f819ffff640 1 -- 172.21.15.150:0/3227566320 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a8017070 con 0x7f81b80b57d0 2024-03-17T23:48:24.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.562+0000 7f819ffff640 1 -- 172.21.15.150:0/3227566320 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81a80052f0 con 0x7f81b80b57d0 2024-03-17T23:48:24.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.562+0000 7f81bdc12640 1 -- 172.21.15.150:0/3227566320 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81b80cc3a0 con 0x7f81b80b57d0 2024-03-17T23:48:24.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.562+0000 7f819ffff640 1 -- 172.21.15.150:0/3227566320 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a8003830 con 0x7f81b80b57d0 2024-03-17T23:48:24.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.562+0000 7f81bdc12640 1 -- 172.21.15.150:0/3227566320 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81b80cc960 con 0x7f81b80b57d0 2024-03-17T23:48:24.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.563+0000 7f81bdc12640 1 -- 172.21.15.150:0/3227566320 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81b80688b0 con 0x7f81b80b57d0 2024-03-17T23:48:24.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.564+0000 7f819ffff640 1 -- 172.21.15.150:0/3227566320 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f81a801a070 con 0x7f81b80b57d0 2024-03-17T23:48:24.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.564+0000 7f819ffff640 1 --2- 172.21.15.150:0/3227566320 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f81a406bc00 0x7f81a406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:24.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.564+0000 7f819ffff640 1 -- 172.21.15.150:0/3227566320 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6496+0+0 (secure 0 0 0) 0x7f81a808a120 con 0x7f81b80b57d0 2024-03-17T23:48:24.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.564+0000 7f81b6ffd640 1 --2- 172.21.15.150:0/3227566320 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f81a406bc00 0x7f81a406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:24.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.565+0000 7f81b6ffd640 1 --2- 172.21.15.150:0/3227566320 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f81a406bc00 0x7f81a406e0c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f81b80aa7f0 tx=0x7f81ac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:24.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.569+0000 7f819ffff640 1 -- 172.21.15.150:0/3227566320 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f81a8058990 con 0x7f81b80b57d0 2024-03-17T23:48:24.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.780+0000 7f81bdc12640 1 -- 172.21.15.150:0/3227566320 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f81b8161760 con 0x7f81b80b57d0 2024-03-17T23:48:24.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.780+0000 7f819ffff640 1 -- 172.21.15.150:0/3227566320 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5195 (secure 0 0 0) 0x7f81a8058290 con 0x7f81b80b57d0 2024-03-17T23:48:24.781 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:24.781 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5177,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6834/3126446677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":3126446677},{"type":"v1","addr":"172.21.15.157:6834","nonce":3126446677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1992779366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1992779366},{"type":"v1","addr":"172.21.15.150:6836","nonce":1992779366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5487,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/3615467865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6833","nonce":3615467865},{"type":"v1","addr":"172.21.15.157:6835","nonce":3615467865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-17T23:48:10.730994+0000","modified":"2024-03-17T23:48:23.827674+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5515":{"gid":5515,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6837/1413722826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1413722826},{"type":"v1","addr":"172.21.15.150:6837","nonce":1413722826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5515,"qdb_cluster":[5515]},"id":2}]} 2024-03-17T23:48:24.781 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 16 2024-03-17T23:48:24.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.782+0000 7f81bdc12640 1 -- 172.21.15.150:0/3227566320 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f81a406bc00 msgr2=0x7f81a406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:24.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.782+0000 7f81bdc12640 1 --2- 172.21.15.150:0/3227566320 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f81a406bc00 0x7f81a406e0c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f81b80aa7f0 tx=0x7f81ac009040 comp rx=0 tx=0).stop 2024-03-17T23:48:24.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.782+0000 7f81bdc12640 1 -- 172.21.15.150:0/3227566320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81b80b57d0 msgr2=0x7f81b815a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:24.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.782+0000 7f81bdc12640 1 --2- 172.21.15.150:0/3227566320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81b80b57d0 0x7f81b815a600 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f81a800c430 tx=0x7f81a800c8f0 comp rx=0 tx=0).stop 2024-03-17T23:48:24.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.782+0000 7f81bdc12640 1 -- 172.21.15.150:0/3227566320 shutdown_connections 2024-03-17T23:48:24.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.782+0000 7f81bdc12640 1 --2- 172.21.15.150:0/3227566320 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f81a406bc00 0x7f81a406e0c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:24.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.782+0000 7f81bdc12640 1 --2- 172.21.15.150:0/3227566320 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f81b815e060 0x7f81b80aafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:24.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.782+0000 7f81bdc12640 1 --2- 172.21.15.150:0/3227566320 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f81b80b5c40 0x7f81b81f4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:24.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.782+0000 7f81bdc12640 1 --2- 172.21.15.150:0/3227566320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81b80b57d0 0x7f81b815a600 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:24.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.782+0000 7f81bdc12640 1 -- 172.21.15.150:0/3227566320 >> 172.21.15.150:0/3227566320 conn(0x7f81b8073200 msgr2=0x7f81b80d1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:24.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.782+0000 7f81bdc12640 1 -- 172.21.15.150:0/3227566320 shutdown_connections 2024-03-17T23:48:24.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.782+0000 7f81bdc12640 1 -- 172.21.15.150:0/3227566320 wait complete. 2024-03-17T23:48:24.795 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:48:24.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.897+0000 7f7b0b753640 1 Processor -- start 2024-03-17T23:48:24.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.897+0000 7f7b0b753640 1 -- start start 2024-03-17T23:48:24.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.898+0000 7f7b0b753640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b0415de30 0x7f7b041619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:24.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.898+0000 7f7b0b753640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7b040b9860 0x7f7b040cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:24.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.898+0000 7f7b0b753640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b040b9cc0 con 0x7f7b040cef50 2024-03-17T23:48:24.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.898+0000 7f7b0b753640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f7b040cf3c0 con 0x7f7b040b9860 2024-03-17T23:48:24.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.898+0000 7f7b0b753640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7b04162940 con 0x7f7b0415de30 2024-03-17T23:48:24.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.898+0000 7f7b09cf0640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f7b040cef50 0x7f7b0406c790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:48792/0 (socket says 172.21.15.150:48792) 2024-03-17T23:48:24.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.898+0000 7f7b09cf0640 1 -- 172.21.15.150:0/3941914002 learned_addr learned my addr 172.21.15.150:0/3941914002 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:24.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.899+0000 7f7b08cee640 1 --2- 172.21.15.150:0/3941914002 >> v2:172.21.15.157:3300/0 conn(0x7f7b040b9860 0x7f7b040cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.899+0000 7f7b094ef640 1 --2- 172.21.15.150:0/3941914002 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b0415de30 0x7f7b041619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.899+0000 7f7af7fff640 1 -- 172.21.15.150:0/3941914002 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2113150815 0 0) 0x7f7b040b9cc0 con 0x7f7b040cef50 2024-03-17T23:48:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.899+0000 7f7af7fff640 1 -- 172.21.15.150:0/3941914002 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7af0003520 con 0x7f7b040cef50 2024-03-17T23:48:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.899+0000 7f7af7fff640 1 -- 172.21.15.150:0/3941914002 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 498605602 0 0) 0x7f7af0003520 con 0x7f7b040cef50 2024-03-17T23:48:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.899+0000 7f7af7fff640 1 -- 172.21.15.150:0/3941914002 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b0415de30 msgr2=0x7f7b041619a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.899+0000 7f7af7fff640 1 --2- 172.21.15.150:0/3941914002 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b0415de30 0x7f7b041619a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.899+0000 7f7af7fff640 1 -- 172.21.15.150:0/3941914002 >> v2:172.21.15.157:3300/0 conn(0x7f7b040b9860 msgr2=0x7f7b040cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.899+0000 7f7af7fff640 1 --2- 172.21.15.150:0/3941914002 >> v2:172.21.15.157:3300/0 conn(0x7f7b040b9860 0x7f7b040cea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.899+0000 7f7af7fff640 1 -- 172.21.15.150:0/3941914002 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b041620a0 con 0x7f7b040cef50 2024-03-17T23:48:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.899+0000 7f7af7fff640 1 -- 172.21.15.150:0/3941914002 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f7afc0019f0 con 0x7f7b040cef50 2024-03-17T23:48:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.899+0000 7f7af7fff640 1 -- 172.21.15.150:0/3941914002 >> v1:172.21.15.150:6789/0 conn(0x7f7b040cef50 legacy=0x7f7b0406c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.899+0000 7f7af7fff640 1 --2- 172.21.15.150:0/3941914002 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b0415de30 0x7f7af00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.899+0000 7f7af7fff640 1 --2- 172.21.15.150:0/3941914002 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7af0003f30 0x7f7af0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.899+0000 7f7af7fff640 1 --2- 172.21.15.150:0/3941914002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7af0004d40 0x7f7af00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.899+0000 7f7af7fff640 1 -- 172.21.15.150:0/3941914002 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7b040cf3c0 con 0x7f7af0003f30 2024-03-17T23:48:24.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.899+0000 7f7af7fff640 1 -- 172.21.15.150:0/3941914002 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7b04162940 con 0x7f7af0004d40 2024-03-17T23:48:24.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.899+0000 7f7af7fff640 1 -- 172.21.15.150:0/3941914002 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7af000ba10 con 0x7f7b0415de30 2024-03-17T23:48:24.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.899+0000 7f7b08cee640 1 --2- 172.21.15.150:0/3941914002 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7af0003f30 0x7f7af0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:24.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.899+0000 7f7b094ef640 1 --2- 172.21.15.150:0/3941914002 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b0415de30 0x7f7af00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:24.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.899+0000 7f7b09cf0640 1 --2- 172.21.15.150:0/3941914002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7af0004d40 0x7f7af00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:24.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.899+0000 7f7b08cee640 1 -- 172.21.15.150:0/3941914002 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b0415de30 msgr2=0x7f7af00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:24.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.899+0000 7f7b08cee640 1 --2- 172.21.15.150:0/3941914002 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b0415de30 0x7f7af00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:24.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.899+0000 7f7b08cee640 1 -- 172.21.15.150:0/3941914002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7af0004d40 msgr2=0x7f7af00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:24.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.899+0000 7f7b08cee640 1 --2- 172.21.15.150:0/3941914002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7af0004d40 0x7f7af00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:24.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.899+0000 7f7b08cee640 1 -- 172.21.15.150:0/3941914002 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af0003520 con 0x7f7af0003f30 2024-03-17T23:48:24.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.900+0000 7f7b08cee640 1 --2- 172.21.15.150:0/3941914002 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7af0003f30 0x7f7af0004800 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f7af8009af0 tx=0x7f7af0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ff579399a02567 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:24.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.900+0000 7f7af7fff640 1 -- 172.21.15.150:0/3941914002 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af8039070 con 0x7f7af0003f30 2024-03-17T23:48:24.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.900+0000 7f7af7fff640 1 -- 172.21.15.150:0/3941914002 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7af802c710 con 0x7f7af0003f30 2024-03-17T23:48:24.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.900+0000 7f7af7fff640 1 -- 172.21.15.150:0/3941914002 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af8009690 con 0x7f7af0003f30 2024-03-17T23:48:24.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.901+0000 7f7b0b753640 1 -- 172.21.15.150:0/3941914002 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7af0003f30 msgr2=0x7f7af0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:24.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.901+0000 7f7b0b753640 1 --2- 172.21.15.150:0/3941914002 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7af0003f30 0x7f7af0004800 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f7af8009af0 tx=0x7f7af0005830 comp rx=0 tx=0).stop 2024-03-17T23:48:24.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.901+0000 7f7b09cf0640 1 -- 172.21.15.150:0/3941914002 reap_dead start 2024-03-17T23:48:24.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.901+0000 7f7b0b753640 1 -- 172.21.15.150:0/3941914002 shutdown_connections 2024-03-17T23:48:24.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.901+0000 7f7b0b753640 1 -- 172.21.15.150:0/3941914002 >> 172.21.15.150:0/3941914002 conn(0x7f7b040730b0 msgr2=0x7f7b040d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:24.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.901+0000 7f7b0b753640 1 -- 172.21.15.150:0/3941914002 shutdown_connections 2024-03-17T23:48:24.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.901+0000 7f7b0b753640 1 -- 172.21.15.150:0/3941914002 wait complete. 2024-03-17T23:48:24.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.902+0000 7f7b0b753640 1 Processor -- start 2024-03-17T23:48:24.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.902+0000 7f7b0b753640 1 -- start start 2024-03-17T23:48:24.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.903+0000 7f7b0b753640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7af0003f30 0x7f7b040abf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:24.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.903+0000 7f7b0b753640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b04160de0 0x7f7b040aa730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:24.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.903+0000 7f7b0b753640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b0414e050 0x7f7b040ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:24.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.903+0000 7f7b0b753640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7b040d2170 con 0x7f7b04160de0 2024-03-17T23:48:24.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.903+0000 7f7b0b753640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7b04162660 con 0x7f7af0003f30 2024-03-17T23:48:24.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.903+0000 7f7b0b753640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7b0415e210 con 0x7f7b0414e050 2024-03-17T23:48:24.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.903+0000 7f7b08cee640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b04160de0 0x7f7b040aa730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:24.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.903+0000 7f7b08cee640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b04160de0 0x7f7b040aa730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36798/0 (socket says 172.21.15.150:36798) 2024-03-17T23:48:24.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.903+0000 7f7b08cee640 1 -- 172.21.15.150:0/1028771354 learned_addr learned my addr 172.21.15.150:0/1028771354 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:24.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.903+0000 7f7b094ef640 1 --2- 172.21.15.150:0/1028771354 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7af0003f30 0x7f7b040abf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:24.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.903+0000 7f7b09cf0640 1 --2- 172.21.15.150:0/1028771354 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b0414e050 0x7f7b040ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:24.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.903+0000 7f7b08cee640 1 -- 172.21.15.150:0/1028771354 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b0414e050 msgr2=0x7f7b040ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:24.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.903+0000 7f7b08cee640 1 --2- 172.21.15.150:0/1028771354 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b0414e050 0x7f7b040ccfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:24.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.903+0000 7f7b08cee640 1 -- 172.21.15.150:0/1028771354 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7af0003f30 msgr2=0x7f7b040abf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:24.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.904+0000 7f7b08cee640 1 --2- 172.21.15.150:0/1028771354 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7af0003f30 0x7f7b040abf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:24.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.904+0000 7f7b08cee640 1 -- 172.21.15.150:0/1028771354 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b040aaea0 con 0x7f7b04160de0 2024-03-17T23:48:24.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.904+0000 7f7b09cf0640 1 --2- 172.21.15.150:0/1028771354 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b0414e050 0x7f7b040ccfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:24.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.904+0000 7f7b08cee640 1 --2- 172.21.15.150:0/1028771354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b04160de0 0x7f7b040aa730 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f7af8009af0 tx=0x7f7af8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:24.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.904+0000 7f7af5ffb640 1 -- 172.21.15.150:0/1028771354 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af8039050 con 0x7f7b04160de0 2024-03-17T23:48:24.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.904+0000 7f7af5ffb640 1 -- 172.21.15.150:0/1028771354 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7af8045070 con 0x7f7b04160de0 2024-03-17T23:48:24.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.904+0000 7f7b0b753640 1 -- 172.21.15.150:0/1028771354 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b040ab180 con 0x7f7b04160de0 2024-03-17T23:48:24.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.904+0000 7f7af5ffb640 1 -- 172.21.15.150:0/1028771354 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af8003a50 con 0x7f7b04160de0 2024-03-17T23:48:24.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.904+0000 7f7b0b753640 1 -- 172.21.15.150:0/1028771354 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b040ab740 con 0x7f7b04160de0 2024-03-17T23:48:24.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.905+0000 7f7b0b753640 1 -- 172.21.15.150:0/1028771354 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b040681a0 con 0x7f7b04160de0 2024-03-17T23:48:24.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.906+0000 7f7af5ffb640 1 -- 172.21.15.150:0/1028771354 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7af8003bf0 con 0x7f7b04160de0 2024-03-17T23:48:24.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.906+0000 7f7af5ffb640 1 --2- 172.21.15.150:0/1028771354 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7ae806bbd0 0x7f7ae806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:24.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.907+0000 7f7af5ffb640 1 -- 172.21.15.150:0/1028771354 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6496+0+0 (secure 0 0 0) 0x7f7af80ac9b0 con 0x7f7b04160de0 2024-03-17T23:48:24.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.907+0000 7f7b094ef640 1 --2- 172.21.15.150:0/1028771354 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7ae806bbd0 0x7f7ae806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:24.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.907+0000 7f7b094ef640 1 --2- 172.21.15.150:0/1028771354 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7ae806bbd0 0x7f7ae806e090 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7b040cdc20 tx=0x7f7b00008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:24.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:24.911+0000 7f7af5ffb640 1 -- 172.21.15.150:0/1028771354 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7af804e020 con 0x7f7b04160de0 2024-03-17T23:48:25.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.089+0000 7f7b0b753640 1 -- 172.21.15.150:0/1028771354 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7b040d1910 con 0x7f7b04160de0 2024-03-17T23:48:25.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.091+0000 7f7af5ffb640 1 -- 172.21.15.150:0/1028771354 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+18803 (secure 0 0 0) 0x7f7af807b2e0 con 0x7f7b04160de0 2024-03-17T23:48:25.092 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:25.092 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":42,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:48:23.827466+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:48:08.957879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-17T23:48:09.698808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-17T23:48:22.562269+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:48:25.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.093+0000 7f7b0b753640 1 -- 172.21.15.150:0/1028771354 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7ae806bbd0 msgr2=0x7f7ae806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:25.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.093+0000 7f7b0b753640 1 --2- 172.21.15.150:0/1028771354 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7ae806bbd0 0x7f7ae806e090 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7b040cdc20 tx=0x7f7b00008040 comp rx=0 tx=0).stop 2024-03-17T23:48:25.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.093+0000 7f7b0b753640 1 -- 172.21.15.150:0/1028771354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b04160de0 msgr2=0x7f7b040aa730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:25.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.093+0000 7f7b0b753640 1 --2- 172.21.15.150:0/1028771354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b04160de0 0x7f7b040aa730 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f7af8009af0 tx=0x7f7af8004730 comp rx=0 tx=0).stop 2024-03-17T23:48:25.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.093+0000 7f7b0b753640 1 -- 172.21.15.150:0/1028771354 shutdown_connections 2024-03-17T23:48:25.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.093+0000 7f7b0b753640 1 --2- 172.21.15.150:0/1028771354 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7b0414e050 0x7f7b040ccfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:25.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.093+0000 7f7b0b753640 1 --2- 172.21.15.150:0/1028771354 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b04160de0 0x7f7b040aa730 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:25.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.093+0000 7f7b0b753640 1 --2- 172.21.15.150:0/1028771354 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7ae806bbd0 0x7f7ae806e090 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:25.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.093+0000 7f7b0b753640 1 --2- 172.21.15.150:0/1028771354 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7af0003f30 0x7f7b040abf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:25.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.093+0000 7f7b0b753640 1 -- 172.21.15.150:0/1028771354 >> 172.21.15.150:0/1028771354 conn(0x7f7b040730b0 msgr2=0x7f7b040d44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:25.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.093+0000 7f7b0b753640 1 -- 172.21.15.150:0/1028771354 shutdown_connections 2024-03-17T23:48:25.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.093+0000 7f7b0b753640 1 -- 172.21.15.150:0/1028771354 wait complete. 2024-03-17T23:48:25.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:25.105 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch layoutfile) 2024-03-17T23:48:25.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:25.124 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir layoutdir) 2024-03-17T23:48:25.181 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds 'allow rw' mon 'allow r' osd 'allow rw pool=cephfs_data,allow rw pool=data_new' 2024-03-17T23:48:25.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.329+0000 7faa2ab22640 1 Processor -- start 2024-03-17T23:48:25.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.329+0000 7faa2ab22640 1 -- start start 2024-03-17T23:48:25.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.330+0000 7faa2ab22640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faa240d0ce0 0x7faa240b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:25.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.330+0000 7faa2ab22640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7faa240b2bf0 0x7faa240d1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:25.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.330+0000 7faa2ab22640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa24161d30 con 0x7faa24161520 2024-03-17T23:48:25.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.330+0000 7faa2ab22640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7faa240b9bf0 con 0x7faa240b2bf0 2024-03-17T23:48:25.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.330+0000 7faa2ab22640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7faa24162940 con 0x7faa240d0ce0 2024-03-17T23:48:25.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.331+0000 7faa288be640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7faa24161520 0x7faa240d05d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:48794/0 (socket says 172.21.15.150:48794) 2024-03-17T23:48:25.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.331+0000 7faa288be640 1 -- 172.21.15.150:0/3540166705 learned_addr learned my addr 172.21.15.150:0/3540166705 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:25.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.331+0000 7faa23fff640 1 --2- 172.21.15.150:0/3540166705 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faa240d0ce0 0x7faa240b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:25.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.331+0000 7faa290bf640 1 --2- 172.21.15.150:0/3540166705 >> v2:172.21.15.157:3300/0 conn(0x7faa240b2bf0 0x7faa240d1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:25.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.331+0000 7faa237fe640 1 -- 172.21.15.150:0/3540166705 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 88078413 0 0) 0x7faa24161d30 con 0x7faa24161520 2024-03-17T23:48:25.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.331+0000 7faa237fe640 1 -- 172.21.15.150:0/3540166705 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa14003520 con 0x7faa24161520 2024-03-17T23:48:25.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.331+0000 7faa290bf640 1 -- 172.21.15.150:0/3540166705 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faa240d0ce0 msgr2=0x7faa240b96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:25.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.331+0000 7faa290bf640 1 --2- 172.21.15.150:0/3540166705 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faa240d0ce0 0x7faa240b96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:25.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.331+0000 7faa290bf640 1 -- 172.21.15.150:0/3540166705 >> v1:172.21.15.150:6789/0 conn(0x7faa24161520 legacy=0x7faa240d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:25.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.331+0000 7faa290bf640 1 -- 172.21.15.150:0/3540166705 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa24162010 con 0x7faa240b2bf0 2024-03-17T23:48:25.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.331+0000 7faa23fff640 1 --2- 172.21.15.150:0/3540166705 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faa240d0ce0 0x7faa240b96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:48:25.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.332+0000 7faa290bf640 1 --2- 172.21.15.150:0/3540166705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa240b2bf0 0x7faa240d1fa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009d90 tx=0x7faa1c02bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98c883b8c56026bd server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:25.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.332+0000 7faa237fe640 1 -- 172.21.15.150:0/3540166705 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c039070 con 0x7faa240b2bf0 2024-03-17T23:48:25.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.332+0000 7faa237fe640 1 -- 172.21.15.150:0/3540166705 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa1c02cb30 con 0x7faa240b2bf0 2024-03-17T23:48:25.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.332+0000 7faa237fe640 1 -- 172.21.15.150:0/3540166705 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c034440 con 0x7faa240b2bf0 2024-03-17T23:48:25.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.332+0000 7faa2ab22640 1 -- 172.21.15.150:0/3540166705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa240b2bf0 msgr2=0x7faa240d1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:25.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.332+0000 7faa2ab22640 1 --2- 172.21.15.150:0/3540166705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa240b2bf0 0x7faa240d1fa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009d90 tx=0x7faa1c02bd80 comp rx=0 tx=0).stop 2024-03-17T23:48:25.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.333+0000 7faa2ab22640 1 -- 172.21.15.150:0/3540166705 shutdown_connections 2024-03-17T23:48:25.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.333+0000 7faa2ab22640 1 --2- 172.21.15.150:0/3540166705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa240b2bf0 0x7faa240d1fa0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:25.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.333+0000 7faa2ab22640 1 --2- 172.21.15.150:0/3540166705 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faa240d0ce0 0x7faa240b96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:25.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.333+0000 7faa2ab22640 1 -- 172.21.15.150:0/3540166705 >> 172.21.15.150:0/3540166705 conn(0x7faa2406ca00 msgr2=0x7faa2406ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:25.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.333+0000 7faa2ab22640 1 -- 172.21.15.150:0/3540166705 shutdown_connections 2024-03-17T23:48:25.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.333+0000 7faa2ab22640 1 -- 172.21.15.150:0/3540166705 wait complete. 2024-03-17T23:48:25.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.334+0000 7faa2ab22640 1 Processor -- start 2024-03-17T23:48:25.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.334+0000 7faa2ab22640 1 -- start start 2024-03-17T23:48:25.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.335+0000 7faa2ab22640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faa240b2bf0 0x7faa241f40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:25.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.335+0000 7faa2ab22640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa240d0ce0 0x7faa240b1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:25.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.335+0000 7faa2ab22640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa24161520 0x7faa240cc760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:25.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.335+0000 7faa2ab22640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7faa24161900 con 0x7faa24161520 2024-03-17T23:48:25.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.335+0000 7faa2ab22640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7faa241625d0 con 0x7faa240d0ce0 2024-03-17T23:48:25.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.335+0000 7faa2ab22640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7faa240d24e0 con 0x7faa240b2bf0 2024-03-17T23:48:25.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.335+0000 7faa290bf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa24161520 0x7faa240cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:25.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.335+0000 7faa290bf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa24161520 0x7faa240cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36806/0 (socket says 172.21.15.150:36806) 2024-03-17T23:48:25.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.335+0000 7faa290bf640 1 -- 172.21.15.150:0/1526456224 learned_addr learned my addr 172.21.15.150:0/1526456224 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:25.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.335+0000 7faa288be640 1 --2- 172.21.15.150:0/1526456224 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faa240b2bf0 0x7faa241f40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:25.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.335+0000 7faa23fff640 1 --2- 172.21.15.150:0/1526456224 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa240d0ce0 0x7faa240b1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:25.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.335+0000 7faa290bf640 1 -- 172.21.15.150:0/1526456224 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faa240b2bf0 msgr2=0x7faa241f40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:25.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.335+0000 7faa290bf640 1 --2- 172.21.15.150:0/1526456224 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faa240b2bf0 0x7faa241f40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:25.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.335+0000 7faa290bf640 1 -- 172.21.15.150:0/1526456224 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa240d0ce0 msgr2=0x7faa240b1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:25.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.335+0000 7faa290bf640 1 --2- 172.21.15.150:0/1526456224 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa240d0ce0 0x7faa240b1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:25.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.335+0000 7faa290bf640 1 -- 172.21.15.150:0/1526456224 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa241f45e0 con 0x7faa24161520 2024-03-17T23:48:25.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.335+0000 7faa290bf640 1 --2- 172.21.15.150:0/1526456224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa24161520 0x7faa240cc760 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009d90 tx=0x7faa1c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:25.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.335+0000 7faa217fa640 1 -- 172.21.15.150:0/1526456224 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c039050 con 0x7faa24161520 2024-03-17T23:48:25.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.335+0000 7faa217fa640 1 -- 172.21.15.150:0/1526456224 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa1c045070 con 0x7faa24161520 2024-03-17T23:48:25.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.336+0000 7faa2ab22640 1 -- 172.21.15.150:0/1526456224 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa240b42d0 con 0x7faa24161520 2024-03-17T23:48:25.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.336+0000 7faa217fa640 1 -- 172.21.15.150:0/1526456224 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c03e830 con 0x7faa24161520 2024-03-17T23:48:25.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.336+0000 7faa2ab22640 1 -- 172.21.15.150:0/1526456224 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa240b4f70 con 0x7faa24161520 2024-03-17T23:48:25.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.337+0000 7faa2ab22640 1 -- 172.21.15.150:0/1526456224 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa240d4b40 con 0x7faa24161520 2024-03-17T23:48:25.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.337+0000 7faa217fa640 1 -- 172.21.15.150:0/1526456224 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7faa1c03e9d0 con 0x7faa24161520 2024-03-17T23:48:25.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.337+0000 7faa217fa640 1 --2- 172.21.15.150:0/1526456224 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7faa0806bbd0 0x7faa0806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:25.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.338+0000 7faa217fa640 1 -- 172.21.15.150:0/1526456224 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6496+0+0 (secure 0 0 0) 0x7faa1c0ad960 con 0x7faa24161520 2024-03-17T23:48:25.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.338+0000 7faa288be640 1 --2- 172.21.15.150:0/1526456224 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7faa0806bbd0 0x7faa0806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:25.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.338+0000 7faa288be640 1 --2- 172.21.15.150:0/1526456224 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7faa0806bbd0 0x7faa0806e090 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7faa18005880 tx=0x7faa18009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:25.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.342+0000 7faa217fa640 1 -- 172.21.15.150:0/1526456224 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7faa240d4b40 con 0x7faa24161520 2024-03-17T23:48:25.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.651+0000 7faa2ab22640 1 -- 172.21.15.150:0/1526456224 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rw", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]} v 0) -- 0x7faa240681a0 con 0x7faa24161520 2024-03-17T23:48:25.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.653+0000 7faa217fa640 1 -- 172.21.15.150:0/1526456224 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rw", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]}]=0 updated caps for client.0 v14) ==== 210+0+167 (secure 0 0 0) 0x7faa1c07bd60 con 0x7faa24161520 2024-03-17T23:48:25.653 INFO:teuthology.orchestra.run.smithi150.stdout:[client.0] 2024-03-17T23:48:25.653 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQC3gPdlvlPJNRAAkUfetPJK4hZQG5rNiSDozw== 2024-03-17T23:48:25.654 INFO:teuthology.orchestra.run.smithi150.stdout: caps mds = "allow rw" 2024-03-17T23:48:25.654 INFO:teuthology.orchestra.run.smithi150.stdout: caps mon = "allow r" 2024-03-17T23:48:25.654 INFO:teuthology.orchestra.run.smithi150.stdout: caps osd = "allow rw pool=cephfs_data,allow rw pool=data_new" 2024-03-17T23:48:25.654 INFO:teuthology.orchestra.run.smithi150.stderr:updated caps for client.0 2024-03-17T23:48:25.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.655+0000 7faa2ab22640 1 -- 172.21.15.150:0/1526456224 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7faa0806bbd0 msgr2=0x7faa0806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:25.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.655+0000 7faa2ab22640 1 --2- 172.21.15.150:0/1526456224 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7faa0806bbd0 0x7faa0806e090 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7faa18005880 tx=0x7faa18009040 comp rx=0 tx=0).stop 2024-03-17T23:48:25.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.655+0000 7faa2ab22640 1 -- 172.21.15.150:0/1526456224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa24161520 msgr2=0x7faa240cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:25.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.655+0000 7faa2ab22640 1 --2- 172.21.15.150:0/1526456224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa24161520 0x7faa240cc760 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009d90 tx=0x7faa1c004a40 comp rx=0 tx=0).stop 2024-03-17T23:48:25.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.655+0000 7faa2ab22640 1 -- 172.21.15.150:0/1526456224 shutdown_connections 2024-03-17T23:48:25.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.655+0000 7faa2ab22640 1 --2- 172.21.15.150:0/1526456224 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faa24161520 0x7faa240cc760 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:25.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.655+0000 7faa2ab22640 1 --2- 172.21.15.150:0/1526456224 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7faa0806bbd0 0x7faa0806e090 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:25.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.655+0000 7faa2ab22640 1 --2- 172.21.15.150:0/1526456224 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa240d0ce0 0x7faa240b1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:25.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.655+0000 7faa2ab22640 1 --2- 172.21.15.150:0/1526456224 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faa240b2bf0 0x7faa241f40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:25.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.655+0000 7faa2ab22640 1 -- 172.21.15.150:0/1526456224 >> 172.21.15.150:0/1526456224 conn(0x7faa2406ca00 msgr2=0x7faa240b8ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:25.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.656+0000 7faa2ab22640 1 -- 172.21.15.150:0/1526456224 shutdown_connections 2024-03-17T23:48:25.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.656+0000 7faa2ab22640 1 -- 172.21.15.150:0/1526456224 wait complete. 2024-03-17T23:48:25.667 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:48:25.667 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:48:25.683 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:48:25.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.825+0000 7f0154e4b640 1 Processor -- start 2024-03-17T23:48:25.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.825+0000 7f0154e4b640 1 -- start start 2024-03-17T23:48:25.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.826+0000 7f0154e4b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f015015ad60 0x7f01500ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:25.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.826+0000 7f0154e4b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f015015e850 0x7f015015ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:25.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.826+0000 7f0154e4b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f01500c6310 0x7f01500c6760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:25.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.826+0000 7f0154e4b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f01500c6ca0 con 0x7f01500c6310 2024-03-17T23:48:25.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.826+0000 7f0154e4b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f01500b0540 con 0x7f015015ad60 2024-03-17T23:48:25.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.826+0000 7f0154e4b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f015015b140 con 0x7f015015e850 2024-03-17T23:48:25.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.827+0000 7f014ed9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f01500c6310 0x7f01500c6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:25.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.827+0000 7f014ed9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f01500c6310 0x7f01500c6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47296/0 (socket says 172.21.15.150:47296) 2024-03-17T23:48:25.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.827+0000 7f014ed9d640 1 -- 172.21.15.150:0/3246320184 learned_addr learned my addr 172.21.15.150:0/3246320184 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:25.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.827+0000 7f014dd9b640 1 --2- 172.21.15.150:0/3246320184 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f015015e850 0x7f015015ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:25.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.827+0000 7f014e59c640 1 --2- 172.21.15.150:0/3246320184 >> v2:172.21.15.157:3300/0 conn(0x7f015015ad60 0x7f01500ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:25.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.827+0000 7f014ed9d640 1 -- 172.21.15.150:0/3246320184 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f015015e850 msgr2=0x7f015015ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:25.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.827+0000 7f014ed9d640 1 --2- 172.21.15.150:0/3246320184 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f015015e850 0x7f015015ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:25.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.827+0000 7f014ed9d640 1 -- 172.21.15.150:0/3246320184 >> v2:172.21.15.157:3300/0 conn(0x7f015015ad60 msgr2=0x7f01500ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:25.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.827+0000 7f014ed9d640 1 --2- 172.21.15.150:0/3246320184 >> v2:172.21.15.157:3300/0 conn(0x7f015015ad60 0x7f01500ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:25.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.827+0000 7f014ed9d640 1 -- 172.21.15.150:0/3246320184 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f015015f310 con 0x7f01500c6310 2024-03-17T23:48:25.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.828+0000 7f014ed9d640 1 --2- 172.21.15.150:0/3246320184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01500c6310 0x7f01500c6760 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f015006e5a0 tx=0x7f014002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9814e612a7c8a209 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:25.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.828+0000 7f014d59a640 1 -- 172.21.15.150:0/3246320184 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014002ca70 con 0x7f01500c6310 2024-03-17T23:48:25.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.828+0000 7f014d59a640 1 -- 172.21.15.150:0/3246320184 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f014002cc10 con 0x7f01500c6310 2024-03-17T23:48:25.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.828+0000 7f014d59a640 1 -- 172.21.15.150:0/3246320184 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01400343f0 con 0x7f01500c6310 2024-03-17T23:48:25.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.828+0000 7f0154e4b640 1 -- 172.21.15.150:0/3246320184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01500c6310 msgr2=0x7f01500c6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:25.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.828+0000 7f0154e4b640 1 --2- 172.21.15.150:0/3246320184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01500c6310 0x7f01500c6760 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f015006e5a0 tx=0x7f014002bed0 comp rx=0 tx=0).stop 2024-03-17T23:48:25.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.829+0000 7f0154e4b640 1 -- 172.21.15.150:0/3246320184 shutdown_connections 2024-03-17T23:48:25.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.829+0000 7f0154e4b640 1 --2- 172.21.15.150:0/3246320184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01500c6310 0x7f01500c6760 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:25.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.829+0000 7f0154e4b640 1 --2- 172.21.15.150:0/3246320184 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f015015e850 0x7f015015ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:25.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.829+0000 7f0154e4b640 1 --2- 172.21.15.150:0/3246320184 >> v2:172.21.15.157:3300/0 conn(0x7f015015ad60 0x7f01500ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:25.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.829+0000 7f0154e4b640 1 -- 172.21.15.150:0/3246320184 >> 172.21.15.150:0/3246320184 conn(0x7f0150099a70 msgr2=0x7f0150099e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:25.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.829+0000 7f0154e4b640 1 -- 172.21.15.150:0/3246320184 shutdown_connections 2024-03-17T23:48:25.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.829+0000 7f0154e4b640 1 -- 172.21.15.150:0/3246320184 wait complete. 2024-03-17T23:48:25.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.831+0000 7f0154e4b640 1 Processor -- start 2024-03-17T23:48:25.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.831+0000 7f0154e4b640 1 -- start start 2024-03-17T23:48:25.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.832+0000 7f0154e4b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01500c6310 0x7f0150149b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:25.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.832+0000 7f0154e4b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f015015ad60 0x7f01500cb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:25.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.832+0000 7f0154e4b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f015015e850 0x7f015006b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:25.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.832+0000 7f0154e4b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f01500b0a70 con 0x7f015015ad60 2024-03-17T23:48:25.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.832+0000 7f0154e4b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f01500b0790 con 0x7f01500c6310 2024-03-17T23:48:25.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.832+0000 7f0154e4b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f01500c5b40 con 0x7f015015e850 2024-03-17T23:48:25.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.832+0000 7f014dd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f015015ad60 0x7f01500cb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:25.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.832+0000 7f014dd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f015015ad60 0x7f01500cb160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47302/0 (socket says 172.21.15.150:47302) 2024-03-17T23:48:25.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.832+0000 7f014dd9b640 1 -- 172.21.15.150:0/3640358073 learned_addr learned my addr 172.21.15.150:0/3640358073 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:25.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.832+0000 7f014ed9d640 1 --2- 172.21.15.150:0/3640358073 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f015015e850 0x7f015006b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:25.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.832+0000 7f014e59c640 1 --2- 172.21.15.150:0/3640358073 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01500c6310 0x7f0150149b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:25.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.832+0000 7f014dd9b640 1 -- 172.21.15.150:0/3640358073 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f015015e850 msgr2=0x7f015006b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:25.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.832+0000 7f014dd9b640 1 --2- 172.21.15.150:0/3640358073 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f015015e850 0x7f015006b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:25.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.832+0000 7f014dd9b640 1 -- 172.21.15.150:0/3640358073 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01500c6310 msgr2=0x7f0150149b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:25.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.832+0000 7f014dd9b640 1 --2- 172.21.15.150:0/3640358073 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01500c6310 0x7f0150149b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:25.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.832+0000 7f014dd9b640 1 -- 172.21.15.150:0/3640358073 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01500cc750 con 0x7f015015ad60 2024-03-17T23:48:25.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.833+0000 7f014dd9b640 1 --2- 172.21.15.150:0/3640358073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f015015ad60 0x7f01500cb160 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f013800d620 tx=0x7f013800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:25.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.833+0000 7f0146ffd640 1 -- 172.21.15.150:0/3640358073 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013800b620 con 0x7f015015ad60 2024-03-17T23:48:25.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.833+0000 7f0146ffd640 1 -- 172.21.15.150:0/3640358073 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0138019070 con 0x7f015015ad60 2024-03-17T23:48:25.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.833+0000 7f0154e4b640 1 -- 172.21.15.150:0/3640358073 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01500cbb40 con 0x7f015015ad60 2024-03-17T23:48:25.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.833+0000 7f0146ffd640 1 -- 172.21.15.150:0/3640358073 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f013800ba00 con 0x7f015015ad60 2024-03-17T23:48:25.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.833+0000 7f0154e4b640 1 -- 172.21.15.150:0/3640358073 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01500cc100 con 0x7f015015ad60 2024-03-17T23:48:25.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.834+0000 7f0154e4b640 1 -- 172.21.15.150:0/3640358073 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01500688b0 con 0x7f015015ad60 2024-03-17T23:48:25.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.834+0000 7f0146ffd640 1 -- 172.21.15.150:0/3640358073 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f013800bba0 con 0x7f015015ad60 2024-03-17T23:48:25.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.835+0000 7f0146ffd640 1 --2- 172.21.15.150:0/3640358073 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f013006bbd0 0x7f013006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:25.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.835+0000 7f014e59c640 1 --2- 172.21.15.150:0/3640358073 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f013006bbd0 0x7f013006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:25.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.835+0000 7f0146ffd640 1 -- 172.21.15.150:0/3640358073 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6496+0+0 (secure 0 0 0) 0x7f013808b070 con 0x7f015015ad60 2024-03-17T23:48:25.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.835+0000 7f014e59c640 1 --2- 172.21.15.150:0/3640358073 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f013006bbd0 0x7f013006e090 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0148003610 tx=0x7f0148008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:25.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:25.839+0000 7f0146ffd640 1 -- 172.21.15.150:0/3640358073 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0138059200 con 0x7f015015ad60 2024-03-17T23:48:26.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.014+0000 7f0154e4b640 1 -- 172.21.15.150:0/3640358073 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0150073000 con 0x7f015015ad60 2024-03-17T23:48:26.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.015+0000 7f0146ffd640 1 -- 172.21.15.150:0/3640358073 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) ==== 74+0+18790 (secure 0 0 0) 0x7f013805d2a0 con 0x7f015015ad60 2024-03-17T23:48:26.016 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:26.016 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":43,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:48:25.427736+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:48:08.957879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-17T23:48:09.698808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-17T23:48:22.562269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:48:26.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.017+0000 7f0154e4b640 1 -- 172.21.15.150:0/3640358073 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f013006bbd0 msgr2=0x7f013006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:26.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.017+0000 7f0154e4b640 1 --2- 172.21.15.150:0/3640358073 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f013006bbd0 0x7f013006e090 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0148003610 tx=0x7f0148008040 comp rx=0 tx=0).stop 2024-03-17T23:48:26.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.017+0000 7f0154e4b640 1 -- 172.21.15.150:0/3640358073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f015015ad60 msgr2=0x7f01500cb160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:26.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.017+0000 7f0154e4b640 1 --2- 172.21.15.150:0/3640358073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f015015ad60 0x7f01500cb160 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f013800d620 tx=0x7f013800dae0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.017+0000 7f0154e4b640 1 -- 172.21.15.150:0/3640358073 shutdown_connections 2024-03-17T23:48:26.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.017+0000 7f0154e4b640 1 --2- 172.21.15.150:0/3640358073 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f015015e850 0x7f015006b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.017+0000 7f0154e4b640 1 --2- 172.21.15.150:0/3640358073 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f015015ad60 0x7f01500cb160 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.017+0000 7f0154e4b640 1 --2- 172.21.15.150:0/3640358073 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f013006bbd0 0x7f013006e090 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.017+0000 7f0154e4b640 1 --2- 172.21.15.150:0/3640358073 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01500c6310 0x7f0150149b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.017+0000 7f0154e4b640 1 -- 172.21.15.150:0/3640358073 >> 172.21.15.150:0/3640358073 conn(0x7f0150099a70 msgr2=0x7f015006f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:26.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.017+0000 7f0154e4b640 1 -- 172.21.15.150:0/3640358073 shutdown_connections 2024-03-17T23:48:26.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.017+0000 7f0154e4b640 1 -- 172.21.15.150:0/3640358073 wait complete. 2024-03-17T23:48:26.033 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:48:08.957879+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:48:09.698808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-17T23:48:22.562269+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-17T23:48:26.033 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:48:26.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.137+0000 7f94c395b640 1 Processor -- start 2024-03-17T23:48:26.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.137+0000 7f94c395b640 1 -- start start 2024-03-17T23:48:26.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.138+0000 7f94c395b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f94bc156460 0x7f94bc15f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:26.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.138+0000 7f94c395b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f94bc15cad0 0x7f94bc072270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:26.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.138+0000 7f94c395b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f94bc0727b0 0x7f94bc06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:26.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.138+0000 7f94c395b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f94bc15fe00 con 0x7f94bc0727b0 2024-03-17T23:48:26.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.138+0000 7f94c395b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f94bc1600e0 con 0x7f94bc15cad0 2024-03-17T23:48:26.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.138+0000 7f94c395b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f94bc1603c0 con 0x7f94bc156460 2024-03-17T23:48:26.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.138+0000 7f94c1ef8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f94bc0727b0 0x7f94bc06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:26.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.138+0000 7f94c1ef8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f94bc0727b0 0x7f94bc06ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47318/0 (socket says 172.21.15.150:47318) 2024-03-17T23:48:26.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.138+0000 7f94c1ef8640 1 -- 172.21.15.150:0/528297685 learned_addr learned my addr 172.21.15.150:0/528297685 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:26.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.138+0000 7f94c16f7640 1 --2- 172.21.15.150:0/528297685 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f94bc156460 0x7f94bc15f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:26.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.138+0000 7f94c0ef6640 1 --2- 172.21.15.150:0/528297685 >> v2:172.21.15.157:3300/0 conn(0x7f94bc15cad0 0x7f94bc072270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:26.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.139+0000 7f94c1ef8640 1 -- 172.21.15.150:0/528297685 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f94bc156460 msgr2=0x7f94bc15f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:26.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.139+0000 7f94c1ef8640 1 --2- 172.21.15.150:0/528297685 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f94bc156460 0x7f94bc15f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.139+0000 7f94c1ef8640 1 -- 172.21.15.150:0/528297685 >> v2:172.21.15.157:3300/0 conn(0x7f94bc15cad0 msgr2=0x7f94bc072270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:26.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.139+0000 7f94c1ef8640 1 --2- 172.21.15.150:0/528297685 >> v2:172.21.15.157:3300/0 conn(0x7f94bc15cad0 0x7f94bc072270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.139+0000 7f94c1ef8640 1 -- 172.21.15.150:0/528297685 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94bc1606a0 con 0x7f94bc0727b0 2024-03-17T23:48:26.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.139+0000 7f94c0ef6640 1 --2- 172.21.15.150:0/528297685 >> v2:172.21.15.157:3300/0 conn(0x7f94bc15cad0 0x7f94bc072270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:26.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.139+0000 7f94c16f7640 1 --2- 172.21.15.150:0/528297685 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f94bc156460 0x7f94bc15f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:26.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.140+0000 7f94c1ef8640 1 --2- 172.21.15.150:0/528297685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94bc0727b0 0x7f94bc06ded0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f94bc067330 tx=0x7f94b402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daddc885c1bbf9ba server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:26.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.140+0000 7f94b3fff640 1 -- 172.21.15.150:0/528297685 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b402ca70 con 0x7f94bc0727b0 2024-03-17T23:48:26.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.140+0000 7f94b3fff640 1 -- 172.21.15.150:0/528297685 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94b402cc10 con 0x7f94bc0727b0 2024-03-17T23:48:26.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.140+0000 7f94b3fff640 1 -- 172.21.15.150:0/528297685 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b40343f0 con 0x7f94bc0727b0 2024-03-17T23:48:26.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.141+0000 7f94c395b640 1 -- 172.21.15.150:0/528297685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94bc0727b0 msgr2=0x7f94bc06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:26.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.141+0000 7f94c395b640 1 --2- 172.21.15.150:0/528297685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94bc0727b0 0x7f94bc06ded0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f94bc067330 tx=0x7f94b402bed0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.141+0000 7f94c395b640 1 -- 172.21.15.150:0/528297685 shutdown_connections 2024-03-17T23:48:26.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.141+0000 7f94c395b640 1 --2- 172.21.15.150:0/528297685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94bc0727b0 0x7f94bc06ded0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.141+0000 7f94c395b640 1 --2- 172.21.15.150:0/528297685 >> v2:172.21.15.157:3300/0 conn(0x7f94bc15cad0 0x7f94bc072270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.141+0000 7f94c395b640 1 --2- 172.21.15.150:0/528297685 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f94bc156460 0x7f94bc15f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.141+0000 7f94c395b640 1 -- 172.21.15.150:0/528297685 >> 172.21.15.150:0/528297685 conn(0x7f94bc06b370 msgr2=0x7f94bc06b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:26.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.141+0000 7f94c395b640 1 -- 172.21.15.150:0/528297685 shutdown_connections 2024-03-17T23:48:26.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.141+0000 7f94c395b640 1 -- 172.21.15.150:0/528297685 wait complete. 2024-03-17T23:48:26.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.142+0000 7f94c395b640 1 Processor -- start 2024-03-17T23:48:26.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.142+0000 7f94c395b640 1 -- start start 2024-03-17T23:48:26.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.143+0000 7f94c395b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94bc0727b0 0x7f94bc149750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:26.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.143+0000 7f94c395b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94bc156460 0x7f94bc0afb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:26.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.143+0000 7f94c395b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f94bc15cad0 0x7f94bc0ad390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:26.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.143+0000 7f94c395b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc15d0d0 con 0x7f94bc0727b0 2024-03-17T23:48:26.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.143+0000 7f94c395b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc15cf50 con 0x7f94bc156460 2024-03-17T23:48:26.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.143+0000 7f94c395b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f94bc06e410 con 0x7f94bc15cad0 2024-03-17T23:48:26.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.143+0000 7f94c16f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94bc0727b0 0x7f94bc149750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:26.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.143+0000 7f94c16f7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94bc0727b0 0x7f94bc149750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47332/0 (socket says 172.21.15.150:47332) 2024-03-17T23:48:26.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.143+0000 7f94c16f7640 1 -- 172.21.15.150:0/4247947138 learned_addr learned my addr 172.21.15.150:0/4247947138 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:26.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.143+0000 7f94c0ef6640 1 --2- 172.21.15.150:0/4247947138 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94bc156460 0x7f94bc0afb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:26.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.143+0000 7f94c1ef8640 1 --2- 172.21.15.150:0/4247947138 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f94bc15cad0 0x7f94bc0ad390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:26.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.143+0000 7f94c16f7640 1 -- 172.21.15.150:0/4247947138 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f94bc15cad0 msgr2=0x7f94bc0ad390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:26.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.143+0000 7f94c16f7640 1 --2- 172.21.15.150:0/4247947138 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f94bc15cad0 0x7f94bc0ad390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.143+0000 7f94c16f7640 1 -- 172.21.15.150:0/4247947138 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94bc156460 msgr2=0x7f94bc0afb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:26.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.143+0000 7f94c16f7640 1 --2- 172.21.15.150:0/4247947138 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94bc156460 0x7f94bc0afb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.143+0000 7f94c16f7640 1 -- 172.21.15.150:0/4247947138 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94bc156e70 con 0x7f94bc0727b0 2024-03-17T23:48:26.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.143+0000 7f94c16f7640 1 --2- 172.21.15.150:0/4247947138 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94bc0727b0 0x7f94bc149750 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f94ac00e840 tx=0x7f94ac00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:26.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.143+0000 7f94b1ffb640 1 -- 172.21.15.150:0/4247947138 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ac00f040 con 0x7f94bc0727b0 2024-03-17T23:48:26.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.143+0000 7f94b1ffb640 1 -- 172.21.15.150:0/4247947138 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94ac004030 con 0x7f94bc0727b0 2024-03-17T23:48:26.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.143+0000 7f94c395b640 1 -- 172.21.15.150:0/4247947138 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94bc0b0470 con 0x7f94bc0727b0 2024-03-17T23:48:26.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.143+0000 7f94b1ffb640 1 -- 172.21.15.150:0/4247947138 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ac004400 con 0x7f94bc0727b0 2024-03-17T23:48:26.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.144+0000 7f94c395b640 1 -- 172.21.15.150:0/4247947138 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94bc0b0a30 con 0x7f94bc0727b0 2024-03-17T23:48:26.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.145+0000 7f94c395b640 1 -- 172.21.15.150:0/4247947138 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94bc0681a0 con 0x7f94bc0727b0 2024-03-17T23:48:26.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.145+0000 7f94b1ffb640 1 -- 172.21.15.150:0/4247947138 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f94ac00c6d0 con 0x7f94bc0727b0 2024-03-17T23:48:26.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.145+0000 7f94b1ffb640 1 --2- 172.21.15.150:0/4247947138 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f94a006bbd0 0x7f94a006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:26.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.146+0000 7f94c0ef6640 1 --2- 172.21.15.150:0/4247947138 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f94a006bbd0 0x7f94a006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:26.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.146+0000 7f94b1ffb640 1 -- 172.21.15.150:0/4247947138 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6496+0+0 (secure 0 0 0) 0x7f94ac08a9f0 con 0x7f94bc0727b0 2024-03-17T23:48:26.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.146+0000 7f94c0ef6640 1 --2- 172.21.15.150:0/4247947138 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f94a006bbd0 0x7f94a006e090 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f94bc151fc0 tx=0x7f94b8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:26.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.150+0000 7f94b1ffb640 1 -- 172.21.15.150:0/4247947138 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f94ac017020 con 0x7f94bc0727b0 2024-03-17T23:48:26.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.328+0000 7f94c395b640 1 -- 172.21.15.150:0/4247947138 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f94bc06f140 con 0x7f94bc0727b0 2024-03-17T23:48:26.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.329+0000 7f94b1ffb640 1 -- 172.21.15.150:0/4247947138 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v43) ==== 93+0+10 (secure 0 0 0) 0x7f94ac0591d0 con 0x7f94bc0727b0 2024-03-17T23:48:26.329 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:48:26.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.331+0000 7f94c395b640 1 -- 172.21.15.150:0/4247947138 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f94a006bbd0 msgr2=0x7f94a006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:26.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.331+0000 7f94c395b640 1 --2- 172.21.15.150:0/4247947138 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f94a006bbd0 0x7f94a006e090 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f94bc151fc0 tx=0x7f94b8008040 comp rx=0 tx=0).stop 2024-03-17T23:48:26.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.331+0000 7f94c395b640 1 -- 172.21.15.150:0/4247947138 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94bc0727b0 msgr2=0x7f94bc149750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:26.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.331+0000 7f94c395b640 1 --2- 172.21.15.150:0/4247947138 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94bc0727b0 0x7f94bc149750 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f94ac00e840 tx=0x7f94ac00ed00 comp rx=0 tx=0).stop 2024-03-17T23:48:26.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.331+0000 7f94c395b640 1 -- 172.21.15.150:0/4247947138 shutdown_connections 2024-03-17T23:48:26.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.331+0000 7f94c395b640 1 --2- 172.21.15.150:0/4247947138 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f94bc15cad0 0x7f94bc0ad390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.331+0000 7f94c395b640 1 --2- 172.21.15.150:0/4247947138 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f94a006bbd0 0x7f94a006e090 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.331+0000 7f94c395b640 1 --2- 172.21.15.150:0/4247947138 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94bc156460 0x7f94bc0afb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.331+0000 7f94c395b640 1 --2- 172.21.15.150:0/4247947138 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94bc0727b0 0x7f94bc149750 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.331+0000 7f94c395b640 1 -- 172.21.15.150:0/4247947138 >> 172.21.15.150:0/4247947138 conn(0x7f94bc06b370 msgr2=0x7f94bc0ae1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:26.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.331+0000 7f94c395b640 1 -- 172.21.15.150:0/4247947138 shutdown_connections 2024-03-17T23:48:26.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.331+0000 7f94c395b640 1 -- 172.21.15.150:0/4247947138 wait complete. 2024-03-17T23:48:26.342 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:48:26.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.455+0000 7f5c3a23f640 1 Processor -- start 2024-03-17T23:48:26.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.455+0000 7f5c3a23f640 1 -- start start 2024-03-17T23:48:26.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.456+0000 7f5c3a23f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5c340cdb20 0x7f5c340cdf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:26.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.456+0000 7f5c3a23f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5c340c6070 0x7f5c340cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:26.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.456+0000 7f5c3a23f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c340c64f0 con 0x7f5c340c5c00 2024-03-17T23:48:26.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.456+0000 7f5c3a23f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f5c3415b0e0 con 0x7f5c340c6070 2024-03-17T23:48:26.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.456+0000 7f5c3a23f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f5c340c67d0 con 0x7f5c340cdb20 2024-03-17T23:48:26.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.456+0000 7f5c33fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5c340c6070 0x7f5c340cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:26.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.456+0000 7f5c32ffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f5c340c5c00 0x7f5c3406f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:57490/0 (socket says 172.21.15.150:57490) 2024-03-17T23:48:26.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.456+0000 7f5c32ffd640 1 -- 172.21.15.150:0/3612895593 learned_addr learned my addr 172.21.15.150:0/3612895593 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:26.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.457+0000 7f5c33fff640 1 -- 172.21.15.150:0/3612895593 >> v2:172.21.15.157:3300/0 conn(0x7f5c340cdb20 msgr2=0x7f5c340cdf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:26.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.457+0000 7f5c337fe640 1 --2- 172.21.15.150:0/3612895593 >> v2:172.21.15.157:3300/0 conn(0x7f5c340cdb20 0x7f5c340cdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:26.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.457+0000 7f5c327fc640 1 -- 172.21.15.150:0/3612895593 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4117342641 0 0) 0x7f5c340c64f0 con 0x7f5c340c5c00 2024-03-17T23:48:26.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.457+0000 7f5c33fff640 1 --2- 172.21.15.150:0/3612895593 >> v2:172.21.15.157:3300/0 conn(0x7f5c340cdb20 0x7f5c340cdf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.457+0000 7f5c33fff640 1 -- 172.21.15.150:0/3612895593 >> v1:172.21.15.150:6789/0 conn(0x7f5c340c5c00 legacy=0x7f5c3406f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:26.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.457+0000 7f5c33fff640 1 -- 172.21.15.150:0/3612895593 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c340c6ab0 con 0x7f5c340c6070 2024-03-17T23:48:26.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.457+0000 7f5c337fe640 1 --2- 172.21.15.150:0/3612895593 >> v2:172.21.15.157:3300/0 conn(0x7f5c340cdb20 0x7f5c340cdf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:26.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.457+0000 7f5c33fff640 1 --2- 172.21.15.150:0/3612895593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c340c6070 0x7f5c340cb0a0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f5c34071780 tx=0x7f5c2c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=254aefe3a06b6bd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:26.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.458+0000 7f5c327fc640 1 -- 172.21.15.150:0/3612895593 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c039070 con 0x7f5c340c6070 2024-03-17T23:48:26.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.458+0000 7f5c327fc640 1 -- 172.21.15.150:0/3612895593 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c2c02cc80 con 0x7f5c340c6070 2024-03-17T23:48:26.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.458+0000 7f5c327fc640 1 -- 172.21.15.150:0/3612895593 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c034660 con 0x7f5c340c6070 2024-03-17T23:48:26.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.458+0000 7f5c3a23f640 1 -- 172.21.15.150:0/3612895593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c340c6070 msgr2=0x7f5c340cb0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:26.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.458+0000 7f5c3a23f640 1 --2- 172.21.15.150:0/3612895593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c340c6070 0x7f5c340cb0a0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f5c34071780 tx=0x7f5c2c02bed0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.458+0000 7f5c3a23f640 1 -- 172.21.15.150:0/3612895593 shutdown_connections 2024-03-17T23:48:26.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.458+0000 7f5c3a23f640 1 --2- 172.21.15.150:0/3612895593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c340c6070 0x7f5c340cb0a0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.458+0000 7f5c3a23f640 1 --2- 172.21.15.150:0/3612895593 >> v2:172.21.15.157:3300/0 conn(0x7f5c340cdb20 0x7f5c340cdf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.458+0000 7f5c3a23f640 1 -- 172.21.15.150:0/3612895593 >> 172.21.15.150:0/3612895593 conn(0x7f5c34099a70 msgr2=0x7f5c34099e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:26.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.458+0000 7f5c3a23f640 1 -- 172.21.15.150:0/3612895593 shutdown_connections 2024-03-17T23:48:26.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.458+0000 7f5c3a23f640 1 -- 172.21.15.150:0/3612895593 wait complete. 2024-03-17T23:48:26.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.459+0000 7f5c3a23f640 1 Processor -- start 2024-03-17T23:48:26.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.460+0000 7f5c3a23f640 1 -- start start 2024-03-17T23:48:26.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.460+0000 7f5c3a23f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c340c5c00 0x7f5c3414d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:26.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.460+0000 7f5c3a23f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c340c6070 0x7f5c340b05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:26.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.460+0000 7f5c3a23f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c340cdb20 0x7f5c340b0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:26.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.460+0000 7f5c3a23f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5c340c7590 con 0x7f5c340c5c00 2024-03-17T23:48:26.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.460+0000 7f5c3a23f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5c340c7070 con 0x7f5c340cdb20 2024-03-17T23:48:26.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.460+0000 7f5c3a23f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5c3406ffa0 con 0x7f5c340c6070 2024-03-17T23:48:26.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.460+0000 7f5c337fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c340c5c00 0x7f5c3414d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:26.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.460+0000 7f5c337fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c340c5c00 0x7f5c3414d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47360/0 (socket says 172.21.15.150:47360) 2024-03-17T23:48:26.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.460+0000 7f5c337fe640 1 -- 172.21.15.150:0/2350571157 learned_addr learned my addr 172.21.15.150:0/2350571157 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:26.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.460+0000 7f5c33fff640 1 --2- 172.21.15.150:0/2350571157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c340cdb20 0x7f5c340b0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:26.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.460+0000 7f5c32ffd640 1 --2- 172.21.15.150:0/2350571157 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c340c6070 0x7f5c340b05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:26.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.461+0000 7f5c337fe640 1 -- 172.21.15.150:0/2350571157 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c340c6070 msgr2=0x7f5c340b05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:26.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.461+0000 7f5c337fe640 1 --2- 172.21.15.150:0/2350571157 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c340c6070 0x7f5c340b05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.461+0000 7f5c337fe640 1 -- 172.21.15.150:0/2350571157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c340cdb20 msgr2=0x7f5c340b0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:26.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.461+0000 7f5c337fe640 1 --2- 172.21.15.150:0/2350571157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c340cdb20 0x7f5c340b0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.461+0000 7f5c337fe640 1 -- 172.21.15.150:0/2350571157 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c34158bd0 con 0x7f5c340c5c00 2024-03-17T23:48:26.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.461+0000 7f5c337fe640 1 --2- 172.21.15.150:0/2350571157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c340c5c00 0x7f5c3414d530 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f5c2800c7f0 tx=0x7f5c2800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:26.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.461+0000 7f5c1bfff640 1 -- 172.21.15.150:0/2350571157 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c28015070 con 0x7f5c340c5c00 2024-03-17T23:48:26.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.461+0000 7f5c1bfff640 1 -- 172.21.15.150:0/2350571157 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c280077c0 con 0x7f5c340c5c00 2024-03-17T23:48:26.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.461+0000 7f5c3a23f640 1 -- 172.21.15.150:0/2350571157 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c3414da70 con 0x7f5c340c5c00 2024-03-17T23:48:26.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.461+0000 7f5c1bfff640 1 -- 172.21.15.150:0/2350571157 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c28007b70 con 0x7f5c340c5c00 2024-03-17T23:48:26.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.461+0000 7f5c3a23f640 1 -- 172.21.15.150:0/2350571157 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c340b3d20 con 0x7f5c340c5c00 2024-03-17T23:48:26.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.462+0000 7f5c3a23f640 1 -- 172.21.15.150:0/2350571157 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c340688b0 con 0x7f5c340c5c00 2024-03-17T23:48:26.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.462+0000 7f5c1bfff640 1 -- 172.21.15.150:0/2350571157 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5c28018070 con 0x7f5c340c5c00 2024-03-17T23:48:26.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.463+0000 7f5c1bfff640 1 --2- 172.21.15.150:0/2350571157 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5c2006bbd0 0x7f5c2006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:26.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.463+0000 7f5c32ffd640 1 --2- 172.21.15.150:0/2350571157 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5c2006bbd0 0x7f5c2006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:26.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.463+0000 7f5c1bfff640 1 -- 172.21.15.150:0/2350571157 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6496+0+0 (secure 0 0 0) 0x7f5c2808ab70 con 0x7f5c340c5c00 2024-03-17T23:48:26.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.463+0000 7f5c32ffd640 1 --2- 172.21.15.150:0/2350571157 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5c2006bbd0 0x7f5c2006e090 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5c340b1df0 tx=0x7f5c24007b70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:26.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.468+0000 7f5c1bfff640 1 -- 172.21.15.150:0/2350571157 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5c3414da70 con 0x7f5c340c5c00 2024-03-17T23:48:26.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.646+0000 7f5c3a23f640 1 -- 172.21.15.150:0/2350571157 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5c341598b0 con 0x7f5c340c5c00 2024-03-17T23:48:26.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.646+0000 7f5c1bfff640 1 -- 172.21.15.150:0/2350571157 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v43) ==== 104+0+11 (secure 0 0 0) 0x7f5c2808a020 con 0x7f5c340c5c00 2024-03-17T23:48:26.647 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:48:26.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.648+0000 7f5c3a23f640 1 -- 172.21.15.150:0/2350571157 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5c2006bbd0 msgr2=0x7f5c2006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:26.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.648+0000 7f5c3a23f640 1 --2- 172.21.15.150:0/2350571157 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5c2006bbd0 0x7f5c2006e090 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5c340b1df0 tx=0x7f5c24007b70 comp rx=0 tx=0).stop 2024-03-17T23:48:26.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.648+0000 7f5c3a23f640 1 -- 172.21.15.150:0/2350571157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c340c5c00 msgr2=0x7f5c3414d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:26.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.648+0000 7f5c3a23f640 1 --2- 172.21.15.150:0/2350571157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c340c5c00 0x7f5c3414d530 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f5c2800c7f0 tx=0x7f5c2800ccb0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.648+0000 7f5c3a23f640 1 -- 172.21.15.150:0/2350571157 shutdown_connections 2024-03-17T23:48:26.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.648+0000 7f5c3a23f640 1 --2- 172.21.15.150:0/2350571157 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5c2006bbd0 0x7f5c2006e090 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.648+0000 7f5c3a23f640 1 --2- 172.21.15.150:0/2350571157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c340cdb20 0x7f5c340b0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.648+0000 7f5c3a23f640 1 --2- 172.21.15.150:0/2350571157 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5c340c6070 0x7f5c340b05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.648+0000 7f5c3a23f640 1 --2- 172.21.15.150:0/2350571157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c340c5c00 0x7f5c3414d530 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.648+0000 7f5c3a23f640 1 -- 172.21.15.150:0/2350571157 >> 172.21.15.150:0/2350571157 conn(0x7f5c34099a70 msgr2=0x7f5c340ad980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:26.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.648+0000 7f5c3a23f640 1 -- 172.21.15.150:0/2350571157 shutdown_connections 2024-03-17T23:48:26.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.648+0000 7f5c3a23f640 1 -- 172.21.15.150:0/2350571157 wait complete. 2024-03-17T23:48:26.660 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:48:26.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.766+0000 7f0f69d59640 1 Processor -- start 2024-03-17T23:48:26.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.766+0000 7f0f69d59640 1 -- start start 2024-03-17T23:48:26.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.767+0000 7f0f69d59640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0f641615d0 0x7f0f640d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:26.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.767+0000 7f0f69d59640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f640d2870 con 0x7f0f640d0e30 2024-03-17T23:48:26.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.767+0000 7f0f69d59640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f640d2b50 con 0x7f0f640d3910 2024-03-17T23:48:26.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.767+0000 7f0f69d59640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f0f641619b0 con 0x7f0f641615d0 2024-03-17T23:48:26.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.767+0000 7f0f637fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0f641615d0 0x7f0f640d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:26.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.767+0000 7f0f63fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f0f640d0e30 0x7f0f640b6e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:57494/0 (socket says 172.21.15.150:57494) 2024-03-17T23:48:26.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.767+0000 7f0f63fff640 1 -- 172.21.15.150:0/4040081500 learned_addr learned my addr 172.21.15.150:0/4040081500 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:26.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.768+0000 7f0f637fe640 1 -- 172.21.15.150:0/4040081500 >> v1:172.21.15.157:6789/0 conn(0x7f0f640d3910 legacy=0x7f0f640d0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:26.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.768+0000 7f0f637fe640 1 -- 172.21.15.150:0/4040081500 >> v1:172.21.15.150:6789/0 conn(0x7f0f640d0e30 legacy=0x7f0f640b6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:26.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.768+0000 7f0f637fe640 1 -- 172.21.15.150:0/4040081500 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f640d2e30 con 0x7f0f641615d0 2024-03-17T23:48:26.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.768+0000 7f0f627fc640 1 -- 172.21.15.150:0/4040081500 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 40465730 0 0) 0x7f0f640d2870 con 0x7f0f640d0e30 2024-03-17T23:48:26.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.768+0000 7f0f637fe640 1 --2- 172.21.15.150:0/4040081500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f641615d0 0x7f0f640d2020 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f0f580097a0 tx=0x7f0f5802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15c2c902772aba32 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:26.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.768+0000 7f0f627fc640 1 -- 172.21.15.150:0/4040081500 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f58034070 con 0x7f0f641615d0 2024-03-17T23:48:26.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.769+0000 7f0f627fc640 1 -- 172.21.15.150:0/4040081500 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f58030610 con 0x7f0f641615d0 2024-03-17T23:48:26.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.769+0000 7f0f627fc640 1 -- 172.21.15.150:0/4040081500 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f580309e0 con 0x7f0f641615d0 2024-03-17T23:48:26.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.769+0000 7f0f69d59640 1 -- 172.21.15.150:0/4040081500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f641615d0 msgr2=0x7f0f640d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:26.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.769+0000 7f0f69d59640 1 --2- 172.21.15.150:0/4040081500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f641615d0 0x7f0f640d2020 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f0f580097a0 tx=0x7f0f5802b960 comp rx=0 tx=0).stop 2024-03-17T23:48:26.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.769+0000 7f0f69d59640 1 -- 172.21.15.150:0/4040081500 shutdown_connections 2024-03-17T23:48:26.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.769+0000 7f0f69d59640 1 --2- 172.21.15.150:0/4040081500 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f641615d0 0x7f0f640d2020 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.769+0000 7f0f69d59640 1 -- 172.21.15.150:0/4040081500 >> 172.21.15.150:0/4040081500 conn(0x7f0f640730b0 msgr2=0x7f0f640b5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:26.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.769+0000 7f0f69d59640 1 -- 172.21.15.150:0/4040081500 shutdown_connections 2024-03-17T23:48:26.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.769+0000 7f0f69d59640 1 -- 172.21.15.150:0/4040081500 wait complete. 2024-03-17T23:48:26.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.770+0000 7f0f69d59640 1 Processor -- start 2024-03-17T23:48:26.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.770+0000 7f0f69d59640 1 -- start start 2024-03-17T23:48:26.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.771+0000 7f0f69d59640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f640d0e30 0x7f0f64154cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:26.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.771+0000 7f0f69d59640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f640d3910 0x7f0f641569b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:26.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.771+0000 7f0f69d59640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f641615d0 0x7f0f64154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:26.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.771+0000 7f0f69d59640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0f640d2560 con 0x7f0f641615d0 2024-03-17T23:48:26.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.771+0000 7f0f69d59640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0f640d33f0 con 0x7f0f640d0e30 2024-03-17T23:48:26.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.771+0000 7f0f69d59640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0f640b9660 con 0x7f0f640d3910 2024-03-17T23:48:26.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.771+0000 7f0f63fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f641615d0 0x7f0f64154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:26.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.771+0000 7f0f63fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f641615d0 0x7f0f64154350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47380/0 (socket says 172.21.15.150:47380) 2024-03-17T23:48:26.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.771+0000 7f0f63fff640 1 -- 172.21.15.150:0/2507430474 learned_addr learned my addr 172.21.15.150:0/2507430474 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:26.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.771+0000 7f0f62ffd640 1 --2- 172.21.15.150:0/2507430474 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f640d3910 0x7f0f641569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:26.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.771+0000 7f0f637fe640 1 --2- 172.21.15.150:0/2507430474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f640d0e30 0x7f0f64154cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:26.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.772+0000 7f0f63fff640 1 -- 172.21.15.150:0/2507430474 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f640d3910 msgr2=0x7f0f641569b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:26.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.772+0000 7f0f63fff640 1 --2- 172.21.15.150:0/2507430474 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f640d3910 0x7f0f641569b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.772+0000 7f0f63fff640 1 -- 172.21.15.150:0/2507430474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f640d0e30 msgr2=0x7f0f64154cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:26.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.772+0000 7f0f63fff640 1 --2- 172.21.15.150:0/2507430474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f640d0e30 0x7f0f64154cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:26.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.772+0000 7f0f63fff640 1 -- 172.21.15.150:0/2507430474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f64156ef0 con 0x7f0f641615d0 2024-03-17T23:48:26.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.772+0000 7f0f63fff640 1 --2- 172.21.15.150:0/2507430474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f641615d0 0x7f0f64154350 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c00c7a0 tx=0x7f0f5c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:26.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.772+0000 7f0f4bfff640 1 -- 172.21.15.150:0/2507430474 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c018070 con 0x7f0f641615d0 2024-03-17T23:48:26.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.772+0000 7f0f4bfff640 1 -- 172.21.15.150:0/2507430474 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f5c00a640 con 0x7f0f641615d0 2024-03-17T23:48:26.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.772+0000 7f0f69d59640 1 -- 172.21.15.150:0/2507430474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f641f75f0 con 0x7f0f641615d0 2024-03-17T23:48:26.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.772+0000 7f0f4bfff640 1 -- 172.21.15.150:0/2507430474 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c00aab0 con 0x7f0f641615d0 2024-03-17T23:48:26.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.772+0000 7f0f69d59640 1 -- 172.21.15.150:0/2507430474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f641f7bb0 con 0x7f0f641615d0 2024-03-17T23:48:26.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.773+0000 7f0f69d59640 1 -- 172.21.15.150:0/2507430474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f640688b0 con 0x7f0f641615d0 2024-03-17T23:48:26.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.773+0000 7f0f4bfff640 1 -- 172.21.15.150:0/2507430474 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0f5c01b070 con 0x7f0f641615d0 2024-03-17T23:48:26.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.774+0000 7f0f4bfff640 1 --2- 172.21.15.150:0/2507430474 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0f5006bc00 0x7f0f5006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:26.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.774+0000 7f0f4bfff640 1 -- 172.21.15.150:0/2507430474 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6496+0+0 (secure 0 0 0) 0x7f0f5c08a1e0 con 0x7f0f641615d0 2024-03-17T23:48:26.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.774+0000 7f0f637fe640 1 --2- 172.21.15.150:0/2507430474 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0f5006bc00 0x7f0f5006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:26.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.774+0000 7f0f637fe640 1 --2- 172.21.15.150:0/2507430474 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0f5006bc00 0x7f0f5006e0c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0f58036000 tx=0x7f0f58008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:26.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:26.778+0000 7f0f4bfff640 1 -- 172.21.15.150:0/2507430474 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0f5c058400 con 0x7f0f641615d0 2024-03-17T23:48:27.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.033+0000 7f0f69d59640 1 -- 172.21.15.150:0/2507430474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0f6415f3e0 con 0x7f0f641615d0 2024-03-17T23:48:27.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.033+0000 7f0f4bfff640 1 -- 172.21.15.150:0/2507430474 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v43) ==== 100+0+11 (secure 0 0 0) 0x7f0f5c05c4a0 con 0x7f0f641615d0 2024-03-17T23:48:27.034 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:48:27.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.035+0000 7f0f69d59640 1 -- 172.21.15.150:0/2507430474 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0f5006bc00 msgr2=0x7f0f5006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:27.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.035+0000 7f0f69d59640 1 --2- 172.21.15.150:0/2507430474 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0f5006bc00 0x7f0f5006e0c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0f58036000 tx=0x7f0f58008030 comp rx=0 tx=0).stop 2024-03-17T23:48:27.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.035+0000 7f0f69d59640 1 -- 172.21.15.150:0/2507430474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f641615d0 msgr2=0x7f0f64154350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:27.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.035+0000 7f0f69d59640 1 --2- 172.21.15.150:0/2507430474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f641615d0 0x7f0f64154350 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c00c7a0 tx=0x7f0f5c00cc60 comp rx=0 tx=0).stop 2024-03-17T23:48:27.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.036+0000 7f0f69d59640 1 -- 172.21.15.150:0/2507430474 shutdown_connections 2024-03-17T23:48:27.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.036+0000 7f0f69d59640 1 --2- 172.21.15.150:0/2507430474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f641615d0 0x7f0f64154350 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.036+0000 7f0f69d59640 1 --2- 172.21.15.150:0/2507430474 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0f640d3910 0x7f0f641569b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.036+0000 7f0f69d59640 1 --2- 172.21.15.150:0/2507430474 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0f5006bc00 0x7f0f5006e0c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.036+0000 7f0f69d59640 1 --2- 172.21.15.150:0/2507430474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f640d0e30 0x7f0f64154cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.036+0000 7f0f69d59640 1 -- 172.21.15.150:0/2507430474 >> 172.21.15.150:0/2507430474 conn(0x7f0f640730b0 msgr2=0x7f0f6415e280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:27.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.036+0000 7f0f69d59640 1 -- 172.21.15.150:0/2507430474 shutdown_connections 2024-03-17T23:48:27.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.036+0000 7f0f69d59640 1 -- 172.21.15.150:0/2507430474 wait complete. 2024-03-17T23:48:27.050 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-17T23:48:27.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.154+0000 7fb4f7110640 1 Processor -- start 2024-03-17T23:48:27.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.154+0000 7fb4f7110640 1 -- start start 2024-03-17T23:48:27.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.155+0000 7fb4f7110640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb4f00d3910 0x7fb4f00d0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:27.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.155+0000 7fb4f7110640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb4f00b9660 0x7fb4f00d0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:27.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.155+0000 7fb4f7110640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4f00d2870 con 0x7fb4f01615d0 2024-03-17T23:48:27.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.155+0000 7fb4f7110640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fb4f01619b0 con 0x7fb4f00d3910 2024-03-17T23:48:27.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.155+0000 7fb4f7110640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb4f00b7e80 con 0x7fb4f00b9660 2024-03-17T23:48:27.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.155+0000 7fb4effff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb4f00d3910 0x7fb4f00d0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:27.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.155+0000 7fb4effff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb4f00d3910 0x7fb4f00d0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47386/0 (socket says 172.21.15.150:47386) 2024-03-17T23:48:27.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.155+0000 7fb4effff640 1 -- 172.21.15.150:0/3647538608 learned_addr learned my addr 172.21.15.150:0/3647538608 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:27.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.155+0000 7fb4f56ad640 1 --2- 172.21.15.150:0/3647538608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb4f00b9660 0x7fb4f00d0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:27.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.155+0000 7fb4effff640 1 -- 172.21.15.150:0/3647538608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb4f00b9660 msgr2=0x7fb4f00d0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:27.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.155+0000 7fb4effff640 1 --2- 172.21.15.150:0/3647538608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb4f00b9660 0x7fb4f00d0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.155+0000 7fb4effff640 1 -- 172.21.15.150:0/3647538608 >> v1:172.21.15.157:6789/0 conn(0x7fb4f01615d0 legacy=0x7fb4f00d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:27.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.155+0000 7fb4effff640 1 -- 172.21.15.150:0/3647538608 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4f00d2b50 con 0x7fb4f00d3910 2024-03-17T23:48:27.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.156+0000 7fb4effff640 1 --2- 172.21.15.150:0/3647538608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4f00d3910 0x7fb4f00d0720 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0009870 tx=0x7fb4e002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42211acd0e49f0b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:27.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.156+0000 7fb4ef7fe640 1 -- 172.21.15.150:0/3647538608 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e0035020 con 0x7fb4f00d3910 2024-03-17T23:48:27.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.156+0000 7fb4ef7fe640 1 -- 172.21.15.150:0/3647538608 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4e0031640 con 0x7fb4f00d3910 2024-03-17T23:48:27.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.156+0000 7fb4ef7fe640 1 -- 172.21.15.150:0/3647538608 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e002d040 con 0x7fb4f00d3910 2024-03-17T23:48:27.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.157+0000 7fb4f7110640 1 -- 172.21.15.150:0/3647538608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4f00d3910 msgr2=0x7fb4f00d0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:27.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.157+0000 7fb4f7110640 1 --2- 172.21.15.150:0/3647538608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4f00d3910 0x7fb4f00d0720 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0009870 tx=0x7fb4e002b8f0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.157+0000 7fb4f7110640 1 -- 172.21.15.150:0/3647538608 shutdown_connections 2024-03-17T23:48:27.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.157+0000 7fb4f7110640 1 --2- 172.21.15.150:0/3647538608 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb4f00b9660 0x7fb4f00d0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.157+0000 7fb4f7110640 1 --2- 172.21.15.150:0/3647538608 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4f00d3910 0x7fb4f00d0720 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.157+0000 7fb4f7110640 1 -- 172.21.15.150:0/3647538608 >> 172.21.15.150:0/3647538608 conn(0x7fb4f00730b0 msgr2=0x7fb4f00b5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:27.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.157+0000 7fb4f7110640 1 -- 172.21.15.150:0/3647538608 shutdown_connections 2024-03-17T23:48:27.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.157+0000 7fb4f7110640 1 -- 172.21.15.150:0/3647538608 wait complete. 2024-03-17T23:48:27.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.158+0000 7fb4f7110640 1 Processor -- start 2024-03-17T23:48:27.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.158+0000 7fb4f7110640 1 -- start start 2024-03-17T23:48:27.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.159+0000 7fb4f7110640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb4f00b9660 0x7fb4f01574b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:27.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.159+0000 7fb4f7110640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb4f00d3910 0x7fb4f00b4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:27.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.159+0000 7fb4f7110640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4f01615d0 0x7fb4f0154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:27.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.159+0000 7fb4f7110640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f00d33f0 con 0x7fb4f01615d0 2024-03-17T23:48:27.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.159+0000 7fb4f7110640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f00d3110 con 0x7fb4f00b9660 2024-03-17T23:48:27.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.159+0000 7fb4f7110640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb4f00b6e70 con 0x7fb4f00d3910 2024-03-17T23:48:27.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.159+0000 7fb4f56ad640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4f01615d0 0x7fb4f0154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:27.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.159+0000 7fb4f56ad640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4f01615d0 0x7fb4f0154070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47388/0 (socket says 172.21.15.150:47388) 2024-03-17T23:48:27.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.159+0000 7fb4f56ad640 1 -- 172.21.15.150:0/1728733911 learned_addr learned my addr 172.21.15.150:0/1728733911 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:27.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.159+0000 7fb4effff640 1 --2- 172.21.15.150:0/1728733911 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb4f00d3910 0x7fb4f00b4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:27.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.159+0000 7fb4f4eac640 1 --2- 172.21.15.150:0/1728733911 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb4f00b9660 0x7fb4f01574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:27.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.159+0000 7fb4f56ad640 1 -- 172.21.15.150:0/1728733911 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb4f00d3910 msgr2=0x7fb4f00b4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:27.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.159+0000 7fb4f56ad640 1 --2- 172.21.15.150:0/1728733911 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb4f00d3910 0x7fb4f00b4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.159+0000 7fb4f56ad640 1 -- 172.21.15.150:0/1728733911 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb4f00b9660 msgr2=0x7fb4f01574b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:27.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.159+0000 7fb4f56ad640 1 --2- 172.21.15.150:0/1728733911 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb4f00b9660 0x7fb4f01574b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.159+0000 7fb4f56ad640 1 -- 172.21.15.150:0/1728733911 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4f01f4c80 con 0x7fb4f01615d0 2024-03-17T23:48:27.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.159+0000 7fb4f56ad640 1 --2- 172.21.15.150:0/1728733911 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4f01615d0 0x7fb4f0154070 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb4e800efc0 tx=0x7fb4e800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:27.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.160+0000 7fb4ed7fa640 1 -- 172.21.15.150:0/1728733911 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e8018020 con 0x7fb4f01615d0 2024-03-17T23:48:27.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.160+0000 7fb4ed7fa640 1 -- 172.21.15.150:0/1728733911 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4e8014480 con 0x7fb4f01615d0 2024-03-17T23:48:27.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.160+0000 7fb4f7110640 1 -- 172.21.15.150:0/1728733911 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4f0155b90 con 0x7fb4f01615d0 2024-03-17T23:48:27.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.160+0000 7fb4ed7fa640 1 -- 172.21.15.150:0/1728733911 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e8014870 con 0x7fb4f01615d0 2024-03-17T23:48:27.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.160+0000 7fb4f7110640 1 -- 172.21.15.150:0/1728733911 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4f01f4f20 con 0x7fb4f01615d0 2024-03-17T23:48:27.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.161+0000 7fb4f7110640 1 -- 172.21.15.150:0/1728733911 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4f00688b0 con 0x7fb4f01615d0 2024-03-17T23:48:27.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.161+0000 7fb4ed7fa640 1 -- 172.21.15.150:0/1728733911 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb4e8005d10 con 0x7fb4f01615d0 2024-03-17T23:48:27.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.161+0000 7fb4ed7fa640 1 --2- 172.21.15.150:0/1728733911 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb4d006bbd0 0x7fb4d006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:27.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.161+0000 7fb4ed7fa640 1 -- 172.21.15.150:0/1728733911 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6496+0+0 (secure 0 0 0) 0x7fb4e808b9c0 con 0x7fb4f01615d0 2024-03-17T23:48:27.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.162+0000 7fb4f4eac640 1 --2- 172.21.15.150:0/1728733911 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb4d006bbd0 0x7fb4d006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:27.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.162+0000 7fb4f4eac640 1 --2- 172.21.15.150:0/1728733911 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb4d006bbd0 0x7fb4d006e090 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb4d80016e0 tx=0x7fb4d8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:27.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.166+0000 7fb4ed7fa640 1 -- 172.21.15.150:0/1728733911 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb4e808de20 con 0x7fb4f01615d0 2024-03-17T23:48:27.356 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-17T23:48:27.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.356+0000 7fb4f7110640 1 -- 172.21.15.150:0/1728733911 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7fb4f015ef30 con 0x7fb4f01615d0 2024-03-17T23:48:27.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.356+0000 7fb4ed7fa640 1 -- 172.21.15.150:0/1728733911 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v43) ==== 97+0+11 (secure 0 0 0) 0x7fb4e8022030 con 0x7fb4f01615d0 2024-03-17T23:48:27.357 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:48:27.359 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-17T23:48:27.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.358+0000 7fb4f7110640 1 -- 172.21.15.150:0/1728733911 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb4d006bbd0 msgr2=0x7fb4d006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:27.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.358+0000 7fb4f7110640 1 --2- 172.21.15.150:0/1728733911 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb4d006bbd0 0x7fb4d006e090 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb4d80016e0 tx=0x7fb4d8009040 comp rx=0 tx=0).stop 2024-03-17T23:48:27.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.358+0000 7fb4f7110640 1 -- 172.21.15.150:0/1728733911 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4f01615d0 msgr2=0x7fb4f0154070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:27.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.358+0000 7fb4f7110640 1 --2- 172.21.15.150:0/1728733911 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4f01615d0 0x7fb4f0154070 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb4e800efc0 tx=0x7fb4e800c490 comp rx=0 tx=0).stop 2024-03-17T23:48:27.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.358+0000 7fb4f7110640 1 -- 172.21.15.150:0/1728733911 shutdown_connections 2024-03-17T23:48:27.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.358+0000 7fb4f7110640 1 --2- 172.21.15.150:0/1728733911 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4f01615d0 0x7fb4f0154070 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.358+0000 7fb4f7110640 1 --2- 172.21.15.150:0/1728733911 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb4f00d3910 0x7fb4f00b4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.359+0000 7fb4f7110640 1 --2- 172.21.15.150:0/1728733911 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb4d006bbd0 0x7fb4d006e090 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.359+0000 7fb4f7110640 1 --2- 172.21.15.150:0/1728733911 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb4f00b9660 0x7fb4f01574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.359+0000 7fb4f7110640 1 -- 172.21.15.150:0/1728733911 >> 172.21.15.150:0/1728733911 conn(0x7fb4f00730b0 msgr2=0x7fb4f015f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:27.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.359+0000 7fb4f7110640 1 -- 172.21.15.150:0/1728733911 shutdown_connections 2024-03-17T23:48:27.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.359+0000 7fb4f7110640 1 -- 172.21.15.150:0/1728733911 wait complete. 2024-03-17T23:48:27.371 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-17T23:48:27.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.556+0000 7f2aa0e68640 1 Processor -- start 2024-03-17T23:48:27.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.556+0000 7f2aa0e68640 1 -- start start 2024-03-17T23:48:27.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.557+0000 7f2aa0e68640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f2a9c0d2100 0x7f2a9c0d2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:27.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.557+0000 7f2aa0e68640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a9c15ec90 con 0x7f2a9c0b97f0 2024-03-17T23:48:27.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.557+0000 7f2aa0e68640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a9c15ef70 con 0x7f2a9c15df40 2024-03-17T23:48:27.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.557+0000 7f2aa0e68640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f2a9c0b6d40 con 0x7f2a9c0d2100 2024-03-17T23:48:27.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.557+0000 7f2a93fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f2a9c0b97f0 0x7f2a9c162160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:57498/0 (socket says 172.21.15.150:57498) 2024-03-17T23:48:27.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.557+0000 7f2a93fff640 1 -- 172.21.15.150:0/2374688206 learned_addr learned my addr 172.21.15.150:0/2374688206 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:27.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.557+0000 7f2a99d9b640 1 -- 172.21.15.150:0/2374688206 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2001029084 0 0) 0x7f2a9c15ec90 con 0x7f2a9c0b97f0 2024-03-17T23:48:27.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.557+0000 7f2a99d9b640 1 -- 172.21.15.150:0/2374688206 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a88003520 con 0x7f2a9c0b97f0 2024-03-17T23:48:27.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.557+0000 7f2a9ad9d640 1 --2- 172.21.15.150:0/2374688206 >> v2:172.21.15.157:3300/0 conn(0x7f2a9c0d2100 0x7f2a9c0d2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:27.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.557+0000 7f2a99d9b640 1 -- 172.21.15.150:0/2374688206 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2041986482 0 0) 0x7f2a88003520 con 0x7f2a9c0b97f0 2024-03-17T23:48:27.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.557+0000 7f2a99d9b640 1 -- 172.21.15.150:0/2374688206 >> v2:172.21.15.157:3300/0 conn(0x7f2a9c0d2100 msgr2=0x7f2a9c0d2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:27.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.557+0000 7f2a99d9b640 1 --2- 172.21.15.150:0/2374688206 >> v2:172.21.15.157:3300/0 conn(0x7f2a9c0d2100 0x7f2a9c0d2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.557+0000 7f2a99d9b640 1 -- 172.21.15.150:0/2374688206 >> v1:172.21.15.157:6789/0 conn(0x7f2a9c15df40 legacy=0x7f2a9c15e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:27.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.557+0000 7f2a99d9b640 1 -- 172.21.15.150:0/2374688206 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a9c15f250 con 0x7f2a9c0b97f0 2024-03-17T23:48:27.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.557+0000 7f2a99d9b640 1 -- 172.21.15.150:0/2374688206 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f2a84001670 con 0x7f2a9c0b97f0 2024-03-17T23:48:27.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.557+0000 7f2a99d9b640 1 -- 172.21.15.150:0/2374688206 >> v1:172.21.15.150:6789/0 conn(0x7f2a9c0b97f0 legacy=0x7f2a9c162160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:27.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.557+0000 7f2a99d9b640 1 --2- 172.21.15.150:0/2374688206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a880039b0 0x7f2a88003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:27.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.557+0000 7f2a99d9b640 1 --2- 172.21.15.150:0/2374688206 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2a88004800 0x7f2a88004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:27.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.557+0000 7f2a99d9b640 1 --2- 172.21.15.150:0/2374688206 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2a880051c0 0x7f2a8800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:27.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.557+0000 7f2a99d9b640 1 -- 172.21.15.150:0/2374688206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8800c040 con 0x7f2a880039b0 2024-03-17T23:48:27.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.558+0000 7f2a99d9b640 1 -- 172.21.15.150:0/2374688206 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8800c1f0 con 0x7f2a88004800 2024-03-17T23:48:27.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.558+0000 7f2a99d9b640 1 -- 172.21.15.150:0/2374688206 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2a8800c3f0 con 0x7f2a880051c0 2024-03-17T23:48:27.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.558+0000 7f2a9a59c640 1 --2- 172.21.15.150:0/2374688206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a880039b0 0x7f2a88003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:27.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.558+0000 7f2a93fff640 1 --2- 172.21.15.150:0/2374688206 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2a88004800 0x7f2a88004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:27.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.558+0000 7f2a9ad9d640 1 --2- 172.21.15.150:0/2374688206 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2a880051c0 0x7f2a8800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:27.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.558+0000 7f2a9a59c640 1 -- 172.21.15.150:0/2374688206 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2a880051c0 msgr2=0x7f2a8800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:27.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.558+0000 7f2a9a59c640 1 --2- 172.21.15.150:0/2374688206 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2a880051c0 0x7f2a8800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.558+0000 7f2a9a59c640 1 -- 172.21.15.150:0/2374688206 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2a88004800 msgr2=0x7f2a88004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:27.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.558+0000 7f2a9a59c640 1 --2- 172.21.15.150:0/2374688206 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2a88004800 0x7f2a88004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.558+0000 7f2a9a59c640 1 -- 172.21.15.150:0/2374688206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a88003520 con 0x7f2a880039b0 2024-03-17T23:48:27.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.558+0000 7f2a9ad9d640 1 --2- 172.21.15.150:0/2374688206 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2a880051c0 0x7f2a8800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:27.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.558+0000 7f2a9ad9d640 1 -- 172.21.15.150:0/2374688206 reap_dead start 2024-03-17T23:48:27.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.559+0000 7f2a9a59c640 1 --2- 172.21.15.150:0/2374688206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a880039b0 0x7f2a88003dd0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c001750 tx=0x7f2a88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc5de719c086c36 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:27.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.559+0000 7f2a99d9b640 1 -- 172.21.15.150:0/2374688206 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c001370 con 0x7f2a880039b0 2024-03-17T23:48:27.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.559+0000 7f2a99d9b640 1 -- 172.21.15.150:0/2374688206 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a8c0052d0 con 0x7f2a880039b0 2024-03-17T23:48:27.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.559+0000 7f2a99d9b640 1 -- 172.21.15.150:0/2374688206 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a8c003830 con 0x7f2a880039b0 2024-03-17T23:48:27.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.559+0000 7f2aa0e68640 1 -- 172.21.15.150:0/2374688206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a880039b0 msgr2=0x7f2a88003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:27.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.559+0000 7f2aa0e68640 1 --2- 172.21.15.150:0/2374688206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a880039b0 0x7f2a88003dd0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c001750 tx=0x7f2a88005830 comp rx=0 tx=0).stop 2024-03-17T23:48:27.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.559+0000 7f2aa0e68640 1 -- 172.21.15.150:0/2374688206 shutdown_connections 2024-03-17T23:48:27.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.559+0000 7f2aa0e68640 1 --2- 172.21.15.150:0/2374688206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a880039b0 0x7f2a88003dd0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.559+0000 7f2aa0e68640 1 -- 172.21.15.150:0/2374688206 >> 172.21.15.150:0/2374688206 conn(0x7f2a9c072ef0 msgr2=0x7f2a9c0d3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:27.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.559+0000 7f2aa0e68640 1 -- 172.21.15.150:0/2374688206 shutdown_connections 2024-03-17T23:48:27.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.559+0000 7f2aa0e68640 1 -- 172.21.15.150:0/2374688206 wait complete. 2024-03-17T23:48:27.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.561+0000 7f2aa0e68640 1 Processor -- start 2024-03-17T23:48:27.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.561+0000 7f2aa0e68640 1 -- start start 2024-03-17T23:48:27.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.561+0000 7f2aa0e68640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2a880039b0 0x7f2a9c1f2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:27.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.561+0000 7f2aa0e68640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2a9c13f850 0x7f2a9c1f33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:27.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.561+0000 7f2aa0e68640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a9c1453a0 0x7f2a9c0cc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:27.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.561+0000 7f2aa0e68640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9c0d2550 con 0x7f2a9c1453a0 2024-03-17T23:48:27.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.561+0000 7f2aa0e68640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9c0b9ca0 con 0x7f2a9c13f850 2024-03-17T23:48:27.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.561+0000 7f2aa0e68640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2a9c0d26d0 con 0x7f2a880039b0 2024-03-17T23:48:27.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.561+0000 7f2a9ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a9c1453a0 0x7f2a9c0cc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:27.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.561+0000 7f2a9ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a9c1453a0 0x7f2a9c0cc9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47410/0 (socket says 172.21.15.150:47410) 2024-03-17T23:48:27.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.561+0000 7f2a9ad9d640 1 -- 172.21.15.150:0/3794901869 learned_addr learned my addr 172.21.15.150:0/3794901869 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:27.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.562+0000 7f2a9a59c640 1 --2- 172.21.15.150:0/3794901869 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2a880039b0 0x7f2a9c1f2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:27.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.562+0000 7f2a9ad9d640 1 -- 172.21.15.150:0/3794901869 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2a880039b0 msgr2=0x7f2a9c1f2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:27.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.562+0000 7f2a93fff640 1 --2- 172.21.15.150:0/3794901869 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2a9c13f850 0x7f2a9c1f33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:27.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.562+0000 7f2a9ad9d640 1 --2- 172.21.15.150:0/3794901869 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2a880039b0 0x7f2a9c1f2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.562+0000 7f2a9ad9d640 1 -- 172.21.15.150:0/3794901869 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2a9c13f850 msgr2=0x7f2a9c1f33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:27.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.562+0000 7f2a9ad9d640 1 --2- 172.21.15.150:0/3794901869 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2a9c13f850 0x7f2a9c1f33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.562+0000 7f2a9ad9d640 1 -- 172.21.15.150:0/3794901869 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a9c0b6190 con 0x7f2a9c1453a0 2024-03-17T23:48:27.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.562+0000 7f2a9ad9d640 1 --2- 172.21.15.150:0/3794901869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a9c1453a0 0x7f2a9c0cc9b0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c0b6870 tx=0x7f2a94010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:27.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.562+0000 7f2a92ffd640 1 -- 172.21.15.150:0/3794901869 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a94010da0 con 0x7f2a9c1453a0 2024-03-17T23:48:27.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.562+0000 7f2a92ffd640 1 -- 172.21.15.150:0/3794901869 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a94003990 con 0x7f2a9c1453a0 2024-03-17T23:48:27.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.562+0000 7f2aa0e68640 1 -- 172.21.15.150:0/3794901869 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a9c0b6470 con 0x7f2a9c1453a0 2024-03-17T23:48:27.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.562+0000 7f2a92ffd640 1 -- 172.21.15.150:0/3794901869 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a94003d80 con 0x7f2a9c1453a0 2024-03-17T23:48:27.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.562+0000 7f2aa0e68640 1 -- 172.21.15.150:0/3794901869 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a9c1f3920 con 0x7f2a9c1453a0 2024-03-17T23:48:27.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.563+0000 7f2aa0e68640 1 -- 172.21.15.150:0/3794901869 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a9c0681a0 con 0x7f2a9c1453a0 2024-03-17T23:48:27.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.564+0000 7f2a92ffd640 1 -- 172.21.15.150:0/3794901869 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2a9401e020 con 0x7f2a9c1453a0 2024-03-17T23:48:27.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.564+0000 7f2a92ffd640 1 --2- 172.21.15.150:0/3794901869 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2a7c06bbd0 0x7f2a7c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:27.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.564+0000 7f2a9a59c640 1 --2- 172.21.15.150:0/3794901869 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2a7c06bbd0 0x7f2a7c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:27.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.564+0000 7f2a92ffd640 1 -- 172.21.15.150:0/3794901869 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6496+0+0 (secure 0 0 0) 0x7f2a9408b370 con 0x7f2a9c1453a0 2024-03-17T23:48:27.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.565+0000 7f2a9a59c640 1 --2- 172.21.15.150:0/3794901869 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2a7c06bbd0 0x7f2a7c06e090 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c033000 tx=0x7f2a8c009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:27.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.568+0000 7f2a92ffd640 1 -- 172.21.15.150:0/3794901869 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2a94059be0 con 0x7f2a9c1453a0 2024-03-17T23:48:27.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.779+0000 7f2aa0e68640 1 -- 172.21.15.150:0/3794901869 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2a9c0d19d0 con 0x7f2a9c1453a0 2024-03-17T23:48:27.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.779+0000 7f2a92ffd640 1 -- 172.21.15.150:0/3794901869 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v43) ==== 80+0+0 (secure 0 0 0) 0x7f2a940594e0 con 0x7f2a9c1453a0 2024-03-17T23:48:27.780 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-03-17T23:48:27.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.782+0000 7f2aa0e68640 1 -- 172.21.15.150:0/3794901869 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2a7c06bbd0 msgr2=0x7f2a7c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:27.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.782+0000 7f2aa0e68640 1 --2- 172.21.15.150:0/3794901869 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2a7c06bbd0 0x7f2a7c06e090 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c033000 tx=0x7f2a8c009030 comp rx=0 tx=0).stop 2024-03-17T23:48:27.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.782+0000 7f2aa0e68640 1 -- 172.21.15.150:0/3794901869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a9c1453a0 msgr2=0x7f2a9c0cc9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:27.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.782+0000 7f2aa0e68640 1 --2- 172.21.15.150:0/3794901869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a9c1453a0 0x7f2a9c0cc9b0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f2a9c0b6870 tx=0x7f2a94010590 comp rx=0 tx=0).stop 2024-03-17T23:48:27.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.782+0000 7f2aa0e68640 1 -- 172.21.15.150:0/3794901869 shutdown_connections 2024-03-17T23:48:27.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.782+0000 7f2aa0e68640 1 --2- 172.21.15.150:0/3794901869 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2a9c1453a0 0x7f2a9c0cc9b0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.782+0000 7f2aa0e68640 1 --2- 172.21.15.150:0/3794901869 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2a7c06bbd0 0x7f2a7c06e090 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.782+0000 7f2aa0e68640 1 --2- 172.21.15.150:0/3794901869 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2a9c13f850 0x7f2a9c1f33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.782+0000 7f2aa0e68640 1 --2- 172.21.15.150:0/3794901869 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2a880039b0 0x7f2a9c1f2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.782+0000 7f2aa0e68640 1 -- 172.21.15.150:0/3794901869 >> 172.21.15.150:0/3794901869 conn(0x7f2a9c072ef0 msgr2=0x7f2a9c15f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:27.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.782+0000 7f2aa0e68640 1 -- 172.21.15.150:0/3794901869 shutdown_connections 2024-03-17T23:48:27.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.782+0000 7f2aa0e68640 1 -- 172.21.15.150:0/3794901869 wait complete. 2024-03-17T23:48:27.794 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:48:27.794 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:48:27.810 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:48:27.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.952+0000 7efd0a6d3640 1 Processor -- start 2024-03-17T23:48:27.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.952+0000 7efd0a6d3640 1 -- start start 2024-03-17T23:48:27.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.953+0000 7efd0a6d3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd0415c030 0x7efd040cf910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:27.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.953+0000 7efd0a6d3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efd040ad890 0x7efd0406b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:27.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.953+0000 7efd0a6d3640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd040cfff0 con 0x7efd0415bc50 2024-03-17T23:48:27.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.953+0000 7efd0a6d3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7efd04067340 con 0x7efd040ad890 2024-03-17T23:48:27.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.953+0000 7efd0a6d3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efd040d0340 con 0x7efd0415c030 2024-03-17T23:48:27.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.953+0000 7efd08c70640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efd040ad890 0x7efd0406b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:27.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.953+0000 7efd08c70640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efd040ad890 0x7efd0406b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47422/0 (socket says 172.21.15.150:47422) 2024-03-17T23:48:27.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.953+0000 7efd08c70640 1 -- 172.21.15.150:0/818040507 learned_addr learned my addr 172.21.15.150:0/818040507 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:27.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.953+0000 7efd037fe640 1 --2- 172.21.15.150:0/818040507 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd0415c030 0x7efd040cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:27.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.953+0000 7efd08c70640 1 -- 172.21.15.150:0/818040507 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd0415c030 msgr2=0x7efd040cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:27.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.953+0000 7efd08c70640 1 --2- 172.21.15.150:0/818040507 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd0415c030 0x7efd040cf910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.953+0000 7efd08c70640 1 -- 172.21.15.150:0/818040507 >> v1:172.21.15.157:6789/0 conn(0x7efd0415bc50 legacy=0x7efd040ad180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:27.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.953+0000 7efd02ffd640 1 -- 172.21.15.150:0/818040507 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3594687258 0 0) 0x7efd040cfff0 con 0x7efd0415bc50 2024-03-17T23:48:27.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.953+0000 7efd08c70640 1 -- 172.21.15.150:0/818040507 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd04067590 con 0x7efd040ad890 2024-03-17T23:48:27.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.954+0000 7efd08c70640 1 --2- 172.21.15.150:0/818040507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd040ad890 0x7efd0406b190 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009e50 tx=0x7efcfc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c36a3b27d920477 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:27.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.954+0000 7efd02ffd640 1 -- 172.21.15.150:0/818040507 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc02ca70 con 0x7efd040ad890 2024-03-17T23:48:27.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.954+0000 7efd02ffd640 1 -- 172.21.15.150:0/818040507 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcfc02cc10 con 0x7efd040ad890 2024-03-17T23:48:27.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.954+0000 7efd02ffd640 1 -- 172.21.15.150:0/818040507 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc0343f0 con 0x7efd040ad890 2024-03-17T23:48:27.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.955+0000 7efd0a6d3640 1 -- 172.21.15.150:0/818040507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd040ad890 msgr2=0x7efd0406b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:27.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.955+0000 7efd0a6d3640 1 --2- 172.21.15.150:0/818040507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd040ad890 0x7efd0406b190 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009e50 tx=0x7efcfc02bed0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.955+0000 7efd0a6d3640 1 -- 172.21.15.150:0/818040507 shutdown_connections 2024-03-17T23:48:27.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.955+0000 7efd0a6d3640 1 --2- 172.21.15.150:0/818040507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd040ad890 0x7efd0406b190 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.955+0000 7efd0a6d3640 1 --2- 172.21.15.150:0/818040507 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd0415c030 0x7efd040cf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.955+0000 7efd0a6d3640 1 -- 172.21.15.150:0/818040507 >> 172.21.15.150:0/818040507 conn(0x7efd040b3da0 msgr2=0x7efd040b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:27.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.955+0000 7efd0a6d3640 1 -- 172.21.15.150:0/818040507 shutdown_connections 2024-03-17T23:48:27.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.955+0000 7efd0a6d3640 1 -- 172.21.15.150:0/818040507 wait complete. 2024-03-17T23:48:27.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.956+0000 7efd0a6d3640 1 Processor -- start 2024-03-17T23:48:27.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.956+0000 7efd0a6d3640 1 -- start start 2024-03-17T23:48:27.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.957+0000 7efd0a6d3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd040ad890 0x7efd0413fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:27.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.957+0000 7efd0a6d3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd0415bc50 0x7efd0415df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:27.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.957+0000 7efd0a6d3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd0415e4c0 0x7efd0414e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:27.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.957+0000 7efd0a6d3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7efd04099a70 con 0x7efd040ad890 2024-03-17T23:48:27.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.957+0000 7efd0a6d3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7efd04072d30 con 0x7efd0415bc50 2024-03-17T23:48:27.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.957+0000 7efd0a6d3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efd04099bf0 con 0x7efd0415e4c0 2024-03-17T23:48:27.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.957+0000 7efd03fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd040ad890 0x7efd0413fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:27.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.957+0000 7efd03fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd040ad890 0x7efd0413fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47436/0 (socket says 172.21.15.150:47436) 2024-03-17T23:48:27.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.957+0000 7efd03fff640 1 -- 172.21.15.150:0/2849048098 learned_addr learned my addr 172.21.15.150:0/2849048098 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:27.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.957+0000 7efd037fe640 1 --2- 172.21.15.150:0/2849048098 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd0415bc50 0x7efd0415df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:27.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.957+0000 7efd08c70640 1 --2- 172.21.15.150:0/2849048098 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd0415e4c0 0x7efd0414e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:27.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.957+0000 7efd03fff640 1 -- 172.21.15.150:0/2849048098 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd0415e4c0 msgr2=0x7efd0414e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:27.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.957+0000 7efd03fff640 1 --2- 172.21.15.150:0/2849048098 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd0415e4c0 0x7efd0414e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.957+0000 7efd03fff640 1 -- 172.21.15.150:0/2849048098 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd0415bc50 msgr2=0x7efd0415df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:27.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.957+0000 7efd03fff640 1 --2- 172.21.15.150:0/2849048098 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd0415bc50 0x7efd0415df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:27.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.957+0000 7efd03fff640 1 -- 172.21.15.150:0/2849048098 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd040b0850 con 0x7efd040ad890 2024-03-17T23:48:27.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.957+0000 7efd03fff640 1 --2- 172.21.15.150:0/2849048098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd040ad890 0x7efd0413fb50 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7efcf800c4e0 tx=0x7efcf800c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:27.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.958+0000 7efd00ff9640 1 -- 172.21.15.150:0/2849048098 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf8017070 con 0x7efd040ad890 2024-03-17T23:48:27.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.958+0000 7efd00ff9640 1 -- 172.21.15.150:0/2849048098 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcf80035f0 con 0x7efd040ad890 2024-03-17T23:48:27.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.958+0000 7efd0a6d3640 1 -- 172.21.15.150:0/2849048098 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd040b0b30 con 0x7efd040ad890 2024-03-17T23:48:27.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.958+0000 7efd00ff9640 1 -- 172.21.15.150:0/2849048098 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf80039e0 con 0x7efd040ad890 2024-03-17T23:48:27.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.958+0000 7efd0a6d3640 1 -- 172.21.15.150:0/2849048098 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd040b10f0 con 0x7efd040ad890 2024-03-17T23:48:27.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.959+0000 7efd0a6d3640 1 -- 172.21.15.150:0/2849048098 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd040681a0 con 0x7efd040ad890 2024-03-17T23:48:27.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.959+0000 7efd00ff9640 1 -- 172.21.15.150:0/2849048098 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7efcf801a070 con 0x7efd040ad890 2024-03-17T23:48:27.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.960+0000 7efd00ff9640 1 --2- 172.21.15.150:0/2849048098 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7efcf006bc00 0x7efcf006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:27.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.960+0000 7efd00ff9640 1 -- 172.21.15.150:0/2849048098 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6496+0+0 (secure 0 0 0) 0x7efcf8089fa0 con 0x7efd040ad890 2024-03-17T23:48:27.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.960+0000 7efd037fe640 1 --2- 172.21.15.150:0/2849048098 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7efcf006bc00 0x7efcf006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:27.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.960+0000 7efd037fe640 1 --2- 172.21.15.150:0/2849048098 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7efcf006bc00 0x7efcf006e0c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7efd040b5780 tx=0x7efcf4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:27.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:27.964+0000 7efd00ff9640 1 -- 172.21.15.150:0/2849048098 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7efcf8058130 con 0x7efd040ad890 2024-03-17T23:48:28.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.137+0000 7efd0a6d3640 1 -- 172.21.15.150:0/2849048098 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efd040cf4e0 con 0x7efd040ad890 2024-03-17T23:48:28.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.139+0000 7efd00ff9640 1 -- 172.21.15.150:0/2849048098 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) ==== 74+0+18790 (secure 0 0 0) 0x7efcf805c1d0 con 0x7efd040ad890 2024-03-17T23:48:28.140 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:28.140 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":43,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:48:25.427736+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:48:08.957879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-17T23:48:09.698808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-17T23:48:22.562269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:48:28.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.141+0000 7efd0a6d3640 1 -- 172.21.15.150:0/2849048098 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7efcf006bc00 msgr2=0x7efcf006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:28.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.141+0000 7efd0a6d3640 1 --2- 172.21.15.150:0/2849048098 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7efcf006bc00 0x7efcf006e0c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7efd040b5780 tx=0x7efcf4009190 comp rx=0 tx=0).stop 2024-03-17T23:48:28.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.141+0000 7efd0a6d3640 1 -- 172.21.15.150:0/2849048098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd040ad890 msgr2=0x7efd0413fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:28.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.141+0000 7efd0a6d3640 1 --2- 172.21.15.150:0/2849048098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd040ad890 0x7efd0413fb50 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7efcf800c4e0 tx=0x7efcf800c9a0 comp rx=0 tx=0).stop 2024-03-17T23:48:28.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.141+0000 7efd0a6d3640 1 -- 172.21.15.150:0/2849048098 shutdown_connections 2024-03-17T23:48:28.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.141+0000 7efd0a6d3640 1 --2- 172.21.15.150:0/2849048098 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efd0415e4c0 0x7efd0414e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:28.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.141+0000 7efd0a6d3640 1 --2- 172.21.15.150:0/2849048098 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7efcf006bc00 0x7efcf006e0c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:28.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.141+0000 7efd0a6d3640 1 --2- 172.21.15.150:0/2849048098 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd0415bc50 0x7efd0415df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:28.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.141+0000 7efd0a6d3640 1 --2- 172.21.15.150:0/2849048098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd040ad890 0x7efd0413fb50 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:28.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.141+0000 7efd0a6d3640 1 -- 172.21.15.150:0/2849048098 >> 172.21.15.150:0/2849048098 conn(0x7efd040b3da0 msgr2=0x7efd04159710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:28.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.142+0000 7efd0a6d3640 1 -- 172.21.15.150:0/2849048098 shutdown_connections 2024-03-17T23:48:28.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.142+0000 7efd0a6d3640 1 -- 172.21.15.150:0/2849048098 wait complete. 2024-03-17T23:48:28.153 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:48:08.957879+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:48:09.698808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-17T23:48:22.562269+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-17T23:48:28.154 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:48:28.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.258+0000 7fa7efd10640 1 Processor -- start 2024-03-17T23:48:28.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.258+0000 7fa7efd10640 1 -- start start 2024-03-17T23:48:28.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.259+0000 7fa7efd10640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa7e80d0880 0x7fa7e80cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:28.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.259+0000 7fa7efd10640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa7e80cc770 0x7fa7e81603c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:28.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.259+0000 7fa7efd10640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7e80ccc50 con 0x7fa7e815f3d0 2024-03-17T23:48:28.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.259+0000 7fa7efd10640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fa7e815f7b0 con 0x7fa7e80cc770 2024-03-17T23:48:28.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.259+0000 7fa7efd10640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa7e8160900 con 0x7fa7e80d0880 2024-03-17T23:48:28.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.259+0000 7fa7ee2ad640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa7e80cc770 0x7fa7e81603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:28.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.259+0000 7fa7ee2ad640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa7e80cc770 0x7fa7e81603c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47450/0 (socket says 172.21.15.150:47450) 2024-03-17T23:48:28.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.259+0000 7fa7ee2ad640 1 -- 172.21.15.150:0/2115333249 learned_addr learned my addr 172.21.15.150:0/2115333249 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:28.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.259+0000 7fa7ed2ab640 1 --2- 172.21.15.150:0/2115333249 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa7e80d0880 0x7fa7e80cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:28.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.260+0000 7fa7ee2ad640 1 -- 172.21.15.150:0/2115333249 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa7e80d0880 msgr2=0x7fa7e80cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:28.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.260+0000 7fa7ee2ad640 1 --2- 172.21.15.150:0/2115333249 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa7e80d0880 0x7fa7e80cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:28.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.260+0000 7fa7ee2ad640 1 -- 172.21.15.150:0/2115333249 >> v1:172.21.15.157:6789/0 conn(0x7fa7e815f3d0 legacy=0x7fa7e80d0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:28.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.260+0000 7fa7ee2ad640 1 -- 172.21.15.150:0/2115333249 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e80ccf30 con 0x7fa7e80cc770 2024-03-17T23:48:28.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.260+0000 7fa7ee2ad640 1 --2- 172.21.15.150:0/2115333249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7e80cc770 0x7fa7e81603c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0009e50 tx=0x7fa7e002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f28be74db3e5507 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:28.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.260+0000 7fa7ecaaa640 1 -- 172.21.15.150:0/2115333249 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e002cb00 con 0x7fa7e80cc770 2024-03-17T23:48:28.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.261+0000 7fa7ecaaa640 1 -- 172.21.15.150:0/2115333249 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7e002cca0 con 0x7fa7e80cc770 2024-03-17T23:48:28.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.261+0000 7fa7ecaaa640 1 -- 172.21.15.150:0/2115333249 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0034610 con 0x7fa7e80cc770 2024-03-17T23:48:28.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.261+0000 7fa7efd10640 1 -- 172.21.15.150:0/2115333249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7e80cc770 msgr2=0x7fa7e81603c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:28.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.261+0000 7fa7efd10640 1 --2- 172.21.15.150:0/2115333249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7e80cc770 0x7fa7e81603c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0009e50 tx=0x7fa7e002bf60 comp rx=0 tx=0).stop 2024-03-17T23:48:28.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.261+0000 7fa7efd10640 1 -- 172.21.15.150:0/2115333249 shutdown_connections 2024-03-17T23:48:28.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.261+0000 7fa7efd10640 1 --2- 172.21.15.150:0/2115333249 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7e80cc770 0x7fa7e81603c0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:28.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.261+0000 7fa7efd10640 1 --2- 172.21.15.150:0/2115333249 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa7e80d0880 0x7fa7e80cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:28.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.261+0000 7fa7efd10640 1 -- 172.21.15.150:0/2115333249 >> 172.21.15.150:0/2115333249 conn(0x7fa7e80b36c0 msgr2=0x7fa7e80b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:28.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.261+0000 7fa7efd10640 1 -- 172.21.15.150:0/2115333249 shutdown_connections 2024-03-17T23:48:28.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.261+0000 7fa7efd10640 1 -- 172.21.15.150:0/2115333249 wait complete. 2024-03-17T23:48:28.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.263+0000 7fa7efd10640 1 Processor -- start 2024-03-17T23:48:28.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.263+0000 7fa7efd10640 1 -- start start 2024-03-17T23:48:28.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.263+0000 7fa7efd10640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa7e80cc770 0x7fa7e8155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:28.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.263+0000 7fa7efd10640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7e80d0880 0x7fa7e80c7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:28.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.263+0000 7fa7efd10640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7e815f3d0 0x7fa7e80c5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:28.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.263+0000 7fa7efd10640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e80cd7d0 con 0x7fa7e80d0880 2024-03-17T23:48:28.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.263+0000 7fa7efd10640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e80cd4f0 con 0x7fa7e815f3d0 2024-03-17T23:48:28.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.263+0000 7fa7efd10640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa7e80cfa30 con 0x7fa7e80cc770 2024-03-17T23:48:28.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.263+0000 7fa7ed2ab640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7e80d0880 0x7fa7e80c7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:28.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.263+0000 7fa7ed2ab640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7e80d0880 0x7fa7e80c7950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47458/0 (socket says 172.21.15.150:47458) 2024-03-17T23:48:28.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.263+0000 7fa7ed2ab640 1 -- 172.21.15.150:0/3922232033 learned_addr learned my addr 172.21.15.150:0/3922232033 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:28.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.263+0000 7fa7edaac640 1 --2- 172.21.15.150:0/3922232033 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa7e80cc770 0x7fa7e8155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:28.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.263+0000 7fa7ee2ad640 1 --2- 172.21.15.150:0/3922232033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7e815f3d0 0x7fa7e80c5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:28.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.264+0000 7fa7edaac640 1 -- 172.21.15.150:0/3922232033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7e815f3d0 msgr2=0x7fa7e80c5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:28.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.264+0000 7fa7edaac640 1 --2- 172.21.15.150:0/3922232033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7e815f3d0 0x7fa7e80c5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:28.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.264+0000 7fa7edaac640 1 -- 172.21.15.150:0/3922232033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7e80d0880 msgr2=0x7fa7e80c7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:28.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.264+0000 7fa7edaac640 1 --2- 172.21.15.150:0/3922232033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7e80d0880 0x7fa7e80c7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:28.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.264+0000 7fa7edaac640 1 -- 172.21.15.150:0/3922232033 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e80c83d0 con 0x7fa7e80cc770 2024-03-17T23:48:28.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.264+0000 7fa7ed2ab640 1 --2- 172.21.15.150:0/3922232033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7e80d0880 0x7fa7e80c7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:48:28.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.264+0000 7fa7edaac640 1 --2- 172.21.15.150:0/3922232033 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa7e80cc770 0x7fa7e8155b40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa7d800c370 tx=0x7fa7d800c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:28.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.264+0000 7fa7de7fc640 1 -- 172.21.15.150:0/3922232033 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d80052d0 con 0x7fa7e80cc770 2024-03-17T23:48:28.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.264+0000 7fa7de7fc640 1 -- 172.21.15.150:0/3922232033 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7d8005490 con 0x7fa7e80cc770 2024-03-17T23:48:28.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.264+0000 7fa7de7fc640 1 -- 172.21.15.150:0/3922232033 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8007260 con 0x7fa7e80cc770 2024-03-17T23:48:28.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.264+0000 7fa7efd10640 1 -- 172.21.15.150:0/3922232033 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e80b4a40 con 0x7fa7e80cc770 2024-03-17T23:48:28.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.264+0000 7fa7efd10640 1 -- 172.21.15.150:0/3922232033 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7e80b5000 con 0x7fa7e80cc770 2024-03-17T23:48:28.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.265+0000 7fa7efd10640 1 -- 172.21.15.150:0/3922232033 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7e80b5200 con 0x7fa7e80cc770 2024-03-17T23:48:28.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.266+0000 7fa7de7fc640 1 -- 172.21.15.150:0/3922232033 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa7d8007400 con 0x7fa7e80cc770 2024-03-17T23:48:28.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.266+0000 7fa7de7fc640 1 --2- 172.21.15.150:0/3922232033 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa7cc06bc00 0x7fa7cc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:28.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.266+0000 7fa7ed2ab640 1 --2- 172.21.15.150:0/3922232033 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa7cc06bc00 0x7fa7cc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:28.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.266+0000 7fa7de7fc640 1 -- 172.21.15.150:0/3922232033 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(43..43 src has 1..43) ==== 6496+0+0 (secure 0 0 0) 0x7fa7d808a170 con 0x7fa7e80cc770 2024-03-17T23:48:28.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.266+0000 7fa7ed2ab640 1 --2- 172.21.15.150:0/3922232033 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa7cc06bc00 0x7fa7cc06e0c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa7e80cee70 tx=0x7fa7e4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:28.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.271+0000 7fa7de7fc640 1 -- 172.21.15.150:0/3922232033 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa7e80b5200 con 0x7fa7e80cc770 2024-03-17T23:48:28.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.562+0000 7fa7efd10640 1 -- 172.21.15.150:0/3922232033 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa7e80688b0 con 0x7fa7e80cc770 2024-03-17T23:48:28.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.563+0000 7fa7de7fc640 1 -- 172.21.15.150:0/3922232033 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v43) ==== 93+0+10 (secure 0 0 0) 0x7fa7d8058310 con 0x7fa7e80cc770 2024-03-17T23:48:28.564 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:48:28.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.565+0000 7fa7efd10640 1 -- 172.21.15.150:0/3922232033 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa7cc06bc00 msgr2=0x7fa7cc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:28.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.565+0000 7fa7efd10640 1 --2- 172.21.15.150:0/3922232033 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa7cc06bc00 0x7fa7cc06e0c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa7e80cee70 tx=0x7fa7e4009190 comp rx=0 tx=0).stop 2024-03-17T23:48:28.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.565+0000 7fa7efd10640 1 -- 172.21.15.150:0/3922232033 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa7e80cc770 msgr2=0x7fa7e8155b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:28.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.565+0000 7fa7efd10640 1 --2- 172.21.15.150:0/3922232033 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa7e80cc770 0x7fa7e8155b40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa7d800c370 tx=0x7fa7d800c830 comp rx=0 tx=0).stop 2024-03-17T23:48:28.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.565+0000 7fa7efd10640 1 -- 172.21.15.150:0/3922232033 shutdown_connections 2024-03-17T23:48:28.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.565+0000 7fa7efd10640 1 --2- 172.21.15.150:0/3922232033 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa7cc06bc00 0x7fa7cc06e0c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:28.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.565+0000 7fa7efd10640 1 --2- 172.21.15.150:0/3922232033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa7e815f3d0 0x7fa7e80c5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:28.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.565+0000 7fa7efd10640 1 --2- 172.21.15.150:0/3922232033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa7e80d0880 0x7fa7e80c7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:28.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.565+0000 7fa7efd10640 1 --2- 172.21.15.150:0/3922232033 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa7e80cc770 0x7fa7e8155b40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:28.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.565+0000 7fa7efd10640 1 -- 172.21.15.150:0/3922232033 >> 172.21.15.150:0/3922232033 conn(0x7fa7e80b36c0 msgr2=0x7fa7e8073b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:28.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.565+0000 7fa7efd10640 1 -- 172.21.15.150:0/3922232033 shutdown_connections 2024-03-17T23:48:28.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.565+0000 7fa7efd10640 1 -- 172.21.15.150:0/3922232033 wait complete. 2024-03-17T23:48:28.580 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:48:28.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.684+0000 7f1e45612640 1 Processor -- start 2024-03-17T23:48:28.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.684+0000 7f1e45612640 1 -- start start 2024-03-17T23:48:28.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.685+0000 7f1e45612640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1e40161520 0x7f1e400b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:28.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.685+0000 7f1e45612640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1e400b2bf0 0x7f1e400d05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:28.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.685+0000 7f1e45612640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1e400d0b10 0x7f1e400b57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:28.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.685+0000 7f1e45612640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f1e400b9bf0 con 0x7f1e40161520 2024-03-17T23:48:28.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.685+0000 7f1e45612640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f1e40162940 con 0x7f1e400b2bf0 2024-03-17T23:48:28.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.685+0000 7f1e45612640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1e400d0f40 con 0x7f1e400d0b10 2024-03-17T23:48:28.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.685+0000 7f1e3effd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1e40161520 0x7f1e400b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:28.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.685+0000 7f1e3effd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1e40161520 0x7f1e400b96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47460/0 (socket says 172.21.15.150:47460) 2024-03-17T23:48:28.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.685+0000 7f1e3effd640 1 -- 172.21.15.150:0/1467372198 learned_addr learned my addr 172.21.15.150:0/1467372198 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:28.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.685+0000 7f1e3f7fe640 1 --2- 172.21.15.150:0/1467372198 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1e400d0b10 0x7f1e400b57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:28.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.685+0000 7f1e3e7fc640 1 --2- 172.21.15.150:0/1467372198 >> v2:172.21.15.157:3300/0 conn(0x7f1e400b2bf0 0x7f1e400d05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:28.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.685+0000 7f1e3effd640 1 -- 172.21.15.150:0/1467372198 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1e400d0b10 msgr2=0x7f1e400b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:28.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.685+0000 7f1e3effd640 1 --2- 172.21.15.150:0/1467372198 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1e400d0b10 0x7f1e400b57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:28.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.685+0000 7f1e3effd640 1 -- 172.21.15.150:0/1467372198 >> v2:172.21.15.157:3300/0 conn(0x7f1e400b2bf0 msgr2=0x7f1e400d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:28.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.685+0000 7f1e3effd640 1 --2- 172.21.15.150:0/1467372198 >> v2:172.21.15.157:3300/0 conn(0x7f1e400b2bf0 0x7f1e400d05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:28.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.685+0000 7f1e3effd640 1 -- 172.21.15.150:0/1467372198 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e40161d30 con 0x7f1e40161520 2024-03-17T23:48:28.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.687+0000 7f1e3effd640 1 --2- 172.21.15.150:0/1467372198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e40161520 0x7f1e400b96b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f1e400ce7a0 tx=0x7f1e3402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33f2e8f8516f39eb server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:28.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.687+0000 7f1e3dffb640 1 -- 172.21.15.150:0/1467372198 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e34038070 con 0x7f1e40161520 2024-03-17T23:48:28.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.687+0000 7f1e3dffb640 1 -- 172.21.15.150:0/1467372198 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e3402f050 con 0x7f1e40161520 2024-03-17T23:48:28.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.687+0000 7f1e3dffb640 1 -- 172.21.15.150:0/1467372198 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e34031970 con 0x7f1e40161520 2024-03-17T23:48:28.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.687+0000 7f1e45612640 1 -- 172.21.15.150:0/1467372198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e40161520 msgr2=0x7f1e400b96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:28.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.687+0000 7f1e45612640 1 --2- 172.21.15.150:0/1467372198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e40161520 0x7f1e400b96b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f1e400ce7a0 tx=0x7f1e3402b7f0 comp rx=0 tx=0).stop 2024-03-17T23:48:28.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.687+0000 7f1e45612640 1 -- 172.21.15.150:0/1467372198 shutdown_connections 2024-03-17T23:48:28.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.687+0000 7f1e45612640 1 --2- 172.21.15.150:0/1467372198 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1e400d0b10 0x7f1e400b57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:28.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.687+0000 7f1e45612640 1 --2- 172.21.15.150:0/1467372198 >> v2:172.21.15.157:3300/0 conn(0x7f1e400b2bf0 0x7f1e400d05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:28.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.687+0000 7f1e45612640 1 --2- 172.21.15.150:0/1467372198 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e40161520 0x7f1e400b96b0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:28.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.687+0000 7f1e45612640 1 -- 172.21.15.150:0/1467372198 >> 172.21.15.150:0/1467372198 conn(0x7f1e4006ca00 msgr2=0x7f1e4006ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:28.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.687+0000 7f1e45612640 1 -- 172.21.15.150:0/1467372198 shutdown_connections 2024-03-17T23:48:28.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.687+0000 7f1e45612640 1 -- 172.21.15.150:0/1467372198 wait complete. 2024-03-17T23:48:28.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.689+0000 7f1e45612640 1 Processor -- start 2024-03-17T23:48:28.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.689+0000 7f1e45612640 1 -- start start 2024-03-17T23:48:28.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.689+0000 7f1e45612640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1e400b2bf0 0x7f1e40151120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:28.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.689+0000 7f1e45612640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e400d0b10 0x7f1e401f3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:28.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.689+0000 7f1e45612640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e40161520 0x7f1e4015fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:28.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.689+0000 7f1e45612640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1e401625d0 con 0x7f1e400d0b10 2024-03-17T23:48:28.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.689+0000 7f1e45612640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1e401622f0 con 0x7f1e40161520 2024-03-17T23:48:28.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.689+0000 7f1e45612640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1e40161900 con 0x7f1e400b2bf0 2024-03-17T23:48:28.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.690+0000 7f1e3e7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e400d0b10 0x7f1e401f3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:28.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.690+0000 7f1e3e7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e400d0b10 0x7f1e401f3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47464/0 (socket says 172.21.15.150:47464) 2024-03-17T23:48:28.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.690+0000 7f1e3e7fc640 1 -- 172.21.15.150:0/2120165851 learned_addr learned my addr 172.21.15.150:0/2120165851 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:28.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.690+0000 7f1e3effd640 1 --2- 172.21.15.150:0/2120165851 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1e400b2bf0 0x7f1e40151120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:28.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.690+0000 7f1e3f7fe640 1 --2- 172.21.15.150:0/2120165851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e40161520 0x7f1e4015fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:28.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.690+0000 7f1e3e7fc640 1 -- 172.21.15.150:0/2120165851 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1e400b2bf0 msgr2=0x7f1e40151120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:28.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.690+0000 7f1e3e7fc640 1 --2- 172.21.15.150:0/2120165851 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1e400b2bf0 0x7f1e40151120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:28.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.690+0000 7f1e3e7fc640 1 -- 172.21.15.150:0/2120165851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e40161520 msgr2=0x7f1e4015fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:28.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.690+0000 7f1e3e7fc640 1 --2- 172.21.15.150:0/2120165851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e40161520 0x7f1e4015fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:28.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.690+0000 7f1e3e7fc640 1 -- 172.21.15.150:0/2120165851 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e40151660 con 0x7f1e400d0b10 2024-03-17T23:48:28.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.690+0000 7f1e3effd640 1 --2- 172.21.15.150:0/2120165851 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1e400b2bf0 0x7f1e40151120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:48:28.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.690+0000 7f1e3e7fc640 1 --2- 172.21.15.150:0/2120165851 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e400d0b10 0x7f1e401f3a80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f1e3000e870 tx=0x7f1e3000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:28.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.690+0000 7f1e277fe640 1 -- 172.21.15.150:0/2120165851 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e30017070 con 0x7f1e400d0b10 2024-03-17T23:48:28.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.690+0000 7f1e277fe640 1 -- 172.21.15.150:0/2120165851 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e3000ee60 con 0x7f1e400d0b10 2024-03-17T23:48:28.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.690+0000 7f1e45612640 1 -- 172.21.15.150:0/2120165851 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e401f4540 con 0x7f1e400d0b10 2024-03-17T23:48:28.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.690+0000 7f1e277fe640 1 -- 172.21.15.150:0/2120165851 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3000cb80 con 0x7f1e400d0b10 2024-03-17T23:48:28.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.690+0000 7f1e45612640 1 -- 172.21.15.150:0/2120165851 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e401f4dc0 con 0x7f1e400d0b10 2024-03-17T23:48:28.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.692+0000 7f1e45612640 1 -- 172.21.15.150:0/2120165851 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e400681a0 con 0x7f1e400d0b10 2024-03-17T23:48:28.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.692+0000 7f1e277fe640 1 -- 172.21.15.150:0/2120165851 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1e3001a070 con 0x7f1e400d0b10 2024-03-17T23:48:28.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.692+0000 7f1e277fe640 1 --2- 172.21.15.150:0/2120165851 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1e2c06bc00 0x7f1e2c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:28.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.692+0000 7f1e277fe640 1 -- 172.21.15.150:0/2120165851 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6496+0+0 (secure 0 0 0) 0x7f1e3008ab50 con 0x7f1e400d0b10 2024-03-17T23:48:28.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.692+0000 7f1e3effd640 1 --2- 172.21.15.150:0/2120165851 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1e2c06bc00 0x7f1e2c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:28.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.693+0000 7f1e3effd640 1 --2- 172.21.15.150:0/2120165851 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1e2c06bc00 0x7f1e2c06e0c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1e34035000 tx=0x7f1e34008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:28.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.697+0000 7f1e277fe640 1 -- 172.21.15.150:0/2120165851 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1e30058ce0 con 0x7f1e400d0b10 2024-03-17T23:48:28.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.876+0000 7f1e45612640 1 -- 172.21.15.150:0/2120165851 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1e400d19c0 con 0x7f1e400d0b10 2024-03-17T23:48:28.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.876+0000 7f1e277fe640 1 -- 172.21.15.150:0/2120165851 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v43) ==== 104+0+11 (secure 0 0 0) 0x7f1e3005cd80 con 0x7f1e400d0b10 2024-03-17T23:48:28.877 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:48:28.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.878+0000 7f1e45612640 1 -- 172.21.15.150:0/2120165851 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1e2c06bc00 msgr2=0x7f1e2c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:28.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.878+0000 7f1e45612640 1 --2- 172.21.15.150:0/2120165851 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1e2c06bc00 0x7f1e2c06e0c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1e34035000 tx=0x7f1e34008030 comp rx=0 tx=0).stop 2024-03-17T23:48:28.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.878+0000 7f1e45612640 1 -- 172.21.15.150:0/2120165851 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e400d0b10 msgr2=0x7f1e401f3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:28.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.878+0000 7f1e45612640 1 --2- 172.21.15.150:0/2120165851 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e400d0b10 0x7f1e401f3a80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f1e3000e870 tx=0x7f1e3000ed30 comp rx=0 tx=0).stop 2024-03-17T23:48:28.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.878+0000 7f1e45612640 1 -- 172.21.15.150:0/2120165851 shutdown_connections 2024-03-17T23:48:28.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.878+0000 7f1e45612640 1 --2- 172.21.15.150:0/2120165851 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1e2c06bc00 0x7f1e2c06e0c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:28.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.878+0000 7f1e45612640 1 --2- 172.21.15.150:0/2120165851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1e40161520 0x7f1e4015fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:28.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.878+0000 7f1e45612640 1 --2- 172.21.15.150:0/2120165851 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e400d0b10 0x7f1e401f3a80 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:28.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.878+0000 7f1e45612640 1 --2- 172.21.15.150:0/2120165851 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1e400b2bf0 0x7f1e40151120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:28.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.878+0000 7f1e45612640 1 -- 172.21.15.150:0/2120165851 >> 172.21.15.150:0/2120165851 conn(0x7f1e4006ca00 msgr2=0x7f1e400b9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:28.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.878+0000 7f1e45612640 1 -- 172.21.15.150:0/2120165851 shutdown_connections 2024-03-17T23:48:28.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.878+0000 7f1e45612640 1 -- 172.21.15.150:0/2120165851 wait complete. 2024-03-17T23:48:28.890 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:48:28.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.994+0000 7f6a0a387640 1 Processor -- start 2024-03-17T23:48:28.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.994+0000 7f6a0a387640 1 -- start start 2024-03-17T23:48:28.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.995+0000 7f6a0a387640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6a04161520 0x7f6a040b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:28.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.995+0000 7f6a0a387640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a040d0ce0 0x7f6a040d1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:28.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.995+0000 7f6a0a387640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a04161d30 con 0x7f6a040b2bf0 2024-03-17T23:48:28.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.995+0000 7f6a0a387640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f6a040b9bf0 con 0x7f6a04161520 2024-03-17T23:48:28.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.995+0000 7f6a0a387640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6a04162940 con 0x7f6a040d0ce0 2024-03-17T23:48:28.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.995+0000 7f6a037fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f6a040b2bf0 0x7f6a040d05d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:57510/0 (socket says 172.21.15.150:57510) 2024-03-17T23:48:28.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.995+0000 7f6a037fe640 1 -- 172.21.15.150:0/2819642518 learned_addr learned my addr 172.21.15.150:0/2819642518 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:28.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.995+0000 7f6a08924640 1 --2- 172.21.15.150:0/2819642518 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a040d0ce0 0x7f6a040d1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:28.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.995+0000 7f6a03fff640 1 --2- 172.21.15.150:0/2819642518 >> v2:172.21.15.157:3300/0 conn(0x7f6a04161520 0x7f6a040b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:28.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.996+0000 7f6a02ffd640 1 -- 172.21.15.150:0/2819642518 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4026067393 0 0) 0x7f6a04161d30 con 0x7f6a040b2bf0 2024-03-17T23:48:28.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.996+0000 7f6a02ffd640 1 -- 172.21.15.150:0/2819642518 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69ec003520 con 0x7f6a040b2bf0 2024-03-17T23:48:28.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.996+0000 7f6a02ffd640 1 -- 172.21.15.150:0/2819642518 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3221749453 0 0) 0x7f69ec003520 con 0x7f6a040b2bf0 2024-03-17T23:48:28.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.996+0000 7f6a02ffd640 1 -- 172.21.15.150:0/2819642518 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a040d0ce0 msgr2=0x7f6a040d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:29.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.996+0000 7f6a02ffd640 1 --2- 172.21.15.150:0/2819642518 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a040d0ce0 0x7f6a040d1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.996+0000 7f6a02ffd640 1 -- 172.21.15.150:0/2819642518 >> v2:172.21.15.157:3300/0 conn(0x7f6a04161520 msgr2=0x7f6a040b96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:29.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.996+0000 7f6a02ffd640 1 --2- 172.21.15.150:0/2819642518 >> v2:172.21.15.157:3300/0 conn(0x7f6a04161520 0x7f6a040b96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.996+0000 7f6a02ffd640 1 -- 172.21.15.150:0/2819642518 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a04162010 con 0x7f6a040b2bf0 2024-03-17T23:48:29.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.996+0000 7f6a02ffd640 1 -- 172.21.15.150:0/2819642518 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f69f4001670 con 0x7f6a040b2bf0 2024-03-17T23:48:29.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.996+0000 7f6a08924640 1 --2- 172.21.15.150:0/2819642518 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a040d0ce0 0x7f6a040d1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:48:29.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.996+0000 7f6a02ffd640 1 -- 172.21.15.150:0/2819642518 >> v1:172.21.15.150:6789/0 conn(0x7f6a040b2bf0 legacy=0x7f6a040d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:29.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.996+0000 7f6a02ffd640 1 --2- 172.21.15.150:0/2819642518 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a040d0ce0 0x7f69ec0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:29.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.996+0000 7f6a02ffd640 1 --2- 172.21.15.150:0/2819642518 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69ec003ef0 0x7f69ec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:29.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.996+0000 7f6a02ffd640 1 --2- 172.21.15.150:0/2819642518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69ec004d40 0x7f69ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:29.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.996+0000 7f6a02ffd640 1 -- 172.21.15.150:0/2819642518 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6a040b9bf0 con 0x7f69ec004d40 2024-03-17T23:48:29.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.996+0000 7f6a02ffd640 1 -- 172.21.15.150:0/2819642518 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6a04162940 con 0x7f69ec003ef0 2024-03-17T23:48:29.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.996+0000 7f6a02ffd640 1 -- 172.21.15.150:0/2819642518 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f69ec00ba70 con 0x7f6a040d0ce0 2024-03-17T23:48:29.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.996+0000 7f6a03fff640 1 --2- 172.21.15.150:0/2819642518 >> v2:172.21.15.157:3300/0 conn(0x7f6a04161520 0x7f6a040b96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:48:29.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.996+0000 7f6a08924640 1 --2- 172.21.15.150:0/2819642518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69ec004d40 0x7f69ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:29.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.996+0000 7f6a037fe640 1 --2- 172.21.15.150:0/2819642518 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69ec003ef0 0x7f69ec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:29.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.996+0000 7f6a03fff640 1 --2- 172.21.15.150:0/2819642518 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a040d0ce0 0x7f69ec0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:29.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.996+0000 7f6a08924640 1 -- 172.21.15.150:0/2819642518 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a040d0ce0 msgr2=0x7f69ec0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:29.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.996+0000 7f6a08924640 1 --2- 172.21.15.150:0/2819642518 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a040d0ce0 0x7f69ec0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.996+0000 7f6a08924640 1 -- 172.21.15.150:0/2819642518 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69ec003ef0 msgr2=0x7f69ec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:29.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.996+0000 7f6a08924640 1 --2- 172.21.15.150:0/2819642518 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f69ec003ef0 0x7f69ec004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.996+0000 7f6a08924640 1 -- 172.21.15.150:0/2819642518 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69ec003520 con 0x7f69ec004d40 2024-03-17T23:48:29.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.997+0000 7f6a08924640 1 --2- 172.21.15.150:0/2819642518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69ec004d40 0x7f69ec0051c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f69fc00e320 tx=0x7f69ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2388d20149382f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:29.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.997+0000 7f6a02ffd640 1 -- 172.21.15.150:0/2819642518 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69fc039070 con 0x7f69ec004d40 2024-03-17T23:48:29.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.997+0000 7f6a02ffd640 1 -- 172.21.15.150:0/2819642518 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69fc02cd60 con 0x7f69ec004d40 2024-03-17T23:48:29.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.997+0000 7f6a02ffd640 1 -- 172.21.15.150:0/2819642518 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69fc009de0 con 0x7f69ec004d40 2024-03-17T23:48:29.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.998+0000 7f6a0a387640 1 -- 172.21.15.150:0/2819642518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69ec004d40 msgr2=0x7f69ec0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:29.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.998+0000 7f6a0a387640 1 --2- 172.21.15.150:0/2819642518 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69ec004d40 0x7f69ec0051c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f69fc00e320 tx=0x7f69ec005830 comp rx=0 tx=0).stop 2024-03-17T23:48:29.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.998+0000 7f6a08924640 1 -- 172.21.15.150:0/2819642518 reap_dead start 2024-03-17T23:48:29.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.998+0000 7f6a0a387640 1 -- 172.21.15.150:0/2819642518 shutdown_connections 2024-03-17T23:48:29.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.998+0000 7f6a0a387640 1 -- 172.21.15.150:0/2819642518 >> 172.21.15.150:0/2819642518 conn(0x7f6a0406ca00 msgr2=0x7f6a0406ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:29.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.998+0000 7f6a0a387640 1 -- 172.21.15.150:0/2819642518 shutdown_connections 2024-03-17T23:48:29.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:28.998+0000 7f6a0a387640 1 -- 172.21.15.150:0/2819642518 wait complete. 2024-03-17T23:48:29.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.000+0000 7f6a0a387640 1 Processor -- start 2024-03-17T23:48:29.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.000+0000 7f6a0a387640 1 -- start start 2024-03-17T23:48:29.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.001+0000 7f6a0a387640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a0414a5e0 0x7f6a0415de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:29.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.001+0000 7f6a0a387640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a0415f6d0 0x7f6a040c86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:29.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.001+0000 7f6a0a387640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a040c8bf0 0x7f6a0414ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:29.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.001+0000 7f6a0a387640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6a04161900 con 0x7f6a0415f6d0 2024-03-17T23:48:29.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.001+0000 7f6a0a387640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6a041625d0 con 0x7f6a040c8bf0 2024-03-17T23:48:29.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.001+0000 7f6a0a387640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6a040d1fa0 con 0x7f6a0414a5e0 2024-03-17T23:48:29.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.001+0000 7f6a037fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a0415f6d0 0x7f6a040c86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:29.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.001+0000 7f6a037fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a0415f6d0 0x7f6a040c86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47484/0 (socket says 172.21.15.150:47484) 2024-03-17T23:48:29.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.001+0000 7f6a037fe640 1 -- 172.21.15.150:0/3671617793 learned_addr learned my addr 172.21.15.150:0/3671617793 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:29.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.001+0000 7f6a03fff640 1 --2- 172.21.15.150:0/3671617793 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a0414a5e0 0x7f6a0415de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:29.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.001+0000 7f6a08924640 1 --2- 172.21.15.150:0/3671617793 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a040c8bf0 0x7f6a0414ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:29.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.001+0000 7f6a037fe640 1 -- 172.21.15.150:0/3671617793 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a0414a5e0 msgr2=0x7f6a0415de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:29.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.001+0000 7f6a037fe640 1 --2- 172.21.15.150:0/3671617793 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a0414a5e0 0x7f6a0415de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.001+0000 7f6a037fe640 1 -- 172.21.15.150:0/3671617793 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a040c8bf0 msgr2=0x7f6a0414ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:29.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.001+0000 7f6a037fe640 1 --2- 172.21.15.150:0/3671617793 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a040c8bf0 0x7f6a0414ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.001+0000 7f6a037fe640 1 -- 172.21.15.150:0/3671617793 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a0415feb0 con 0x7f6a0415f6d0 2024-03-17T23:48:29.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.002+0000 7f6a037fe640 1 --2- 172.21.15.150:0/3671617793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a0415f6d0 0x7f6a040c86b0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f69f4001d00 tx=0x7f69f4007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:29.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.002+0000 7f6a00ff9640 1 -- 172.21.15.150:0/3671617793 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f400f9c0 con 0x7f6a0415f6d0 2024-03-17T23:48:29.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.002+0000 7f6a00ff9640 1 -- 172.21.15.150:0/3671617793 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69f4018070 con 0x7f6a0415f6d0 2024-03-17T23:48:29.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.002+0000 7f6a0a387640 1 -- 172.21.15.150:0/3671617793 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a04160190 con 0x7f6a0415f6d0 2024-03-17T23:48:29.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.002+0000 7f6a00ff9640 1 -- 172.21.15.150:0/3671617793 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f400fde0 con 0x7f6a0415f6d0 2024-03-17T23:48:29.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.002+0000 7f6a0a387640 1 -- 172.21.15.150:0/3671617793 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a0414b090 con 0x7f6a0415f6d0 2024-03-17T23:48:29.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.003+0000 7f6a0a387640 1 -- 172.21.15.150:0/3671617793 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a0414b290 con 0x7f6a0415f6d0 2024-03-17T23:48:29.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.003+0000 7f6a00ff9640 1 -- 172.21.15.150:0/3671617793 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f69f401d460 con 0x7f6a0415f6d0 2024-03-17T23:48:29.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.004+0000 7f6a00ff9640 1 --2- 172.21.15.150:0/3671617793 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f69f006bc00 0x7f69f006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:29.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.004+0000 7f6a00ff9640 1 -- 172.21.15.150:0/3671617793 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6496+0+0 (secure 0 0 0) 0x7f69f408ade0 con 0x7f6a0415f6d0 2024-03-17T23:48:29.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.004+0000 7f6a03fff640 1 --2- 172.21.15.150:0/3671617793 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f69f006bc00 0x7f69f006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:29.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.004+0000 7f6a03fff640 1 --2- 172.21.15.150:0/3671617793 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f69f006bc00 0x7f69f006e0c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f69f8009dc0 tx=0x7f69f8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:29.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.008+0000 7f6a00ff9640 1 -- 172.21.15.150:0/3671617793 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6a0414b290 con 0x7f6a0415f6d0 2024-03-17T23:48:29.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.186+0000 7f6a0a387640 1 -- 172.21.15.150:0/3671617793 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6a040681a0 con 0x7f6a0415f6d0 2024-03-17T23:48:29.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.187+0000 7f6a00ff9640 1 -- 172.21.15.150:0/3671617793 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v43) ==== 100+0+11 (secure 0 0 0) 0x7f69f4058ef0 con 0x7f6a0415f6d0 2024-03-17T23:48:29.187 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:48:29.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.189+0000 7f6a0a387640 1 -- 172.21.15.150:0/3671617793 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f69f006bc00 msgr2=0x7f69f006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:29.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.189+0000 7f6a0a387640 1 --2- 172.21.15.150:0/3671617793 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f69f006bc00 0x7f69f006e0c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f69f8009dc0 tx=0x7f69f8009190 comp rx=0 tx=0).stop 2024-03-17T23:48:29.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.189+0000 7f6a0a387640 1 -- 172.21.15.150:0/3671617793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a0415f6d0 msgr2=0x7f6a040c86b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:29.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.189+0000 7f6a0a387640 1 --2- 172.21.15.150:0/3671617793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a0415f6d0 0x7f6a040c86b0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f69f4001d00 tx=0x7f69f4007720 comp rx=0 tx=0).stop 2024-03-17T23:48:29.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.189+0000 7f6a0a387640 1 -- 172.21.15.150:0/3671617793 shutdown_connections 2024-03-17T23:48:29.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.189+0000 7f6a0a387640 1 --2- 172.21.15.150:0/3671617793 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f69f006bc00 0x7f69f006e0c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.189+0000 7f6a0a387640 1 --2- 172.21.15.150:0/3671617793 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a040c8bf0 0x7f6a0414ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.189+0000 7f6a0a387640 1 --2- 172.21.15.150:0/3671617793 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6a0415f6d0 0x7f6a040c86b0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.189+0000 7f6a0a387640 1 --2- 172.21.15.150:0/3671617793 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6a0414a5e0 0x7f6a0415de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.189+0000 7f6a0a387640 1 -- 172.21.15.150:0/3671617793 >> 172.21.15.150:0/3671617793 conn(0x7f6a0406ca00 msgr2=0x7f6a040c79f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:29.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.189+0000 7f6a0a387640 1 -- 172.21.15.150:0/3671617793 shutdown_connections 2024-03-17T23:48:29.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.189+0000 7f6a0a387640 1 -- 172.21.15.150:0/3671617793 wait complete. 2024-03-17T23:48:29.200 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-17T23:48:29.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.304+0000 7f2d7e109640 1 Processor -- start 2024-03-17T23:48:29.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.305+0000 7f2d7e109640 1 -- start start 2024-03-17T23:48:29.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.306+0000 7f2d7e109640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2d78161520 0x7f2d780b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:29.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.306+0000 7f2d7e109640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d78161d30 con 0x7f2d780d0ce0 2024-03-17T23:48:29.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.306+0000 7f2d7e109640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d78162010 con 0x7f2d780b2bf0 2024-03-17T23:48:29.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.306+0000 7f2d7e109640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f2d780b9bf0 con 0x7f2d78161520 2024-03-17T23:48:29.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.306+0000 7f2d77fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f2d780d0ce0 0x7f2d780d1fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:57512/0 (socket says 172.21.15.150:57512) 2024-03-17T23:48:29.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.306+0000 7f2d777fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2d78161520 0x7f2d780b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:29.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.306+0000 7f2d77fff640 1 -- 172.21.15.150:0/1454448346 learned_addr learned my addr 172.21.15.150:0/1454448346 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:29.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.306+0000 7f2d767fc640 1 -- 172.21.15.150:0/1454448346 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3432375819 0 0) 0x7f2d78161d30 con 0x7f2d780d0ce0 2024-03-17T23:48:29.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.306+0000 7f2d767fc640 1 -- 172.21.15.150:0/1454448346 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d60003520 con 0x7f2d780d0ce0 2024-03-17T23:48:29.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.306+0000 7f2d777fe640 1 -- 172.21.15.150:0/1454448346 >> v1:172.21.15.157:6789/0 conn(0x7f2d780b2bf0 legacy=0x7f2d780d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:29.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.306+0000 7f2d777fe640 1 -- 172.21.15.150:0/1454448346 >> v1:172.21.15.150:6789/0 conn(0x7f2d780d0ce0 legacy=0x7f2d780d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:29.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.306+0000 7f2d767fc640 1 -- 172.21.15.150:0/1454448346 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2900166594 0 0) 0x7f2d78162010 con 0x7f2d780b2bf0 2024-03-17T23:48:29.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.306+0000 7f2d777fe640 1 -- 172.21.15.150:0/1454448346 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d781622f0 con 0x7f2d78161520 2024-03-17T23:48:29.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.307+0000 7f2d777fe640 1 --2- 172.21.15.150:0/1454448346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d78161520 0x7f2d780b96b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f2d680097a0 tx=0x7f2d6802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15a3f6202857dd19 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.307+0000 7f2d767fc640 1 -- 172.21.15.150:0/1454448346 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d680315a0 con 0x7f2d78161520 2024-03-17T23:48:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.307+0000 7f2d767fc640 1 -- 172.21.15.150:0/1454448346 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d68037070 con 0x7f2d78161520 2024-03-17T23:48:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.307+0000 7f2d767fc640 1 -- 172.21.15.150:0/1454448346 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d68031960 con 0x7f2d78161520 2024-03-17T23:48:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.308+0000 7f2d7e109640 1 -- 172.21.15.150:0/1454448346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d78161520 msgr2=0x7f2d780b96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.308+0000 7f2d7e109640 1 --2- 172.21.15.150:0/1454448346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d78161520 0x7f2d780b96b0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f2d680097a0 tx=0x7f2d6802b840 comp rx=0 tx=0).stop 2024-03-17T23:48:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.308+0000 7f2d7e109640 1 -- 172.21.15.150:0/1454448346 shutdown_connections 2024-03-17T23:48:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.308+0000 7f2d7e109640 1 --2- 172.21.15.150:0/1454448346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d78161520 0x7f2d780b96b0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.308+0000 7f2d7e109640 1 -- 172.21.15.150:0/1454448346 >> 172.21.15.150:0/1454448346 conn(0x7f2d7806ca00 msgr2=0x7f2d7806ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.308+0000 7f2d7e109640 1 -- 172.21.15.150:0/1454448346 shutdown_connections 2024-03-17T23:48:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.308+0000 7f2d7e109640 1 -- 172.21.15.150:0/1454448346 wait complete. 2024-03-17T23:48:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.309+0000 7f2d7e109640 1 Processor -- start 2024-03-17T23:48:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.309+0000 7f2d7e109640 1 -- start start 2024-03-17T23:48:29.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.310+0000 7f2d7e109640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d780b2bf0 0x7f2d781f4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:29.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.310+0000 7f2d7e109640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d780d0ce0 0x7f2d781f6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:29.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.310+0000 7f2d7e109640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d78161520 0x7f2d780b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:29.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.310+0000 7f2d7e109640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2d78161900 con 0x7f2d780b2bf0 2024-03-17T23:48:29.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.310+0000 7f2d7e109640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2d78162940 con 0x7f2d78161520 2024-03-17T23:48:29.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.310+0000 7f2d7e109640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2d780d26b0 con 0x7f2d780d0ce0 2024-03-17T23:48:29.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.310+0000 7f2d777fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d780b2bf0 0x7f2d781f4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:29.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.310+0000 7f2d777fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d780b2bf0 0x7f2d781f4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47500/0 (socket says 172.21.15.150:47500) 2024-03-17T23:48:29.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.310+0000 7f2d777fe640 1 -- 172.21.15.150:0/1995971100 learned_addr learned my addr 172.21.15.150:0/1995971100 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:29.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.310+0000 7f2d77fff640 1 --2- 172.21.15.150:0/1995971100 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d78161520 0x7f2d780b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:29.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.310+0000 7f2d76ffd640 1 --2- 172.21.15.150:0/1995971100 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d780d0ce0 0x7f2d781f6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:29.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.310+0000 7f2d777fe640 1 -- 172.21.15.150:0/1995971100 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d780d0ce0 msgr2=0x7f2d781f6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:29.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.310+0000 7f2d777fe640 1 --2- 172.21.15.150:0/1995971100 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d780d0ce0 0x7f2d781f6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.310+0000 7f2d777fe640 1 -- 172.21.15.150:0/1995971100 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d78161520 msgr2=0x7f2d780b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:29.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.310+0000 7f2d777fe640 1 --2- 172.21.15.150:0/1995971100 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d78161520 0x7f2d780b46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.310+0000 7f2d777fe640 1 -- 172.21.15.150:0/1995971100 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d780b5040 con 0x7f2d780b2bf0 2024-03-17T23:48:29.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.311+0000 7f2d777fe640 1 --2- 172.21.15.150:0/1995971100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d780b2bf0 0x7f2d781f4030 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f2d680098c0 tx=0x7f2d680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:29.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.311+0000 7f2d5ffff640 1 -- 172.21.15.150:0/1995971100 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d6804a070 con 0x7f2d780b2bf0 2024-03-17T23:48:29.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.311+0000 7f2d5ffff640 1 -- 172.21.15.150:0/1995971100 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d68032430 con 0x7f2d780b2bf0 2024-03-17T23:48:29.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.311+0000 7f2d7e109640 1 -- 172.21.15.150:0/1995971100 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d780b5320 con 0x7f2d780b2bf0 2024-03-17T23:48:29.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.311+0000 7f2d5ffff640 1 -- 172.21.15.150:0/1995971100 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d68032800 con 0x7f2d780b2bf0 2024-03-17T23:48:29.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.311+0000 7f2d7e109640 1 -- 172.21.15.150:0/1995971100 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d780cadf0 con 0x7f2d780b2bf0 2024-03-17T23:48:29.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.312+0000 7f2d7e109640 1 -- 172.21.15.150:0/1995971100 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d780b7cd0 con 0x7f2d780b2bf0 2024-03-17T23:48:29.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.312+0000 7f2d5ffff640 1 -- 172.21.15.150:0/1995971100 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2d680329a0 con 0x7f2d780b2bf0 2024-03-17T23:48:29.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.312+0000 7f2d5ffff640 1 --2- 172.21.15.150:0/1995971100 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2d6406bbd0 0x7f2d6406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:29.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.313+0000 7f2d76ffd640 1 --2- 172.21.15.150:0/1995971100 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2d6406bbd0 0x7f2d6406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:29.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.313+0000 7f2d5ffff640 1 -- 172.21.15.150:0/1995971100 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6496+0+0 (secure 0 0 0) 0x7f2d6802f020 con 0x7f2d780b2bf0 2024-03-17T23:48:29.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.313+0000 7f2d76ffd640 1 --2- 172.21.15.150:0/1995971100 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2d6406bbd0 0x7f2d6406e090 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2d780b7550 tx=0x7f2d6c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:29.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.321+0000 7f2d5ffff640 1 -- 172.21.15.150:0/1995971100 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2d780b7cd0 con 0x7f2d780b2bf0 2024-03-17T23:48:29.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.504+0000 7f2d7e109640 1 -- 172.21.15.150:0/1995971100 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7f2d780681a0 con 0x7f2d780b2bf0 2024-03-17T23:48:29.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.504+0000 7f2d5ffff640 1 -- 172.21.15.150:0/1995971100 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v43) ==== 97+0+11 (secure 0 0 0) 0x7f2d68040070 con 0x7f2d780b2bf0 2024-03-17T23:48:29.505 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:48:29.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.507+0000 7f2d7e109640 1 -- 172.21.15.150:0/1995971100 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2d6406bbd0 msgr2=0x7f2d6406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:29.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.507+0000 7f2d7e109640 1 --2- 172.21.15.150:0/1995971100 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2d6406bbd0 0x7f2d6406e090 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2d780b7550 tx=0x7f2d6c009040 comp rx=0 tx=0).stop 2024-03-17T23:48:29.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.507+0000 7f2d7e109640 1 -- 172.21.15.150:0/1995971100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d780b2bf0 msgr2=0x7f2d781f4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:29.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.507+0000 7f2d7e109640 1 --2- 172.21.15.150:0/1995971100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d780b2bf0 0x7f2d781f4030 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f2d680098c0 tx=0x7f2d680046c0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.507+0000 7f2d7e109640 1 -- 172.21.15.150:0/1995971100 shutdown_connections 2024-03-17T23:48:29.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.507+0000 7f2d7e109640 1 --2- 172.21.15.150:0/1995971100 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2d6406bbd0 0x7f2d6406e090 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.507+0000 7f2d7e109640 1 --2- 172.21.15.150:0/1995971100 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2d78161520 0x7f2d780b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.507+0000 7f2d7e109640 1 --2- 172.21.15.150:0/1995971100 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2d780d0ce0 0x7f2d781f6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.507+0000 7f2d7e109640 1 --2- 172.21.15.150:0/1995971100 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d780b2bf0 0x7f2d781f4030 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.507+0000 7f2d7e109640 1 -- 172.21.15.150:0/1995971100 >> 172.21.15.150:0/1995971100 conn(0x7f2d7806ca00 msgr2=0x7f2d780c80b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:29.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.507+0000 7f2d7e109640 1 -- 172.21.15.150:0/1995971100 shutdown_connections 2024-03-17T23:48:29.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.507+0000 7f2d7e109640 1 -- 172.21.15.150:0/1995971100 wait complete. 2024-03-17T23:48:29.519 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-17T23:48:29.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.620+0000 7f061c3f3640 1 Processor -- start 2024-03-17T23:48:29.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.620+0000 7f061c3f3640 1 -- start start 2024-03-17T23:48:29.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.621+0000 7f061c3f3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06140d0ac0 0x7f06140d0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:29.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.621+0000 7f061c3f3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06140cb8a0 con 0x7f061415ae40 2024-03-17T23:48:29.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.621+0000 7f061c3f3640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06140cd2d0 con 0x7f06140cb440 2024-03-17T23:48:29.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.621+0000 7f061c3f3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f06140cf9b0 con 0x7f06140d0ac0 2024-03-17T23:48:29.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.622+0000 7f061a18f640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f061415ae40 0x7f06140cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:57526/0 (socket says 172.21.15.150:57526) 2024-03-17T23:48:29.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.622+0000 7f061a18f640 1 -- 172.21.15.150:0/1285617019 learned_addr learned my addr 172.21.15.150:0/1285617019 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:29.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.622+0000 7f061a990640 1 --2- 172.21.15.150:0/1285617019 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06140d0ac0 0x7f06140d0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:29.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.622+0000 7f061918d640 1 -- 172.21.15.150:0/1285617019 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2298374167 0 0) 0x7f0610001110 con 0x7f061415ae40 2024-03-17T23:48:29.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.622+0000 7f061918d640 1 -- 172.21.15.150:0/1285617019 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0600003520 con 0x7f061415ae40 2024-03-17T23:48:29.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.622+0000 7f061918d640 1 -- 172.21.15.150:0/1285617019 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3169524834 0 0) 0x7f06140cd2d0 con 0x7f06140cb440 2024-03-17T23:48:29.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.622+0000 7f061918d640 1 -- 172.21.15.150:0/1285617019 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0610001110 con 0x7f06140cb440 2024-03-17T23:48:29.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.622+0000 7f061918d640 1 -- 172.21.15.150:0/1285617019 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3090245518 0 0) 0x7f0600003520 con 0x7f061415ae40 2024-03-17T23:48:29.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.622+0000 7f061918d640 1 -- 172.21.15.150:0/1285617019 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06140d0ac0 msgr2=0x7f06140d0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:29.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.622+0000 7f061918d640 1 --2- 172.21.15.150:0/1285617019 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06140d0ac0 0x7f06140d0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.622+0000 7f061918d640 1 -- 172.21.15.150:0/1285617019 >> v1:172.21.15.157:6789/0 conn(0x7f06140cb440 legacy=0x7f06140ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:29.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.622+0000 7f061918d640 1 -- 172.21.15.150:0/1285617019 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06140cbb20 con 0x7f061415ae40 2024-03-17T23:48:29.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.622+0000 7f061918d640 1 -- 172.21.15.150:0/1285617019 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f0610001840 con 0x7f061415ae40 2024-03-17T23:48:29.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.622+0000 7f061918d640 1 -- 172.21.15.150:0/1285617019 >> v1:172.21.15.150:6789/0 conn(0x7f061415ae40 legacy=0x7f06140cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:29.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.622+0000 7f061918d640 1 --2- 172.21.15.150:0/1285617019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0600003b70 0x7f0600003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:29.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.622+0000 7f061918d640 1 --2- 172.21.15.150:0/1285617019 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0600004800 0x7f0600004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:29.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.622+0000 7f061918d640 1 --2- 172.21.15.150:0/1285617019 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06140d0ac0 0x7f06000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:29.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.622+0000 7f061918d640 1 -- 172.21.15.150:0/1285617019 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f06140cf9b0 con 0x7f0600003b70 2024-03-17T23:48:29.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.622+0000 7f061918d640 1 -- 172.21.15.150:0/1285617019 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f060000ba70 con 0x7f0600004800 2024-03-17T23:48:29.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.622+0000 7f061918d640 1 -- 172.21.15.150:0/1285617019 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f060000bc70 con 0x7f06140d0ac0 2024-03-17T23:48:29.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.622+0000 7f061a18f640 1 --2- 172.21.15.150:0/1285617019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0600003b70 0x7f0600003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:29.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.622+0000 7f061998e640 1 --2- 172.21.15.150:0/1285617019 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0600004800 0x7f0600004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:29.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.622+0000 7f061a990640 1 --2- 172.21.15.150:0/1285617019 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06140d0ac0 0x7f06000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:29.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.623+0000 7f061a18f640 1 -- 172.21.15.150:0/1285617019 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06140d0ac0 msgr2=0x7f06000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:29.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.623+0000 7f061a18f640 1 --2- 172.21.15.150:0/1285617019 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06140d0ac0 0x7f06000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.623+0000 7f061a18f640 1 -- 172.21.15.150:0/1285617019 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0600004800 msgr2=0x7f0600004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:29.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.623+0000 7f061a18f640 1 --2- 172.21.15.150:0/1285617019 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0600004800 0x7f0600004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.623+0000 7f061a18f640 1 -- 172.21.15.150:0/1285617019 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0600003520 con 0x7f0600003b70 2024-03-17T23:48:29.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.623+0000 7f061a990640 1 --2- 172.21.15.150:0/1285617019 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06140d0ac0 0x7f06000051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:29.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.623+0000 7f061a18f640 1 --2- 172.21.15.150:0/1285617019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0600003b70 0x7f0600003f70 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f061000c2a0 tx=0x7f0600005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee548cc4287cc342 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:29.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.623+0000 7f061918d640 1 -- 172.21.15.150:0/1285617019 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0610007de0 con 0x7f0600003b70 2024-03-17T23:48:29.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.624+0000 7f061918d640 1 -- 172.21.15.150:0/1285617019 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0610030070 con 0x7f0600003b70 2024-03-17T23:48:29.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.624+0000 7f061918d640 1 -- 172.21.15.150:0/1285617019 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f061002cdf0 con 0x7f0600003b70 2024-03-17T23:48:29.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.624+0000 7f061c3f3640 1 -- 172.21.15.150:0/1285617019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0600003b70 msgr2=0x7f0600003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:29.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.624+0000 7f061c3f3640 1 --2- 172.21.15.150:0/1285617019 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0600003b70 0x7f0600003f70 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f061000c2a0 tx=0x7f0600005830 comp rx=0 tx=0).stop 2024-03-17T23:48:29.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.624+0000 7f061a990640 1 -- 172.21.15.150:0/1285617019 reap_dead start 2024-03-17T23:48:29.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.624+0000 7f061c3f3640 1 -- 172.21.15.150:0/1285617019 shutdown_connections 2024-03-17T23:48:29.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.624+0000 7f061c3f3640 1 -- 172.21.15.150:0/1285617019 >> 172.21.15.150:0/1285617019 conn(0x7f0614099a70 msgr2=0x7f0614099e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:29.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.624+0000 7f061c3f3640 1 -- 172.21.15.150:0/1285617019 shutdown_connections 2024-03-17T23:48:29.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.624+0000 7f061c3f3640 1 -- 172.21.15.150:0/1285617019 wait complete. 2024-03-17T23:48:29.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.625+0000 7f061c3f3640 1 Processor -- start 2024-03-17T23:48:29.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.625+0000 7f061c3f3640 1 -- start start 2024-03-17T23:48:29.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.626+0000 7f061c3f3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0600003b70 0x7f06141f3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:29.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.626+0000 7f061c3f3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f061409acf0 0x7f061406b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:29.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.626+0000 7f061c3f3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f061409d410 0x7f061409d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:29.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.626+0000 7f061c3f3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f06140cc0e0 con 0x7f061409d410 2024-03-17T23:48:29.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.626+0000 7f061c3f3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f06140cc730 con 0x7f061409acf0 2024-03-17T23:48:29.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.626+0000 7f061c3f3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f06140cc3c0 con 0x7f0600003b70 2024-03-17T23:48:29.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.626+0000 7f061a990640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f061409d410 0x7f061409d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:29.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.626+0000 7f061a990640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f061409d410 0x7f061409d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47508/0 (socket says 172.21.15.150:47508) 2024-03-17T23:48:29.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.626+0000 7f061a990640 1 -- 172.21.15.150:0/177997501 learned_addr learned my addr 172.21.15.150:0/177997501 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:29.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.626+0000 7f061998e640 1 --2- 172.21.15.150:0/177997501 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f061409acf0 0x7f061406b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:29.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.626+0000 7f061a18f640 1 --2- 172.21.15.150:0/177997501 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0600003b70 0x7f06141f3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:29.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.626+0000 7f061a990640 1 -- 172.21.15.150:0/177997501 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0600003b70 msgr2=0x7f06141f3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:29.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.626+0000 7f061a990640 1 --2- 172.21.15.150:0/177997501 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0600003b70 0x7f06141f3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.626+0000 7f061a990640 1 -- 172.21.15.150:0/177997501 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f061409acf0 msgr2=0x7f061406b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:29.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.626+0000 7f061a990640 1 --2- 172.21.15.150:0/177997501 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f061409acf0 0x7f061406b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.626+0000 7f061a990640 1 -- 172.21.15.150:0/177997501 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06141f4220 con 0x7f061409d410 2024-03-17T23:48:29.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.627+0000 7f061a990640 1 --2- 172.21.15.150:0/177997501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f061409d410 0x7f061409d890 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f06140ce650 tx=0x7f060c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:29.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.627+0000 7f060affd640 1 -- 172.21.15.150:0/177997501 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060c01a030 con 0x7f061409d410 2024-03-17T23:48:29.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.627+0000 7f060affd640 1 -- 172.21.15.150:0/177997501 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f060c010e50 con 0x7f061409d410 2024-03-17T23:48:29.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.627+0000 7f060affd640 1 -- 172.21.15.150:0/177997501 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060c003bd0 con 0x7f061409d410 2024-03-17T23:48:29.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.627+0000 7f061c3f3640 1 -- 172.21.15.150:0/177997501 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06141f4500 con 0x7f061409d410 2024-03-17T23:48:29.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.627+0000 7f061c3f3640 1 -- 172.21.15.150:0/177997501 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f061409b170 con 0x7f061409d410 2024-03-17T23:48:29.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.628+0000 7f061c3f3640 1 -- 172.21.15.150:0/177997501 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06140688b0 con 0x7f061409d410 2024-03-17T23:48:29.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.628+0000 7f060affd640 1 -- 172.21.15.150:0/177997501 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f060c01d070 con 0x7f061409d410 2024-03-17T23:48:29.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.629+0000 7f060affd640 1 --2- 172.21.15.150:0/177997501 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f05f806bc00 0x7f05f806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:29.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.629+0000 7f061a18f640 1 --2- 172.21.15.150:0/177997501 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f05f806bc00 0x7f05f806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:29.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.629+0000 7f060affd640 1 -- 172.21.15.150:0/177997501 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6496+0+0 (secure 0 0 0) 0x7f060c08b7d0 con 0x7f061409d410 2024-03-17T23:48:29.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.629+0000 7f061a18f640 1 --2- 172.21.15.150:0/177997501 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f05f806bc00 0x7f05f806e0c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0610034000 tx=0x7f0610009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:29.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.633+0000 7f060affd640 1 -- 172.21.15.150:0/177997501 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f060c0599f0 con 0x7f061409d410 2024-03-17T23:48:29.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.824+0000 7f061c3f3640 1 -- 172.21.15.150:0/177997501 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f06140c9c80 con 0x7f061409d410 2024-03-17T23:48:29.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.825+0000 7f060affd640 1 -- 172.21.15.150:0/177997501 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v43) ==== 80+0+0 (secure 0 0 0) 0x7f060c05da90 con 0x7f061409d410 2024-03-17T23:48:29.826 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-03-17T23:48:29.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.827+0000 7f061c3f3640 1 -- 172.21.15.150:0/177997501 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f05f806bc00 msgr2=0x7f05f806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:29.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.827+0000 7f061c3f3640 1 --2- 172.21.15.150:0/177997501 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f05f806bc00 0x7f05f806e0c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0610034000 tx=0x7f0610009030 comp rx=0 tx=0).stop 2024-03-17T23:48:29.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.827+0000 7f061c3f3640 1 -- 172.21.15.150:0/177997501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f061409d410 msgr2=0x7f061409d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:29.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.827+0000 7f061c3f3640 1 --2- 172.21.15.150:0/177997501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f061409d410 0x7f061409d890 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f06140ce650 tx=0x7f060c009f90 comp rx=0 tx=0).stop 2024-03-17T23:48:29.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.827+0000 7f061c3f3640 1 -- 172.21.15.150:0/177997501 shutdown_connections 2024-03-17T23:48:29.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.827+0000 7f061c3f3640 1 --2- 172.21.15.150:0/177997501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f061409d410 0x7f061409d890 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.827+0000 7f061c3f3640 1 --2- 172.21.15.150:0/177997501 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f05f806bc00 0x7f05f806e0c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.827+0000 7f061c3f3640 1 --2- 172.21.15.150:0/177997501 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f061409acf0 0x7f061406b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.827+0000 7f061c3f3640 1 --2- 172.21.15.150:0/177997501 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0600003b70 0x7f06141f3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:29.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.827+0000 7f061c3f3640 1 -- 172.21.15.150:0/177997501 >> 172.21.15.150:0/177997501 conn(0x7f0614099a70 msgr2=0x7f06140c7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:29.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.827+0000 7f061c3f3640 1 -- 172.21.15.150:0/177997501 shutdown_connections 2024-03-17T23:48:29.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:29.827+0000 7f061c3f3640 1 -- 172.21.15.150:0/177997501 wait complete. 2024-03-17T23:48:29.838 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi150.front.sepia.ceph.com... 2024-03-17T23:48:29.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:29.839 DEBUG:teuthology.orchestra.run.smithi150:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-17T23:48:29.864 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:ceph-fuse[54700]: fuse finished with error 0 and tester_r 0 2024-03-17T23:48:29.866 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:48:35.967 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi150.front.sepia.ceph.com 2024-03-17T23:48:35.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:35.968 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-17T23:48:35.983 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-17T23:48:36.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.126+0000 7f7c25e36640 1 Processor -- start 2024-03-17T23:48:36.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.126+0000 7f7c25e36640 1 -- start start 2024-03-17T23:48:36.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.127+0000 7f7c25e36640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7c2015de30 0x7f7c201619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:36.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.127+0000 7f7c25e36640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7c200cf120 0x7f7c200d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:36.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.127+0000 7f7c25e36640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c200b9cc0 con 0x7f7c200b9860 2024-03-17T23:48:36.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.127+0000 7f7c25e36640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f7c20162940 con 0x7f7c200cf120 2024-03-17T23:48:36.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.127+0000 7f7c25e36640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7c201620a0 con 0x7f7c2015de30 2024-03-17T23:48:36.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.127+0000 7f7c1effd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f7c200b9860 0x7f7c200cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37896/0 (socket says 172.21.15.150:37896) 2024-03-17T23:48:36.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.127+0000 7f7c1effd640 1 -- 172.21.15.150:0/336149611 learned_addr learned my addr 172.21.15.150:0/336149611 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:36.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.128+0000 7f7c1ffff640 1 --2- 172.21.15.150:0/336149611 >> v2:172.21.15.157:3300/0 conn(0x7f7c200cf120 0x7f7c200d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:36.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.128+0000 7f7c1f7fe640 1 --2- 172.21.15.150:0/336149611 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7c2015de30 0x7f7c201619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:36.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.128+0000 7f7c1e7fc640 1 -- 172.21.15.150:0/336149611 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1628249189 0 0) 0x7f7c200b9cc0 con 0x7f7c200b9860 2024-03-17T23:48:36.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.128+0000 7f7c1e7fc640 1 -- 172.21.15.150:0/336149611 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c08003520 con 0x7f7c200b9860 2024-03-17T23:48:36.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.128+0000 7f7c1e7fc640 1 -- 172.21.15.150:0/336149611 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1249995815 0 0) 0x7f7c08003520 con 0x7f7c200b9860 2024-03-17T23:48:36.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.128+0000 7f7c1e7fc640 1 -- 172.21.15.150:0/336149611 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7c2015de30 msgr2=0x7f7c201619a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:36.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.128+0000 7f7c1e7fc640 1 --2- 172.21.15.150:0/336149611 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7c2015de30 0x7f7c201619a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:36.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.128+0000 7f7c1e7fc640 1 -- 172.21.15.150:0/336149611 >> v2:172.21.15.157:3300/0 conn(0x7f7c200cf120 msgr2=0x7f7c200d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:36.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.128+0000 7f7c1e7fc640 1 --2- 172.21.15.150:0/336149611 >> v2:172.21.15.157:3300/0 conn(0x7f7c200cf120 0x7f7c200d2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:36.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.128+0000 7f7c1e7fc640 1 -- 172.21.15.150:0/336149611 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c20162380 con 0x7f7c200b9860 2024-03-17T23:48:36.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.128+0000 7f7c1e7fc640 1 -- 172.21.15.150:0/336149611 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f7c10001670 con 0x7f7c200b9860 2024-03-17T23:48:36.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.128+0000 7f7c1e7fc640 1 -- 172.21.15.150:0/336149611 >> v1:172.21.15.150:6789/0 conn(0x7f7c200b9860 legacy=0x7f7c200cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:36.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.128+0000 7f7c1e7fc640 1 --2- 172.21.15.150:0/336149611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7c080039b0 0x7f7c08003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:36.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.128+0000 7f7c1e7fc640 1 --2- 172.21.15.150:0/336149611 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7c2015de30 0x7f7c08004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:36.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.128+0000 7f7c1e7fc640 1 --2- 172.21.15.150:0/336149611 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c08004d40 0x7f7c080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:36.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.128+0000 7f7c1e7fc640 1 -- 172.21.15.150:0/336149611 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7c20162940 con 0x7f7c08004d40 2024-03-17T23:48:36.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.128+0000 7f7c1e7fc640 1 -- 172.21.15.150:0/336149611 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7c201620a0 con 0x7f7c080039b0 2024-03-17T23:48:36.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.128+0000 7f7c1e7fc640 1 -- 172.21.15.150:0/336149611 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7c0800b9c0 con 0x7f7c2015de30 2024-03-17T23:48:36.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.128+0000 7f7c1f7fe640 1 --2- 172.21.15.150:0/336149611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7c080039b0 0x7f7c08003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:36.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.128+0000 7f7c1ffff640 1 --2- 172.21.15.150:0/336149611 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c08004d40 0x7f7c080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:36.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.128+0000 7f7c1effd640 1 --2- 172.21.15.150:0/336149611 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7c2015de30 0x7f7c08004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:36.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.129+0000 7f7c1f7fe640 1 -- 172.21.15.150:0/336149611 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7c2015de30 msgr2=0x7f7c08004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:36.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.129+0000 7f7c1f7fe640 1 --2- 172.21.15.150:0/336149611 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7c2015de30 0x7f7c08004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:36.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.129+0000 7f7c1f7fe640 1 -- 172.21.15.150:0/336149611 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c08004d40 msgr2=0x7f7c080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:36.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.129+0000 7f7c1f7fe640 1 --2- 172.21.15.150:0/336149611 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c08004d40 0x7f7c080051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:36.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.129+0000 7f7c1f7fe640 1 -- 172.21.15.150:0/336149611 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c08003520 con 0x7f7c080039b0 2024-03-17T23:48:36.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.129+0000 7f7c1ffff640 1 --2- 172.21.15.150:0/336149611 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c08004d40 0x7f7c080051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:48:36.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.130+0000 7f7c1f7fe640 1 --2- 172.21.15.150:0/336149611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7c080039b0 0x7f7c08003dd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7c14009ac0 tx=0x7f7c08005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0443f732035965c server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:36.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.130+0000 7f7c1e7fc640 1 -- 172.21.15.150:0/336149611 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c14039070 con 0x7f7c080039b0 2024-03-17T23:48:36.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.130+0000 7f7c1e7fc640 1 -- 172.21.15.150:0/336149611 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c1402c6b0 con 0x7f7c080039b0 2024-03-17T23:48:36.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.130+0000 7f7c1e7fc640 1 -- 172.21.15.150:0/336149611 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c14009690 con 0x7f7c080039b0 2024-03-17T23:48:36.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.131+0000 7f7c25e36640 1 -- 172.21.15.150:0/336149611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7c080039b0 msgr2=0x7f7c08003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:36.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.131+0000 7f7c25e36640 1 --2- 172.21.15.150:0/336149611 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7c080039b0 0x7f7c08003dd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7c14009ac0 tx=0x7f7c08005830 comp rx=0 tx=0).stop 2024-03-17T23:48:36.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.131+0000 7f7c1ffff640 1 -- 172.21.15.150:0/336149611 reap_dead start 2024-03-17T23:48:36.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.131+0000 7f7c25e36640 1 -- 172.21.15.150:0/336149611 shutdown_connections 2024-03-17T23:48:36.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.131+0000 7f7c25e36640 1 -- 172.21.15.150:0/336149611 >> 172.21.15.150:0/336149611 conn(0x7f7c200730b0 msgr2=0x7f7c200d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:36.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.131+0000 7f7c25e36640 1 -- 172.21.15.150:0/336149611 shutdown_connections 2024-03-17T23:48:36.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.131+0000 7f7c25e36640 1 -- 172.21.15.150:0/336149611 wait complete. 2024-03-17T23:48:36.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.132+0000 7f7c25e36640 1 Processor -- start 2024-03-17T23:48:36.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.132+0000 7f7c25e36640 1 -- start start 2024-03-17T23:48:36.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.133+0000 7f7c25e36640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7c080039b0 0x7f7c200cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:36.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.133+0000 7f7c25e36640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7c2015a3c0 0x7f7c2015a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:36.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.133+0000 7f7c25e36640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c201f3830 0x7f7c201f3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:36.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.133+0000 7f7c25e36640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2015e210 con 0x7f7c201f3830 2024-03-17T23:48:36.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.133+0000 7f7c25e36640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7c200d2170 con 0x7f7c2015a3c0 2024-03-17T23:48:36.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.133+0000 7f7c25e36640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7c201619a0 con 0x7f7c080039b0 2024-03-17T23:48:36.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.133+0000 7f7c1ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c201f3830 0x7f7c201f3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:36.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.133+0000 7f7c1ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c201f3830 0x7f7c201f3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44696/0 (socket says 172.21.15.150:44696) 2024-03-17T23:48:36.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.133+0000 7f7c1ffff640 1 -- 172.21.15.150:0/4178593318 learned_addr learned my addr 172.21.15.150:0/4178593318 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:36.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.133+0000 7f7c1f7fe640 1 --2- 172.21.15.150:0/4178593318 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7c080039b0 0x7f7c200cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:36.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.133+0000 7f7c1effd640 1 --2- 172.21.15.150:0/4178593318 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7c2015a3c0 0x7f7c2015a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:36.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.133+0000 7f7c1ffff640 1 -- 172.21.15.150:0/4178593318 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7c080039b0 msgr2=0x7f7c200cd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:36.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.133+0000 7f7c1ffff640 1 --2- 172.21.15.150:0/4178593318 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7c080039b0 0x7f7c200cd5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:36.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.133+0000 7f7c1ffff640 1 -- 172.21.15.150:0/4178593318 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7c2015a3c0 msgr2=0x7f7c2015a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:36.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.133+0000 7f7c1ffff640 1 --2- 172.21.15.150:0/4178593318 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7c2015a3c0 0x7f7c2015a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:36.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.133+0000 7f7c1ffff640 1 -- 172.21.15.150:0/4178593318 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c201f41d0 con 0x7f7c201f3830 2024-03-17T23:48:36.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.133+0000 7f7c1ffff640 1 --2- 172.21.15.150:0/4178593318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c201f3830 0x7f7c201f3c90 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f7c200d0400 tx=0x7f7c1800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:36.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.133+0000 7f7c07fff640 1 -- 172.21.15.150:0/4178593318 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c18019070 con 0x7f7c201f3830 2024-03-17T23:48:36.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.133+0000 7f7c07fff640 1 -- 172.21.15.150:0/4178593318 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c18003990 con 0x7f7c201f3830 2024-03-17T23:48:36.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.133+0000 7f7c25e36640 1 -- 172.21.15.150:0/4178593318 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c201f4490 con 0x7f7c201f3830 2024-03-17T23:48:36.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.134+0000 7f7c07fff640 1 -- 172.21.15.150:0/4178593318 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c18009c10 con 0x7f7c201f3830 2024-03-17T23:48:36.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.134+0000 7f7c25e36640 1 -- 172.21.15.150:0/4178593318 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c201f4da0 con 0x7f7c201f3830 2024-03-17T23:48:36.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.134+0000 7f7c25e36640 1 -- 172.21.15.150:0/4178593318 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c200681a0 con 0x7f7c201f3830 2024-03-17T23:48:36.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.135+0000 7f7c07fff640 1 -- 172.21.15.150:0/4178593318 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7c1801c070 con 0x7f7c201f3830 2024-03-17T23:48:36.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.136+0000 7f7c07fff640 1 --2- 172.21.15.150:0/4178593318 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7c0c06bbd0 0x7f7c0c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:36.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.136+0000 7f7c07fff640 1 -- 172.21.15.150:0/4178593318 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6496+0+0 (secure 0 0 0) 0x7f7c1808b470 con 0x7f7c201f3830 2024-03-17T23:48:36.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.136+0000 7f7c1f7fe640 1 --2- 172.21.15.150:0/4178593318 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7c0c06bbd0 0x7f7c0c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:36.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.136+0000 7f7c1f7fe640 1 --2- 172.21.15.150:0/4178593318 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7c0c06bbd0 0x7f7c0c06e090 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7c14036000 tx=0x7f7c14030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:36.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.140+0000 7f7c07fff640 1 -- 172.21.15.150:0/4178593318 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7c18010040 con 0x7f7c201f3830 2024-03-17T23:48:36.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.352+0000 7f7c25e36640 1 -- 172.21.15.150:0/4178593318 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f7c200cdc20 con 0x7f7c201f3830 2024-03-17T23:48:36.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.352+0000 7f7c07fff640 1 -- 172.21.15.150:0/4178593318 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+82 (secure 0 0 0) 0x7f7c18059690 con 0x7f7c201f3830 2024-03-17T23:48:36.353 INFO:teuthology.orchestra.run.smithi150.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data data_new ] 2024-03-17T23:48:36.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.354+0000 7f7c25e36640 1 -- 172.21.15.150:0/4178593318 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7c0c06bbd0 msgr2=0x7f7c0c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:36.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.354+0000 7f7c25e36640 1 --2- 172.21.15.150:0/4178593318 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7c0c06bbd0 0x7f7c0c06e090 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7c14036000 tx=0x7f7c14030000 comp rx=0 tx=0).stop 2024-03-17T23:48:36.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.354+0000 7f7c25e36640 1 -- 172.21.15.150:0/4178593318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c201f3830 msgr2=0x7f7c201f3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:36.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.354+0000 7f7c25e36640 1 --2- 172.21.15.150:0/4178593318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c201f3830 0x7f7c201f3c90 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f7c200d0400 tx=0x7f7c1800f580 comp rx=0 tx=0).stop 2024-03-17T23:48:36.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.354+0000 7f7c25e36640 1 -- 172.21.15.150:0/4178593318 shutdown_connections 2024-03-17T23:48:36.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.354+0000 7f7c25e36640 1 --2- 172.21.15.150:0/4178593318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c201f3830 0x7f7c201f3c90 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:36.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.354+0000 7f7c25e36640 1 --2- 172.21.15.150:0/4178593318 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7c0c06bbd0 0x7f7c0c06e090 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:36.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.354+0000 7f7c25e36640 1 --2- 172.21.15.150:0/4178593318 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7c2015a3c0 0x7f7c2015a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:36.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.354+0000 7f7c25e36640 1 --2- 172.21.15.150:0/4178593318 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7c080039b0 0x7f7c200cd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:36.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.354+0000 7f7c25e36640 1 -- 172.21.15.150:0/4178593318 >> 172.21.15.150:0/4178593318 conn(0x7f7c200730b0 msgr2=0x7f7c200d4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:36.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.355+0000 7f7c25e36640 1 -- 172.21.15.150:0/4178593318 shutdown_connections 2024-03-17T23:48:36.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36.355+0000 7f7c25e36640 1 -- 172.21.15.150:0/4178593318 wait complete. 2024-03-17T23:48:36.366 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-17T23:48:36.366 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-17T23:48:36.366 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi150.front.sepia.ceph.com 2024-03-17T23:48:36.367 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-17T23:48:36.367 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-17T23:48:36.367 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-17T23:48:36.367 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-17T23:48:36.367 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-17T23:48:36.367 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-17T23:48:36.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:36.367 DEBUG:teuthology.orchestra.run.smithi150:> ip addr 2024-03-17T23:48:36.383 INFO:teuthology.orchestra.run.smithi150.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-17T23:48:36.383 INFO:teuthology.orchestra.run.smithi150.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-17T23:48:36.384 INFO:teuthology.orchestra.run.smithi150.stdout: inet 127.0.0.1/8 scope host lo 2024-03-17T23:48:36.384 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:48:36.384 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 ::1/128 scope host 2024-03-17T23:48:36.384 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:48:36.384 INFO:teuthology.orchestra.run.smithi150.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:48:36.384 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bc brd ff:ff:ff:ff:ff:ff 2024-03-17T23:48:36.384 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f0 2024-03-17T23:48:36.384 INFO:teuthology.orchestra.run.smithi150.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:48:36.384 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bd brd ff:ff:ff:ff:ff:ff 2024-03-17T23:48:36.384 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f1 2024-03-17T23:48:36.384 INFO:teuthology.orchestra.run.smithi150.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:48:36.384 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:34 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:48:36.385 INFO:teuthology.orchestra.run.smithi150.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-17T23:48:36.385 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:35 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:48:36.385 INFO:teuthology.orchestra.run.smithi150.stdout: inet 172.21.15.150/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-17T23:48:36.385 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft 42338sec preferred_lft 42338sec 2024-03-17T23:48:36.385 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::ec4:7aff:fe88:6e35/64 scope link 2024-03-17T23:48:36.385 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:48:36.385 INFO:teuthology.orchestra.run.smithi150.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-17T23:48:36.385 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether f6:2d:aa:e3:d6:a3 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:48:36.385 INFO:teuthology.orchestra.run.smithi150.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-17T23:48:36.385 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:48:36.385 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::bc2f:65ff:fe6d:abc3/64 scope link 2024-03-17T23:48:36.385 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:48:36.385 INFO:teuthology.orchestra.run.smithi150.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-17T23:48:36.385 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether f6:2d:aa:e3:d6:a3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:48:36.386 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::f42d:aaff:fee3:d6a3/64 scope link 2024-03-17T23:48:36.386 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:48:36.386 INFO:teuthology.orchestra.run.smithi150.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-17T23:48:36.386 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether fa:95:93:4e:f1:dd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:48:36.386 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::f895:93ff:fe4e:f1dd/64 scope link 2024-03-17T23:48:36.386 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:48:36.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:36.386 DEBUG:teuthology.orchestra.run.smithi150:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-17T23:48:36.448 INFO:teuthology.orchestra.run.smithi150.stdout:1 2024-03-17T23:48:36.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:36.450 DEBUG:teuthology.orchestra.run.smithi150:> route 2024-03-17T23:48:36.505 INFO:teuthology.orchestra.run.smithi150.stdout:Kernel IP routing table 2024-03-17T23:48:36.506 INFO:teuthology.orchestra.run.smithi150.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-17T23:48:36.506 INFO:teuthology.orchestra.run.smithi150.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-17T23:48:36.506 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-17T23:48:36.506 INFO:teuthology.orchestra.run.smithi150.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-17T23:48:36.506 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:48:36.506 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-03-17T23:48:36.506 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-17T23:48:36.506 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-17T23:48:36.506 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-17T23:48:36.507 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-03-17T23:48:36.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:48:36.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:48:36.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:36.642 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-03-17T23:48:36.657 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-17T23:48:36.657 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-17T23:48:36.658 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-17T23:48:36.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:36.658 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set brx.0 up 2024-03-17T23:48:36.726 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-17T23:48:36.726 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-17T23:48:36.726 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-17T23:48:36.781 INFO:teuthology.orchestra.run.smithi150.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-17T23:48:36.782 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-17T23:48:36.782 DEBUG:teuthology.orchestra.run.smithi150:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-17T23:48:36.836 DEBUG:teuthology.orchestra.run.smithi150:> sudo modprobe fuse 2024-03-17T23:48:36.904 DEBUG:teuthology.orchestra.run.smithi150:> cat /proc/self/mounts | awk '{print $2}' 2024-03-17T23:48:36.959 INFO:teuthology.orchestra.run.smithi150.stdout:/proc 2024-03-17T23:48:36.960 INFO:teuthology.orchestra.run.smithi150.stdout:/sys 2024-03-17T23:48:36.960 INFO:teuthology.orchestra.run.smithi150.stdout:/dev 2024-03-17T23:48:36.960 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/security 2024-03-17T23:48:36.960 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/shm 2024-03-17T23:48:36.960 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/pts 2024-03-17T23:48:36.960 INFO:teuthology.orchestra.run.smithi150.stdout:/run 2024-03-17T23:48:36.960 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/cgroup 2024-03-17T23:48:36.960 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/pstore 2024-03-17T23:48:36.960 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/bpf 2024-03-17T23:48:36.960 INFO:teuthology.orchestra.run.smithi150.stdout:/ 2024-03-17T23:48:36.960 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/selinux 2024-03-17T23:48:36.960 INFO:teuthology.orchestra.run.smithi150.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:48:36.960 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/hugepages 2024-03-17T23:48:36.961 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/mqueue 2024-03-17T23:48:36.961 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/debug 2024-03-17T23:48:36.961 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/tracing 2024-03-17T23:48:36.961 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-17T23:48:36.961 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/fuse/connections 2024-03-17T23:48:36.961 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/config 2024-03-17T23:48:36.961 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-sysctl.service 2024-03-17T23:48:36.961 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-17T23:48:36.961 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/nfs/rpc_pipefs 2024-03-17T23:48:36.961 INFO:teuthology.orchestra.run.smithi150.stdout:/run/user/1000 2024-03-17T23:48:36.961 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph 2024-03-17T23:48:36.961 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-0 2024-03-17T23:48:36.961 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-1 2024-03-17T23:48:36.961 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-2 2024-03-17T23:48:36.962 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-3 2024-03-17T23:48:36.962 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns 2024-03-17T23:48:36.962 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:48:36.962 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:48:36.962 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:48:36.962 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:48:36.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:36.962 DEBUG:teuthology.orchestra.run.smithi150:> ls /sys/fs/fuse/connections 2024-03-17T23:48:37.015 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-17T23:48:37.015 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-17T23:48:37.057 DEBUG:teuthology.orchestra.run.smithi150:> sudo modprobe fuse 2024-03-17T23:48:37.085 DEBUG:teuthology.orchestra.run.smithi150:> cat /proc/self/mounts | awk '{print $2}' 2024-03-17T23:48:37.143 INFO:teuthology.orchestra.run.smithi150.stdout:/proc 2024-03-17T23:48:37.143 INFO:teuthology.orchestra.run.smithi150.stdout:/sys 2024-03-17T23:48:37.143 INFO:teuthology.orchestra.run.smithi150.stdout:/dev 2024-03-17T23:48:37.143 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/security 2024-03-17T23:48:37.144 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/shm 2024-03-17T23:48:37.144 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/pts 2024-03-17T23:48:37.144 INFO:teuthology.orchestra.run.smithi150.stdout:/run 2024-03-17T23:48:37.144 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/cgroup 2024-03-17T23:48:37.144 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/pstore 2024-03-17T23:48:37.144 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/bpf 2024-03-17T23:48:37.144 INFO:teuthology.orchestra.run.smithi150.stdout:/ 2024-03-17T23:48:37.144 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/selinux 2024-03-17T23:48:37.144 INFO:teuthology.orchestra.run.smithi150.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:48:37.144 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/hugepages 2024-03-17T23:48:37.144 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/mqueue 2024-03-17T23:48:37.144 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/debug 2024-03-17T23:48:37.144 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/tracing 2024-03-17T23:48:37.144 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-17T23:48:37.145 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/fuse/connections 2024-03-17T23:48:37.145 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/config 2024-03-17T23:48:37.145 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-sysctl.service 2024-03-17T23:48:37.145 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-17T23:48:37.145 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/nfs/rpc_pipefs 2024-03-17T23:48:37.145 INFO:teuthology.orchestra.run.smithi150.stdout:/run/user/1000 2024-03-17T23:48:37.145 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph 2024-03-17T23:48:37.145 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-0 2024-03-17T23:48:37.145 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-1 2024-03-17T23:48:37.145 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-2 2024-03-17T23:48:37.145 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-3 2024-03-17T23:48:37.145 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns 2024-03-17T23:48:37.145 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:48:37.145 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:48:37.146 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:48:37.146 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:48:37.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:37.146 DEBUG:teuthology.orchestra.run.smithi150:> ls /sys/fs/fuse/connections 2024-03-17T23:48:37.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:2024-03-17T23:48:37.142+0000 7fe026a3a480 -1 init, newargv = 0x55835b855770 newargc=15 2024-03-17T23:48:37.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:ceph-fuse[55953]: starting ceph client 2024-03-17T23:48:37.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:ceph-fuse[55953]: starting fuse 2024-03-17T23:48:37.162 INFO:teuthology.orchestra.run.smithi150.stdout:43 2024-03-17T23:48:37.162 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-17T23:48:37.162 DEBUG:teuthology.orchestra.run.smithi150:> sudo stdin-killer -- python3 -c ' 2024-03-17T23:48:37.162 DEBUG:teuthology.orchestra.run.smithi150:> import glob 2024-03-17T23:48:37.163 DEBUG:teuthology.orchestra.run.smithi150:> import re 2024-03-17T23:48:37.163 DEBUG:teuthology.orchestra.run.smithi150:> import os 2024-03-17T23:48:37.163 DEBUG:teuthology.orchestra.run.smithi150:> import subprocess 2024-03-17T23:48:37.163 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:48:37.163 DEBUG:teuthology.orchestra.run.smithi150:> def _find_admin_socket(client_name): 2024-03-17T23:48:37.163 DEBUG:teuthology.orchestra.run.smithi150:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-17T23:48:37.163 DEBUG:teuthology.orchestra.run.smithi150:> files = glob.glob(asok_path) 2024-03-17T23:48:37.163 DEBUG:teuthology.orchestra.run.smithi150:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-17T23:48:37.163 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:48:37.163 DEBUG:teuthology.orchestra.run.smithi150:> # Given a non-glob path, it better be there 2024-03-17T23:48:37.163 DEBUG:teuthology.orchestra.run.smithi150:> if "*" not in asok_path: 2024-03-17T23:48:37.163 DEBUG:teuthology.orchestra.run.smithi150:> assert(len(files) == 1) 2024-03-17T23:48:37.163 DEBUG:teuthology.orchestra.run.smithi150:> return files[0] 2024-03-17T23:48:37.163 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:48:37.163 DEBUG:teuthology.orchestra.run.smithi150:> for f in files: 2024-03-17T23:48:37.164 DEBUG:teuthology.orchestra.run.smithi150:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-17T23:48:37.164 DEBUG:teuthology.orchestra.run.smithi150:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-17T23:48:37.164 DEBUG:teuthology.orchestra.run.smithi150:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-17T23:48:37.164 DEBUG:teuthology.orchestra.run.smithi150:> contents = proc_f.read() 2024-03-17T23:48:37.164 DEBUG:teuthology.orchestra.run.smithi150:> if mountpoint in contents: 2024-03-17T23:48:37.164 DEBUG:teuthology.orchestra.run.smithi150:> return f 2024-03-17T23:48:37.164 DEBUG:teuthology.orchestra.run.smithi150:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-17T23:48:37.164 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:48:37.164 DEBUG:teuthology.orchestra.run.smithi150:> print(_find_admin_socket("client.0")) 2024-03-17T23:48:37.164 DEBUG:teuthology.orchestra.run.smithi150:> ' 2024-03-17T23:48:37.282 INFO:teuthology.orchestra.run.smithi150.stdout:/var/run/ceph/ceph-client.0.55953.asok 2024-03-17T23:48:37.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:48:37.292 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.55953.asok 2024-03-17T23:48:37.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:37.293 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.55953.asok status 2024-03-17T23:48:37.431 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-17T23:48:37.431 INFO:teuthology.orchestra.run.smithi150.stdout: "metadata": { 2024-03-17T23:48:37.431 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-17T23:48:37.431 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-17T23:48:37.431 INFO:teuthology.orchestra.run.smithi150.stdout: "entity_id": "0", 2024-03-17T23:48:37.431 INFO:teuthology.orchestra.run.smithi150.stdout: "hostname": "smithi150", 2024-03-17T23:48:37.431 INFO:teuthology.orchestra.run.smithi150.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-17T23:48:37.431 INFO:teuthology.orchestra.run.smithi150.stdout: "pid": "55953", 2024-03-17T23:48:37.431 INFO:teuthology.orchestra.run.smithi150.stdout: "root": "/" 2024-03-17T23:48:37.431 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:37.431 INFO:teuthology.orchestra.run.smithi150.stdout: "dentry_count": 0, 2024-03-17T23:48:37.432 INFO:teuthology.orchestra.run.smithi150.stdout: "dentry_pinned_count": 0, 2024-03-17T23:48:37.432 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 5835, 2024-03-17T23:48:37.432 INFO:teuthology.orchestra.run.smithi150.stdout: "inst": { 2024-03-17T23:48:37.432 INFO:teuthology.orchestra.run.smithi150.stdout: "name": { 2024-03-17T23:48:37.432 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "client", 2024-03-17T23:48:37.432 INFO:teuthology.orchestra.run.smithi150.stdout: "num": 5835 2024-03-17T23:48:37.432 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:37.432 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": { 2024-03-17T23:48:37.432 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v1", 2024-03-17T23:48:37.432 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "192.168.0.1:0", 2024-03-17T23:48:37.432 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1134914913 2024-03-17T23:48:37.432 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:37.432 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:37.432 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": { 2024-03-17T23:48:37.433 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v1", 2024-03-17T23:48:37.433 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "192.168.0.1:0", 2024-03-17T23:48:37.433 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1134914913 2024-03-17T23:48:37.433 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:37.433 INFO:teuthology.orchestra.run.smithi150.stdout: "inst_str": "client.5835 192.168.0.1:0/1134914913", 2024-03-17T23:48:37.433 INFO:teuthology.orchestra.run.smithi150.stdout: "addr_str": "192.168.0.1:0/1134914913", 2024-03-17T23:48:37.433 INFO:teuthology.orchestra.run.smithi150.stdout: "inode_count": 1, 2024-03-17T23:48:37.433 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_epoch": 16, 2024-03-17T23:48:37.433 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_epoch": 44, 2024-03-17T23:48:37.433 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_epoch_barrier": 0, 2024-03-17T23:48:37.433 INFO:teuthology.orchestra.run.smithi150.stdout: "blocklisted": false, 2024-03-17T23:48:37.433 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "cephfs" 2024-03-17T23:48:37.433 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-17T23:48:37.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:37.440 DEBUG:teuthology.orchestra.run.smithi150:> stat --file-system '--printf=%T 2024-03-17T23:48:37.440 DEBUG:teuthology.orchestra.run.smithi150:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-17T23:48:37.455 INFO:teuthology.orchestra.run.smithi150.stdout:fuseblk 2024-03-17T23:48:37.455 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-17T23:48:37.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:37.455 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-17T23:48:37.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:37.523 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout.pool -v data_new layoutfile) 2024-03-17T23:48:37.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-17T23:48:37.580 INFO:teuthology.orchestra.run.smithi150.stderr:setfattr: layoutfile: Permission denied 2024-03-17T23:48:37.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:37.580 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.layout.pool -v data_new layoutdir) 2024-03-17T23:48:37.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-17T23:48:37.638 INFO:teuthology.orchestra.run.smithi150.stderr:setfattr: layoutdir: Permission denied 2024-03-17T23:48:37.638 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:48:37.638 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:48:37.697 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:48:37.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.840+0000 7f3856012640 1 Processor -- start 2024-03-17T23:48:37.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.840+0000 7f3856012640 1 -- start start 2024-03-17T23:48:37.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.841+0000 7f3856012640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38500c8210 0x7f38500cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:37.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.841+0000 7f3856012640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f38500ccb80 0x7f38500cd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:37.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.841+0000 7f3856012640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38500cd550 con 0x7f385015be40 2024-03-17T23:48:37.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.841+0000 7f3856012640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f38500c5d70 con 0x7f38500ccb80 2024-03-17T23:48:37.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.841+0000 7f3856012640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f385009b260 con 0x7f38500c8210 2024-03-17T23:48:37.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.841+0000 7f384f7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f385015be40 0x7f385009a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37912/0 (socket says 172.21.15.150:37912) 2024-03-17T23:48:37.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.841+0000 7f384f7fe640 1 -- 172.21.15.150:0/1914773733 learned_addr learned my addr 172.21.15.150:0/1914773733 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:37.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384e7fc640 1 -- 172.21.15.150:0/1914773733 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3682661360 0 0) 0x7f38500cd550 con 0x7f385015be40 2024-03-17T23:48:37.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384e7fc640 1 -- 172.21.15.150:0/1914773733 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3838003520 con 0x7f385015be40 2024-03-17T23:48:37.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384effd640 1 --2- 172.21.15.150:0/1914773733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38500c8210 0x7f38500cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:37.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384ffff640 1 --2- 172.21.15.150:0/1914773733 >> v2:172.21.15.157:3300/0 conn(0x7f38500ccb80 0x7f38500cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:37.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384e7fc640 1 -- 172.21.15.150:0/1914773733 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1086230031 0 0) 0x7f3838003520 con 0x7f385015be40 2024-03-17T23:48:37.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384e7fc640 1 -- 172.21.15.150:0/1914773733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38500c8210 msgr2=0x7f38500cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:37.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384e7fc640 1 --2- 172.21.15.150:0/1914773733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38500c8210 0x7f38500cc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:37.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384e7fc640 1 -- 172.21.15.150:0/1914773733 >> v2:172.21.15.157:3300/0 conn(0x7f38500ccb80 msgr2=0x7f38500cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:37.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384e7fc640 1 --2- 172.21.15.150:0/1914773733 >> v2:172.21.15.157:3300/0 conn(0x7f38500ccb80 0x7f38500cd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:37.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384e7fc640 1 -- 172.21.15.150:0/1914773733 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f385009af80 con 0x7f385015be40 2024-03-17T23:48:37.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384e7fc640 1 -- 172.21.15.150:0/1914773733 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f3840001670 con 0x7f385015be40 2024-03-17T23:48:37.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384e7fc640 1 -- 172.21.15.150:0/1914773733 >> v1:172.21.15.150:6789/0 conn(0x7f385015be40 legacy=0x7f385009a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:37.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384e7fc640 1 --2- 172.21.15.150:0/1914773733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38380039b0 0x7f3838003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:37.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384e7fc640 1 --2- 172.21.15.150:0/1914773733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38500c8210 0x7f3838004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:37.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384e7fc640 1 --2- 172.21.15.150:0/1914773733 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3838004d40 0x7f38380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:37.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384e7fc640 1 -- 172.21.15.150:0/1914773733 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f38500c5d70 con 0x7f38380039b0 2024-03-17T23:48:37.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384e7fc640 1 -- 172.21.15.150:0/1914773733 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f385009b260 con 0x7f3838004d40 2024-03-17T23:48:37.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384e7fc640 1 -- 172.21.15.150:0/1914773733 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f383800b9c0 con 0x7f38500c8210 2024-03-17T23:48:37.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384f7fe640 1 --2- 172.21.15.150:0/1914773733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38380039b0 0x7f3838003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:37.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384effd640 1 --2- 172.21.15.150:0/1914773733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38500c8210 0x7f3838004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:37.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384ffff640 1 --2- 172.21.15.150:0/1914773733 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3838004d40 0x7f38380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:37.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384f7fe640 1 -- 172.21.15.150:0/1914773733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38500c8210 msgr2=0x7f3838004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:37.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384f7fe640 1 --2- 172.21.15.150:0/1914773733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38500c8210 0x7f3838004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:37.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384f7fe640 1 -- 172.21.15.150:0/1914773733 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3838004d40 msgr2=0x7f38380051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:37.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384f7fe640 1 --2- 172.21.15.150:0/1914773733 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3838004d40 0x7f38380051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:37.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384f7fe640 1 -- 172.21.15.150:0/1914773733 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3838003520 con 0x7f38380039b0 2024-03-17T23:48:37.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.842+0000 7f384effd640 1 --2- 172.21.15.150:0/1914773733 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38500c8210 0x7f3838004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:37.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.843+0000 7f384f7fe640 1 --2- 172.21.15.150:0/1914773733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38380039b0 0x7f3838003dd0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f384000c210 tx=0x7f3838005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af4e57be438e9994 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:37.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.843+0000 7f384e7fc640 1 -- 172.21.15.150:0/1914773733 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3840007d50 con 0x7f38380039b0 2024-03-17T23:48:37.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.844+0000 7f384e7fc640 1 -- 172.21.15.150:0/1914773733 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3840030070 con 0x7f38380039b0 2024-03-17T23:48:37.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.844+0000 7f384e7fc640 1 -- 172.21.15.150:0/1914773733 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f384002cd80 con 0x7f38380039b0 2024-03-17T23:48:37.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.844+0000 7f3856012640 1 -- 172.21.15.150:0/1914773733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38380039b0 msgr2=0x7f3838003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:37.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.844+0000 7f3856012640 1 --2- 172.21.15.150:0/1914773733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38380039b0 0x7f3838003dd0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f384000c210 tx=0x7f3838005830 comp rx=0 tx=0).stop 2024-03-17T23:48:37.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.844+0000 7f384ffff640 1 -- 172.21.15.150:0/1914773733 reap_dead start 2024-03-17T23:48:37.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.844+0000 7f3856012640 1 -- 172.21.15.150:0/1914773733 shutdown_connections 2024-03-17T23:48:37.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.844+0000 7f3856012640 1 -- 172.21.15.150:0/1914773733 >> 172.21.15.150:0/1914773733 conn(0x7f38500b3620 msgr2=0x7f38500b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:37.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.844+0000 7f3856012640 1 -- 172.21.15.150:0/1914773733 shutdown_connections 2024-03-17T23:48:37.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.844+0000 7f3856012640 1 -- 172.21.15.150:0/1914773733 wait complete. 2024-03-17T23:48:37.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.845+0000 7f3856012640 1 Processor -- start 2024-03-17T23:48:37.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.845+0000 7f3856012640 1 -- start start 2024-03-17T23:48:37.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.846+0000 7f3856012640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38380039b0 0x7f38501f16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.846+0000 7f3856012640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3850145950 0x7f3850154480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.846+0000 7f3856012640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38501549c0 0x7f385015e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.846+0000 7f3856012640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f385015c220 con 0x7f38501549c0 2024-03-17T23:48:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.846+0000 7f3856012640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f38500c5a00 con 0x7f3850145950 2024-03-17T23:48:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.846+0000 7f3856012640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f38500cc640 con 0x7f38380039b0 2024-03-17T23:48:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.846+0000 7f384ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38501549c0 0x7f385015e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.846+0000 7f384ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38501549c0 0x7f385015e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44704/0 (socket says 172.21.15.150:44704) 2024-03-17T23:48:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.846+0000 7f384ffff640 1 -- 172.21.15.150:0/305786603 learned_addr learned my addr 172.21.15.150:0/305786603 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.846+0000 7f384f7fe640 1 --2- 172.21.15.150:0/305786603 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38380039b0 0x7f38501f16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.846+0000 7f384effd640 1 --2- 172.21.15.150:0/305786603 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3850145950 0x7f3850154480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.847+0000 7f384ffff640 1 -- 172.21.15.150:0/305786603 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38380039b0 msgr2=0x7f38501f16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.847+0000 7f384ffff640 1 --2- 172.21.15.150:0/305786603 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38380039b0 0x7f38501f16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.847+0000 7f384ffff640 1 -- 172.21.15.150:0/305786603 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3850145950 msgr2=0x7f3850154480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.847+0000 7f384ffff640 1 --2- 172.21.15.150:0/305786603 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3850145950 0x7f3850154480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:37.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.847+0000 7f384ffff640 1 -- 172.21.15.150:0/305786603 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3850155240 con 0x7f38501549c0 2024-03-17T23:48:37.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.847+0000 7f384effd640 1 --2- 172.21.15.150:0/305786603 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3850145950 0x7f3850154480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:37.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.847+0000 7f384f7fe640 1 --2- 172.21.15.150:0/305786603 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38380039b0 0x7f38501f16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:37.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.847+0000 7f384ffff640 1 --2- 172.21.15.150:0/305786603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38501549c0 0x7f385015e570 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f38500b7be0 tx=0x7f38480115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:37.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.847+0000 7f3837fff640 1 -- 172.21.15.150:0/305786603 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3848011de0 con 0x7f38501549c0 2024-03-17T23:48:37.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.847+0000 7f3837fff640 1 -- 172.21.15.150:0/305786603 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f384801b070 con 0x7f38501549c0 2024-03-17T23:48:37.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.847+0000 7f3837fff640 1 -- 172.21.15.150:0/305786603 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3848003990 con 0x7f38501549c0 2024-03-17T23:48:37.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.847+0000 7f3856012640 1 -- 172.21.15.150:0/305786603 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3850155520 con 0x7f38501549c0 2024-03-17T23:48:37.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.847+0000 7f3856012640 1 -- 172.21.15.150:0/305786603 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3850155ae0 con 0x7f38501549c0 2024-03-17T23:48:37.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.848+0000 7f3837fff640 1 -- 172.21.15.150:0/305786603 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3848005d10 con 0x7f38501549c0 2024-03-17T23:48:37.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.848+0000 7f3856012640 1 -- 172.21.15.150:0/305786603 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38500688b0 con 0x7f38501549c0 2024-03-17T23:48:37.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.849+0000 7f3837fff640 1 --2- 172.21.15.150:0/305786603 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f383c06bbd0 0x7f383c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:37.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.849+0000 7f384f7fe640 1 --2- 172.21.15.150:0/305786603 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f383c06bbd0 0x7f383c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:37.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.849+0000 7f3837fff640 1 -- 172.21.15.150:0/305786603 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7f384808b330 con 0x7f38501549c0 2024-03-17T23:48:37.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.849+0000 7f384f7fe640 1 --2- 172.21.15.150:0/305786603 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f383c06bbd0 0x7f383c06e090 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3840034000 tx=0x7f3840007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:37.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:37.853+0000 7f3837fff640 1 -- 172.21.15.150:0/305786603 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3848025020 con 0x7f38501549c0 2024-03-17T23:48:38.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.033+0000 7f3856012640 1 -- 172.21.15.150:0/305786603 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f38500c6f10 con 0x7f38501549c0 2024-03-17T23:48:38.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.034+0000 7f3837fff640 1 -- 172.21.15.150:0/305786603 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+18839 (secure 0 0 0) 0x7f3848014350 con 0x7f38501549c0 2024-03-17T23:48:38.036 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:38.036 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":45,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:48:37.677776+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:48:08.957879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-17T23:48:09.698808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-17T23:48:22.562269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.3d","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:48:38.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.036+0000 7f3856012640 1 -- 172.21.15.150:0/305786603 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f383c06bbd0 msgr2=0x7f383c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:38.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.036+0000 7f3856012640 1 --2- 172.21.15.150:0/305786603 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f383c06bbd0 0x7f383c06e090 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3840034000 tx=0x7f3840007000 comp rx=0 tx=0).stop 2024-03-17T23:48:38.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.036+0000 7f3856012640 1 -- 172.21.15.150:0/305786603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38501549c0 msgr2=0x7f385015e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:38.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.036+0000 7f3856012640 1 --2- 172.21.15.150:0/305786603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38501549c0 0x7f385015e570 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f38500b7be0 tx=0x7f38480115a0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.036+0000 7f3856012640 1 -- 172.21.15.150:0/305786603 shutdown_connections 2024-03-17T23:48:38.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.036+0000 7f3856012640 1 --2- 172.21.15.150:0/305786603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f38501549c0 0x7f385015e570 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.036+0000 7f3856012640 1 --2- 172.21.15.150:0/305786603 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f383c06bbd0 0x7f383c06e090 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.036+0000 7f3856012640 1 --2- 172.21.15.150:0/305786603 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3850145950 0x7f3850154480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.036+0000 7f3856012640 1 --2- 172.21.15.150:0/305786603 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38380039b0 0x7f38501f16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.036+0000 7f3856012640 1 -- 172.21.15.150:0/305786603 >> 172.21.15.150:0/305786603 conn(0x7f38500b3620 msgr2=0x7f38500cde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:38.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.037+0000 7f3856012640 1 -- 172.21.15.150:0/305786603 shutdown_connections 2024-03-17T23:48:38.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.037+0000 7f3856012640 1 -- 172.21.15.150:0/305786603 wait complete. 2024-03-17T23:48:38.050 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:48:08.957879+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:48:09.698808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-17T23:48:22.562269+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-17T23:48:38.050 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:48:38.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.154+0000 7fd412d1d640 1 Processor -- start 2024-03-17T23:48:38.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.154+0000 7fd412d1d640 1 -- start start 2024-03-17T23:48:38.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.154+0000 7fd412d1d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd40c15f3d0 0x7fd40c0cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:38.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.154+0000 7fd412d1d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd40c0d0880 0x7fd40c1603c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:38.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.154+0000 7fd412d1d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd40c0ccc50 con 0x7fd40c0cc770 2024-03-17T23:48:38.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.154+0000 7fd412d1d640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd40c15f7b0 con 0x7fd40c0d0880 2024-03-17T23:48:38.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.154+0000 7fd412d1d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd40c160900 con 0x7fd40c15f3d0 2024-03-17T23:48:38.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.155+0000 7fd4112ba640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd40c0d0880 0x7fd40c1603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:38.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.155+0000 7fd410ab9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd40c15f3d0 0x7fd40c0cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:38.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.155+0000 7fd40bfff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fd40c0cc770 0x7fd40c0d0170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:43528/0 (socket says 172.21.15.150:43528) 2024-03-17T23:48:38.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.155+0000 7fd40bfff640 1 -- 172.21.15.150:0/2193580017 learned_addr learned my addr 172.21.15.150:0/2193580017 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:38.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.155+0000 7fd4112ba640 1 -- 172.21.15.150:0/2193580017 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd40c15f3d0 msgr2=0x7fd40c0cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:38.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.155+0000 7fd4112ba640 1 --2- 172.21.15.150:0/2193580017 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd40c15f3d0 0x7fd40c0cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.155+0000 7fd4112ba640 1 -- 172.21.15.150:0/2193580017 >> v1:172.21.15.157:6789/0 conn(0x7fd40c0cc770 legacy=0x7fd40c0d0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:38.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.155+0000 7fd4112ba640 1 -- 172.21.15.150:0/2193580017 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd40c0ccf30 con 0x7fd40c0d0880 2024-03-17T23:48:38.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.155+0000 7fd410ab9640 1 --2- 172.21.15.150:0/2193580017 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd40c15f3d0 0x7fd40c0cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:48:38.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.156+0000 7fd4112ba640 1 --2- 172.21.15.150:0/2193580017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd40c0d0880 0x7fd40c1603c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd404009d90 tx=0x7fd40402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d75a9325999789f server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:38.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.156+0000 7fd40b7fe640 1 -- 172.21.15.150:0/2193580017 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd404039070 con 0x7fd40c0d0880 2024-03-17T23:48:38.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.156+0000 7fd40b7fe640 1 -- 172.21.15.150:0/2193580017 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd40402cc50 con 0x7fd40c0d0880 2024-03-17T23:48:38.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.156+0000 7fd40b7fe640 1 -- 172.21.15.150:0/2193580017 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd404034440 con 0x7fd40c0d0880 2024-03-17T23:48:38.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.157+0000 7fd412d1d640 1 -- 172.21.15.150:0/2193580017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd40c0d0880 msgr2=0x7fd40c1603c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:38.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.157+0000 7fd412d1d640 1 --2- 172.21.15.150:0/2193580017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd40c0d0880 0x7fd40c1603c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd404009d90 tx=0x7fd40402bea0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.157+0000 7fd412d1d640 1 -- 172.21.15.150:0/2193580017 shutdown_connections 2024-03-17T23:48:38.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.157+0000 7fd412d1d640 1 --2- 172.21.15.150:0/2193580017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd40c0d0880 0x7fd40c1603c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.157+0000 7fd412d1d640 1 --2- 172.21.15.150:0/2193580017 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd40c15f3d0 0x7fd40c0cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.157+0000 7fd412d1d640 1 -- 172.21.15.150:0/2193580017 >> 172.21.15.150:0/2193580017 conn(0x7fd40c0b36c0 msgr2=0x7fd40c0b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:38.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.157+0000 7fd412d1d640 1 -- 172.21.15.150:0/2193580017 shutdown_connections 2024-03-17T23:48:38.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.157+0000 7fd412d1d640 1 -- 172.21.15.150:0/2193580017 wait complete. 2024-03-17T23:48:38.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.158+0000 7fd412d1d640 1 Processor -- start 2024-03-17T23:48:38.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.158+0000 7fd412d1d640 1 -- start start 2024-03-17T23:48:38.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.159+0000 7fd412d1d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd40c0cc770 0x7fd40c0ca750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:38.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.159+0000 7fd412d1d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd40c0d0880 0x7fd40c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:38.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.159+0000 7fd412d1d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd40c15f3d0 0x7fd40c15d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:38.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.159+0000 7fd412d1d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd40c0cd7d0 con 0x7fd40c15f3d0 2024-03-17T23:48:38.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.159+0000 7fd412d1d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd40c0cd4f0 con 0x7fd40c0d0880 2024-03-17T23:48:38.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.159+0000 7fd412d1d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd40c0cfa30 con 0x7fd40c0cc770 2024-03-17T23:48:38.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.159+0000 7fd4112ba640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd40c15f3d0 0x7fd40c15d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:38.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.159+0000 7fd4112ba640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd40c15f3d0 0x7fd40c15d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44712/0 (socket says 172.21.15.150:44712) 2024-03-17T23:48:38.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.159+0000 7fd4112ba640 1 -- 172.21.15.150:0/1991569802 learned_addr learned my addr 172.21.15.150:0/1991569802 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:38.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.159+0000 7fd410ab9640 1 --2- 172.21.15.150:0/1991569802 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd40c0cc770 0x7fd40c0ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:38.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.159+0000 7fd40bfff640 1 --2- 172.21.15.150:0/1991569802 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd40c0d0880 0x7fd40c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:38.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.160+0000 7fd4112ba640 1 -- 172.21.15.150:0/1991569802 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd40c0cc770 msgr2=0x7fd40c0ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:38.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.160+0000 7fd4112ba640 1 --2- 172.21.15.150:0/1991569802 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd40c0cc770 0x7fd40c0ca750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.160+0000 7fd4112ba640 1 -- 172.21.15.150:0/1991569802 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd40c0d0880 msgr2=0x7fd40c06dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:38.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.160+0000 7fd4112ba640 1 --2- 172.21.15.150:0/1991569802 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd40c0d0880 0x7fd40c06dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.160+0000 7fd4112ba640 1 -- 172.21.15.150:0/1991569802 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd40c0cb5d0 con 0x7fd40c15f3d0 2024-03-17T23:48:38.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.160+0000 7fd410ab9640 1 --2- 172.21.15.150:0/1991569802 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd40c0cc770 0x7fd40c0ca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:48:38.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.160+0000 7fd4112ba640 1 --2- 172.21.15.150:0/1991569802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd40c15f3d0 0x7fd40c15d9d0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fd404009d90 tx=0x7fd404004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:38.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.160+0000 7fd4097fa640 1 -- 172.21.15.150:0/1991569802 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd404039050 con 0x7fd40c15f3d0 2024-03-17T23:48:38.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.160+0000 7fd4097fa640 1 -- 172.21.15.150:0/1991569802 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd404045070 con 0x7fd40c15f3d0 2024-03-17T23:48:38.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.160+0000 7fd4097fa640 1 -- 172.21.15.150:0/1991569802 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd404009450 con 0x7fd40c15f3d0 2024-03-17T23:48:38.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.160+0000 7fd412d1d640 1 -- 172.21.15.150:0/1991569802 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd40c06e260 con 0x7fd40c15f3d0 2024-03-17T23:48:38.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.160+0000 7fd412d1d640 1 -- 172.21.15.150:0/1991569802 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd40c0cb010 con 0x7fd40c15f3d0 2024-03-17T23:48:38.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.161+0000 7fd412d1d640 1 -- 172.21.15.150:0/1991569802 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd40c0688b0 con 0x7fd40c15f3d0 2024-03-17T23:48:38.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.162+0000 7fd4097fa640 1 -- 172.21.15.150:0/1991569802 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd40403e610 con 0x7fd40c15f3d0 2024-03-17T23:48:38.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.162+0000 7fd4097fa640 1 --2- 172.21.15.150:0/1991569802 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd3f006bbd0 0x7fd3f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:38.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.162+0000 7fd4097fa640 1 -- 172.21.15.150:0/1991569802 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7fd4040ad700 con 0x7fd40c15f3d0 2024-03-17T23:48:38.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.162+0000 7fd410ab9640 1 --2- 172.21.15.150:0/1991569802 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd3f006bbd0 0x7fd3f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:38.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.163+0000 7fd410ab9640 1 --2- 172.21.15.150:0/1991569802 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd3f006bbd0 0x7fd3f006e090 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd400009560 tx=0x7fd400009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:38.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.167+0000 7fd4097fa640 1 -- 172.21.15.150:0/1991569802 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd40407b870 con 0x7fd40c15f3d0 2024-03-17T23:48:38.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.344+0000 7fd412d1d640 1 -- 172.21.15.150:0/1991569802 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd40c0b7b50 con 0x7fd40c15f3d0 2024-03-17T23:48:38.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.345+0000 7fd4097fa640 1 -- 172.21.15.150:0/1991569802 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v45) ==== 93+0+10 (secure 0 0 0) 0x7fd40407f910 con 0x7fd40c15f3d0 2024-03-17T23:48:38.346 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:48:38.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.347+0000 7fd412d1d640 1 -- 172.21.15.150:0/1991569802 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd3f006bbd0 msgr2=0x7fd3f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:38.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.347+0000 7fd412d1d640 1 --2- 172.21.15.150:0/1991569802 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd3f006bbd0 0x7fd3f006e090 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd400009560 tx=0x7fd400009190 comp rx=0 tx=0).stop 2024-03-17T23:48:38.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.347+0000 7fd412d1d640 1 -- 172.21.15.150:0/1991569802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd40c15f3d0 msgr2=0x7fd40c15d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:38.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.347+0000 7fd412d1d640 1 --2- 172.21.15.150:0/1991569802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd40c15f3d0 0x7fd40c15d9d0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fd404009d90 tx=0x7fd404004a40 comp rx=0 tx=0).stop 2024-03-17T23:48:38.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.347+0000 7fd412d1d640 1 -- 172.21.15.150:0/1991569802 shutdown_connections 2024-03-17T23:48:38.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.347+0000 7fd412d1d640 1 --2- 172.21.15.150:0/1991569802 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd40c15f3d0 0x7fd40c15d9d0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.347+0000 7fd412d1d640 1 --2- 172.21.15.150:0/1991569802 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd3f006bbd0 0x7fd3f006e090 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.347+0000 7fd412d1d640 1 --2- 172.21.15.150:0/1991569802 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd40c0d0880 0x7fd40c06dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.347+0000 7fd412d1d640 1 --2- 172.21.15.150:0/1991569802 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd40c0cc770 0x7fd40c0ca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.347+0000 7fd412d1d640 1 -- 172.21.15.150:0/1991569802 >> 172.21.15.150:0/1991569802 conn(0x7fd40c0b36c0 msgr2=0x7fd40c070780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:38.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.347+0000 7fd412d1d640 1 -- 172.21.15.150:0/1991569802 shutdown_connections 2024-03-17T23:48:38.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.347+0000 7fd412d1d640 1 -- 172.21.15.150:0/1991569802 wait complete. 2024-03-17T23:48:38.359 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:48:38.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.463+0000 7fcc21352640 1 Processor -- start 2024-03-17T23:48:38.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.463+0000 7fcc21352640 1 -- start start 2024-03-17T23:48:38.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.464+0000 7fcc21352640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcc1c161520 0x7fcc1c0b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:38.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.464+0000 7fcc21352640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fcc1c0d0ce0 0x7fcc1c0d1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:38.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.464+0000 7fcc21352640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc1c161d30 con 0x7fcc1c0b2bf0 2024-03-17T23:48:38.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.464+0000 7fcc21352640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fcc1c0b9bf0 con 0x7fcc1c161520 2024-03-17T23:48:38.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.464+0000 7fcc21352640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fcc1c162940 con 0x7fcc1c0d0ce0 2024-03-17T23:48:38.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.464+0000 7fcc1affd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcc1c161520 0x7fcc1c0b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:38.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.464+0000 7fcc1affd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcc1c161520 0x7fcc1c0b96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44722/0 (socket says 172.21.15.150:44722) 2024-03-17T23:48:38.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.464+0000 7fcc1a7fc640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fcc1c0b2bf0 0x7fcc1c0d05d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37924/0 (socket says 172.21.15.150:37924) 2024-03-17T23:48:38.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.464+0000 7fcc1affd640 1 -- 172.21.15.150:0/2109070960 learned_addr learned my addr 172.21.15.150:0/2109070960 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:38.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.465+0000 7fcc1b7fe640 1 --2- 172.21.15.150:0/2109070960 >> v2:172.21.15.157:3300/0 conn(0x7fcc1c0d0ce0 0x7fcc1c0d1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:38.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.465+0000 7fcc1affd640 1 -- 172.21.15.150:0/2109070960 >> v2:172.21.15.157:3300/0 conn(0x7fcc1c0d0ce0 msgr2=0x7fcc1c0d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:38.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.465+0000 7fcc1affd640 1 --2- 172.21.15.150:0/2109070960 >> v2:172.21.15.157:3300/0 conn(0x7fcc1c0d0ce0 0x7fcc1c0d1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.465+0000 7fcc1affd640 1 -- 172.21.15.150:0/2109070960 >> v1:172.21.15.150:6789/0 conn(0x7fcc1c0b2bf0 legacy=0x7fcc1c0d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:38.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.465+0000 7fcc1affd640 1 -- 172.21.15.150:0/2109070960 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc1c162010 con 0x7fcc1c161520 2024-03-17T23:48:38.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.465+0000 7fcc19ffb640 1 -- 172.21.15.150:0/2109070960 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1845899823 0 0) 0x7fcc1c161d30 con 0x7fcc1c0b2bf0 2024-03-17T23:48:38.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.466+0000 7fcc1affd640 1 --2- 172.21.15.150:0/2109070960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc1c161520 0x7fcc1c0b96b0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c0d16a0 tx=0x7fcc0c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d846e2fb4aa7bd89 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:38.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.466+0000 7fcc19ffb640 1 -- 172.21.15.150:0/2109070960 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c02e070 con 0x7fcc1c161520 2024-03-17T23:48:38.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.466+0000 7fcc19ffb640 1 -- 172.21.15.150:0/2109070960 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc0c037070 con 0x7fcc1c161520 2024-03-17T23:48:38.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.466+0000 7fcc19ffb640 1 -- 172.21.15.150:0/2109070960 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c031a00 con 0x7fcc1c161520 2024-03-17T23:48:38.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.466+0000 7fcc21352640 1 -- 172.21.15.150:0/2109070960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc1c161520 msgr2=0x7fcc1c0b96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:38.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.466+0000 7fcc21352640 1 --2- 172.21.15.150:0/2109070960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc1c161520 0x7fcc1c0b96b0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c0d16a0 tx=0x7fcc0c02b880 comp rx=0 tx=0).stop 2024-03-17T23:48:38.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.466+0000 7fcc21352640 1 -- 172.21.15.150:0/2109070960 shutdown_connections 2024-03-17T23:48:38.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.466+0000 7fcc21352640 1 --2- 172.21.15.150:0/2109070960 >> v2:172.21.15.157:3300/0 conn(0x7fcc1c0d0ce0 0x7fcc1c0d1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.466+0000 7fcc21352640 1 --2- 172.21.15.150:0/2109070960 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc1c161520 0x7fcc1c0b96b0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.466+0000 7fcc21352640 1 -- 172.21.15.150:0/2109070960 >> 172.21.15.150:0/2109070960 conn(0x7fcc1c06ca00 msgr2=0x7fcc1c06ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:38.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.466+0000 7fcc21352640 1 -- 172.21.15.150:0/2109070960 shutdown_connections 2024-03-17T23:48:38.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.466+0000 7fcc21352640 1 -- 172.21.15.150:0/2109070960 wait complete. 2024-03-17T23:48:38.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.468+0000 7fcc21352640 1 Processor -- start 2024-03-17T23:48:38.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.468+0000 7fcc21352640 1 -- start start 2024-03-17T23:48:38.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.468+0000 7fcc21352640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc1c0b2bf0 0x7fcc1c152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:38.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.468+0000 7fcc21352640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc1c0d0ce0 0x7fcc1c1544c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:38.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.468+0000 7fcc21352640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc1c161520 0x7fcc1c154a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:38.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.468+0000 7fcc21352640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1c161900 con 0x7fcc1c161520 2024-03-17T23:48:38.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.468+0000 7fcc21352640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1c1625d0 con 0x7fcc1c0b2bf0 2024-03-17T23:48:38.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.468+0000 7fcc21352640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fcc1c0d24e0 con 0x7fcc1c0d0ce0 2024-03-17T23:48:38.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.468+0000 7fcc1b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc1c161520 0x7fcc1c154a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:38.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.468+0000 7fcc1b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc1c161520 0x7fcc1c154a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44724/0 (socket says 172.21.15.150:44724) 2024-03-17T23:48:38.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.468+0000 7fcc1b7fe640 1 -- 172.21.15.150:0/2091113891 learned_addr learned my addr 172.21.15.150:0/2091113891 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:38.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.469+0000 7fcc1a7fc640 1 --2- 172.21.15.150:0/2091113891 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc1c0d0ce0 0x7fcc1c1544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:38.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.469+0000 7fcc1affd640 1 --2- 172.21.15.150:0/2091113891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc1c0b2bf0 0x7fcc1c152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:38.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.469+0000 7fcc1b7fe640 1 -- 172.21.15.150:0/2091113891 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc1c0d0ce0 msgr2=0x7fcc1c1544c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:38.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.469+0000 7fcc1b7fe640 1 --2- 172.21.15.150:0/2091113891 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc1c0d0ce0 0x7fcc1c1544c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.469+0000 7fcc1b7fe640 1 -- 172.21.15.150:0/2091113891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc1c0b2bf0 msgr2=0x7fcc1c152210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:38.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.469+0000 7fcc1b7fe640 1 --2- 172.21.15.150:0/2091113891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc1c0b2bf0 0x7fcc1c152210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.469+0000 7fcc1b7fe640 1 -- 172.21.15.150:0/2091113891 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc1c1f3dd0 con 0x7fcc1c161520 2024-03-17T23:48:38.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.469+0000 7fcc1b7fe640 1 --2- 172.21.15.150:0/2091113891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc1c161520 0x7fcc1c154a00 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fcc1400efc0 tx=0x7fcc1400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:38.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.469+0000 7fcc037fe640 1 -- 172.21.15.150:0/2091113891 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc14018020 con 0x7fcc1c161520 2024-03-17T23:48:38.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.469+0000 7fcc037fe640 1 -- 172.21.15.150:0/2091113891 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc14014480 con 0x7fcc1c161520 2024-03-17T23:48:38.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.469+0000 7fcc21352640 1 -- 172.21.15.150:0/2091113891 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc1c155060 con 0x7fcc1c161520 2024-03-17T23:48:38.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.469+0000 7fcc037fe640 1 -- 172.21.15.150:0/2091113891 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc14014870 con 0x7fcc1c161520 2024-03-17T23:48:38.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.469+0000 7fcc21352640 1 -- 172.21.15.150:0/2091113891 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc1c155620 con 0x7fcc1c161520 2024-03-17T23:48:38.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.470+0000 7fcc21352640 1 -- 172.21.15.150:0/2091113891 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc1c0681a0 con 0x7fcc1c161520 2024-03-17T23:48:38.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.471+0000 7fcc037fe640 1 -- 172.21.15.150:0/2091113891 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fcc14005d10 con 0x7fcc1c161520 2024-03-17T23:48:38.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.471+0000 7fcc037fe640 1 --2- 172.21.15.150:0/2091113891 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcc0806bbd0 0x7fcc0806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:38.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.471+0000 7fcc037fe640 1 -- 172.21.15.150:0/2091113891 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7fcc1408ba50 con 0x7fcc1c161520 2024-03-17T23:48:38.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.471+0000 7fcc1affd640 1 --2- 172.21.15.150:0/2091113891 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcc0806bbd0 0x7fcc0806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:38.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.471+0000 7fcc1affd640 1 --2- 172.21.15.150:0/2091113891 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcc0806bbd0 0x7fcc0806e090 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c036000 tx=0x7fcc0c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:38.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.475+0000 7fcc037fe640 1 -- 172.21.15.150:0/2091113891 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fcc14022030 con 0x7fcc1c161520 2024-03-17T23:48:38.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.647+0000 7fcc21352640 1 -- 172.21.15.150:0/2091113891 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fcc1c0d19c0 con 0x7fcc1c161520 2024-03-17T23:48:38.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.647+0000 7fcc037fe640 1 -- 172.21.15.150:0/2091113891 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v45) ==== 104+0+11 (secure 0 0 0) 0x7fcc1c1f3dd0 con 0x7fcc1c161520 2024-03-17T23:48:38.649 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:48:38.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.649+0000 7fcc21352640 1 -- 172.21.15.150:0/2091113891 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcc0806bbd0 msgr2=0x7fcc0806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:38.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.649+0000 7fcc21352640 1 --2- 172.21.15.150:0/2091113891 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcc0806bbd0 0x7fcc0806e090 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c036000 tx=0x7fcc0c008030 comp rx=0 tx=0).stop 2024-03-17T23:48:38.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.649+0000 7fcc21352640 1 -- 172.21.15.150:0/2091113891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc1c161520 msgr2=0x7fcc1c154a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:38.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.649+0000 7fcc21352640 1 --2- 172.21.15.150:0/2091113891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc1c161520 0x7fcc1c154a00 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fcc1400efc0 tx=0x7fcc1400c490 comp rx=0 tx=0).stop 2024-03-17T23:48:38.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.649+0000 7fcc21352640 1 -- 172.21.15.150:0/2091113891 shutdown_connections 2024-03-17T23:48:38.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.649+0000 7fcc21352640 1 --2- 172.21.15.150:0/2091113891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcc1c161520 0x7fcc1c154a00 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.649+0000 7fcc21352640 1 --2- 172.21.15.150:0/2091113891 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcc1c0d0ce0 0x7fcc1c1544c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.649+0000 7fcc21352640 1 --2- 172.21.15.150:0/2091113891 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcc0806bbd0 0x7fcc0806e090 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.649+0000 7fcc21352640 1 --2- 172.21.15.150:0/2091113891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc1c0b2bf0 0x7fcc1c152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.649+0000 7fcc21352640 1 -- 172.21.15.150:0/2091113891 >> 172.21.15.150:0/2091113891 conn(0x7fcc1c06ca00 msgr2=0x7fcc1c0ce7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:38.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.649+0000 7fcc21352640 1 -- 172.21.15.150:0/2091113891 shutdown_connections 2024-03-17T23:48:38.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.649+0000 7fcc21352640 1 -- 172.21.15.150:0/2091113891 wait complete. 2024-03-17T23:48:38.662 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:48:38.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.766+0000 7effe9359640 1 Processor -- start 2024-03-17T23:48:38.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.766+0000 7effe9359640 1 -- start start 2024-03-17T23:48:38.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.767+0000 7effe9359640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7effe415e460 0x7effe406c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:38.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.767+0000 7effe9359640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effe406c8c0 0x7effe4167ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:38.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.767+0000 7effe9359640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effe415e990 con 0x7effe4099e20 2024-03-17T23:48:38.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.767+0000 7effe9359640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7effe415ec70 con 0x7effe415e460 2024-03-17T23:48:38.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.767+0000 7effe9359640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7effe415ef50 con 0x7effe406c8c0 2024-03-17T23:48:38.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.767+0000 7effe27fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7effe415e460 0x7effe406c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:38.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.767+0000 7effe27fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7effe415e460 0x7effe406c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44730/0 (socket says 172.21.15.150:44730) 2024-03-17T23:48:38.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.767+0000 7effe27fc640 1 -- 172.21.15.150:0/2119645589 learned_addr learned my addr 172.21.15.150:0/2119645589 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:38.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.767+0000 7effe37fe640 1 --2- 172.21.15.150:0/2119645589 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effe406c8c0 0x7effe4167ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:38.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.767+0000 7effe27fc640 1 -- 172.21.15.150:0/2119645589 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effe406c8c0 msgr2=0x7effe4167ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:38.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.767+0000 7effe27fc640 1 --2- 172.21.15.150:0/2119645589 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effe406c8c0 0x7effe4167ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.767+0000 7effe27fc640 1 -- 172.21.15.150:0/2119645589 >> v1:172.21.15.157:6789/0 conn(0x7effe4099e20 legacy=0x7effe409a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:38.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.767+0000 7effe27fc640 1 -- 172.21.15.150:0/2119645589 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effe415f230 con 0x7effe415e460 2024-03-17T23:48:38.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.768+0000 7effe27fc640 1 --2- 172.21.15.150:0/2119645589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe415e460 0x7effe406c380 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7effd4009870 tx=0x7effd402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=196506ee09c0267f server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:38.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.768+0000 7effe1ffb640 1 -- 172.21.15.150:0/2119645589 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd4035020 con 0x7effe415e460 2024-03-17T23:48:38.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.768+0000 7effe1ffb640 1 -- 172.21.15.150:0/2119645589 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effd4031640 con 0x7effe415e460 2024-03-17T23:48:38.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.768+0000 7effe1ffb640 1 -- 172.21.15.150:0/2119645589 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd402d040 con 0x7effe415e460 2024-03-17T23:48:38.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.769+0000 7effe9359640 1 -- 172.21.15.150:0/2119645589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe415e460 msgr2=0x7effe406c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:38.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.769+0000 7effe9359640 1 --2- 172.21.15.150:0/2119645589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe415e460 0x7effe406c380 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7effd4009870 tx=0x7effd402b8f0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.769+0000 7effe9359640 1 -- 172.21.15.150:0/2119645589 shutdown_connections 2024-03-17T23:48:38.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.769+0000 7effe9359640 1 --2- 172.21.15.150:0/2119645589 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effe406c8c0 0x7effe4167ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.769+0000 7effe9359640 1 --2- 172.21.15.150:0/2119645589 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe415e460 0x7effe406c380 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.769+0000 7effe9359640 1 -- 172.21.15.150:0/2119645589 >> 172.21.15.150:0/2119645589 conn(0x7effe40b3630 msgr2=0x7effe40b3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:38.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.769+0000 7effe9359640 1 -- 172.21.15.150:0/2119645589 shutdown_connections 2024-03-17T23:48:38.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.769+0000 7effe9359640 1 -- 172.21.15.150:0/2119645589 wait complete. 2024-03-17T23:48:38.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.770+0000 7effe9359640 1 Processor -- start 2024-03-17T23:48:38.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.770+0000 7effe9359640 1 -- start start 2024-03-17T23:48:38.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.771+0000 7effe9359640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7effe4099e20 0x7effe414efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:38.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.771+0000 7effe9359640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe415e460 0x7effe40b4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:38.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.771+0000 7effe9359640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effe40b4ed0 0x7effe40cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:38.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.771+0000 7effe9359640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7effe40d1210 con 0x7effe415e460 2024-03-17T23:48:38.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.771+0000 7effe9359640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7effe406d570 con 0x7effe4099e20 2024-03-17T23:48:38.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.771+0000 7effe9359640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7effe40d1390 con 0x7effe40b4ed0 2024-03-17T23:48:38.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.771+0000 7effe27fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe415e460 0x7effe40b4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:38.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.771+0000 7effe27fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe415e460 0x7effe40b4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44746/0 (socket says 172.21.15.150:44746) 2024-03-17T23:48:38.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.771+0000 7effe27fc640 1 -- 172.21.15.150:0/2037553985 learned_addr learned my addr 172.21.15.150:0/2037553985 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:38.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.771+0000 7effe2ffd640 1 --2- 172.21.15.150:0/2037553985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7effe4099e20 0x7effe414efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:38.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.771+0000 7effe37fe640 1 --2- 172.21.15.150:0/2037553985 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effe40b4ed0 0x7effe40cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:38.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.771+0000 7effe27fc640 1 -- 172.21.15.150:0/2037553985 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effe40b4ed0 msgr2=0x7effe40cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:38.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.771+0000 7effe27fc640 1 --2- 172.21.15.150:0/2037553985 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effe40b4ed0 0x7effe40cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.771+0000 7effe27fc640 1 -- 172.21.15.150:0/2037553985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7effe4099e20 msgr2=0x7effe414efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:38.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.771+0000 7effe27fc640 1 --2- 172.21.15.150:0/2037553985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7effe4099e20 0x7effe414efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.771+0000 7effe27fc640 1 -- 172.21.15.150:0/2037553985 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effe40b5350 con 0x7effe415e460 2024-03-17T23:48:38.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.771+0000 7effe27fc640 1 --2- 172.21.15.150:0/2037553985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe415e460 0x7effe40b4990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7effd402b930 tx=0x7effd40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:38.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.772+0000 7effcb7fe640 1 -- 172.21.15.150:0/2037553985 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd4040070 con 0x7effe415e460 2024-03-17T23:48:38.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.772+0000 7effcb7fe640 1 -- 172.21.15.150:0/2037553985 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effd40399f0 con 0x7effe415e460 2024-03-17T23:48:38.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.772+0000 7effe9359640 1 -- 172.21.15.150:0/2037553985 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effe40b5630 con 0x7effe415e460 2024-03-17T23:48:38.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.772+0000 7effcb7fe640 1 -- 172.21.15.150:0/2037553985 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd4035040 con 0x7effe415e460 2024-03-17T23:48:38.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.772+0000 7effe9359640 1 -- 172.21.15.150:0/2037553985 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effe40b5bf0 con 0x7effe415e460 2024-03-17T23:48:38.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.773+0000 7effe9359640 1 -- 172.21.15.150:0/2037553985 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effe40681a0 con 0x7effe415e460 2024-03-17T23:48:38.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.773+0000 7effcb7fe640 1 -- 172.21.15.150:0/2037553985 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7effd404b060 con 0x7effe415e460 2024-03-17T23:48:38.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.773+0000 7effcb7fe640 1 --2- 172.21.15.150:0/2037553985 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7effd006bc00 0x7effd006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:38.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.773+0000 7effcb7fe640 1 -- 172.21.15.150:0/2037553985 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7effd40ac4a0 con 0x7effe415e460 2024-03-17T23:48:38.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.774+0000 7effe2ffd640 1 --2- 172.21.15.150:0/2037553985 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7effd006bc00 0x7effd006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:38.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.774+0000 7effe2ffd640 1 --2- 172.21.15.150:0/2037553985 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7effd006bc00 0x7effd006e0c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7effe40ce480 tx=0x7effd8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:38.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.778+0000 7effcb7fe640 1 -- 172.21.15.150:0/2037553985 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7effd407a6a0 con 0x7effe415e460 2024-03-17T23:48:38.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.956+0000 7effe9359640 1 -- 172.21.15.150:0/2037553985 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7effe415de60 con 0x7effe415e460 2024-03-17T23:48:38.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.956+0000 7effcb7fe640 1 -- 172.21.15.150:0/2037553985 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v45) ==== 100+0+11 (secure 0 0 0) 0x7effd407e740 con 0x7effe415e460 2024-03-17T23:48:38.958 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:48:38.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.958+0000 7effe9359640 1 -- 172.21.15.150:0/2037553985 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7effd006bc00 msgr2=0x7effd006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:38.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.958+0000 7effe9359640 1 --2- 172.21.15.150:0/2037553985 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7effd006bc00 0x7effd006e0c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7effe40ce480 tx=0x7effd8009040 comp rx=0 tx=0).stop 2024-03-17T23:48:38.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.958+0000 7effe9359640 1 -- 172.21.15.150:0/2037553985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe415e460 msgr2=0x7effe40b4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:38.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.958+0000 7effe9359640 1 --2- 172.21.15.150:0/2037553985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe415e460 0x7effe40b4990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7effd402b930 tx=0x7effd40046c0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.959+0000 7effe9359640 1 -- 172.21.15.150:0/2037553985 shutdown_connections 2024-03-17T23:48:38.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.959+0000 7effe9359640 1 --2- 172.21.15.150:0/2037553985 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7effe40b4ed0 0x7effe40cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.959+0000 7effe9359640 1 --2- 172.21.15.150:0/2037553985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effe415e460 0x7effe40b4990 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.959+0000 7effe9359640 1 --2- 172.21.15.150:0/2037553985 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7effd006bc00 0x7effd006e0c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.959+0000 7effe9359640 1 --2- 172.21.15.150:0/2037553985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7effe4099e20 0x7effe414efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:38.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.959+0000 7effe9359640 1 -- 172.21.15.150:0/2037553985 >> 172.21.15.150:0/2037553985 conn(0x7effe40b3630 msgr2=0x7effe4070b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:38.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.959+0000 7effe9359640 1 -- 172.21.15.150:0/2037553985 shutdown_connections 2024-03-17T23:48:38.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:38.959+0000 7effe9359640 1 -- 172.21.15.150:0/2037553985 wait complete. 2024-03-17T23:48:38.971 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-17T23:48:39.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.072+0000 7f446f5b3640 1 Processor -- start 2024-03-17T23:48:39.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.072+0000 7f446f5b3640 1 -- start start 2024-03-17T23:48:39.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.073+0000 7f446f5b3640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f44680d3910 0x7f44680d0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:39.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.073+0000 7f446f5b3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f44680b9660 0x7f44680d0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:39.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.073+0000 7f446f5b3640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44680d2870 con 0x7f44681615d0 2024-03-17T23:48:39.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.073+0000 7f446f5b3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f44681619b0 con 0x7f44680b9660 2024-03-17T23:48:39.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.073+0000 7f446f5b3640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f44680b7e80 con 0x7f44680d3910 2024-03-17T23:48:39.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.073+0000 7f446db50640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f44680b9660 0x7f44680d0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:39.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.073+0000 7f446db50640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f44680b9660 0x7f44680d0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44756/0 (socket says 172.21.15.150:44756) 2024-03-17T23:48:39.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.073+0000 7f446db50640 1 -- 172.21.15.150:0/2510601857 learned_addr learned my addr 172.21.15.150:0/2510601857 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:39.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.074+0000 7f446cb4e640 1 --2- 172.21.15.150:0/2510601857 >> v2:172.21.15.157:3300/0 conn(0x7f44680d3910 0x7f44680d0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:39.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.074+0000 7f446db50640 1 -- 172.21.15.150:0/2510601857 >> v2:172.21.15.157:3300/0 conn(0x7f44680d3910 msgr2=0x7f44680d0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:39.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.074+0000 7f446db50640 1 --2- 172.21.15.150:0/2510601857 >> v2:172.21.15.157:3300/0 conn(0x7f44680d3910 0x7f44680d0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.074+0000 7f446db50640 1 -- 172.21.15.150:0/2510601857 >> v1:172.21.15.157:6789/0 conn(0x7f44681615d0 legacy=0x7f44680d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:39.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.074+0000 7f446db50640 1 -- 172.21.15.150:0/2510601857 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44680d2b50 con 0x7f44680b9660 2024-03-17T23:48:39.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.074+0000 7f446db50640 1 --2- 172.21.15.150:0/2510601857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44680b9660 0x7f44680d0c60 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f44680d0620 tx=0x7f446002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ed1e5ae713669dc server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:39.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.075+0000 7f445ffff640 1 -- 172.21.15.150:0/2510601857 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446002ca80 con 0x7f44680b9660 2024-03-17T23:48:39.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.075+0000 7f445ffff640 1 -- 172.21.15.150:0/2510601857 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f446002cc20 con 0x7f44680b9660 2024-03-17T23:48:39.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.075+0000 7f445ffff640 1 -- 172.21.15.150:0/2510601857 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44600343f0 con 0x7f44680b9660 2024-03-17T23:48:39.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.075+0000 7f446f5b3640 1 -- 172.21.15.150:0/2510601857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44680b9660 msgr2=0x7f44680d0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:39.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.075+0000 7f446f5b3640 1 --2- 172.21.15.150:0/2510601857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44680b9660 0x7f44680d0c60 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f44680d0620 tx=0x7f446002bee0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.075+0000 7f446f5b3640 1 -- 172.21.15.150:0/2510601857 shutdown_connections 2024-03-17T23:48:39.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.075+0000 7f446f5b3640 1 --2- 172.21.15.150:0/2510601857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44680b9660 0x7f44680d0c60 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.075+0000 7f446f5b3640 1 --2- 172.21.15.150:0/2510601857 >> v2:172.21.15.157:3300/0 conn(0x7f44680d3910 0x7f44680d0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.075+0000 7f446f5b3640 1 -- 172.21.15.150:0/2510601857 >> 172.21.15.150:0/2510601857 conn(0x7f44680730b0 msgr2=0x7f44680b5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:39.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.075+0000 7f446f5b3640 1 -- 172.21.15.150:0/2510601857 shutdown_connections 2024-03-17T23:48:39.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.075+0000 7f446f5b3640 1 -- 172.21.15.150:0/2510601857 wait complete. 2024-03-17T23:48:39.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.077+0000 7f446f5b3640 1 Processor -- start 2024-03-17T23:48:39.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.077+0000 7f446f5b3640 1 -- start start 2024-03-17T23:48:39.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.078+0000 7f446f5b3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44680b9660 0x7f44680cc400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:39.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.078+0000 7f446f5b3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44680d3910 0x7f4468159670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:39.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.078+0000 7f446f5b3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44681615d0 0x7f44681f3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:39.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.078+0000 7f446f5b3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f44680d33f0 con 0x7f44681615d0 2024-03-17T23:48:39.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.078+0000 7f446f5b3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f44680d3110 con 0x7f44680b9660 2024-03-17T23:48:39.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.078+0000 7f446f5b3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f44680b6e70 con 0x7f44680d3910 2024-03-17T23:48:39.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.079+0000 7f446db50640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44681615d0 0x7f44681f3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:39.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.079+0000 7f446db50640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44681615d0 0x7f44681f3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44758/0 (socket says 172.21.15.150:44758) 2024-03-17T23:48:39.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.079+0000 7f446db50640 1 -- 172.21.15.150:0/1182668840 learned_addr learned my addr 172.21.15.150:0/1182668840 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:39.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.079+0000 7f446db50640 1 -- 172.21.15.150:0/1182668840 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44680d3910 msgr2=0x7f4468159670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:39.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.079+0000 7f446db50640 1 --2- 172.21.15.150:0/1182668840 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44680d3910 0x7f4468159670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.079+0000 7f446db50640 1 -- 172.21.15.150:0/1182668840 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44680b9660 msgr2=0x7f44680cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:39.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.079+0000 7f446db50640 1 --2- 172.21.15.150:0/1182668840 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44680b9660 0x7f44680cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.079+0000 7f446db50640 1 -- 172.21.15.150:0/1182668840 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44680cc940 con 0x7f44681615d0 2024-03-17T23:48:39.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.079+0000 7f446db50640 1 --2- 172.21.15.150:0/1182668840 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44681615d0 0x7f44681f3aa0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f4460009f40 tx=0x7f4460004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:39.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.079+0000 7f445dffb640 1 -- 172.21.15.150:0/1182668840 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4460043020 con 0x7f44681615d0 2024-03-17T23:48:39.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.079+0000 7f445dffb640 1 -- 172.21.15.150:0/1182668840 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f446003e450 con 0x7f44681615d0 2024-03-17T23:48:39.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.079+0000 7f446f5b3640 1 -- 172.21.15.150:0/1182668840 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44680cb3c0 con 0x7f44681615d0 2024-03-17T23:48:39.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.079+0000 7f445dffb640 1 -- 172.21.15.150:0/1182668840 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4460009510 con 0x7f44681615d0 2024-03-17T23:48:39.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.079+0000 7f446f5b3640 1 -- 172.21.15.150:0/1182668840 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44680cbc00 con 0x7f44681615d0 2024-03-17T23:48:39.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.080+0000 7f446f5b3640 1 -- 172.21.15.150:0/1182668840 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44680688b0 con 0x7f44681615d0 2024-03-17T23:48:39.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.081+0000 7f445dffb640 1 -- 172.21.15.150:0/1182668840 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4460046020 con 0x7f44681615d0 2024-03-17T23:48:39.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.081+0000 7f445dffb640 1 --2- 172.21.15.150:0/1182668840 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f444c06bbd0 0x7f444c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:39.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.081+0000 7f445dffb640 1 -- 172.21.15.150:0/1182668840 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7f44600ad670 con 0x7f44681615d0 2024-03-17T23:48:39.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.081+0000 7f446d34f640 1 --2- 172.21.15.150:0/1182668840 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f444c06bbd0 0x7f444c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:39.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.082+0000 7f446d34f640 1 --2- 172.21.15.150:0/1182668840 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f444c06bbd0 0x7f444c06e090 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4464006fd0 tx=0x7f4464006bc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:39.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.086+0000 7f445dffb640 1 -- 172.21.15.150:0/1182668840 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f446004d030 con 0x7f44681615d0 2024-03-17T23:48:39.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.261+0000 7f446f5b3640 1 -- 172.21.15.150:0/1182668840 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7f44680ca1d0 con 0x7f44681615d0 2024-03-17T23:48:39.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.262+0000 7f445dffb640 1 -- 172.21.15.150:0/1182668840 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v45) ==== 97+0+11 (secure 0 0 0) 0x7f44680cb3c0 con 0x7f44681615d0 2024-03-17T23:48:39.263 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:48:39.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.264+0000 7f446f5b3640 1 -- 172.21.15.150:0/1182668840 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f444c06bbd0 msgr2=0x7f444c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:39.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.264+0000 7f446f5b3640 1 --2- 172.21.15.150:0/1182668840 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f444c06bbd0 0x7f444c06e090 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4464006fd0 tx=0x7f4464006bc0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.264+0000 7f446f5b3640 1 -- 172.21.15.150:0/1182668840 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44681615d0 msgr2=0x7f44681f3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:39.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.264+0000 7f446f5b3640 1 --2- 172.21.15.150:0/1182668840 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44681615d0 0x7f44681f3aa0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f4460009f40 tx=0x7f4460004a40 comp rx=0 tx=0).stop 2024-03-17T23:48:39.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.264+0000 7f446f5b3640 1 -- 172.21.15.150:0/1182668840 shutdown_connections 2024-03-17T23:48:39.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.264+0000 7f446f5b3640 1 --2- 172.21.15.150:0/1182668840 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44681615d0 0x7f44681f3aa0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.264+0000 7f446f5b3640 1 --2- 172.21.15.150:0/1182668840 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44680d3910 0x7f4468159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.264+0000 7f446f5b3640 1 --2- 172.21.15.150:0/1182668840 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f444c06bbd0 0x7f444c06e090 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.264+0000 7f446f5b3640 1 --2- 172.21.15.150:0/1182668840 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44680b9660 0x7f44680cc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.264+0000 7f446f5b3640 1 -- 172.21.15.150:0/1182668840 >> 172.21.15.150:0/1182668840 conn(0x7f44680730b0 msgr2=0x7f44680b5450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:39.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.264+0000 7f446f5b3640 1 -- 172.21.15.150:0/1182668840 shutdown_connections 2024-03-17T23:48:39.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.264+0000 7f446f5b3640 1 -- 172.21.15.150:0/1182668840 wait complete. 2024-03-17T23:48:39.279 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-17T23:48:39.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.380+0000 7faf10110640 1 Processor -- start 2024-03-17T23:48:39.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.380+0000 7faf10110640 1 -- start start 2024-03-17T23:48:39.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.381+0000 7faf10110640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7faf0815e100 0x7faf080d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:39.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.381+0000 7faf10110640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7faf080d2370 0x7faf0806c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:39.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.381+0000 7faf10110640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf0806d290 con 0x7faf0806ce80 2024-03-17T23:48:39.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.381+0000 7faf10110640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7faf080d28f0 con 0x7faf080d2370 2024-03-17T23:48:39.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.381+0000 7faf10110640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7faf080d2bd0 con 0x7faf0815e100 2024-03-17T23:48:39.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.381+0000 7faf0d6ab640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7faf080d2370 0x7faf0806c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:39.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.381+0000 7faf0d6ab640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7faf080d2370 0x7faf0806c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44770/0 (socket says 172.21.15.150:44770) 2024-03-17T23:48:39.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.381+0000 7faf0d6ab640 1 -- 172.21.15.150:0/2609299076 learned_addr learned my addr 172.21.15.150:0/2609299076 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:39.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.381+0000 7faf0e6ad640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7faf0806ce80 0x7faf080b5770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37932/0 (socket says 172.21.15.150:37932) 2024-03-17T23:48:39.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.381+0000 7faf0d6ab640 1 -- 172.21.15.150:0/2609299076 >> v2:172.21.15.157:3300/0 conn(0x7faf0815e100 msgr2=0x7faf080d1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:39.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.381+0000 7faf0deac640 1 --2- 172.21.15.150:0/2609299076 >> v2:172.21.15.157:3300/0 conn(0x7faf0815e100 0x7faf080d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:39.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.381+0000 7faf0d6ab640 1 --2- 172.21.15.150:0/2609299076 >> v2:172.21.15.157:3300/0 conn(0x7faf0815e100 0x7faf080d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.381+0000 7faf0d6ab640 1 -- 172.21.15.150:0/2609299076 >> v1:172.21.15.150:6789/0 conn(0x7faf0806ce80 legacy=0x7faf080b5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:39.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.381+0000 7faf0ceaa640 1 -- 172.21.15.150:0/2609299076 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2678636330 0 0) 0x7faf00001490 con 0x7faf0806ce80 2024-03-17T23:48:39.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.381+0000 7faf0d6ab640 1 -- 172.21.15.150:0/2609299076 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf080d2eb0 con 0x7faf080d2370 2024-03-17T23:48:39.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.382+0000 7faf0d6ab640 1 --2- 172.21.15.150:0/2609299076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf080d2370 0x7faf0806c940 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7faf0806d4e0 tx=0x7faef802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb5d1e556f73e64e server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:39.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.383+0000 7faf0ceaa640 1 -- 172.21.15.150:0/2609299076 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef802f850 con 0x7faf080d2370 2024-03-17T23:48:39.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.383+0000 7faf0ceaa640 1 -- 172.21.15.150:0/2609299076 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faef802f9f0 con 0x7faf080d2370 2024-03-17T23:48:39.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.383+0000 7faf0ceaa640 1 -- 172.21.15.150:0/2609299076 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef802fdc0 con 0x7faf080d2370 2024-03-17T23:48:39.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.383+0000 7faf10110640 1 -- 172.21.15.150:0/2609299076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf080d2370 msgr2=0x7faf0806c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:39.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.383+0000 7faf10110640 1 --2- 172.21.15.150:0/2609299076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf080d2370 0x7faf0806c940 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7faf0806d4e0 tx=0x7faef802ba30 comp rx=0 tx=0).stop 2024-03-17T23:48:39.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.383+0000 7faf10110640 1 -- 172.21.15.150:0/2609299076 shutdown_connections 2024-03-17T23:48:39.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.383+0000 7faf10110640 1 --2- 172.21.15.150:0/2609299076 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf080d2370 0x7faf0806c940 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.383+0000 7faf10110640 1 --2- 172.21.15.150:0/2609299076 >> v2:172.21.15.157:3300/0 conn(0x7faf0815e100 0x7faf080d1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.383+0000 7faf10110640 1 -- 172.21.15.150:0/2609299076 >> 172.21.15.150:0/2609299076 conn(0x7faf08072ce0 msgr2=0x7faf080730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:39.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.383+0000 7faf10110640 1 -- 172.21.15.150:0/2609299076 shutdown_connections 2024-03-17T23:48:39.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.383+0000 7faf10110640 1 -- 172.21.15.150:0/2609299076 wait complete. 2024-03-17T23:48:39.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.384+0000 7faf10110640 1 Processor -- start 2024-03-17T23:48:39.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.384+0000 7faf10110640 1 -- start start 2024-03-17T23:48:39.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.385+0000 7faf10110640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf0806ce80 0x7faf081501e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:39.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.385+0000 7faf10110640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf080d2370 0x7faf080b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:39.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.385+0000 7faf10110640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf0815e100 0x7faf080b5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:39.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.385+0000 7faf10110640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7faf080d3990 con 0x7faf0815e100 2024-03-17T23:48:39.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.385+0000 7faf10110640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7faf080d3470 con 0x7faf0806ce80 2024-03-17T23:48:39.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.385+0000 7faf10110640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7faf080b9780 con 0x7faf080d2370 2024-03-17T23:48:39.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.385+0000 7faf0e6ad640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf0815e100 0x7faf080b5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:39.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.385+0000 7faf0e6ad640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf0815e100 0x7faf080b5280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44776/0 (socket says 172.21.15.150:44776) 2024-03-17T23:48:39.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.385+0000 7faf0e6ad640 1 -- 172.21.15.150:0/3677617176 learned_addr learned my addr 172.21.15.150:0/3677617176 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:39.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.385+0000 7faf0deac640 1 --2- 172.21.15.150:0/3677617176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf0806ce80 0x7faf081501e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:39.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.385+0000 7faf0d6ab640 1 --2- 172.21.15.150:0/3677617176 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf080d2370 0x7faf080b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:39.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.385+0000 7faf0e6ad640 1 -- 172.21.15.150:0/3677617176 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf080d2370 msgr2=0x7faf080b4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:39.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.385+0000 7faf0e6ad640 1 --2- 172.21.15.150:0/3677617176 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf080d2370 0x7faf080b4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.385+0000 7faf0e6ad640 1 -- 172.21.15.150:0/3677617176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf0806ce80 msgr2=0x7faf081501e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:39.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.385+0000 7faf0e6ad640 1 --2- 172.21.15.150:0/3677617176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf0806ce80 0x7faf081501e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.385+0000 7faf0e6ad640 1 -- 172.21.15.150:0/3677617176 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf080cd740 con 0x7faf0815e100 2024-03-17T23:48:39.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.385+0000 7faf0d6ab640 1 --2- 172.21.15.150:0/3677617176 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf080d2370 0x7faf080b4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:39.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.386+0000 7faf0e6ad640 1 --2- 172.21.15.150:0/3677617176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf0815e100 0x7faf080b5280 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7faf0000c7f0 tx=0x7faf0000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:39.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.386+0000 7faefe7fc640 1 -- 172.21.15.150:0/3677617176 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf00018070 con 0x7faf0815e100 2024-03-17T23:48:39.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.386+0000 7faefe7fc640 1 -- 172.21.15.150:0/3677617176 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf0000a5e0 con 0x7faf0815e100 2024-03-17T23:48:39.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.386+0000 7faf10110640 1 -- 172.21.15.150:0/3677617176 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf0815e580 con 0x7faf0815e100 2024-03-17T23:48:39.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.386+0000 7faefe7fc640 1 -- 172.21.15.150:0/3677617176 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf0000aa10 con 0x7faf0815e100 2024-03-17T23:48:39.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.386+0000 7faf10110640 1 -- 172.21.15.150:0/3677617176 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf0815eb40 con 0x7faf0815e100 2024-03-17T23:48:39.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.387+0000 7faf10110640 1 -- 172.21.15.150:0/3677617176 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf080688b0 con 0x7faf0815e100 2024-03-17T23:48:39.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.387+0000 7faefe7fc640 1 -- 172.21.15.150:0/3677617176 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7faf0001b070 con 0x7faf0815e100 2024-03-17T23:48:39.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.388+0000 7faefe7fc640 1 --2- 172.21.15.150:0/3677617176 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7faeec06bbd0 0x7faeec06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:39.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.388+0000 7faf0deac640 1 --2- 172.21.15.150:0/3677617176 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7faeec06bbd0 0x7faeec06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:39.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.388+0000 7faefe7fc640 1 -- 172.21.15.150:0/3677617176 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7faf0008a2b0 con 0x7faf0815e100 2024-03-17T23:48:39.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.388+0000 7faf0deac640 1 --2- 172.21.15.150:0/3677617176 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7faeec06bbd0 0x7faeec06e090 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7faf04006fd0 tx=0x7faf04008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:39.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.392+0000 7faefe7fc640 1 -- 172.21.15.150:0/3677617176 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7faf0806d290 con 0x7faf0815e100 2024-03-17T23:48:39.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.613+0000 7faf10110640 1 -- 172.21.15.150:0/3677617176 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7faf0815ac40 con 0x7faf0815e100 2024-03-17T23:48:39.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.613+0000 7faefe7fc640 1 -- 172.21.15.150:0/3677617176 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v45) ==== 80+0+0 (secure 0 0 0) 0x7faf00058b00 con 0x7faf0815e100 2024-03-17T23:48:39.615 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-03-17T23:48:39.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.615+0000 7faf10110640 1 -- 172.21.15.150:0/3677617176 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7faeec06bbd0 msgr2=0x7faeec06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:39.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.615+0000 7faf10110640 1 --2- 172.21.15.150:0/3677617176 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7faeec06bbd0 0x7faeec06e090 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7faf04006fd0 tx=0x7faf04008040 comp rx=0 tx=0).stop 2024-03-17T23:48:39.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.615+0000 7faf10110640 1 -- 172.21.15.150:0/3677617176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf0815e100 msgr2=0x7faf080b5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:39.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.615+0000 7faf10110640 1 --2- 172.21.15.150:0/3677617176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf0815e100 0x7faf080b5280 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7faf0000c7f0 tx=0x7faf0000ccb0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.615+0000 7faf10110640 1 -- 172.21.15.150:0/3677617176 shutdown_connections 2024-03-17T23:48:39.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.615+0000 7faf10110640 1 --2- 172.21.15.150:0/3677617176 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf0815e100 0x7faf080b5280 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.615+0000 7faf10110640 1 --2- 172.21.15.150:0/3677617176 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faf080d2370 0x7faf080b4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.615+0000 7faf10110640 1 --2- 172.21.15.150:0/3677617176 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7faeec06bbd0 0x7faeec06e090 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.615+0000 7faf10110640 1 --2- 172.21.15.150:0/3677617176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faf0806ce80 0x7faf081501e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.615+0000 7faf10110640 1 -- 172.21.15.150:0/3677617176 >> 172.21.15.150:0/3677617176 conn(0x7faf08072ce0 msgr2=0x7faf080ce4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:39.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.616+0000 7faf10110640 1 -- 172.21.15.150:0/3677617176 shutdown_connections 2024-03-17T23:48:39.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.616+0000 7faf10110640 1 -- 172.21.15.150:0/3677617176 wait complete. 2024-03-17T23:48:39.628 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:48:39.628 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:48:39.643 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:48:39.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.783+0000 7f8a33f77640 1 Processor -- start 2024-03-17T23:48:39.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.783+0000 7f8a33f77640 1 -- start start 2024-03-17T23:48:39.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.784+0000 7f8a33f77640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8a2c0b31c0 0x7f8a2c0b4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:39.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.784+0000 7f8a33f77640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8a2c09a7d0 0x7f8a2c06d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:39.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.784+0000 7f8a33f77640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a2c0cc220 con 0x7f8a2c06ccb0 2024-03-17T23:48:39.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.784+0000 7f8a33f77640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f8a2c09ac00 con 0x7f8a2c0b31c0 2024-03-17T23:48:39.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.784+0000 7f8a33f77640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f8a2c0cc500 con 0x7f8a2c09a7d0 2024-03-17T23:48:39.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.784+0000 7f8a31d13640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8a2c0b31c0 0x7f8a2c0b4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:39.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.784+0000 7f8a31d13640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8a2c0b31c0 0x7f8a2c0b4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44778/0 (socket says 172.21.15.150:44778) 2024-03-17T23:48:39.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.784+0000 7f8a31d13640 1 -- 172.21.15.150:0/4225177456 learned_addr learned my addr 172.21.15.150:0/4225177456 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:39.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.784+0000 7f8a31512640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f8a2c06ccb0 0x7f8a2c09a0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37938/0 (socket says 172.21.15.150:37938) 2024-03-17T23:48:39.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.784+0000 7f8a32514640 1 --2- 172.21.15.150:0/4225177456 >> v2:172.21.15.157:3300/0 conn(0x7f8a2c09a7d0 0x7f8a2c06d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:39.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.784+0000 7f8a31d13640 1 -- 172.21.15.150:0/4225177456 >> v2:172.21.15.157:3300/0 conn(0x7f8a2c09a7d0 msgr2=0x7f8a2c06d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:39.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.784+0000 7f8a31d13640 1 --2- 172.21.15.150:0/4225177456 >> v2:172.21.15.157:3300/0 conn(0x7f8a2c09a7d0 0x7f8a2c06d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.784+0000 7f8a31d13640 1 -- 172.21.15.150:0/4225177456 >> v1:172.21.15.150:6789/0 conn(0x7f8a2c06ccb0 legacy=0x7f8a2c09a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:39.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.784+0000 7f8a30d11640 1 -- 172.21.15.150:0/4225177456 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3867888948 0 0) 0x7f8a2c0cc220 con 0x7f8a2c06ccb0 2024-03-17T23:48:39.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.785+0000 7f8a31d13640 1 -- 172.21.15.150:0/4225177456 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a2c0cc7e0 con 0x7f8a2c0b31c0 2024-03-17T23:48:39.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.785+0000 7f8a32514640 1 --2- 172.21.15.150:0/4225177456 >> v2:172.21.15.157:3300/0 conn(0x7f8a2c09a7d0 0x7f8a2c06d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:39.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.786+0000 7f8a31d13640 1 --2- 172.21.15.150:0/4225177456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a2c0b31c0 0x7f8a2c0b4980 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c0ce560 tx=0x7f8a2802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e2a7181b05c1f3d server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:39.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.786+0000 7f8a30d11640 1 -- 172.21.15.150:0/4225177456 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a28034070 con 0x7f8a2c0b31c0 2024-03-17T23:48:39.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.786+0000 7f8a30d11640 1 -- 172.21.15.150:0/4225177456 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a28030860 con 0x7f8a2c0b31c0 2024-03-17T23:48:39.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.786+0000 7f8a30d11640 1 -- 172.21.15.150:0/4225177456 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a28030c30 con 0x7f8a2c0b31c0 2024-03-17T23:48:39.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.786+0000 7f8a33f77640 1 -- 172.21.15.150:0/4225177456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a2c0b31c0 msgr2=0x7f8a2c0b4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:39.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.786+0000 7f8a33f77640 1 --2- 172.21.15.150:0/4225177456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a2c0b31c0 0x7f8a2c0b4980 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c0ce560 tx=0x7f8a2802b9a0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.786+0000 7f8a33f77640 1 -- 172.21.15.150:0/4225177456 shutdown_connections 2024-03-17T23:48:39.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.786+0000 7f8a33f77640 1 --2- 172.21.15.150:0/4225177456 >> v2:172.21.15.157:3300/0 conn(0x7f8a2c09a7d0 0x7f8a2c06d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.786+0000 7f8a33f77640 1 --2- 172.21.15.150:0/4225177456 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a2c0b31c0 0x7f8a2c0b4980 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.786+0000 7f8a33f77640 1 -- 172.21.15.150:0/4225177456 >> 172.21.15.150:0/4225177456 conn(0x7f8a2c0b3620 msgr2=0x7f8a2c0b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:39.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.786+0000 7f8a33f77640 1 -- 172.21.15.150:0/4225177456 shutdown_connections 2024-03-17T23:48:39.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.787+0000 7f8a33f77640 1 -- 172.21.15.150:0/4225177456 wait complete. 2024-03-17T23:48:39.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.788+0000 7f8a33f77640 1 Processor -- start 2024-03-17T23:48:39.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.788+0000 7f8a33f77640 1 -- start start 2024-03-17T23:48:39.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.788+0000 7f8a33f77640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a2c06ccb0 0x7f8a2c0c8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:39.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.788+0000 7f8a33f77640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a2c09a7d0 0x7f8a2c15dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:39.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.788+0000 7f8a33f77640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a2c0b31c0 0x7f8a2c14ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:39.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.788+0000 7f8a33f77640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2c15d7a0 con 0x7f8a2c06ccb0 2024-03-17T23:48:39.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.788+0000 7f8a33f77640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2c0ccda0 con 0x7f8a2c0b31c0 2024-03-17T23:48:39.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.788+0000 7f8a33f77640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8a2c15d920 con 0x7f8a2c09a7d0 2024-03-17T23:48:39.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.788+0000 7f8a31d13640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a2c06ccb0 0x7f8a2c0c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:39.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.788+0000 7f8a31d13640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a2c06ccb0 0x7f8a2c0c8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44780/0 (socket says 172.21.15.150:44780) 2024-03-17T23:48:39.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.788+0000 7f8a31d13640 1 -- 172.21.15.150:0/1418224592 learned_addr learned my addr 172.21.15.150:0/1418224592 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:39.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.789+0000 7f8a31512640 1 --2- 172.21.15.150:0/1418224592 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a2c09a7d0 0x7f8a2c15dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:39.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.789+0000 7f8a32514640 1 --2- 172.21.15.150:0/1418224592 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a2c0b31c0 0x7f8a2c14ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:39.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.789+0000 7f8a31d13640 1 -- 172.21.15.150:0/1418224592 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a2c09a7d0 msgr2=0x7f8a2c15dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:39.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.789+0000 7f8a31d13640 1 --2- 172.21.15.150:0/1418224592 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a2c09a7d0 0x7f8a2c15dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.789+0000 7f8a31d13640 1 -- 172.21.15.150:0/1418224592 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a2c0b31c0 msgr2=0x7f8a2c14ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:39.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.789+0000 7f8a31d13640 1 --2- 172.21.15.150:0/1418224592 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a2c0b31c0 0x7f8a2c14ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:39.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.789+0000 7f8a31d13640 1 -- 172.21.15.150:0/1418224592 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a2c0ca9e0 con 0x7f8a2c06ccb0 2024-03-17T23:48:39.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.789+0000 7f8a31512640 1 --2- 172.21.15.150:0/1418224592 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a2c09a7d0 0x7f8a2c15dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:39.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.789+0000 7f8a32514640 1 --2- 172.21.15.150:0/1418224592 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a2c0b31c0 0x7f8a2c14ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:39.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.789+0000 7f8a31d13640 1 --2- 172.21.15.150:0/1418224592 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a2c06ccb0 0x7f8a2c0c8030 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8a28000c00 tx=0x7f8a280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:39.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.789+0000 7f8a227fc640 1 -- 172.21.15.150:0/1418224592 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a28034040 con 0x7f8a2c06ccb0 2024-03-17T23:48:39.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.789+0000 7f8a227fc640 1 -- 172.21.15.150:0/1418224592 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a2803ab90 con 0x7f8a2c06ccb0 2024-03-17T23:48:39.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.789+0000 7f8a227fc640 1 -- 172.21.15.150:0/1418224592 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a28004030 con 0x7f8a2c06ccb0 2024-03-17T23:48:39.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.789+0000 7f8a33f77640 1 -- 172.21.15.150:0/1418224592 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a2c0cacc0 con 0x7f8a2c06ccb0 2024-03-17T23:48:39.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.789+0000 7f8a33f77640 1 -- 172.21.15.150:0/1418224592 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a2c0cb280 con 0x7f8a2c06ccb0 2024-03-17T23:48:39.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.790+0000 7f8a33f77640 1 -- 172.21.15.150:0/1418224592 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a2c0688b0 con 0x7f8a2c06ccb0 2024-03-17T23:48:39.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.791+0000 7f8a227fc640 1 -- 172.21.15.150:0/1418224592 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8a28037070 con 0x7f8a2c06ccb0 2024-03-17T23:48:39.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.791+0000 7f8a227fc640 1 --2- 172.21.15.150:0/1418224592 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8a1006bbd0 0x7f8a1006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:39.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.791+0000 7f8a31512640 1 --2- 172.21.15.150:0/1418224592 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8a1006bbd0 0x7f8a1006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:39.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.791+0000 7f8a227fc640 1 -- 172.21.15.150:0/1418224592 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7f8a280acd30 con 0x7f8a2c06ccb0 2024-03-17T23:48:39.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.791+0000 7f8a31512640 1 --2- 172.21.15.150:0/1418224592 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8a1006bbd0 0x7f8a1006e090 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c0b4870 tx=0x7f8a1c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:39.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:39.795+0000 7f8a227fc640 1 -- 172.21.15.150:0/1418224592 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8a2803f070 con 0x7f8a2c06ccb0 2024-03-17T23:48:40.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.072+0000 7f8a33f77640 1 -- 172.21.15.150:0/1418224592 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8a2c15f840 con 0x7f8a2c06ccb0 2024-03-17T23:48:40.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.074+0000 7f8a227fc640 1 -- 172.21.15.150:0/1418224592 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+18839 (secure 0 0 0) 0x7f8a2807b500 con 0x7f8a2c06ccb0 2024-03-17T23:48:40.075 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:40.075 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":45,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:48:37.677776+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:48:08.957879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-17T23:48:09.698808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-17T23:48:22.562269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.3d","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:48:40.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.076+0000 7f8a33f77640 1 -- 172.21.15.150:0/1418224592 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8a1006bbd0 msgr2=0x7f8a1006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:40.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.076+0000 7f8a33f77640 1 --2- 172.21.15.150:0/1418224592 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8a1006bbd0 0x7f8a1006e090 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c0b4870 tx=0x7f8a1c009040 comp rx=0 tx=0).stop 2024-03-17T23:48:40.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.076+0000 7f8a33f77640 1 -- 172.21.15.150:0/1418224592 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a2c06ccb0 msgr2=0x7f8a2c0c8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:40.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.076+0000 7f8a33f77640 1 --2- 172.21.15.150:0/1418224592 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a2c06ccb0 0x7f8a2c0c8030 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8a28000c00 tx=0x7f8a280046c0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.076+0000 7f8a33f77640 1 -- 172.21.15.150:0/1418224592 shutdown_connections 2024-03-17T23:48:40.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.076+0000 7f8a33f77640 1 --2- 172.21.15.150:0/1418224592 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8a1006bbd0 0x7f8a1006e090 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.076+0000 7f8a33f77640 1 --2- 172.21.15.150:0/1418224592 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a2c0b31c0 0x7f8a2c14ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.076+0000 7f8a33f77640 1 --2- 172.21.15.150:0/1418224592 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8a2c09a7d0 0x7f8a2c15dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.076+0000 7f8a33f77640 1 --2- 172.21.15.150:0/1418224592 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8a2c06ccb0 0x7f8a2c0c8030 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.076+0000 7f8a33f77640 1 -- 172.21.15.150:0/1418224592 >> 172.21.15.150:0/1418224592 conn(0x7f8a2c0b3620 msgr2=0x7f8a2c0ce5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:40.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.077+0000 7f8a33f77640 1 -- 172.21.15.150:0/1418224592 shutdown_connections 2024-03-17T23:48:40.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.077+0000 7f8a33f77640 1 -- 172.21.15.150:0/1418224592 wait complete. 2024-03-17T23:48:40.089 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:48:08.957879+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:48:09.698808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-17T23:48:22.562269+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-17T23:48:40.089 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:48:40.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.193+0000 7ff001c43640 1 Processor -- start 2024-03-17T23:48:40.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.193+0000 7ff001c43640 1 -- start start 2024-03-17T23:48:40.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.194+0000 7ff001c43640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7feffc0d3910 0x7feffc0d0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:40.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.194+0000 7ff001c43640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feffc0b9660 0x7feffc0d0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:40.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.194+0000 7ff001c43640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feffc0d2870 con 0x7feffc1615d0 2024-03-17T23:48:40.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.194+0000 7ff001c43640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7feffc1619b0 con 0x7feffc0d3910 2024-03-17T23:48:40.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.194+0000 7ff001c43640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7feffc0b7e80 con 0x7feffc0b9660 2024-03-17T23:48:40.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.194+0000 7feffbfff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feffc0b9660 0x7feffc0d0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:40.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.194+0000 7feffbfff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feffc0b9660 0x7feffc0d0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:47494/0 (socket says 172.21.15.150:47494) 2024-03-17T23:48:40.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.194+0000 7feff3fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7feffc0d3910 0x7feffc0d0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:40.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.194+0000 7feffbfff640 1 -- 172.21.15.150:0/1804610549 learned_addr learned my addr 172.21.15.150:0/1804610549 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:40.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.194+0000 7feffbfff640 1 -- 172.21.15.150:0/1804610549 >> v2:172.21.15.157:3300/0 conn(0x7feffc0d3910 msgr2=0x7feffc0d0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:40.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.194+0000 7feffbfff640 1 --2- 172.21.15.150:0/1804610549 >> v2:172.21.15.157:3300/0 conn(0x7feffc0d3910 0x7feffc0d0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.194+0000 7feffbfff640 1 -- 172.21.15.150:0/1804610549 >> v1:172.21.15.157:6789/0 conn(0x7feffc1615d0 legacy=0x7feffc0d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:40.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.194+0000 7feffbfff640 1 -- 172.21.15.150:0/1804610549 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feffc0d2b50 con 0x7feffc0b9660 2024-03-17T23:48:40.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.195+0000 7feffbfff640 1 --2- 172.21.15.150:0/1804610549 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feffc0b9660 0x7feffc0d0c60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7feff4009e50 tx=0x7feff402bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88cf69c9af19089f server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:40.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.195+0000 7feffaffd640 1 -- 172.21.15.150:0/1804610549 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff402ca70 con 0x7feffc0b9660 2024-03-17T23:48:40.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.195+0000 7feffaffd640 1 -- 172.21.15.150:0/1804610549 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feff402cc10 con 0x7feffc0b9660 2024-03-17T23:48:40.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.195+0000 7feffaffd640 1 -- 172.21.15.150:0/1804610549 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff40343f0 con 0x7feffc0b9660 2024-03-17T23:48:40.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.196+0000 7ff001c43640 1 -- 172.21.15.150:0/1804610549 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feffc0b9660 msgr2=0x7feffc0d0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:40.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.196+0000 7ff001c43640 1 --2- 172.21.15.150:0/1804610549 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feffc0b9660 0x7feffc0d0c60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7feff4009e50 tx=0x7feff402bed0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.196+0000 7ff001c43640 1 -- 172.21.15.150:0/1804610549 shutdown_connections 2024-03-17T23:48:40.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.196+0000 7ff001c43640 1 --2- 172.21.15.150:0/1804610549 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feffc0b9660 0x7feffc0d0c60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.196+0000 7ff001c43640 1 --2- 172.21.15.150:0/1804610549 >> v2:172.21.15.157:3300/0 conn(0x7feffc0d3910 0x7feffc0d0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.196+0000 7ff001c43640 1 -- 172.21.15.150:0/1804610549 >> 172.21.15.150:0/1804610549 conn(0x7feffc0730b0 msgr2=0x7feffc0b5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:40.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.196+0000 7ff001c43640 1 -- 172.21.15.150:0/1804610549 shutdown_connections 2024-03-17T23:48:40.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.196+0000 7ff001c43640 1 -- 172.21.15.150:0/1804610549 wait complete. 2024-03-17T23:48:40.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.197+0000 7ff001c43640 1 Processor -- start 2024-03-17T23:48:40.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.197+0000 7ff001c43640 1 -- start start 2024-03-17T23:48:40.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.198+0000 7ff001c43640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feffc0b9660 0x7feffc156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:40.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.198+0000 7ff001c43640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feffc0d3910 0x7feffc0cc780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:40.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.198+0000 7ff001c43640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feffc1615d0 0x7feffc15d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:40.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.198+0000 7ff001c43640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7feffc0d33f0 con 0x7feffc0b9660 2024-03-17T23:48:40.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.198+0000 7ff001c43640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7feffc0d3110 con 0x7feffc0d3910 2024-03-17T23:48:40.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.198+0000 7ff001c43640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7feffc0b6e70 con 0x7feffc1615d0 2024-03-17T23:48:40.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.198+0000 7feffb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feffc0b9660 0x7feffc156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:40.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.198+0000 7feffb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feffc0b9660 0x7feffc156790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44782/0 (socket says 172.21.15.150:44782) 2024-03-17T23:48:40.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.198+0000 7feffb7fe640 1 -- 172.21.15.150:0/1627035784 learned_addr learned my addr 172.21.15.150:0/1627035784 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:40.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.198+0000 7feff3fff640 1 --2- 172.21.15.150:0/1627035784 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feffc0d3910 0x7feffc0cc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:40.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.198+0000 7feffbfff640 1 --2- 172.21.15.150:0/1627035784 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feffc1615d0 0x7feffc15d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:40.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.198+0000 7feffb7fe640 1 -- 172.21.15.150:0/1627035784 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feffc1615d0 msgr2=0x7feffc15d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:40.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.198+0000 7feffb7fe640 1 --2- 172.21.15.150:0/1627035784 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feffc1615d0 0x7feffc15d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.198+0000 7feffb7fe640 1 -- 172.21.15.150:0/1627035784 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feffc0d3910 msgr2=0x7feffc0cc780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:40.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.198+0000 7feffb7fe640 1 --2- 172.21.15.150:0/1627035784 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feffc0d3910 0x7feffc0cc780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.198+0000 7feffb7fe640 1 -- 172.21.15.150:0/1627035784 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feffc1559d0 con 0x7feffc0b9660 2024-03-17T23:48:40.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.199+0000 7feffb7fe640 1 --2- 172.21.15.150:0/1627035784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feffc0b9660 0x7feffc156790 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fefec00c3c0 tx=0x7fefec00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:40.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.199+0000 7feff8ff9640 1 -- 172.21.15.150:0/1627035784 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec0052d0 con 0x7feffc0b9660 2024-03-17T23:48:40.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.199+0000 7feff8ff9640 1 -- 172.21.15.150:0/1627035784 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefec005490 con 0x7feffc0b9660 2024-03-17T23:48:40.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.199+0000 7ff001c43640 1 -- 172.21.15.150:0/1627035784 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feffc155cb0 con 0x7feffc0b9660 2024-03-17T23:48:40.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.199+0000 7feff8ff9640 1 -- 172.21.15.150:0/1627035784 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec007260 con 0x7feffc0b9660 2024-03-17T23:48:40.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.199+0000 7ff001c43640 1 -- 172.21.15.150:0/1627035784 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feffc156270 con 0x7feffc0b9660 2024-03-17T23:48:40.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.200+0000 7ff001c43640 1 -- 172.21.15.150:0/1627035784 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feffc156470 con 0x7feffc0b9660 2024-03-17T23:48:40.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.200+0000 7feff8ff9640 1 -- 172.21.15.150:0/1627035784 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fefec007400 con 0x7feffc0b9660 2024-03-17T23:48:40.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.201+0000 7feff8ff9640 1 --2- 172.21.15.150:0/1627035784 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fefdc06bbd0 0x7fefdc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:40.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.201+0000 7feff8ff9640 1 -- 172.21.15.150:0/1627035784 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7fefec08a280 con 0x7feffc0b9660 2024-03-17T23:48:40.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.201+0000 7feff3fff640 1 --2- 172.21.15.150:0/1627035784 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fefdc06bbd0 0x7fefdc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:40.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.201+0000 7feff3fff640 1 --2- 172.21.15.150:0/1627035784 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fefdc06bbd0 0x7fefdc06e090 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7feffc0cbff0 tx=0x7fefe4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:40.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.205+0000 7feff8ff9640 1 -- 172.21.15.150:0/1627035784 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7feffc156470 con 0x7feffc0b9660 2024-03-17T23:48:40.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.436+0000 7ff001c43640 1 -- 172.21.15.150:0/1627035784 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7feffc0688b0 con 0x7feffc0b9660 2024-03-17T23:48:40.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.436+0000 7feff8ff9640 1 -- 172.21.15.150:0/1627035784 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v45) ==== 93+0+10 (secure 0 0 0) 0x7fefec058a40 con 0x7feffc0b9660 2024-03-17T23:48:40.438 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:48:40.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.438+0000 7ff001c43640 1 -- 172.21.15.150:0/1627035784 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fefdc06bbd0 msgr2=0x7fefdc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:40.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.438+0000 7ff001c43640 1 --2- 172.21.15.150:0/1627035784 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fefdc06bbd0 0x7fefdc06e090 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7feffc0cbff0 tx=0x7fefe4008040 comp rx=0 tx=0).stop 2024-03-17T23:48:40.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.438+0000 7ff001c43640 1 -- 172.21.15.150:0/1627035784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feffc0b9660 msgr2=0x7feffc156790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:40.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.438+0000 7ff001c43640 1 --2- 172.21.15.150:0/1627035784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feffc0b9660 0x7feffc156790 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fefec00c3c0 tx=0x7fefec00c880 comp rx=0 tx=0).stop 2024-03-17T23:48:40.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.438+0000 7ff001c43640 1 -- 172.21.15.150:0/1627035784 shutdown_connections 2024-03-17T23:48:40.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.438+0000 7ff001c43640 1 --2- 172.21.15.150:0/1627035784 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feffc1615d0 0x7feffc15d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.438+0000 7ff001c43640 1 --2- 172.21.15.150:0/1627035784 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fefdc06bbd0 0x7fefdc06e090 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.438+0000 7ff001c43640 1 --2- 172.21.15.150:0/1627035784 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feffc0d3910 0x7feffc0cc780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.438+0000 7ff001c43640 1 --2- 172.21.15.150:0/1627035784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feffc0b9660 0x7feffc156790 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.438+0000 7ff001c43640 1 -- 172.21.15.150:0/1627035784 >> 172.21.15.150:0/1627035784 conn(0x7feffc0730b0 msgr2=0x7feffc0ce790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:40.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.438+0000 7ff001c43640 1 -- 172.21.15.150:0/1627035784 shutdown_connections 2024-03-17T23:48:40.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.438+0000 7ff001c43640 1 -- 172.21.15.150:0/1627035784 wait complete. 2024-03-17T23:48:40.452 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:48:40.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.555+0000 7f3781681640 1 Processor -- start 2024-03-17T23:48:40.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.555+0000 7f3781681640 1 -- start start 2024-03-17T23:48:40.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.556+0000 7f3781681640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f377c0d3910 0x7f377c0d0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:40.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.556+0000 7f3781681640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f377c0d2870 con 0x7f377c1615d0 2024-03-17T23:48:40.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.556+0000 7f3781681640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f377c0d2b50 con 0x7f377c0b9660 2024-03-17T23:48:40.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.556+0000 7f3781681640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f377c1619b0 con 0x7f377c0d3910 2024-03-17T23:48:40.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.556+0000 7f377affd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f377c1615d0 0x7f377c0d2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37946/0 (socket says 172.21.15.150:37946) 2024-03-17T23:48:40.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.556+0000 7f377affd640 1 -- 172.21.15.150:0/2084277433 learned_addr learned my addr 172.21.15.150:0/2084277433 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:40.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.557+0000 7f377a7fc640 1 --2- 172.21.15.150:0/2084277433 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f377c0d3910 0x7f377c0d0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:40.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.557+0000 7f3779ffb640 1 -- 172.21.15.150:0/2084277433 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1804770292 0 0) 0x7f377c0d2870 con 0x7f377c1615d0 2024-03-17T23:48:40.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.557+0000 7f3779ffb640 1 -- 172.21.15.150:0/2084277433 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3764003520 con 0x7f377c1615d0 2024-03-17T23:48:40.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.557+0000 7f3779ffb640 1 -- 172.21.15.150:0/2084277433 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3539499353 0 0) 0x7f377c0d2b50 con 0x7f377c0b9660 2024-03-17T23:48:40.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.557+0000 7f3779ffb640 1 -- 172.21.15.150:0/2084277433 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f377c0d2870 con 0x7f377c0b9660 2024-03-17T23:48:40.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.557+0000 7f3779ffb640 1 -- 172.21.15.150:0/2084277433 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1582128907 0 0) 0x7f3764003520 con 0x7f377c1615d0 2024-03-17T23:48:40.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.557+0000 7f377a7fc640 1 -- 172.21.15.150:0/2084277433 >> v1:172.21.15.157:6789/0 conn(0x7f377c0b9660 legacy=0x7f377c0b6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:40.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.557+0000 7f377a7fc640 1 -- 172.21.15.150:0/2084277433 >> v1:172.21.15.150:6789/0 conn(0x7f377c1615d0 legacy=0x7f377c0d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:40.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.557+0000 7f377a7fc640 1 -- 172.21.15.150:0/2084277433 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f377c0d2e30 con 0x7f377c0d3910 2024-03-17T23:48:40.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.558+0000 7f377a7fc640 1 --2- 172.21.15.150:0/2084277433 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f377c0d3910 0x7f377c0d0720 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f37700097d0 tx=0x7f377002b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4ea62103bf19109 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:40.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.558+0000 7f3779ffb640 1 -- 172.21.15.150:0/2084277433 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3770037070 con 0x7f377c0d3910 2024-03-17T23:48:40.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.558+0000 7f3779ffb640 1 -- 172.21.15.150:0/2084277433 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3770030510 con 0x7f377c0d3910 2024-03-17T23:48:40.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.558+0000 7f3779ffb640 1 -- 172.21.15.150:0/2084277433 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f376c001670 con 0x7f377c0d3910 2024-03-17T23:48:40.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.558+0000 7f3781681640 1 -- 172.21.15.150:0/2084277433 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f377c0d3910 msgr2=0x7f377c0d0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:40.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.558+0000 7f3781681640 1 --2- 172.21.15.150:0/2084277433 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f377c0d3910 0x7f377c0d0720 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f37700097d0 tx=0x7f377002b800 comp rx=0 tx=0).stop 2024-03-17T23:48:40.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.559+0000 7f3781681640 1 -- 172.21.15.150:0/2084277433 shutdown_connections 2024-03-17T23:48:40.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.559+0000 7f3781681640 1 --2- 172.21.15.150:0/2084277433 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f377c0d3910 0x7f377c0d0720 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.559+0000 7f3781681640 1 -- 172.21.15.150:0/2084277433 >> 172.21.15.150:0/2084277433 conn(0x7f377c0730b0 msgr2=0x7f377c0b5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:40.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.559+0000 7f3781681640 1 -- 172.21.15.150:0/2084277433 shutdown_connections 2024-03-17T23:48:40.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.559+0000 7f3781681640 1 -- 172.21.15.150:0/2084277433 wait complete. 2024-03-17T23:48:40.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.561+0000 7f3781681640 1 Processor -- start 2024-03-17T23:48:40.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.561+0000 7f3781681640 1 -- start start 2024-03-17T23:48:40.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.562+0000 7f3781681640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f377c0b9660 0x7f377c14efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:40.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.562+0000 7f3781681640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f377c0d3910 0x7f377c1594e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:40.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.562+0000 7f3781681640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f377c1615d0 0x7f377c159a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:40.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.562+0000 7f3781681640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f377c0d0c60 con 0x7f377c1615d0 2024-03-17T23:48:40.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.562+0000 7f3781681640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f377c0d33f0 con 0x7f377c0d3910 2024-03-17T23:48:40.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.562+0000 7f3781681640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f377c0d0de0 con 0x7f377c0b9660 2024-03-17T23:48:40.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.562+0000 7f377b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f377c1615d0 0x7f377c159a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:40.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.562+0000 7f377b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f377c1615d0 0x7f377c159a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44788/0 (socket says 172.21.15.150:44788) 2024-03-17T23:48:40.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.562+0000 7f377b7fe640 1 -- 172.21.15.150:0/3805705778 learned_addr learned my addr 172.21.15.150:0/3805705778 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:40.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.562+0000 7f377affd640 1 --2- 172.21.15.150:0/3805705778 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f377c0b9660 0x7f377c14efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:40.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.562+0000 7f377a7fc640 1 --2- 172.21.15.150:0/3805705778 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f377c0d3910 0x7f377c1594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:40.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.562+0000 7f377b7fe640 1 -- 172.21.15.150:0/3805705778 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f377c0b9660 msgr2=0x7f377c14efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:40.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.562+0000 7f377b7fe640 1 --2- 172.21.15.150:0/3805705778 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f377c0b9660 0x7f377c14efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.562+0000 7f377b7fe640 1 -- 172.21.15.150:0/3805705778 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f377c0d3910 msgr2=0x7f377c1594e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:40.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.562+0000 7f377b7fe640 1 --2- 172.21.15.150:0/3805705778 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f377c0d3910 0x7f377c1594e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.562+0000 7f377b7fe640 1 -- 172.21.15.150:0/3805705778 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f377c159f60 con 0x7f377c1615d0 2024-03-17T23:48:40.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.562+0000 7f377b7fe640 1 --2- 172.21.15.150:0/3805705778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f377c1615d0 0x7f377c159a20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f377400c790 tx=0x7f377400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:40.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.563+0000 7f37637fe640 1 -- 172.21.15.150:0/3805705778 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3774003b70 con 0x7f377c1615d0 2024-03-17T23:48:40.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.563+0000 7f37637fe640 1 -- 172.21.15.150:0/3805705778 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3774003d10 con 0x7f377c1615d0 2024-03-17T23:48:40.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.563+0000 7f3781681640 1 -- 172.21.15.150:0/3805705778 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f377c15a210 con 0x7f377c1615d0 2024-03-17T23:48:40.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.563+0000 7f37637fe640 1 -- 172.21.15.150:0/3805705778 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377400aa50 con 0x7f377c1615d0 2024-03-17T23:48:40.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.563+0000 7f3781681640 1 -- 172.21.15.150:0/3805705778 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f377c15a7d0 con 0x7f377c1615d0 2024-03-17T23:48:40.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.564+0000 7f3781681640 1 -- 172.21.15.150:0/3805705778 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f377c0688b0 con 0x7f377c1615d0 2024-03-17T23:48:40.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.564+0000 7f37637fe640 1 -- 172.21.15.150:0/3805705778 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f377400abf0 con 0x7f377c1615d0 2024-03-17T23:48:40.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.565+0000 7f37637fe640 1 --2- 172.21.15.150:0/3805705778 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f376806bbd0 0x7f376806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:40.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.565+0000 7f37637fe640 1 -- 172.21.15.150:0/3805705778 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7f377408ac60 con 0x7f377c1615d0 2024-03-17T23:48:40.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.565+0000 7f377affd640 1 --2- 172.21.15.150:0/3805705778 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f376806bbd0 0x7f376806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:40.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.565+0000 7f377affd640 1 --2- 172.21.15.150:0/3805705778 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f376806bbd0 0x7f376806e090 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f376c0011e0 tx=0x7f376c001690 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:40.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.573+0000 7f37637fe640 1 -- 172.21.15.150:0/3805705778 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3774058d40 con 0x7f377c1615d0 2024-03-17T23:48:40.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.752+0000 7f3781681640 1 -- 172.21.15.150:0/3805705778 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f377c0b9250 con 0x7f377c1615d0 2024-03-17T23:48:40.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.752+0000 7f37637fe640 1 -- 172.21.15.150:0/3805705778 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v45) ==== 104+0+11 (secure 0 0 0) 0x7f377405cde0 con 0x7f377c1615d0 2024-03-17T23:48:40.754 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:48:40.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.754+0000 7f3781681640 1 -- 172.21.15.150:0/3805705778 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f376806bbd0 msgr2=0x7f376806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:40.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.754+0000 7f3781681640 1 --2- 172.21.15.150:0/3805705778 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f376806bbd0 0x7f376806e090 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f376c0011e0 tx=0x7f376c001690 comp rx=0 tx=0).stop 2024-03-17T23:48:40.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.754+0000 7f3781681640 1 -- 172.21.15.150:0/3805705778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f377c1615d0 msgr2=0x7f377c159a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:40.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.754+0000 7f3781681640 1 --2- 172.21.15.150:0/3805705778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f377c1615d0 0x7f377c159a20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f377400c790 tx=0x7f377400cc50 comp rx=0 tx=0).stop 2024-03-17T23:48:40.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.755+0000 7f3781681640 1 -- 172.21.15.150:0/3805705778 shutdown_connections 2024-03-17T23:48:40.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.755+0000 7f3781681640 1 --2- 172.21.15.150:0/3805705778 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f377c1615d0 0x7f377c159a20 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.755+0000 7f3781681640 1 --2- 172.21.15.150:0/3805705778 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f376806bbd0 0x7f376806e090 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.755+0000 7f3781681640 1 --2- 172.21.15.150:0/3805705778 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f377c0d3910 0x7f377c1594e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.755+0000 7f3781681640 1 --2- 172.21.15.150:0/3805705778 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f377c0b9660 0x7f377c14efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.755+0000 7f3781681640 1 -- 172.21.15.150:0/3805705778 >> 172.21.15.150:0/3805705778 conn(0x7f377c0730b0 msgr2=0x7f377c0cf0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:40.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.755+0000 7f3781681640 1 -- 172.21.15.150:0/3805705778 shutdown_connections 2024-03-17T23:48:40.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.755+0000 7f3781681640 1 -- 172.21.15.150:0/3805705778 wait complete. 2024-03-17T23:48:40.767 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:48:40.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.870+0000 7fbaa21df640 1 Processor -- start 2024-03-17T23:48:40.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.870+0000 7fbaa21df640 1 -- start start 2024-03-17T23:48:40.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.871+0000 7fbaa21df640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba9c0b2220 0x7fba9c0cc220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:40.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.871+0000 7fbaa21df640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fba9c0b27f0 0x7fba9c0b2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:40.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.871+0000 7fbaa21df640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fba9c06ccb0 0x7fba9c06d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:40.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.871+0000 7fbaa21df640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fba9c06c910 con 0x7fba9c0b27f0 2024-03-17T23:48:40.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.871+0000 7fbaa21df640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fba9c06d660 con 0x7fba9c06ccb0 2024-03-17T23:48:40.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.871+0000 7fbaa21df640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fba9c09b480 con 0x7fba9c0b2220 2024-03-17T23:48:40.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.871+0000 7fba92ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fba9c0b27f0 0x7fba9c0b2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:40.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.871+0000 7fba92ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fba9c0b27f0 0x7fba9c0b2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44800/0 (socket says 172.21.15.150:44800) 2024-03-17T23:48:40.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.871+0000 7fba92ffd640 1 -- 172.21.15.150:0/380521485 learned_addr learned my addr 172.21.15.150:0/380521485 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:40.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.871+0000 7fba9b7fe640 1 --2- 172.21.15.150:0/380521485 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba9c0b2220 0x7fba9c0cc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:40.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.871+0000 7fba92ffd640 1 -- 172.21.15.150:0/380521485 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba9c0b2220 msgr2=0x7fba9c0cc220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:40.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.871+0000 7fba92ffd640 1 --2- 172.21.15.150:0/380521485 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba9c0b2220 0x7fba9c0cc220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.871+0000 7fba9bfff640 1 --2- 172.21.15.150:0/380521485 >> v2:172.21.15.157:3300/0 conn(0x7fba9c06ccb0 0x7fba9c06d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:40.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.871+0000 7fba92ffd640 1 -- 172.21.15.150:0/380521485 >> v2:172.21.15.157:3300/0 conn(0x7fba9c06ccb0 msgr2=0x7fba9c06d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:40.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.871+0000 7fba92ffd640 1 --2- 172.21.15.150:0/380521485 >> v2:172.21.15.157:3300/0 conn(0x7fba9c06ccb0 0x7fba9c06d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.871+0000 7fba92ffd640 1 -- 172.21.15.150:0/380521485 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba9c0b2600 con 0x7fba9c0b27f0 2024-03-17T23:48:40.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.871+0000 7fba9b7fe640 1 --2- 172.21.15.150:0/380521485 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba9c0b2220 0x7fba9c0cc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:40.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.872+0000 7fba92ffd640 1 --2- 172.21.15.150:0/380521485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba9c0b27f0 0x7fba9c0b2c50 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fba9c0736f0 tx=0x7fba8402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48b50b0628fa92e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:40.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.872+0000 7fba9affd640 1 -- 172.21.15.150:0/380521485 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba840317d0 con 0x7fba9c0b27f0 2024-03-17T23:48:40.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.872+0000 7fba9affd640 1 -- 172.21.15.150:0/380521485 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba84037070 con 0x7fba9c0b27f0 2024-03-17T23:48:40.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.872+0000 7fba9affd640 1 -- 172.21.15.150:0/380521485 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba84031b90 con 0x7fba9c0b27f0 2024-03-17T23:48:40.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.873+0000 7fbaa21df640 1 -- 172.21.15.150:0/380521485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba9c0b27f0 msgr2=0x7fba9c0b2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:40.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.873+0000 7fbaa21df640 1 --2- 172.21.15.150:0/380521485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba9c0b27f0 0x7fba9c0b2c50 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fba9c0736f0 tx=0x7fba8402b870 comp rx=0 tx=0).stop 2024-03-17T23:48:40.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.873+0000 7fbaa21df640 1 -- 172.21.15.150:0/380521485 shutdown_connections 2024-03-17T23:48:40.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.873+0000 7fbaa21df640 1 --2- 172.21.15.150:0/380521485 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba9c0b27f0 0x7fba9c0b2c50 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.873+0000 7fbaa21df640 1 --2- 172.21.15.150:0/380521485 >> v2:172.21.15.157:3300/0 conn(0x7fba9c06ccb0 0x7fba9c06d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.873+0000 7fbaa21df640 1 --2- 172.21.15.150:0/380521485 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba9c0b2220 0x7fba9c0cc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.873+0000 7fbaa21df640 1 -- 172.21.15.150:0/380521485 >> 172.21.15.150:0/380521485 conn(0x7fba9c0b36c0 msgr2=0x7fba9c0b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:40.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.873+0000 7fbaa21df640 1 -- 172.21.15.150:0/380521485 shutdown_connections 2024-03-17T23:48:40.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.873+0000 7fbaa21df640 1 -- 172.21.15.150:0/380521485 wait complete. 2024-03-17T23:48:40.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.875+0000 7fbaa21df640 1 Processor -- start 2024-03-17T23:48:40.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.875+0000 7fbaa21df640 1 -- start start 2024-03-17T23:48:40.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.876+0000 7fbaa21df640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba9c06ccb0 0x7fba9c0c5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:40.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.876+0000 7fbaa21df640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba9c0b2220 0x7fba9c148160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:40.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.876+0000 7fbaa21df640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba9c0b27f0 0x7fba9c1487f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:40.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.876+0000 7fbaa21df640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fba9c0d0180 con 0x7fba9c0b27f0 2024-03-17T23:48:40.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.876+0000 7fbaa21df640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fba9c0cff00 con 0x7fba9c06ccb0 2024-03-17T23:48:40.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.876+0000 7fbaa21df640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fba9c0cc760 con 0x7fba9c0b2220 2024-03-17T23:48:40.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.876+0000 7fba9bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba9c0b27f0 0x7fba9c1487f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:40.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.876+0000 7fba9bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba9c0b27f0 0x7fba9c1487f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44804/0 (socket says 172.21.15.150:44804) 2024-03-17T23:48:40.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.876+0000 7fba9bfff640 1 -- 172.21.15.150:0/3264072468 learned_addr learned my addr 172.21.15.150:0/3264072468 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:40.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.876+0000 7fba9b7fe640 1 --2- 172.21.15.150:0/3264072468 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba9c06ccb0 0x7fba9c0c5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:40.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.876+0000 7fba92ffd640 1 --2- 172.21.15.150:0/3264072468 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba9c0b2220 0x7fba9c148160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:40.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.877+0000 7fba9bfff640 1 -- 172.21.15.150:0/3264072468 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba9c0b2220 msgr2=0x7fba9c148160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:40.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.877+0000 7fba9bfff640 1 --2- 172.21.15.150:0/3264072468 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba9c0b2220 0x7fba9c148160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.877+0000 7fba9bfff640 1 -- 172.21.15.150:0/3264072468 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba9c06ccb0 msgr2=0x7fba9c0c5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:40.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.877+0000 7fba9bfff640 1 --2- 172.21.15.150:0/3264072468 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba9c06ccb0 0x7fba9c0c5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:40.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.877+0000 7fba9bfff640 1 -- 172.21.15.150:0/3264072468 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba9c0c8640 con 0x7fba9c0b27f0 2024-03-17T23:48:40.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.877+0000 7fba92ffd640 1 --2- 172.21.15.150:0/3264072468 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba9c0b2220 0x7fba9c148160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:40.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.877+0000 7fba9bfff640 1 --2- 172.21.15.150:0/3264072468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba9c0b27f0 0x7fba9c1487f0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fba9400cb80 tx=0x7fba9400a4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:40.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.877+0000 7fba98ff9640 1 -- 172.21.15.150:0/3264072468 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba94015070 con 0x7fba9c0b27f0 2024-03-17T23:48:40.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.877+0000 7fba98ff9640 1 -- 172.21.15.150:0/3264072468 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba9400ab70 con 0x7fba9c0b27f0 2024-03-17T23:48:40.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.877+0000 7fbaa21df640 1 -- 172.21.15.150:0/3264072468 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba9c0c5d20 con 0x7fba9c0b27f0 2024-03-17T23:48:40.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.877+0000 7fba98ff9640 1 -- 172.21.15.150:0/3264072468 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9400d040 con 0x7fba9c0b27f0 2024-03-17T23:48:40.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.877+0000 7fbaa21df640 1 -- 172.21.15.150:0/3264072468 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba9c0c62e0 con 0x7fba9c0b27f0 2024-03-17T23:48:40.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.878+0000 7fbaa21df640 1 -- 172.21.15.150:0/3264072468 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba9c0688b0 con 0x7fba9c0b27f0 2024-03-17T23:48:40.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.879+0000 7fba98ff9640 1 -- 172.21.15.150:0/3264072468 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fba94018070 con 0x7fba9c0b27f0 2024-03-17T23:48:40.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.879+0000 7fba98ff9640 1 --2- 172.21.15.150:0/3264072468 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fba7c06bc00 0x7fba7c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:40.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.879+0000 7fba98ff9640 1 -- 172.21.15.150:0/3264072468 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7fba9408a7d0 con 0x7fba9c0b27f0 2024-03-17T23:48:40.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.879+0000 7fba9b7fe640 1 --2- 172.21.15.150:0/3264072468 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fba7c06bc00 0x7fba7c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:40.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.879+0000 7fba9b7fe640 1 --2- 172.21.15.150:0/3264072468 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fba7c06bc00 0x7fba7c06e0c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fba8c00ab60 tx=0x7fba8c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:40.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:40.883+0000 7fba98ff9640 1 -- 172.21.15.150:0/3264072468 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fba94058950 con 0x7fba9c0b27f0 2024-03-17T23:48:41.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.057+0000 7fbaa21df640 1 -- 172.21.15.150:0/3264072468 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fba9c0b1950 con 0x7fba9c0b27f0 2024-03-17T23:48:41.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.057+0000 7fba98ff9640 1 -- 172.21.15.150:0/3264072468 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v45) ==== 100+0+11 (secure 0 0 0) 0x7fba9405c9f0 con 0x7fba9c0b27f0 2024-03-17T23:48:41.059 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:48:41.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.059+0000 7fbaa21df640 1 -- 172.21.15.150:0/3264072468 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fba7c06bc00 msgr2=0x7fba7c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:41.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.059+0000 7fbaa21df640 1 --2- 172.21.15.150:0/3264072468 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fba7c06bc00 0x7fba7c06e0c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fba8c00ab60 tx=0x7fba8c008040 comp rx=0 tx=0).stop 2024-03-17T23:48:41.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.059+0000 7fbaa21df640 1 -- 172.21.15.150:0/3264072468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba9c0b27f0 msgr2=0x7fba9c1487f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:41.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.059+0000 7fbaa21df640 1 --2- 172.21.15.150:0/3264072468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba9c0b27f0 0x7fba9c1487f0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fba9400cb80 tx=0x7fba9400a4c0 comp rx=0 tx=0).stop 2024-03-17T23:48:41.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.059+0000 7fbaa21df640 1 -- 172.21.15.150:0/3264072468 shutdown_connections 2024-03-17T23:48:41.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.059+0000 7fbaa21df640 1 --2- 172.21.15.150:0/3264072468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba9c0b27f0 0x7fba9c1487f0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:41.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.059+0000 7fbaa21df640 1 --2- 172.21.15.150:0/3264072468 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fba9c0b2220 0x7fba9c148160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:41.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.059+0000 7fbaa21df640 1 --2- 172.21.15.150:0/3264072468 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fba7c06bc00 0x7fba7c06e0c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:41.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.059+0000 7fbaa21df640 1 --2- 172.21.15.150:0/3264072468 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba9c06ccb0 0x7fba9c0c5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:41.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.059+0000 7fbaa21df640 1 -- 172.21.15.150:0/3264072468 >> 172.21.15.150:0/3264072468 conn(0x7fba9c0b36c0 msgr2=0x7fba9c15eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:41.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.059+0000 7fbaa21df640 1 -- 172.21.15.150:0/3264072468 shutdown_connections 2024-03-17T23:48:41.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.059+0000 7fbaa21df640 1 -- 172.21.15.150:0/3264072468 wait complete. 2024-03-17T23:48:41.071 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-17T23:48:41.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.175+0000 7fec04297640 1 Processor -- start 2024-03-17T23:48:41.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.175+0000 7fec04297640 1 -- start start 2024-03-17T23:48:41.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.176+0000 7fec04297640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7febfc15de00 0x7febfc0b9970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:41.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.176+0000 7fec04297640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7febfc06c690 0x7febfc0b5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:41.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.176+0000 7fec04297640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7febfc0b5cd0 0x7febfc1619f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:41.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.176+0000 7fec04297640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7febfc0b6cd0 con 0x7febfc06c690 2024-03-17T23:48:41.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.176+0000 7fec04297640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7febfc0b6100 con 0x7febfc15de00 2024-03-17T23:48:41.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.176+0000 7fec04297640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7febfc06cec0 con 0x7febfc0b5cd0 2024-03-17T23:48:41.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.177+0000 7fec01832640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7febfc06c690 0x7febfc0b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:41.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.177+0000 7fec01832640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7febfc06c690 0x7febfc0b5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44808/0 (socket says 172.21.15.150:44808) 2024-03-17T23:48:41.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.177+0000 7fec01832640 1 -- 172.21.15.150:0/3949066001 learned_addr learned my addr 172.21.15.150:0/3949066001 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:41.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.177+0000 7fec02033640 1 --2- 172.21.15.150:0/3949066001 >> v2:172.21.15.157:3300/0 conn(0x7febfc15de00 0x7febfc0b9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:41.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.177+0000 7fec02834640 1 --2- 172.21.15.150:0/3949066001 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7febfc0b5cd0 0x7febfc1619f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:41.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.177+0000 7fec01832640 1 -- 172.21.15.150:0/3949066001 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7febfc0b5cd0 msgr2=0x7febfc1619f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:41.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.177+0000 7fec01832640 1 --2- 172.21.15.150:0/3949066001 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7febfc0b5cd0 0x7febfc1619f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:41.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.177+0000 7fec01832640 1 -- 172.21.15.150:0/3949066001 >> v2:172.21.15.157:3300/0 conn(0x7febfc15de00 msgr2=0x7febfc0b9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:41.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.177+0000 7fec01832640 1 --2- 172.21.15.150:0/3949066001 >> v2:172.21.15.157:3300/0 conn(0x7febfc15de00 0x7febfc0b9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:41.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.177+0000 7fec01832640 1 -- 172.21.15.150:0/3949066001 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febfc06d1a0 con 0x7febfc06c690 2024-03-17T23:48:41.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.178+0000 7fec01832640 1 --2- 172.21.15.150:0/3949066001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febfc06c690 0x7febfc0b5790 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7febfc0d4d70 tx=0x7febf802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a0e6249c475584e server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:41.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.178+0000 7fec01031640 1 -- 172.21.15.150:0/3949066001 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf8035020 con 0x7febfc06c690 2024-03-17T23:48:41.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.178+0000 7fec01031640 1 -- 172.21.15.150:0/3949066001 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febf8031640 con 0x7febfc06c690 2024-03-17T23:48:41.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.178+0000 7fec01031640 1 -- 172.21.15.150:0/3949066001 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf802d040 con 0x7febfc06c690 2024-03-17T23:48:41.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.178+0000 7fec04297640 1 -- 172.21.15.150:0/3949066001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febfc06c690 msgr2=0x7febfc0b5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:41.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.178+0000 7fec04297640 1 --2- 172.21.15.150:0/3949066001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febfc06c690 0x7febfc0b5790 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7febfc0d4d70 tx=0x7febf802b8f0 comp rx=0 tx=0).stop 2024-03-17T23:48:41.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.178+0000 7fec04297640 1 -- 172.21.15.150:0/3949066001 shutdown_connections 2024-03-17T23:48:41.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.178+0000 7fec04297640 1 --2- 172.21.15.150:0/3949066001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febfc06c690 0x7febfc0b5790 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:41.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.178+0000 7fec04297640 1 --2- 172.21.15.150:0/3949066001 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7febfc0b5cd0 0x7febfc1619f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:41.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.178+0000 7fec04297640 1 --2- 172.21.15.150:0/3949066001 >> v2:172.21.15.157:3300/0 conn(0x7febfc15de00 0x7febfc0b9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:41.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.178+0000 7fec04297640 1 -- 172.21.15.150:0/3949066001 >> 172.21.15.150:0/3949066001 conn(0x7febfc0d0790 msgr2=0x7febfc0d0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:41.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.178+0000 7fec04297640 1 -- 172.21.15.150:0/3949066001 shutdown_connections 2024-03-17T23:48:41.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.179+0000 7fec04297640 1 -- 172.21.15.150:0/3949066001 wait complete. 2024-03-17T23:48:41.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.180+0000 7fec04297640 1 Processor -- start 2024-03-17T23:48:41.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.180+0000 7fec04297640 1 -- start start 2024-03-17T23:48:41.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.180+0000 7fec04297640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7febfc06c690 0x7febfc0cb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:41.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.180+0000 7fec04297640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febfc0b5cd0 0x7febfc15a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:41.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.180+0000 7fec04297640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7febfc15de00 0x7febfc15c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:41.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.180+0000 7fec04297640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7febfc15e1e0 con 0x7febfc0b5cd0 2024-03-17T23:48:41.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.180+0000 7fec04297640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7febfc0d2190 con 0x7febfc15de00 2024-03-17T23:48:41.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.180+0000 7fec04297640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7febfc06cb00 con 0x7febfc06c690 2024-03-17T23:48:41.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.180+0000 7fec01832640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febfc0b5cd0 0x7febfc15a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:41.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.181+0000 7fec01832640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febfc0b5cd0 0x7febfc15a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44812/0 (socket says 172.21.15.150:44812) 2024-03-17T23:48:41.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.181+0000 7fec01832640 1 -- 172.21.15.150:0/2121268342 learned_addr learned my addr 172.21.15.150:0/2121268342 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:41.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.181+0000 7fec02033640 1 --2- 172.21.15.150:0/2121268342 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7febfc06c690 0x7febfc0cb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:41.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.181+0000 7fec02834640 1 --2- 172.21.15.150:0/2121268342 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7febfc15de00 0x7febfc15c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:41.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.181+0000 7fec01832640 1 -- 172.21.15.150:0/2121268342 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7febfc06c690 msgr2=0x7febfc0cb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:41.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.181+0000 7fec01832640 1 --2- 172.21.15.150:0/2121268342 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7febfc06c690 0x7febfc0cb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:41.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.181+0000 7fec01832640 1 -- 172.21.15.150:0/2121268342 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7febfc15de00 msgr2=0x7febfc15c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:41.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.181+0000 7fec01832640 1 --2- 172.21.15.150:0/2121268342 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7febfc15de00 0x7febfc15c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:41.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.181+0000 7fec01832640 1 -- 172.21.15.150:0/2121268342 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febfc161100 con 0x7febfc0b5cd0 2024-03-17T23:48:41.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.181+0000 7fec01832640 1 --2- 172.21.15.150:0/2121268342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febfc0b5cd0 0x7febfc15a5c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7febf8000c00 tx=0x7febf80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:41.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.181+0000 7febf2ffd640 1 -- 172.21.15.150:0/2121268342 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf8040070 con 0x7febfc0b5cd0 2024-03-17T23:48:41.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.181+0000 7febf2ffd640 1 -- 172.21.15.150:0/2121268342 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febf80399f0 con 0x7febfc0b5cd0 2024-03-17T23:48:41.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.181+0000 7febf2ffd640 1 -- 172.21.15.150:0/2121268342 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf8035040 con 0x7febfc0b5cd0 2024-03-17T23:48:41.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.181+0000 7fec04297640 1 -- 172.21.15.150:0/2121268342 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febfc0caf10 con 0x7febfc0b5cd0 2024-03-17T23:48:41.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.181+0000 7fec04297640 1 -- 172.21.15.150:0/2121268342 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febfc0cb4d0 con 0x7febfc0b5cd0 2024-03-17T23:48:41.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.182+0000 7fec04297640 1 -- 172.21.15.150:0/2121268342 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febfc0cb6d0 con 0x7febfc0b5cd0 2024-03-17T23:48:41.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.183+0000 7febf2ffd640 1 -- 172.21.15.150:0/2121268342 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7febf804b060 con 0x7febfc0b5cd0 2024-03-17T23:48:41.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.183+0000 7febf2ffd640 1 --2- 172.21.15.150:0/2121268342 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7febe006bc00 0x7febe006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:41.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.184+0000 7febf2ffd640 1 -- 172.21.15.150:0/2121268342 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7febf80ac770 con 0x7febfc0b5cd0 2024-03-17T23:48:41.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.184+0000 7fec02033640 1 --2- 172.21.15.150:0/2121268342 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7febe006bc00 0x7febe006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:41.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.184+0000 7fec02033640 1 --2- 172.21.15.150:0/2121268342 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7febe006bc00 0x7febe006e0c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7febfc15feb0 tx=0x7febec009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:41.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.192+0000 7febf2ffd640 1 -- 172.21.15.150:0/2121268342 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7febfc0cb6d0 con 0x7febfc0b5cd0 2024-03-17T23:48:41.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.452+0000 7fec04297640 1 -- 172.21.15.150:0/2121268342 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7febfc0681a0 con 0x7febfc0b5cd0 2024-03-17T23:48:41.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.452+0000 7febf2ffd640 1 -- 172.21.15.150:0/2121268342 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v45) ==== 97+0+11 (secure 0 0 0) 0x7febf807aa50 con 0x7febfc0b5cd0 2024-03-17T23:48:41.453 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:48:41.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.454+0000 7fec04297640 1 -- 172.21.15.150:0/2121268342 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7febe006bc00 msgr2=0x7febe006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:41.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.454+0000 7fec04297640 1 --2- 172.21.15.150:0/2121268342 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7febe006bc00 0x7febe006e0c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7febfc15feb0 tx=0x7febec009190 comp rx=0 tx=0).stop 2024-03-17T23:48:41.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.454+0000 7fec04297640 1 -- 172.21.15.150:0/2121268342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febfc0b5cd0 msgr2=0x7febfc15a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:41.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.454+0000 7fec04297640 1 --2- 172.21.15.150:0/2121268342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febfc0b5cd0 0x7febfc15a5c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7febf8000c00 tx=0x7febf80046c0 comp rx=0 tx=0).stop 2024-03-17T23:48:41.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.454+0000 7fec04297640 1 -- 172.21.15.150:0/2121268342 shutdown_connections 2024-03-17T23:48:41.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.454+0000 7fec04297640 1 --2- 172.21.15.150:0/2121268342 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7febe006bc00 0x7febe006e0c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:41.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.454+0000 7fec04297640 1 --2- 172.21.15.150:0/2121268342 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7febfc15de00 0x7febfc15c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:41.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.454+0000 7fec04297640 1 --2- 172.21.15.150:0/2121268342 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7febfc0b5cd0 0x7febfc15a5c0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:41.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.454+0000 7fec04297640 1 --2- 172.21.15.150:0/2121268342 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7febfc06c690 0x7febfc0cb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:41.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.454+0000 7fec04297640 1 -- 172.21.15.150:0/2121268342 >> 172.21.15.150:0/2121268342 conn(0x7febfc0d0790 msgr2=0x7febfc0b7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:41.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.454+0000 7fec04297640 1 -- 172.21.15.150:0/2121268342 shutdown_connections 2024-03-17T23:48:41.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.454+0000 7fec04297640 1 -- 172.21.15.150:0/2121268342 wait complete. 2024-03-17T23:48:41.467 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-17T23:48:41.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.568+0000 7f11cd697640 1 Processor -- start 2024-03-17T23:48:41.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.569+0000 7f11cd697640 1 -- start start 2024-03-17T23:48:41.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.569+0000 7f11cd697640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f11c806f420 0x7f11c80cce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:41.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.569+0000 7f11cd697640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11c80cebc0 con 0x7f11c806ecd0 2024-03-17T23:48:41.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.569+0000 7f11cd697640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11c80ceea0 con 0x7f11c815a4f0 2024-03-17T23:48:41.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.569+0000 7f11cd697640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f11c80cf180 con 0x7f11c806f420 2024-03-17T23:48:41.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.569+0000 7f11c67fc640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f11c806ecd0 0x7f11c80b2b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37952/0 (socket says 172.21.15.150:37952) 2024-03-17T23:48:41.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.569+0000 7f11c67fc640 1 -- 172.21.15.150:0/1851306666 learned_addr learned my addr 172.21.15.150:0/1851306666 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:41.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.570+0000 7f11c77fe640 1 --2- 172.21.15.150:0/1851306666 >> v2:172.21.15.157:3300/0 conn(0x7f11c806f420 0x7f11c80cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:41.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.570+0000 7f11c5ffb640 1 -- 172.21.15.150:0/1851306666 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 462024186 0 0) 0x7f11c80cebc0 con 0x7f11c806ecd0 2024-03-17T23:48:41.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.570+0000 7f11c5ffb640 1 -- 172.21.15.150:0/1851306666 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11b0003520 con 0x7f11c806ecd0 2024-03-17T23:48:41.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.570+0000 7f11c5ffb640 1 -- 172.21.15.150:0/1851306666 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1585365299 0 0) 0x7f11c80ceea0 con 0x7f11c815a4f0 2024-03-17T23:48:41.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.570+0000 7f11c5ffb640 1 -- 172.21.15.150:0/1851306666 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11c80cebc0 con 0x7f11c815a4f0 2024-03-17T23:48:41.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.570+0000 7f11c77fe640 1 -- 172.21.15.150:0/1851306666 >> v1:172.21.15.157:6789/0 conn(0x7f11c815a4f0 legacy=0x7f11c806e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:41.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.570+0000 7f11c77fe640 1 -- 172.21.15.150:0/1851306666 >> v1:172.21.15.150:6789/0 conn(0x7f11c806ecd0 legacy=0x7f11c80b2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:41.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.570+0000 7f11c77fe640 1 -- 172.21.15.150:0/1851306666 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11c80cf460 con 0x7f11c806f420 2024-03-17T23:48:41.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.571+0000 7f11c77fe640 1 --2- 172.21.15.150:0/1851306666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11c806f420 0x7f11c80cce30 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f11c0009d90 tx=0x7f11c002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=293a5c4ec81b87c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:41.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.571+0000 7f11c5ffb640 1 -- 172.21.15.150:0/1851306666 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c0039070 con 0x7f11c806f420 2024-03-17T23:48:41.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.571+0000 7f11c5ffb640 1 -- 172.21.15.150:0/1851306666 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11c002cd00 con 0x7f11c806f420 2024-03-17T23:48:41.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.571+0000 7f11c5ffb640 1 -- 172.21.15.150:0/1851306666 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c0034640 con 0x7f11c806f420 2024-03-17T23:48:41.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.571+0000 7f11cd697640 1 -- 172.21.15.150:0/1851306666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11c806f420 msgr2=0x7f11c80cce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:41.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.571+0000 7f11cd697640 1 --2- 172.21.15.150:0/1851306666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11c806f420 0x7f11c80cce30 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f11c0009d90 tx=0x7f11c002bf50 comp rx=0 tx=0).stop 2024-03-17T23:48:41.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.571+0000 7f11cd697640 1 -- 172.21.15.150:0/1851306666 shutdown_connections 2024-03-17T23:48:41.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.571+0000 7f11cd697640 1 --2- 172.21.15.150:0/1851306666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11c806f420 0x7f11c80cce30 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:41.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.571+0000 7f11cd697640 1 -- 172.21.15.150:0/1851306666 >> 172.21.15.150:0/1851306666 conn(0x7f11c806b190 msgr2=0x7f11c806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:41.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.572+0000 7f11cd697640 1 -- 172.21.15.150:0/1851306666 shutdown_connections 2024-03-17T23:48:41.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.572+0000 7f11cd697640 1 -- 172.21.15.150:0/1851306666 wait complete. 2024-03-17T23:48:41.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.573+0000 7f11cd697640 1 Processor -- start 2024-03-17T23:48:41.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.573+0000 7f11cd697640 1 -- start start 2024-03-17T23:48:41.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.574+0000 7f11cd697640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11c806ecd0 0x7f11c813f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:41.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.574+0000 7f11cd697640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11c806f420 0x7f11c813b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:41.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.574+0000 7f11cd697640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11c815a4f0 0x7f11c81489a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:41.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.574+0000 7f11cd697640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f11c80ce9a0 con 0x7f11c815a4f0 2024-03-17T23:48:41.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.574+0000 7f11cd697640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f11c80ce820 con 0x7f11c806f420 2024-03-17T23:48:41.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.574+0000 7f11cd697640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f11c806dd30 con 0x7f11c806ecd0 2024-03-17T23:48:41.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.574+0000 7f11c77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11c815a4f0 0x7f11c81489a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:41.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.574+0000 7f11c77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11c815a4f0 0x7f11c81489a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44826/0 (socket says 172.21.15.150:44826) 2024-03-17T23:48:41.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.574+0000 7f11c77fe640 1 -- 172.21.15.150:0/1264464452 learned_addr learned my addr 172.21.15.150:0/1264464452 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:41.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.574+0000 7f11c6ffd640 1 --2- 172.21.15.150:0/1264464452 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11c806ecd0 0x7f11c813f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:41.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.574+0000 7f11c77fe640 1 -- 172.21.15.150:0/1264464452 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11c806ecd0 msgr2=0x7f11c813f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:41.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.574+0000 7f11c77fe640 1 --2- 172.21.15.150:0/1264464452 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11c806ecd0 0x7f11c813f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:41.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.574+0000 7f11c67fc640 1 --2- 172.21.15.150:0/1264464452 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11c806f420 0x7f11c813b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:41.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.574+0000 7f11c77fe640 1 -- 172.21.15.150:0/1264464452 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11c806f420 msgr2=0x7f11c813b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:41.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.575+0000 7f11c77fe640 1 --2- 172.21.15.150:0/1264464452 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11c806f420 0x7f11c813b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:41.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.575+0000 7f11c77fe640 1 -- 172.21.15.150:0/1264464452 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11c813ba60 con 0x7f11c815a4f0 2024-03-17T23:48:41.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.575+0000 7f11c67fc640 1 --2- 172.21.15.150:0/1264464452 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11c806f420 0x7f11c813b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:41.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.575+0000 7f11c77fe640 1 --2- 172.21.15.150:0/1264464452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11c815a4f0 0x7f11c81489a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f11c0037040 tx=0x7f11c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:41.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.575+0000 7f11af7fe640 1 -- 172.21.15.150:0/1264464452 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c0039050 con 0x7f11c815a4f0 2024-03-17T23:48:41.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.575+0000 7f11af7fe640 1 -- 172.21.15.150:0/1264464452 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11c003a040 con 0x7f11c815a4f0 2024-03-17T23:48:41.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.575+0000 7f11cd697640 1 -- 172.21.15.150:0/1264464452 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11c813f9a0 con 0x7f11c815a4f0 2024-03-17T23:48:41.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.575+0000 7f11af7fe640 1 -- 172.21.15.150:0/1264464452 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c0009450 con 0x7f11c815a4f0 2024-03-17T23:48:41.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.575+0000 7f11cd697640 1 -- 172.21.15.150:0/1264464452 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11c813ff60 con 0x7f11c815a4f0 2024-03-17T23:48:41.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.576+0000 7f11cd697640 1 -- 172.21.15.150:0/1264464452 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11c80681a0 con 0x7f11c815a4f0 2024-03-17T23:48:41.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.576+0000 7f11af7fe640 1 -- 172.21.15.150:0/1264464452 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f11c0047020 con 0x7f11c815a4f0 2024-03-17T23:48:41.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.577+0000 7f11af7fe640 1 --2- 172.21.15.150:0/1264464452 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f11b406bbd0 0x7f11b406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:41.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.577+0000 7f11af7fe640 1 -- 172.21.15.150:0/1264464452 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7f11c00ace70 con 0x7f11c815a4f0 2024-03-17T23:48:41.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.577+0000 7f11c6ffd640 1 --2- 172.21.15.150:0/1264464452 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f11b406bbd0 0x7f11b406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:41.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.577+0000 7f11c6ffd640 1 --2- 172.21.15.150:0/1264464452 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f11b406bbd0 0x7f11b406e090 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f11b8007fd0 tx=0x7f11b8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:41.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.585+0000 7f11af7fe640 1 -- 172.21.15.150:0/1264464452 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f11c004e020 con 0x7f11c815a4f0 2024-03-17T23:48:41.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.790+0000 7f11cd697640 1 -- 172.21.15.150:0/1264464452 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f11c806bf90 con 0x7f11c815a4f0 2024-03-17T23:48:41.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.790+0000 7f11af7fe640 1 -- 172.21.15.150:0/1264464452 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v45) ==== 80+0+0 (secure 0 0 0) 0x7f11c003d360 con 0x7f11c815a4f0 2024-03-17T23:48:41.792 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-03-17T23:48:41.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.793+0000 7f11cd697640 1 -- 172.21.15.150:0/1264464452 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f11b406bbd0 msgr2=0x7f11b406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:41.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.793+0000 7f11cd697640 1 --2- 172.21.15.150:0/1264464452 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f11b406bbd0 0x7f11b406e090 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f11b8007fd0 tx=0x7f11b8009040 comp rx=0 tx=0).stop 2024-03-17T23:48:41.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.793+0000 7f11cd697640 1 -- 172.21.15.150:0/1264464452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11c815a4f0 msgr2=0x7f11c81489a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:41.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.793+0000 7f11cd697640 1 --2- 172.21.15.150:0/1264464452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11c815a4f0 0x7f11c81489a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f11c0037040 tx=0x7f11c0004a40 comp rx=0 tx=0).stop 2024-03-17T23:48:41.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.793+0000 7f11cd697640 1 -- 172.21.15.150:0/1264464452 shutdown_connections 2024-03-17T23:48:41.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.793+0000 7f11cd697640 1 --2- 172.21.15.150:0/1264464452 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11c815a4f0 0x7f11c81489a0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:41.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.793+0000 7f11cd697640 1 --2- 172.21.15.150:0/1264464452 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f11b406bbd0 0x7f11b406e090 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:41.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.793+0000 7f11cd697640 1 --2- 172.21.15.150:0/1264464452 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11c806f420 0x7f11c813b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:41.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.793+0000 7f11cd697640 1 --2- 172.21.15.150:0/1264464452 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11c806ecd0 0x7f11c813f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:41.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.793+0000 7f11cd697640 1 -- 172.21.15.150:0/1264464452 >> 172.21.15.150:0/1264464452 conn(0x7f11c806b190 msgr2=0x7f11c814a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:41.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.793+0000 7f11cd697640 1 -- 172.21.15.150:0/1264464452 shutdown_connections 2024-03-17T23:48:41.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:41.793+0000 7f11cd697640 1 -- 172.21.15.150:0/1264464452 wait complete. 2024-03-17T23:48:41.805 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi150.front.sepia.ceph.com... 2024-03-17T23:48:41.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:41.805 DEBUG:teuthology.orchestra.run.smithi150:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-17T23:48:41.830 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:ceph-fuse[55953]: fuse finished with error 0 and tester_r 0 2024-03-17T23:48:41.832 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:48:47.935 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi150.front.sepia.ceph.com 2024-03-17T23:48:47.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:47.935 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-17T23:48:47.950 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds 'allow rwp' mon 'allow r' osd 'allow rw pool=cephfs_data,allow rw pool=data_new' 2024-03-17T23:48:48.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.135+0000 7f33b339d640 1 Processor -- start 2024-03-17T23:48:48.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.135+0000 7f33b339d640 1 -- start start 2024-03-17T23:48:48.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.136+0000 7f33b339d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f33ac15df40 0x7f33ac0d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:48.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.136+0000 7f33b339d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f33ac0d23d0 0x7f33ac06c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:48.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.136+0000 7f33b339d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33ac15e320 con 0x7f33ac06ce60 2024-03-17T23:48:48.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.136+0000 7f33b339d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f33ac06d2d0 con 0x7f33ac0d23d0 2024-03-17T23:48:48.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.136+0000 7f33b339d640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f33ac0d3460 con 0x7f33ac15df40 2024-03-17T23:48:48.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.136+0000 7f33b1b9a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f33ac0d23d0 0x7f33ac06c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:48.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.136+0000 7f33b1b9a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f33ac0d23d0 0x7f33ac06c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36410/0 (socket says 172.21.15.150:36410) 2024-03-17T23:48:48.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.136+0000 7f33b1b9a640 1 -- 172.21.15.150:0/3437103645 learned_addr learned my addr 172.21.15.150:0/3437103645 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:48.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.136+0000 7f33b1b9a640 1 -- 172.21.15.150:0/3437103645 >> v2:172.21.15.157:3300/0 conn(0x7f33ac15df40 msgr2=0x7f33ac0d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:48.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.136+0000 7f33b239b640 1 --2- 172.21.15.150:0/3437103645 >> v2:172.21.15.157:3300/0 conn(0x7f33ac15df40 0x7f33ac0d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:48.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.136+0000 7f33b1b9a640 1 --2- 172.21.15.150:0/3437103645 >> v2:172.21.15.157:3300/0 conn(0x7f33ac15df40 0x7f33ac0d1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:48.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.136+0000 7f33b1b9a640 1 -- 172.21.15.150:0/3437103645 >> v1:172.21.15.157:6789/0 conn(0x7f33ac06ce60 legacy=0x7f33ac0b5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:48.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.136+0000 7f33b1b9a640 1 -- 172.21.15.150:0/3437103645 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33ac0d2850 con 0x7f33ac0d23d0 2024-03-17T23:48:48.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.137+0000 7f33b1b9a640 1 --2- 172.21.15.150:0/3437103645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33ac0d23d0 0x7f33ac06c920 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f33ac0ce4a0 tx=0x7f339c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50868ca820dff9c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:48.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.137+0000 7f33b1399640 1 -- 172.21.15.150:0/3437103645 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339c034070 con 0x7f33ac0d23d0 2024-03-17T23:48:48.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.137+0000 7f33b1399640 1 -- 172.21.15.150:0/3437103645 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f339c030860 con 0x7f33ac0d23d0 2024-03-17T23:48:48.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.137+0000 7f33b1399640 1 -- 172.21.15.150:0/3437103645 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339c030c30 con 0x7f33ac0d23d0 2024-03-17T23:48:48.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.138+0000 7f33b339d640 1 -- 172.21.15.150:0/3437103645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33ac0d23d0 msgr2=0x7f33ac06c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:48.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.138+0000 7f33b339d640 1 --2- 172.21.15.150:0/3437103645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33ac0d23d0 0x7f33ac06c920 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f33ac0ce4a0 tx=0x7f339c02b9a0 comp rx=0 tx=0).stop 2024-03-17T23:48:48.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.138+0000 7f33b339d640 1 -- 172.21.15.150:0/3437103645 shutdown_connections 2024-03-17T23:48:48.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.138+0000 7f33b339d640 1 --2- 172.21.15.150:0/3437103645 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33ac0d23d0 0x7f33ac06c920 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:48.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.138+0000 7f33b339d640 1 --2- 172.21.15.150:0/3437103645 >> v2:172.21.15.157:3300/0 conn(0x7f33ac15df40 0x7f33ac0d1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:48.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.138+0000 7f33b339d640 1 -- 172.21.15.150:0/3437103645 >> 172.21.15.150:0/3437103645 conn(0x7f33ac072ef0 msgr2=0x7f33ac0d3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:48.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.138+0000 7f33b339d640 1 -- 172.21.15.150:0/3437103645 shutdown_connections 2024-03-17T23:48:48.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.138+0000 7f33b339d640 1 -- 172.21.15.150:0/3437103645 wait complete. 2024-03-17T23:48:48.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.140+0000 7f33b339d640 1 Processor -- start 2024-03-17T23:48:48.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.140+0000 7f33b339d640 1 -- start start 2024-03-17T23:48:48.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.141+0000 7f33b339d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33ac06ce60 0x7f33ac0b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:48.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.141+0000 7f33b339d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f33ac0d23d0 0x7f33ac0b3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:48.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.141+0000 7f33b339d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f33ac15df40 0x7f33ac15fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:48.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.141+0000 7f33b339d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f33ac0d30f0 con 0x7f33ac06ce60 2024-03-17T23:48:48.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.141+0000 7f33b339d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f33ac0d2e10 con 0x7f33ac0d23d0 2024-03-17T23:48:48.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.141+0000 7f33b339d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f33ac0b7de0 con 0x7f33ac15df40 2024-03-17T23:48:48.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.141+0000 7f33b239b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33ac06ce60 0x7f33ac0b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:48.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.141+0000 7f33b239b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33ac06ce60 0x7f33ac0b61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36414/0 (socket says 172.21.15.150:36414) 2024-03-17T23:48:48.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.141+0000 7f33b239b640 1 -- 172.21.15.150:0/1784842693 learned_addr learned my addr 172.21.15.150:0/1784842693 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:48.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.142+0000 7f33b1b9a640 1 --2- 172.21.15.150:0/1784842693 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f33ac0d23d0 0x7f33ac0b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:48.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.142+0000 7f33b2b9c640 1 --2- 172.21.15.150:0/1784842693 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f33ac15df40 0x7f33ac15fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:48.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.142+0000 7f33b239b640 1 -- 172.21.15.150:0/1784842693 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f33ac15df40 msgr2=0x7f33ac15fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:48.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.142+0000 7f33b239b640 1 --2- 172.21.15.150:0/1784842693 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f33ac15df40 0x7f33ac15fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:48.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.142+0000 7f33b239b640 1 -- 172.21.15.150:0/1784842693 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f33ac0d23d0 msgr2=0x7f33ac0b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:48.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.142+0000 7f33b239b640 1 --2- 172.21.15.150:0/1784842693 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f33ac0d23d0 0x7f33ac0b3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:48.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.142+0000 7f33b239b640 1 -- 172.21.15.150:0/1784842693 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33ac0cadc0 con 0x7f33ac06ce60 2024-03-17T23:48:48.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.142+0000 7f33b239b640 1 --2- 172.21.15.150:0/1784842693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33ac06ce60 0x7f33ac0b61f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f33a800c610 tx=0x7f33a800cad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:48.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.142+0000 7f33a2ffd640 1 -- 172.21.15.150:0/1784842693 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a8015070 con 0x7f33ac06ce60 2024-03-17T23:48:48.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.142+0000 7f33a2ffd640 1 -- 172.21.15.150:0/1784842693 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33a8004220 con 0x7f33ac06ce60 2024-03-17T23:48:48.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.142+0000 7f33b339d640 1 -- 172.21.15.150:0/1784842693 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33ac0cb0a0 con 0x7f33ac06ce60 2024-03-17T23:48:48.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.142+0000 7f33a2ffd640 1 -- 172.21.15.150:0/1784842693 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a8003610 con 0x7f33ac06ce60 2024-03-17T23:48:48.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.142+0000 7f33b339d640 1 -- 172.21.15.150:0/1784842693 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33ac0cb660 con 0x7f33ac06ce60 2024-03-17T23:48:48.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.143+0000 7f33b339d640 1 -- 172.21.15.150:0/1784842693 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33ac074c40 con 0x7f33ac06ce60 2024-03-17T23:48:48.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.144+0000 7f33a2ffd640 1 -- 172.21.15.150:0/1784842693 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f33a80037d0 con 0x7f33ac06ce60 2024-03-17T23:48:48.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.144+0000 7f33a2ffd640 1 --2- 172.21.15.150:0/1784842693 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f339006bbd0 0x7f339006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:48.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.144+0000 7f33a2ffd640 1 -- 172.21.15.150:0/1784842693 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7f33a808b000 con 0x7f33ac06ce60 2024-03-17T23:48:48.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.144+0000 7f33b1b9a640 1 --2- 172.21.15.150:0/1784842693 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f339006bbd0 0x7f339006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:48.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.145+0000 7f33b1b9a640 1 --2- 172.21.15.150:0/1784842693 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f339006bbd0 0x7f339006e090 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f339c036000 tx=0x7f339c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:48.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.148+0000 7f33a2ffd640 1 -- 172.21.15.150:0/1784842693 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f33a80598a0 con 0x7f33ac06ce60 2024-03-17T23:48:48.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.399+0000 7f33b339d640 1 -- 172.21.15.150:0/1784842693 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rwp", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]} v 0) -- 0x7f33ac15c8a0 con 0x7f33ac06ce60 2024-03-17T23:48:48.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.401+0000 7f33a2ffd640 1 -- 172.21.15.150:0/1784842693 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rwp", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]}]=0 updated caps for client.0 v15) ==== 211+0+168 (secure 0 0 0) 0x7f33a80591a0 con 0x7f33ac06ce60 2024-03-17T23:48:48.402 INFO:teuthology.orchestra.run.smithi150.stdout:[client.0] 2024-03-17T23:48:48.402 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQC3gPdlvlPJNRAAkUfetPJK4hZQG5rNiSDozw== 2024-03-17T23:48:48.402 INFO:teuthology.orchestra.run.smithi150.stdout: caps mds = "allow rwp" 2024-03-17T23:48:48.402 INFO:teuthology.orchestra.run.smithi150.stdout: caps mon = "allow r" 2024-03-17T23:48:48.402 INFO:teuthology.orchestra.run.smithi150.stdout: caps osd = "allow rw pool=cephfs_data,allow rw pool=data_new" 2024-03-17T23:48:48.402 INFO:teuthology.orchestra.run.smithi150.stderr:updated caps for client.0 2024-03-17T23:48:48.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.403+0000 7f33b339d640 1 -- 172.21.15.150:0/1784842693 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f339006bbd0 msgr2=0x7f339006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:48.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.403+0000 7f33b339d640 1 --2- 172.21.15.150:0/1784842693 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f339006bbd0 0x7f339006e090 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f339c036000 tx=0x7f339c03b040 comp rx=0 tx=0).stop 2024-03-17T23:48:48.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.403+0000 7f33b339d640 1 -- 172.21.15.150:0/1784842693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33ac06ce60 msgr2=0x7f33ac0b61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:48.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.403+0000 7f33b339d640 1 --2- 172.21.15.150:0/1784842693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33ac06ce60 0x7f33ac0b61f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f33a800c610 tx=0x7f33a800cad0 comp rx=0 tx=0).stop 2024-03-17T23:48:48.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.403+0000 7f33b339d640 1 -- 172.21.15.150:0/1784842693 shutdown_connections 2024-03-17T23:48:48.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.403+0000 7f33b339d640 1 --2- 172.21.15.150:0/1784842693 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f33ac15df40 0x7f33ac15fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:48.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.403+0000 7f33b339d640 1 --2- 172.21.15.150:0/1784842693 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f339006bbd0 0x7f339006e090 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:48.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.403+0000 7f33b339d640 1 --2- 172.21.15.150:0/1784842693 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f33ac0d23d0 0x7f33ac0b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:48.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.403+0000 7f33b339d640 1 --2- 172.21.15.150:0/1784842693 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33ac06ce60 0x7f33ac0b61f0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:48.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.403+0000 7f33b339d640 1 -- 172.21.15.150:0/1784842693 >> 172.21.15.150:0/1784842693 conn(0x7f33ac072ef0 msgr2=0x7f33ac15b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:48.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.403+0000 7f33b339d640 1 -- 172.21.15.150:0/1784842693 shutdown_connections 2024-03-17T23:48:48.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.403+0000 7f33b339d640 1 -- 172.21.15.150:0/1784842693 wait complete. 2024-03-17T23:48:48.415 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-17T23:48:48.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.518+0000 7f823add4640 1 Processor -- start 2024-03-17T23:48:48.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.518+0000 7f823add4640 1 -- start start 2024-03-17T23:48:48.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.519+0000 7f823add4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f82340cf120 0x7f82341619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:48.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.519+0000 7f823add4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82340b9860 0x7f82340d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:48.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.519+0000 7f823add4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82341620a0 con 0x7f823415de30 2024-03-17T23:48:48.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.519+0000 7f823add4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f8234162940 con 0x7f82340cf120 2024-03-17T23:48:48.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.519+0000 7f823add4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8234162380 con 0x7f82340b9860 2024-03-17T23:48:48.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.520+0000 7f8238b70640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f823415de30 0x7f82340cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49538/0 (socket says 172.21.15.150:49538) 2024-03-17T23:48:48.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.520+0000 7f8238b70640 1 -- 172.21.15.150:0/1418248622 learned_addr learned my addr 172.21.15.150:0/1418248622 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:48.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.520+0000 7f8239371640 1 --2- 172.21.15.150:0/1418248622 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82340b9860 0x7f82340d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:48.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.520+0000 7f82337fe640 1 -- 172.21.15.150:0/1418248622 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1232324541 0 0) 0x7f82341620a0 con 0x7f823415de30 2024-03-17T23:48:48.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.520+0000 7f8233fff640 1 --2- 172.21.15.150:0/1418248622 >> v2:172.21.15.157:3300/0 conn(0x7f82340cf120 0x7f82341619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:48.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.520+0000 7f82337fe640 1 -- 172.21.15.150:0/1418248622 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8224003520 con 0x7f823415de30 2024-03-17T23:48:48.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.520+0000 7f82337fe640 1 -- 172.21.15.150:0/1418248622 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 984657707 0 0) 0x7f8224003520 con 0x7f823415de30 2024-03-17T23:48:48.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.520+0000 7f82337fe640 1 -- 172.21.15.150:0/1418248622 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82340b9860 msgr2=0x7f82340d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:48.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.520+0000 7f82337fe640 1 --2- 172.21.15.150:0/1418248622 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82340b9860 0x7f82340d2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:48.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.520+0000 7f82337fe640 1 -- 172.21.15.150:0/1418248622 >> v2:172.21.15.157:3300/0 conn(0x7f82340cf120 msgr2=0x7f82341619a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:48.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.520+0000 7f82337fe640 1 --2- 172.21.15.150:0/1418248622 >> v2:172.21.15.157:3300/0 conn(0x7f82340cf120 0x7f82341619a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:48.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.520+0000 7f82337fe640 1 -- 172.21.15.150:0/1418248622 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8234162660 con 0x7f823415de30 2024-03-17T23:48:48.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.520+0000 7f82337fe640 1 -- 172.21.15.150:0/1418248622 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f8228001670 con 0x7f823415de30 2024-03-17T23:48:48.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.520+0000 7f82337fe640 1 -- 172.21.15.150:0/1418248622 >> v1:172.21.15.150:6789/0 conn(0x7f823415de30 legacy=0x7f82340cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:48.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.520+0000 7f82337fe640 1 --2- 172.21.15.150:0/1418248622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82240039b0 0x7f8224003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:48.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.520+0000 7f82337fe640 1 --2- 172.21.15.150:0/1418248622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8224004800 0x7f8224004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:48.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.520+0000 7f82337fe640 1 --2- 172.21.15.150:0/1418248622 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82340b9860 0x7f82240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:48.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.520+0000 7f82337fe640 1 -- 172.21.15.150:0/1418248622 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8234162940 con 0x7f82240039b0 2024-03-17T23:48:48.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.520+0000 7f82337fe640 1 -- 172.21.15.150:0/1418248622 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8234162380 con 0x7f8224004800 2024-03-17T23:48:48.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.520+0000 7f82337fe640 1 -- 172.21.15.150:0/1418248622 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f822400b9c0 con 0x7f82340b9860 2024-03-17T23:48:48.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.520+0000 7f8238b70640 1 --2- 172.21.15.150:0/1418248622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82240039b0 0x7f8224003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:48.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.520+0000 7f8239371640 1 --2- 172.21.15.150:0/1418248622 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82340b9860 0x7f82240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:48.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.520+0000 7f8233fff640 1 --2- 172.21.15.150:0/1418248622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8224004800 0x7f8224004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:48.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.521+0000 7f8238b70640 1 -- 172.21.15.150:0/1418248622 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82340b9860 msgr2=0x7f82240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:48.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.521+0000 7f8238b70640 1 --2- 172.21.15.150:0/1418248622 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82340b9860 0x7f82240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:48.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.521+0000 7f8238b70640 1 -- 172.21.15.150:0/1418248622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8224004800 msgr2=0x7f8224004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:48.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.521+0000 7f8238b70640 1 --2- 172.21.15.150:0/1418248622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8224004800 0x7f8224004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:48.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.521+0000 7f8238b70640 1 -- 172.21.15.150:0/1418248622 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8224003520 con 0x7f82240039b0 2024-03-17T23:48:48.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.521+0000 7f8238b70640 1 --2- 172.21.15.150:0/1418248622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82240039b0 0x7f8224003dd0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f822800c210 tx=0x7f8224005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55df8785b4f81d6b server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:48.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.521+0000 7f82337fe640 1 -- 172.21.15.150:0/1418248622 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8228007d50 con 0x7f82240039b0 2024-03-17T23:48:48.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.522+0000 7f82337fe640 1 -- 172.21.15.150:0/1418248622 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8228030070 con 0x7f82240039b0 2024-03-17T23:48:48.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.522+0000 7f82337fe640 1 -- 172.21.15.150:0/1418248622 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822802cd20 con 0x7f82240039b0 2024-03-17T23:48:48.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.522+0000 7f823add4640 1 -- 172.21.15.150:0/1418248622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82240039b0 msgr2=0x7f8224003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:48.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.522+0000 7f823add4640 1 --2- 172.21.15.150:0/1418248622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f82240039b0 0x7f8224003dd0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f822800c210 tx=0x7f8224005830 comp rx=0 tx=0).stop 2024-03-17T23:48:48.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.522+0000 7f8239371640 1 -- 172.21.15.150:0/1418248622 reap_dead start 2024-03-17T23:48:48.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.522+0000 7f823add4640 1 -- 172.21.15.150:0/1418248622 shutdown_connections 2024-03-17T23:48:48.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.522+0000 7f823add4640 1 -- 172.21.15.150:0/1418248622 >> 172.21.15.150:0/1418248622 conn(0x7f82340730b0 msgr2=0x7f82340d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:48.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.522+0000 7f823add4640 1 -- 172.21.15.150:0/1418248622 shutdown_connections 2024-03-17T23:48:48.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.522+0000 7f823add4640 1 -- 172.21.15.150:0/1418248622 wait complete. 2024-03-17T23:48:48.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.523+0000 7f823add4640 1 Processor -- start 2024-03-17T23:48:48.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.523+0000 7f823add4640 1 -- start start 2024-03-17T23:48:48.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.524+0000 7f823add4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82240039b0 0x7f8234160e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:48.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.524+0000 7f823add4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82340ace10 0x7f823415e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:48.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.524+0000 7f823add4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f823415e8d0 0x7f823415ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:48.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.524+0000 7f823add4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f823415e210 con 0x7f823415e8d0 2024-03-17T23:48:48.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.524+0000 7f823add4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f82340b9c60 con 0x7f82340ace10 2024-03-17T23:48:48.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.524+0000 7f823add4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f82340d2690 con 0x7f82240039b0 2024-03-17T23:48:48.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.524+0000 7f8239371640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f823415e8d0 0x7f823415ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:48.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.524+0000 7f8239371640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f823415e8d0 0x7f823415ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36440/0 (socket says 172.21.15.150:36440) 2024-03-17T23:48:48.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.524+0000 7f8239371640 1 -- 172.21.15.150:0/1384895159 learned_addr learned my addr 172.21.15.150:0/1384895159 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:48.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.525+0000 7f8238b70640 1 --2- 172.21.15.150:0/1384895159 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82240039b0 0x7f8234160e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:48.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.525+0000 7f8233fff640 1 --2- 172.21.15.150:0/1384895159 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82340ace10 0x7f823415e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:48.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.525+0000 7f8239371640 1 -- 172.21.15.150:0/1384895159 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82240039b0 msgr2=0x7f8234160e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:48.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.525+0000 7f8239371640 1 --2- 172.21.15.150:0/1384895159 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82240039b0 0x7f8234160e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:48.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.525+0000 7f8239371640 1 -- 172.21.15.150:0/1384895159 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82340ace10 msgr2=0x7f823415e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:48.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.525+0000 7f8239371640 1 --2- 172.21.15.150:0/1384895159 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82340ace10 0x7f823415e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:48.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.525+0000 7f8239371640 1 -- 172.21.15.150:0/1384895159 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f823414fb50 con 0x7f823415e8d0 2024-03-17T23:48:48.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.525+0000 7f8238b70640 1 --2- 172.21.15.150:0/1384895159 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82240039b0 0x7f8234160e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:48.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.525+0000 7f8239371640 1 --2- 172.21.15.150:0/1384895159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f823415e8d0 0x7f823415ed50 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f82340ce320 tx=0x7f822c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:48.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.525+0000 7f82317fa640 1 -- 172.21.15.150:0/1384895159 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822c00fe10 con 0x7f823415e8d0 2024-03-17T23:48:48.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.525+0000 7f82317fa640 1 -- 172.21.15.150:0/1384895159 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f822c015430 con 0x7f823415e8d0 2024-03-17T23:48:48.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.525+0000 7f82317fa640 1 -- 172.21.15.150:0/1384895159 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822c015820 con 0x7f823415e8d0 2024-03-17T23:48:48.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.525+0000 7f823add4640 1 -- 172.21.15.150:0/1384895159 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f823414fe30 con 0x7f823415e8d0 2024-03-17T23:48:48.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.525+0000 7f823add4640 1 -- 172.21.15.150:0/1384895159 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82341503f0 con 0x7f823415e8d0 2024-03-17T23:48:48.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.526+0000 7f823add4640 1 -- 172.21.15.150:0/1384895159 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82341505f0 con 0x7f823415e8d0 2024-03-17T23:48:48.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.526+0000 7f82317fa640 1 -- 172.21.15.150:0/1384895159 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f822c0159c0 con 0x7f823415e8d0 2024-03-17T23:48:48.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.527+0000 7f82317fa640 1 --2- 172.21.15.150:0/1384895159 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f821806bbd0 0x7f821806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:48.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.527+0000 7f82317fa640 1 -- 172.21.15.150:0/1384895159 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7f822c08bbd0 con 0x7f823415e8d0 2024-03-17T23:48:48.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.527+0000 7f8238b70640 1 --2- 172.21.15.150:0/1384895159 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f821806bbd0 0x7f821806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:48.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.527+0000 7f8238b70640 1 --2- 172.21.15.150:0/1384895159 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f821806bbd0 0x7f821806e090 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8228034000 tx=0x7f8228007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:48.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.535+0000 7f82317fa640 1 -- 172.21.15.150:0/1384895159 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f82341505f0 con 0x7f823415e8d0 2024-03-17T23:48:48.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.752+0000 7f823add4640 1 -- 172.21.15.150:0/1384895159 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f82340681a0 con 0x7f823415e8d0 2024-03-17T23:48:48.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.753+0000 7f82317fa640 1 -- 172.21.15.150:0/1384895159 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+82 (secure 0 0 0) 0x7f822c019030 con 0x7f823415e8d0 2024-03-17T23:48:48.753 INFO:teuthology.orchestra.run.smithi150.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data data_new ] 2024-03-17T23:48:48.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.755+0000 7f823add4640 1 -- 172.21.15.150:0/1384895159 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f821806bbd0 msgr2=0x7f821806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:48.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.755+0000 7f823add4640 1 --2- 172.21.15.150:0/1384895159 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f821806bbd0 0x7f821806e090 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8228034000 tx=0x7f8228007000 comp rx=0 tx=0).stop 2024-03-17T23:48:48.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.755+0000 7f823add4640 1 -- 172.21.15.150:0/1384895159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f823415e8d0 msgr2=0x7f823415ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:48.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.755+0000 7f823add4640 1 --2- 172.21.15.150:0/1384895159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f823415e8d0 0x7f823415ed50 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f82340ce320 tx=0x7f822c00f580 comp rx=0 tx=0).stop 2024-03-17T23:48:48.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.756+0000 7f823add4640 1 -- 172.21.15.150:0/1384895159 shutdown_connections 2024-03-17T23:48:48.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.756+0000 7f823add4640 1 --2- 172.21.15.150:0/1384895159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f823415e8d0 0x7f823415ed50 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:48.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.756+0000 7f823add4640 1 --2- 172.21.15.150:0/1384895159 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f821806bbd0 0x7f821806e090 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:48.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.756+0000 7f823add4640 1 --2- 172.21.15.150:0/1384895159 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82340ace10 0x7f823415e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:48.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.756+0000 7f823add4640 1 --2- 172.21.15.150:0/1384895159 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f82240039b0 0x7f8234160e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:48.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.756+0000 7f823add4640 1 -- 172.21.15.150:0/1384895159 >> 172.21.15.150:0/1384895159 conn(0x7f82340730b0 msgr2=0x7f82340d45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:48.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.756+0000 7f823add4640 1 -- 172.21.15.150:0/1384895159 shutdown_connections 2024-03-17T23:48:48.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48.756+0000 7f823add4640 1 -- 172.21.15.150:0/1384895159 wait complete. 2024-03-17T23:48:48.767 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-17T23:48:48.767 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-17T23:48:48.767 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi150.front.sepia.ceph.com 2024-03-17T23:48:48.767 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-17T23:48:48.767 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-17T23:48:48.767 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-17T23:48:48.767 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-17T23:48:48.767 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-17T23:48:48.767 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-17T23:48:48.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:48.768 DEBUG:teuthology.orchestra.run.smithi150:> ip addr 2024-03-17T23:48:48.783 INFO:teuthology.orchestra.run.smithi150.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-17T23:48:48.783 INFO:teuthology.orchestra.run.smithi150.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-17T23:48:48.783 INFO:teuthology.orchestra.run.smithi150.stdout: inet 127.0.0.1/8 scope host lo 2024-03-17T23:48:48.783 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:48:48.783 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 ::1/128 scope host 2024-03-17T23:48:48.784 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:48:48.784 INFO:teuthology.orchestra.run.smithi150.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:48:48.784 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bc brd ff:ff:ff:ff:ff:ff 2024-03-17T23:48:48.784 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f0 2024-03-17T23:48:48.784 INFO:teuthology.orchestra.run.smithi150.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:48:48.784 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bd brd ff:ff:ff:ff:ff:ff 2024-03-17T23:48:48.784 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f1 2024-03-17T23:48:48.784 INFO:teuthology.orchestra.run.smithi150.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:48:48.784 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:34 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:48:48.784 INFO:teuthology.orchestra.run.smithi150.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-17T23:48:48.784 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:35 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:48:48.785 INFO:teuthology.orchestra.run.smithi150.stdout: inet 172.21.15.150/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-17T23:48:48.785 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft 42326sec preferred_lft 42326sec 2024-03-17T23:48:48.785 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::ec4:7aff:fe88:6e35/64 scope link 2024-03-17T23:48:48.785 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:48:48.785 INFO:teuthology.orchestra.run.smithi150.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-17T23:48:48.785 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether f6:2d:aa:e3:d6:a3 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:48:48.785 INFO:teuthology.orchestra.run.smithi150.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-17T23:48:48.785 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:48:48.785 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::bc2f:65ff:fe6d:abc3/64 scope link 2024-03-17T23:48:48.785 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:48:48.785 INFO:teuthology.orchestra.run.smithi150.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-17T23:48:48.785 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether f6:2d:aa:e3:d6:a3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:48:48.785 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::f42d:aaff:fee3:d6a3/64 scope link 2024-03-17T23:48:48.785 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:48:48.786 INFO:teuthology.orchestra.run.smithi150.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-17T23:48:48.786 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether fa:95:93:4e:f1:dd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:48:48.786 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::f895:93ff:fe4e:f1dd/64 scope link 2024-03-17T23:48:48.786 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:48:48.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:48.787 DEBUG:teuthology.orchestra.run.smithi150:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-17T23:48:48.848 INFO:teuthology.orchestra.run.smithi150.stdout:1 2024-03-17T23:48:48.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:48.850 DEBUG:teuthology.orchestra.run.smithi150:> route 2024-03-17T23:48:48.905 INFO:teuthology.orchestra.run.smithi150.stdout:Kernel IP routing table 2024-03-17T23:48:48.905 INFO:teuthology.orchestra.run.smithi150.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-17T23:48:48.906 INFO:teuthology.orchestra.run.smithi150.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-17T23:48:48.906 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-17T23:48:48.906 INFO:teuthology.orchestra.run.smithi150.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-17T23:48:48.906 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:48:48.906 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-03-17T23:48:48.906 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-17T23:48:48.906 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-17T23:48:48.906 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-17T23:48:48.907 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-03-17T23:48:48.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:48:49.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:48:49.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:49.042 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-03-17T23:48:49.056 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-17T23:48:49.056 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-17T23:48:49.056 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-17T23:48:49.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:49.056 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set brx.0 up 2024-03-17T23:48:49.124 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-17T23:48:49.124 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-17T23:48:49.124 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-17T23:48:49.180 INFO:teuthology.orchestra.run.smithi150.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-17T23:48:49.180 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-17T23:48:49.180 DEBUG:teuthology.orchestra.run.smithi150:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-17T23:48:49.235 DEBUG:teuthology.orchestra.run.smithi150:> sudo modprobe fuse 2024-03-17T23:48:49.305 DEBUG:teuthology.orchestra.run.smithi150:> cat /proc/self/mounts | awk '{print $2}' 2024-03-17T23:48:49.361 INFO:teuthology.orchestra.run.smithi150.stdout:/proc 2024-03-17T23:48:49.361 INFO:teuthology.orchestra.run.smithi150.stdout:/sys 2024-03-17T23:48:49.361 INFO:teuthology.orchestra.run.smithi150.stdout:/dev 2024-03-17T23:48:49.361 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/security 2024-03-17T23:48:49.362 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/shm 2024-03-17T23:48:49.362 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/pts 2024-03-17T23:48:49.362 INFO:teuthology.orchestra.run.smithi150.stdout:/run 2024-03-17T23:48:49.362 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/cgroup 2024-03-17T23:48:49.362 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/pstore 2024-03-17T23:48:49.362 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/bpf 2024-03-17T23:48:49.362 INFO:teuthology.orchestra.run.smithi150.stdout:/ 2024-03-17T23:48:49.362 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/selinux 2024-03-17T23:48:49.362 INFO:teuthology.orchestra.run.smithi150.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:48:49.362 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/hugepages 2024-03-17T23:48:49.362 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/mqueue 2024-03-17T23:48:49.362 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/debug 2024-03-17T23:48:49.362 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/tracing 2024-03-17T23:48:49.362 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-17T23:48:49.363 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/fuse/connections 2024-03-17T23:48:49.363 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/config 2024-03-17T23:48:49.363 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-sysctl.service 2024-03-17T23:48:49.363 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-17T23:48:49.363 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/nfs/rpc_pipefs 2024-03-17T23:48:49.363 INFO:teuthology.orchestra.run.smithi150.stdout:/run/user/1000 2024-03-17T23:48:49.363 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph 2024-03-17T23:48:49.363 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-0 2024-03-17T23:48:49.363 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-1 2024-03-17T23:48:49.363 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-2 2024-03-17T23:48:49.363 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-3 2024-03-17T23:48:49.363 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns 2024-03-17T23:48:49.363 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:48:49.363 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:48:49.364 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:48:49.364 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:48:49.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:49.364 DEBUG:teuthology.orchestra.run.smithi150:> ls /sys/fs/fuse/connections 2024-03-17T23:48:49.416 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-17T23:48:49.416 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-17T23:48:49.458 DEBUG:teuthology.orchestra.run.smithi150:> sudo modprobe fuse 2024-03-17T23:48:49.488 DEBUG:teuthology.orchestra.run.smithi150:> cat /proc/self/mounts | awk '{print $2}' 2024-03-17T23:48:49.544 INFO:teuthology.orchestra.run.smithi150.stdout:/proc 2024-03-17T23:48:49.544 INFO:teuthology.orchestra.run.smithi150.stdout:/sys 2024-03-17T23:48:49.544 INFO:teuthology.orchestra.run.smithi150.stdout:/dev 2024-03-17T23:48:49.544 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/security 2024-03-17T23:48:49.544 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/shm 2024-03-17T23:48:49.544 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/pts 2024-03-17T23:48:49.544 INFO:teuthology.orchestra.run.smithi150.stdout:/run 2024-03-17T23:48:49.544 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/cgroup 2024-03-17T23:48:49.545 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/pstore 2024-03-17T23:48:49.545 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/bpf 2024-03-17T23:48:49.545 INFO:teuthology.orchestra.run.smithi150.stdout:/ 2024-03-17T23:48:49.545 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/selinux 2024-03-17T23:48:49.545 INFO:teuthology.orchestra.run.smithi150.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:48:49.545 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/hugepages 2024-03-17T23:48:49.545 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/mqueue 2024-03-17T23:48:49.545 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/debug 2024-03-17T23:48:49.545 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/tracing 2024-03-17T23:48:49.545 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-17T23:48:49.545 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/fuse/connections 2024-03-17T23:48:49.545 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/config 2024-03-17T23:48:49.545 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-sysctl.service 2024-03-17T23:48:49.545 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-17T23:48:49.546 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/nfs/rpc_pipefs 2024-03-17T23:48:49.546 INFO:teuthology.orchestra.run.smithi150.stdout:/run/user/1000 2024-03-17T23:48:49.546 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph 2024-03-17T23:48:49.546 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-0 2024-03-17T23:48:49.546 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-1 2024-03-17T23:48:49.546 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-2 2024-03-17T23:48:49.546 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-3 2024-03-17T23:48:49.546 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns 2024-03-17T23:48:49.546 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:48:49.546 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:48:49.546 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:48:49.546 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:48:49.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:49.547 DEBUG:teuthology.orchestra.run.smithi150:> ls /sys/fs/fuse/connections 2024-03-17T23:48:49.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:2024-03-17T23:48:49.544+0000 7f19ba160480 -1 init, newargv = 0x55e8f1b779e0 newargc=15 2024-03-17T23:48:49.548 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:ceph-fuse[57019]: starting ceph client 2024-03-17T23:48:49.550 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:ceph-fuse[57019]: starting fuse 2024-03-17T23:48:49.563 INFO:teuthology.orchestra.run.smithi150.stdout:43 2024-03-17T23:48:49.563 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-17T23:48:49.563 DEBUG:teuthology.orchestra.run.smithi150:> sudo stdin-killer -- python3 -c ' 2024-03-17T23:48:49.563 DEBUG:teuthology.orchestra.run.smithi150:> import glob 2024-03-17T23:48:49.563 DEBUG:teuthology.orchestra.run.smithi150:> import re 2024-03-17T23:48:49.563 DEBUG:teuthology.orchestra.run.smithi150:> import os 2024-03-17T23:48:49.563 DEBUG:teuthology.orchestra.run.smithi150:> import subprocess 2024-03-17T23:48:49.563 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:48:49.563 DEBUG:teuthology.orchestra.run.smithi150:> def _find_admin_socket(client_name): 2024-03-17T23:48:49.564 DEBUG:teuthology.orchestra.run.smithi150:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-17T23:48:49.564 DEBUG:teuthology.orchestra.run.smithi150:> files = glob.glob(asok_path) 2024-03-17T23:48:49.564 DEBUG:teuthology.orchestra.run.smithi150:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-17T23:48:49.564 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:48:49.564 DEBUG:teuthology.orchestra.run.smithi150:> # Given a non-glob path, it better be there 2024-03-17T23:48:49.564 DEBUG:teuthology.orchestra.run.smithi150:> if "*" not in asok_path: 2024-03-17T23:48:49.564 DEBUG:teuthology.orchestra.run.smithi150:> assert(len(files) == 1) 2024-03-17T23:48:49.564 DEBUG:teuthology.orchestra.run.smithi150:> return files[0] 2024-03-17T23:48:49.564 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:48:49.564 DEBUG:teuthology.orchestra.run.smithi150:> for f in files: 2024-03-17T23:48:49.564 DEBUG:teuthology.orchestra.run.smithi150:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-17T23:48:49.564 DEBUG:teuthology.orchestra.run.smithi150:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-17T23:48:49.564 DEBUG:teuthology.orchestra.run.smithi150:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-17T23:48:49.564 DEBUG:teuthology.orchestra.run.smithi150:> contents = proc_f.read() 2024-03-17T23:48:49.564 DEBUG:teuthology.orchestra.run.smithi150:> if mountpoint in contents: 2024-03-17T23:48:49.565 DEBUG:teuthology.orchestra.run.smithi150:> return f 2024-03-17T23:48:49.565 DEBUG:teuthology.orchestra.run.smithi150:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-17T23:48:49.565 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:48:49.565 DEBUG:teuthology.orchestra.run.smithi150:> print(_find_admin_socket("client.0")) 2024-03-17T23:48:49.565 DEBUG:teuthology.orchestra.run.smithi150:> ' 2024-03-17T23:48:49.683 INFO:teuthology.orchestra.run.smithi150.stdout:/var/run/ceph/ceph-client.0.57019.asok 2024-03-17T23:48:49.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:48:49.694 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.57019.asok 2024-03-17T23:48:49.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:49.694 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.57019.asok status 2024-03-17T23:48:49.832 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-17T23:48:49.832 INFO:teuthology.orchestra.run.smithi150.stdout: "metadata": { 2024-03-17T23:48:49.832 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-17T23:48:49.833 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-17T23:48:49.833 INFO:teuthology.orchestra.run.smithi150.stdout: "entity_id": "0", 2024-03-17T23:48:49.833 INFO:teuthology.orchestra.run.smithi150.stdout: "hostname": "smithi150", 2024-03-17T23:48:49.833 INFO:teuthology.orchestra.run.smithi150.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-17T23:48:49.833 INFO:teuthology.orchestra.run.smithi150.stdout: "pid": "57019", 2024-03-17T23:48:49.833 INFO:teuthology.orchestra.run.smithi150.stdout: "root": "/" 2024-03-17T23:48:49.833 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:49.833 INFO:teuthology.orchestra.run.smithi150.stdout: "dentry_count": 0, 2024-03-17T23:48:49.833 INFO:teuthology.orchestra.run.smithi150.stdout: "dentry_pinned_count": 0, 2024-03-17T23:48:49.833 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 5931, 2024-03-17T23:48:49.833 INFO:teuthology.orchestra.run.smithi150.stdout: "inst": { 2024-03-17T23:48:49.833 INFO:teuthology.orchestra.run.smithi150.stdout: "name": { 2024-03-17T23:48:49.833 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "client", 2024-03-17T23:48:49.834 INFO:teuthology.orchestra.run.smithi150.stdout: "num": 5931 2024-03-17T23:48:49.834 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:49.834 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": { 2024-03-17T23:48:49.834 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v1", 2024-03-17T23:48:49.834 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "192.168.0.1:0", 2024-03-17T23:48:49.834 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3602818426 2024-03-17T23:48:49.834 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:48:49.834 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:49.834 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": { 2024-03-17T23:48:49.834 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v1", 2024-03-17T23:48:49.834 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "192.168.0.1:0", 2024-03-17T23:48:49.834 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3602818426 2024-03-17T23:48:49.834 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:48:49.835 INFO:teuthology.orchestra.run.smithi150.stdout: "inst_str": "client.5931 192.168.0.1:0/3602818426", 2024-03-17T23:48:49.835 INFO:teuthology.orchestra.run.smithi150.stdout: "addr_str": "192.168.0.1:0/3602818426", 2024-03-17T23:48:49.835 INFO:teuthology.orchestra.run.smithi150.stdout: "inode_count": 1, 2024-03-17T23:48:49.835 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_epoch": 16, 2024-03-17T23:48:49.835 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_epoch": 45, 2024-03-17T23:48:49.835 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_epoch_barrier": 0, 2024-03-17T23:48:49.835 INFO:teuthology.orchestra.run.smithi150.stdout: "blocklisted": false, 2024-03-17T23:48:49.835 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "cephfs" 2024-03-17T23:48:49.835 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-17T23:48:49.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:49.841 DEBUG:teuthology.orchestra.run.smithi150:> stat --file-system '--printf=%T 2024-03-17T23:48:49.841 DEBUG:teuthology.orchestra.run.smithi150:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-17T23:48:49.857 INFO:teuthology.orchestra.run.smithi150.stdout:fuseblk 2024-03-17T23:48:49.857 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-17T23:48:49.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:49.857 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-17T23:48:49.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:49.924 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout.pool -v data_new layoutfile) 2024-03-17T23:48:49.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:49.981 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.layout.pool -v data_new layoutdir) 2024-03-17T23:48:50.041 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:48:50.041 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:48:50.096 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:48:50.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.236+0000 7f4c2a313640 1 Processor -- start 2024-03-17T23:48:50.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.236+0000 7f4c2a313640 1 -- start start 2024-03-17T23:48:50.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.237+0000 7f4c2a313640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c2415de30 0x7f4c241619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:50.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.237+0000 7f4c2a313640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4c240cf120 0x7f4c240d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:50.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.237+0000 7f4c2a313640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c240b9cc0 con 0x7f4c240b9860 2024-03-17T23:48:50.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.237+0000 7f4c2a313640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f4c24162940 con 0x7f4c240cf120 2024-03-17T23:48:50.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.237+0000 7f4c2a313640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4c241620a0 con 0x7f4c2415de30 2024-03-17T23:48:50.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.237+0000 7f4c23fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c2415de30 0x7f4c241619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:50.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.237+0000 7f4c288b0640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4c240cf120 0x7f4c240d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:50.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.237+0000 7f4c237fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f4c240b9860 0x7f4c240cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:54684/0 (socket says 172.21.15.150:54684) 2024-03-17T23:48:50.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.237+0000 7f4c288b0640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4c240cf120 0x7f4c240d2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:53016/0 (socket says 172.21.15.150:53016) 2024-03-17T23:48:50.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.237+0000 7f4c23fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c2415de30 0x7f4c241619a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:54822/0 (socket says 172.21.15.150:54822) 2024-03-17T23:48:50.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.237+0000 7f4c237fe640 1 -- 172.21.15.150:0/2706212861 learned_addr learned my addr 172.21.15.150:0/2706212861 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:50.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.237+0000 7f4c288b0640 1 -- 172.21.15.150:0/2706212861 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c2415de30 msgr2=0x7f4c241619a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:50.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.237+0000 7f4c288b0640 1 --2- 172.21.15.150:0/2706212861 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c2415de30 0x7f4c241619a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.237+0000 7f4c288b0640 1 -- 172.21.15.150:0/2706212861 >> v1:172.21.15.157:6789/0 conn(0x7f4c240b9860 legacy=0x7f4c240cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:50.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.237+0000 7f4c288b0640 1 -- 172.21.15.150:0/2706212861 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c24162380 con 0x7f4c240cf120 2024-03-17T23:48:50.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.237+0000 7f4c23fff640 1 --2- 172.21.15.150:0/2706212861 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c2415de30 0x7f4c241619a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:48:50.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.238+0000 7f4c288b0640 1 --2- 172.21.15.150:0/2706212861 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c240cf120 0x7f4c240d2690 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c009d90 tx=0x7f4c1c02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=182721a98189d1ec server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:50.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.238+0000 7f4c22ffd640 1 -- 172.21.15.150:0/2706212861 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c039070 con 0x7f4c240cf120 2024-03-17T23:48:50.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.238+0000 7f4c22ffd640 1 -- 172.21.15.150:0/2706212861 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c1c02cbc0 con 0x7f4c240cf120 2024-03-17T23:48:50.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.238+0000 7f4c22ffd640 1 -- 172.21.15.150:0/2706212861 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c034440 con 0x7f4c240cf120 2024-03-17T23:48:50.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.239+0000 7f4c2a313640 1 -- 172.21.15.150:0/2706212861 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c240cf120 msgr2=0x7f4c240d2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:50.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.239+0000 7f4c2a313640 1 --2- 172.21.15.150:0/2706212861 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c240cf120 0x7f4c240d2690 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c009d90 tx=0x7f4c1c02be10 comp rx=0 tx=0).stop 2024-03-17T23:48:50.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.239+0000 7f4c2a313640 1 -- 172.21.15.150:0/2706212861 shutdown_connections 2024-03-17T23:48:50.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.239+0000 7f4c2a313640 1 --2- 172.21.15.150:0/2706212861 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c240cf120 0x7f4c240d2690 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.239+0000 7f4c2a313640 1 --2- 172.21.15.150:0/2706212861 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c2415de30 0x7f4c241619a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.239+0000 7f4c2a313640 1 -- 172.21.15.150:0/2706212861 >> 172.21.15.150:0/2706212861 conn(0x7f4c240730b0 msgr2=0x7f4c240d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:50.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.239+0000 7f4c2a313640 1 -- 172.21.15.150:0/2706212861 shutdown_connections 2024-03-17T23:48:50.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.239+0000 7f4c2a313640 1 -- 172.21.15.150:0/2706212861 wait complete. 2024-03-17T23:48:50.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.240+0000 7f4c2a313640 1 Processor -- start 2024-03-17T23:48:50.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.241+0000 7f4c2a313640 1 -- start start 2024-03-17T23:48:50.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.241+0000 7f4c2a313640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c240b9860 0x7f4c241e7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:50.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.241+0000 7f4c2a313640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c240cf120 0x7f4c240cb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:50.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.241+0000 7f4c2a313640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c2415de30 0x7f4c24153f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:50.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.241+0000 7f4c2a313640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2415e210 con 0x7f4c240cf120 2024-03-17T23:48:50.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.241+0000 7f4c2a313640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4c240d2170 con 0x7f4c240b9860 2024-03-17T23:48:50.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.241+0000 7f4c2a313640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4c240d2bd0 con 0x7f4c2415de30 2024-03-17T23:48:50.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.241+0000 7f4c237fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c240cf120 0x7f4c240cb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:50.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.241+0000 7f4c237fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c240cf120 0x7f4c240cb080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36454/0 (socket says 172.21.15.150:36454) 2024-03-17T23:48:50.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.241+0000 7f4c237fe640 1 -- 172.21.15.150:0/2546428478 learned_addr learned my addr 172.21.15.150:0/2546428478 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:50.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.242+0000 7f4c288b0640 1 --2- 172.21.15.150:0/2546428478 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c2415de30 0x7f4c24153f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:50.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.242+0000 7f4c23fff640 1 --2- 172.21.15.150:0/2546428478 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c240b9860 0x7f4c241e7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:50.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.242+0000 7f4c237fe640 1 -- 172.21.15.150:0/2546428478 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c2415de30 msgr2=0x7f4c24153f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:50.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.242+0000 7f4c237fe640 1 --2- 172.21.15.150:0/2546428478 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c2415de30 0x7f4c24153f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.242+0000 7f4c237fe640 1 -- 172.21.15.150:0/2546428478 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c240b9860 msgr2=0x7f4c241e7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:50.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.242+0000 7f4c237fe640 1 --2- 172.21.15.150:0/2546428478 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c240b9860 0x7f4c241e7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.242+0000 7f4c237fe640 1 -- 172.21.15.150:0/2546428478 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c24141270 con 0x7f4c240cf120 2024-03-17T23:48:50.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.242+0000 7f4c237fe640 1 --2- 172.21.15.150:0/2546428478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c240cf120 0x7f4c240cb080 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f4c1400c400 tx=0x7f4c1400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:50.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.242+0000 7f4c20ff9640 1 -- 172.21.15.150:0/2546428478 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c14017070 con 0x7f4c240cf120 2024-03-17T23:48:50.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.242+0000 7f4c2a313640 1 -- 172.21.15.150:0/2546428478 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c24141550 con 0x7f4c240cf120 2024-03-17T23:48:50.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.242+0000 7f4c20ff9640 1 -- 172.21.15.150:0/2546428478 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c14005ce0 con 0x7f4c240cf120 2024-03-17T23:48:50.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.242+0000 7f4c20ff9640 1 -- 172.21.15.150:0/2546428478 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c140072f0 con 0x7f4c240cf120 2024-03-17T23:48:50.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.242+0000 7f4c2a313640 1 -- 172.21.15.150:0/2546428478 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c24141b10 con 0x7f4c240cf120 2024-03-17T23:48:50.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.243+0000 7f4c2a313640 1 -- 172.21.15.150:0/2546428478 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c240681a0 con 0x7f4c240cf120 2024-03-17T23:48:50.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.244+0000 7f4c20ff9640 1 -- 172.21.15.150:0/2546428478 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4c1401a070 con 0x7f4c240cf120 2024-03-17T23:48:50.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.244+0000 7f4c20ff9640 1 --2- 172.21.15.150:0/2546428478 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4c1006bbd0 0x7f4c1006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:50.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.244+0000 7f4c23fff640 1 --2- 172.21.15.150:0/2546428478 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4c1006bbd0 0x7f4c1006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:50.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.244+0000 7f4c20ff9640 1 -- 172.21.15.150:0/2546428478 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7f4c1408a0a0 con 0x7f4c240cf120 2024-03-17T23:48:50.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.245+0000 7f4c23fff640 1 --2- 172.21.15.150:0/2546428478 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4c1006bbd0 0x7f4c1006e090 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4c24160b00 tx=0x7f4c18008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:50.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.248+0000 7f4c20ff9640 1 -- 172.21.15.150:0/2546428478 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4c140588f0 con 0x7f4c240cf120 2024-03-17T23:48:50.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.426+0000 7f4c2a313640 1 -- 172.21.15.150:0/2546428478 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4c240d1910 con 0x7f4c240cf120 2024-03-17T23:48:50.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.427+0000 7f4c20ff9640 1 -- 172.21.15.150:0/2546428478 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+18839 (secure 0 0 0) 0x7f4c140581f0 con 0x7f4c240cf120 2024-03-17T23:48:50.428 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:50.428 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":45,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:48:37.677776+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:48:08.957879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-17T23:48:09.698808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-17T23:48:22.562269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.3d","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:48:50.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.429+0000 7f4c2a313640 1 -- 172.21.15.150:0/2546428478 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4c1006bbd0 msgr2=0x7f4c1006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:50.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.429+0000 7f4c2a313640 1 --2- 172.21.15.150:0/2546428478 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4c1006bbd0 0x7f4c1006e090 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4c24160b00 tx=0x7f4c18008040 comp rx=0 tx=0).stop 2024-03-17T23:48:50.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.429+0000 7f4c2a313640 1 -- 172.21.15.150:0/2546428478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c240cf120 msgr2=0x7f4c240cb080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:50.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.429+0000 7f4c2a313640 1 --2- 172.21.15.150:0/2546428478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c240cf120 0x7f4c240cb080 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f4c1400c400 tx=0x7f4c1400c8c0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.430+0000 7f4c2a313640 1 -- 172.21.15.150:0/2546428478 shutdown_connections 2024-03-17T23:48:50.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.430+0000 7f4c2a313640 1 --2- 172.21.15.150:0/2546428478 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c2415de30 0x7f4c24153f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.430+0000 7f4c2a313640 1 --2- 172.21.15.150:0/2546428478 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c240cf120 0x7f4c240cb080 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.430+0000 7f4c2a313640 1 --2- 172.21.15.150:0/2546428478 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4c1006bbd0 0x7f4c1006e090 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.430+0000 7f4c2a313640 1 --2- 172.21.15.150:0/2546428478 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c240b9860 0x7f4c241e7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.430+0000 7f4c2a313640 1 -- 172.21.15.150:0/2546428478 >> 172.21.15.150:0/2546428478 conn(0x7f4c240730b0 msgr2=0x7f4c240b8720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:50.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.430+0000 7f4c2a313640 1 -- 172.21.15.150:0/2546428478 shutdown_connections 2024-03-17T23:48:50.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.430+0000 7f4c2a313640 1 -- 172.21.15.150:0/2546428478 wait complete. 2024-03-17T23:48:50.442 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:48:08.957879+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:48:09.698808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-17T23:48:22.562269+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-17T23:48:50.442 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:48:50.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.549+0000 7fcd5a590640 1 Processor -- start 2024-03-17T23:48:50.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.550+0000 7fcd5a590640 1 -- start start 2024-03-17T23:48:50.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.550+0000 7fcd5a590640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcd540d0ce0 0x7fcd540b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:50.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.550+0000 7fcd5a590640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcd540b2bf0 0x7fcd540d1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:50.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.550+0000 7fcd5a590640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd54161d30 con 0x7fcd54161520 2024-03-17T23:48:50.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.550+0000 7fcd5a590640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fcd540b9bf0 con 0x7fcd540d0ce0 2024-03-17T23:48:50.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.550+0000 7fcd5a590640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fcd54162940 con 0x7fcd540b2bf0 2024-03-17T23:48:50.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.550+0000 7fcd537fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcd540d0ce0 0x7fcd540b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:50.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.550+0000 7fcd53fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fcd54161520 0x7fcd540d05d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49550/0 (socket says 172.21.15.150:49550) 2024-03-17T23:48:50.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.550+0000 7fcd53fff640 1 -- 172.21.15.150:0/2636266197 learned_addr learned my addr 172.21.15.150:0/2636266197 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:50.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.551+0000 7fcd58b2d640 1 --2- 172.21.15.150:0/2636266197 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcd540b2bf0 0x7fcd540d1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:50.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.551+0000 7fcd537fe640 1 -- 172.21.15.150:0/2636266197 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcd540b2bf0 msgr2=0x7fcd540d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:50.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.551+0000 7fcd537fe640 1 --2- 172.21.15.150:0/2636266197 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcd540b2bf0 0x7fcd540d1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.551+0000 7fcd537fe640 1 -- 172.21.15.150:0/2636266197 >> v1:172.21.15.150:6789/0 conn(0x7fcd54161520 legacy=0x7fcd540d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:50.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.551+0000 7fcd537fe640 1 -- 172.21.15.150:0/2636266197 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd54162010 con 0x7fcd540d0ce0 2024-03-17T23:48:50.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.551+0000 7fcd52ffd640 1 -- 172.21.15.150:0/2636266197 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2090329131 0 0) 0x7fcd54161d30 con 0x7fcd54161520 2024-03-17T23:48:50.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.551+0000 7fcd58b2d640 1 --2- 172.21.15.150:0/2636266197 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcd540b2bf0 0x7fcd540d1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:50.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.552+0000 7fcd537fe640 1 --2- 172.21.15.150:0/2636266197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd540d0ce0 0x7fcd540b96b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fcd480097a0 tx=0x7fcd4802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a579a785162756 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:50.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.552+0000 7fcd52ffd640 1 -- 172.21.15.150:0/2636266197 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd48038070 con 0x7fcd540d0ce0 2024-03-17T23:48:50.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.552+0000 7fcd52ffd640 1 -- 172.21.15.150:0/2636266197 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd48031700 con 0x7fcd540d0ce0 2024-03-17T23:48:50.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.552+0000 7fcd52ffd640 1 -- 172.21.15.150:0/2636266197 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd48031ad0 con 0x7fcd540d0ce0 2024-03-17T23:48:50.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.553+0000 7fcd5a590640 1 -- 172.21.15.150:0/2636266197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd540d0ce0 msgr2=0x7fcd540b96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:50.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.553+0000 7fcd5a590640 1 --2- 172.21.15.150:0/2636266197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd540d0ce0 0x7fcd540b96b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fcd480097a0 tx=0x7fcd4802b7a0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.553+0000 7fcd5a590640 1 -- 172.21.15.150:0/2636266197 shutdown_connections 2024-03-17T23:48:50.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.553+0000 7fcd5a590640 1 --2- 172.21.15.150:0/2636266197 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcd540b2bf0 0x7fcd540d1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.553+0000 7fcd5a590640 1 --2- 172.21.15.150:0/2636266197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd540d0ce0 0x7fcd540b96b0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.553+0000 7fcd5a590640 1 -- 172.21.15.150:0/2636266197 >> 172.21.15.150:0/2636266197 conn(0x7fcd5406ca00 msgr2=0x7fcd5406ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:50.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.553+0000 7fcd5a590640 1 -- 172.21.15.150:0/2636266197 shutdown_connections 2024-03-17T23:48:50.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.553+0000 7fcd5a590640 1 -- 172.21.15.150:0/2636266197 wait complete. 2024-03-17T23:48:50.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.554+0000 7fcd5a590640 1 Processor -- start 2024-03-17T23:48:50.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.554+0000 7fcd5a590640 1 -- start start 2024-03-17T23:48:50.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.555+0000 7fcd5a590640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd540b2bf0 0x7fcd54158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:50.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.555+0000 7fcd5a590640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd540d0ce0 0x7fcd541f4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:50.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.555+0000 7fcd5a590640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcd54161520 0x7fcd541f4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:50.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.555+0000 7fcd5a590640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcd54161900 con 0x7fcd540b2bf0 2024-03-17T23:48:50.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.555+0000 7fcd5a590640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcd541625d0 con 0x7fcd540d0ce0 2024-03-17T23:48:50.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.555+0000 7fcd5a590640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fcd540d24e0 con 0x7fcd54161520 2024-03-17T23:48:50.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.555+0000 7fcd53fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd540b2bf0 0x7fcd54158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:50.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.555+0000 7fcd53fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd540b2bf0 0x7fcd54158cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36468/0 (socket says 172.21.15.150:36468) 2024-03-17T23:48:50.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.555+0000 7fcd53fff640 1 -- 172.21.15.150:0/1856287865 learned_addr learned my addr 172.21.15.150:0/1856287865 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:50.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.555+0000 7fcd58b2d640 1 --2- 172.21.15.150:0/1856287865 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcd54161520 0x7fcd541f4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:50.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.555+0000 7fcd537fe640 1 --2- 172.21.15.150:0/1856287865 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd540d0ce0 0x7fcd541f4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:50.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.555+0000 7fcd53fff640 1 -- 172.21.15.150:0/1856287865 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcd54161520 msgr2=0x7fcd541f4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:50.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.555+0000 7fcd53fff640 1 --2- 172.21.15.150:0/1856287865 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcd54161520 0x7fcd541f4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.555+0000 7fcd53fff640 1 -- 172.21.15.150:0/1856287865 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd540d0ce0 msgr2=0x7fcd541f4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:50.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.555+0000 7fcd53fff640 1 --2- 172.21.15.150:0/1856287865 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd540d0ce0 0x7fcd541f4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.555+0000 7fcd53fff640 1 -- 172.21.15.150:0/1856287865 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd540cdae0 con 0x7fcd540b2bf0 2024-03-17T23:48:50.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.555+0000 7fcd537fe640 1 --2- 172.21.15.150:0/1856287865 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd540d0ce0 0x7fcd541f4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:50.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.555+0000 7fcd53fff640 1 --2- 172.21.15.150:0/1856287865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd540b2bf0 0x7fcd54158cc0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fcd4400c430 tx=0x7fcd4400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:50.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.555+0000 7fcd50ff9640 1 -- 172.21.15.150:0/1856287865 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd440074e0 con 0x7fcd540b2bf0 2024-03-17T23:48:50.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.555+0000 7fcd50ff9640 1 -- 172.21.15.150:0/1856287865 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd440047e0 con 0x7fcd540b2bf0 2024-03-17T23:48:50.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.555+0000 7fcd5a590640 1 -- 172.21.15.150:0/1856287865 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd54157f90 con 0x7fcd540b2bf0 2024-03-17T23:48:50.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.556+0000 7fcd50ff9640 1 -- 172.21.15.150:0/1856287865 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd44004bd0 con 0x7fcd540b2bf0 2024-03-17T23:48:50.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.556+0000 7fcd5a590640 1 -- 172.21.15.150:0/1856287865 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd54158550 con 0x7fcd540b2bf0 2024-03-17T23:48:50.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.557+0000 7fcd5a590640 1 -- 172.21.15.150:0/1856287865 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd540b3140 con 0x7fcd540b2bf0 2024-03-17T23:48:50.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.557+0000 7fcd50ff9640 1 -- 172.21.15.150:0/1856287865 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fcd44011070 con 0x7fcd540b2bf0 2024-03-17T23:48:50.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.558+0000 7fcd50ff9640 1 --2- 172.21.15.150:0/1856287865 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcd4006bbd0 0x7fcd4006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:50.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.558+0000 7fcd537fe640 1 --2- 172.21.15.150:0/1856287865 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcd4006bbd0 0x7fcd4006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:50.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.558+0000 7fcd50ff9640 1 -- 172.21.15.150:0/1856287865 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7fcd4408aa50 con 0x7fcd540b2bf0 2024-03-17T23:48:50.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.558+0000 7fcd537fe640 1 --2- 172.21.15.150:0/1856287865 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcd4006bbd0 0x7fcd4006e090 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fcd48035000 tx=0x7fcd48008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:50.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.562+0000 7fcd50ff9640 1 -- 172.21.15.150:0/1856287865 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fcd540b3140 con 0x7fcd540b2bf0 2024-03-17T23:48:50.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.742+0000 7fcd5a590640 1 -- 172.21.15.150:0/1856287865 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcd540681a0 con 0x7fcd540b2bf0 2024-03-17T23:48:50.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.743+0000 7fcd50ff9640 1 -- 172.21.15.150:0/1856287865 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v45) ==== 93+0+10 (secure 0 0 0) 0x7fcd4400e280 con 0x7fcd540b2bf0 2024-03-17T23:48:50.743 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:48:50.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.745+0000 7fcd5a590640 1 -- 172.21.15.150:0/1856287865 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcd4006bbd0 msgr2=0x7fcd4006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:50.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.745+0000 7fcd5a590640 1 --2- 172.21.15.150:0/1856287865 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcd4006bbd0 0x7fcd4006e090 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fcd48035000 tx=0x7fcd48008030 comp rx=0 tx=0).stop 2024-03-17T23:48:50.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.745+0000 7fcd5a590640 1 -- 172.21.15.150:0/1856287865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd540b2bf0 msgr2=0x7fcd54158cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:50.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.745+0000 7fcd5a590640 1 --2- 172.21.15.150:0/1856287865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd540b2bf0 0x7fcd54158cc0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fcd4400c430 tx=0x7fcd4400c8f0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.745+0000 7fcd5a590640 1 -- 172.21.15.150:0/1856287865 shutdown_connections 2024-03-17T23:48:50.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.745+0000 7fcd5a590640 1 --2- 172.21.15.150:0/1856287865 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcd54161520 0x7fcd541f4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.745+0000 7fcd5a590640 1 --2- 172.21.15.150:0/1856287865 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcd4006bbd0 0x7fcd4006e090 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.745+0000 7fcd5a590640 1 --2- 172.21.15.150:0/1856287865 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd540d0ce0 0x7fcd541f4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.745+0000 7fcd5a590640 1 --2- 172.21.15.150:0/1856287865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd540b2bf0 0x7fcd54158cc0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.745+0000 7fcd5a590640 1 -- 172.21.15.150:0/1856287865 >> 172.21.15.150:0/1856287865 conn(0x7fcd5406ca00 msgr2=0x7fcd540d28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:50.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.745+0000 7fcd5a590640 1 -- 172.21.15.150:0/1856287865 shutdown_connections 2024-03-17T23:48:50.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.745+0000 7fcd5a590640 1 -- 172.21.15.150:0/1856287865 wait complete. 2024-03-17T23:48:50.757 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:48:50.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.867+0000 7fcdaaa12640 1 Processor -- start 2024-03-17T23:48:50.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.868+0000 7fcdaaa12640 1 -- start start 2024-03-17T23:48:50.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.868+0000 7fcdaaa12640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fcda4161520 0x7fcda40b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:50.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.868+0000 7fcdaaa12640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcda40d0ce0 0x7fcda40d1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:50.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.868+0000 7fcdaaa12640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcda4161d30 con 0x7fcda40b2bf0 2024-03-17T23:48:50.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.868+0000 7fcdaaa12640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fcda40b9bf0 con 0x7fcda40d0ce0 2024-03-17T23:48:50.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.868+0000 7fcdaaa12640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fcda4162940 con 0x7fcda4161520 2024-03-17T23:48:50.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.868+0000 7fcda8faf640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcda40d0ce0 0x7fcda40d1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:50.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.869+0000 7fcda8faf640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcda40d0ce0 0x7fcda40d1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36476/0 (socket says 172.21.15.150:36476) 2024-03-17T23:48:50.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.869+0000 7fcda8faf640 1 -- 172.21.15.150:0/3511074571 learned_addr learned my addr 172.21.15.150:0/3511074571 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:50.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.869+0000 7fcda3fff640 1 --2- 172.21.15.150:0/3511074571 >> v2:172.21.15.157:3300/0 conn(0x7fcda4161520 0x7fcda40b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:50.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.869+0000 7fcda8faf640 1 -- 172.21.15.150:0/3511074571 >> v2:172.21.15.157:3300/0 conn(0x7fcda4161520 msgr2=0x7fcda40b96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:50.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.869+0000 7fcda8faf640 1 --2- 172.21.15.150:0/3511074571 >> v2:172.21.15.157:3300/0 conn(0x7fcda4161520 0x7fcda40b96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.869+0000 7fcda8faf640 1 -- 172.21.15.150:0/3511074571 >> v1:172.21.15.157:6789/0 conn(0x7fcda40b2bf0 legacy=0x7fcda40d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:50.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.869+0000 7fcda8faf640 1 -- 172.21.15.150:0/3511074571 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcda4162010 con 0x7fcda40d0ce0 2024-03-17T23:48:50.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.870+0000 7fcda8faf640 1 --2- 172.21.15.150:0/3511074571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcda40d0ce0 0x7fcda40d1fa0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fcda408c940 tx=0x7fcd9c02bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f8e12945293be72 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:50.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.870+0000 7fcda2ffd640 1 -- 172.21.15.150:0/3511074571 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c02c960 con 0x7fcda40d0ce0 2024-03-17T23:48:50.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.870+0000 7fcda2ffd640 1 -- 172.21.15.150:0/3511074571 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd9c02cb00 con 0x7fcda40d0ce0 2024-03-17T23:48:50.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.870+0000 7fcda2ffd640 1 -- 172.21.15.150:0/3511074571 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c0343f0 con 0x7fcda40d0ce0 2024-03-17T23:48:50.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.870+0000 7fcdaaa12640 1 -- 172.21.15.150:0/3511074571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcda40d0ce0 msgr2=0x7fcda40d1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:50.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.870+0000 7fcdaaa12640 1 --2- 172.21.15.150:0/3511074571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcda40d0ce0 0x7fcda40d1fa0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fcda408c940 tx=0x7fcd9c02bdc0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.870+0000 7fcdaaa12640 1 -- 172.21.15.150:0/3511074571 shutdown_connections 2024-03-17T23:48:50.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.870+0000 7fcdaaa12640 1 --2- 172.21.15.150:0/3511074571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcda40d0ce0 0x7fcda40d1fa0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.870+0000 7fcdaaa12640 1 --2- 172.21.15.150:0/3511074571 >> v2:172.21.15.157:3300/0 conn(0x7fcda4161520 0x7fcda40b96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.870+0000 7fcdaaa12640 1 -- 172.21.15.150:0/3511074571 >> 172.21.15.150:0/3511074571 conn(0x7fcda406ca00 msgr2=0x7fcda406ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:50.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.870+0000 7fcdaaa12640 1 -- 172.21.15.150:0/3511074571 shutdown_connections 2024-03-17T23:48:50.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.870+0000 7fcdaaa12640 1 -- 172.21.15.150:0/3511074571 wait complete. 2024-03-17T23:48:50.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.872+0000 7fcdaaa12640 1 Processor -- start 2024-03-17T23:48:50.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.872+0000 7fcdaaa12640 1 -- start start 2024-03-17T23:48:50.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.874+0000 7fcdaaa12640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcda40b2bf0 0x7fcda41605d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:50.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.874+0000 7fcdaaa12640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcda40d0ce0 0x7fcda40cadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:50.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.874+0000 7fcdaaa12640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcda4161520 0x7fcda40cb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:50.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.874+0000 7fcdaaa12640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcda4161900 con 0x7fcda40d0ce0 2024-03-17T23:48:50.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.874+0000 7fcdaaa12640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcda41625d0 con 0x7fcda4161520 2024-03-17T23:48:50.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.874+0000 7fcdaaa12640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fcda40d24e0 con 0x7fcda40b2bf0 2024-03-17T23:48:50.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.874+0000 7fcda37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcda40d0ce0 0x7fcda40cadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:50.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.874+0000 7fcda37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcda40d0ce0 0x7fcda40cadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36484/0 (socket says 172.21.15.150:36484) 2024-03-17T23:48:50.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.874+0000 7fcda37fe640 1 -- 172.21.15.150:0/3063171727 learned_addr learned my addr 172.21.15.150:0/3063171727 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:50.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.874+0000 7fcda3fff640 1 --2- 172.21.15.150:0/3063171727 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcda40b2bf0 0x7fcda41605d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:50.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.874+0000 7fcda8faf640 1 --2- 172.21.15.150:0/3063171727 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcda4161520 0x7fcda40cb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:50.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.874+0000 7fcda37fe640 1 -- 172.21.15.150:0/3063171727 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcda40b2bf0 msgr2=0x7fcda41605d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:50.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.874+0000 7fcda37fe640 1 --2- 172.21.15.150:0/3063171727 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcda40b2bf0 0x7fcda41605d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.874+0000 7fcda37fe640 1 -- 172.21.15.150:0/3063171727 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcda4161520 msgr2=0x7fcda40cb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:50.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.874+0000 7fcda37fe640 1 --2- 172.21.15.150:0/3063171727 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcda4161520 0x7fcda40cb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:50.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.874+0000 7fcda37fe640 1 -- 172.21.15.150:0/3063171727 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcda40cb9b0 con 0x7fcda40d0ce0 2024-03-17T23:48:50.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.874+0000 7fcda37fe640 1 --2- 172.21.15.150:0/3063171727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcda40d0ce0 0x7fcda40cadf0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fcd9400c430 tx=0x7fcd9400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:50.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.874+0000 7fcda0ff9640 1 -- 172.21.15.150:0/3063171727 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd94011790 con 0x7fcda40d0ce0 2024-03-17T23:48:50.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.874+0000 7fcda0ff9640 1 -- 172.21.15.150:0/3063171727 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd94017070 con 0x7fcda40d0ce0 2024-03-17T23:48:50.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.875+0000 7fcda0ff9640 1 -- 172.21.15.150:0/3063171727 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd94011bb0 con 0x7fcda40d0ce0 2024-03-17T23:48:50.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.875+0000 7fcdaaa12640 1 -- 172.21.15.150:0/3063171727 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcda40cbc90 con 0x7fcda40d0ce0 2024-03-17T23:48:50.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.875+0000 7fcdaaa12640 1 -- 172.21.15.150:0/3063171727 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcda40cc250 con 0x7fcda40d0ce0 2024-03-17T23:48:50.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.876+0000 7fcda0ff9640 1 -- 172.21.15.150:0/3063171727 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fcd9401c460 con 0x7fcda40d0ce0 2024-03-17T23:48:50.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.876+0000 7fcdaaa12640 1 -- 172.21.15.150:0/3063171727 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcda40681a0 con 0x7fcda40d0ce0 2024-03-17T23:48:50.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.876+0000 7fcda0ff9640 1 --2- 172.21.15.150:0/3063171727 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcd9006bbd0 0x7fcd9006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:50.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.876+0000 7fcda3fff640 1 --2- 172.21.15.150:0/3063171727 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcd9006bbd0 0x7fcd9006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:50.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.876+0000 7fcda0ff9640 1 -- 172.21.15.150:0/3063171727 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7fcd9408aa90 con 0x7fcda40d0ce0 2024-03-17T23:48:50.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.877+0000 7fcda3fff640 1 --2- 172.21.15.150:0/3063171727 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcd9006bbd0 0x7fcd9006e090 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fcd980079a0 tx=0x7fcd98008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:50.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:50.881+0000 7fcda0ff9640 1 -- 172.21.15.150:0/3063171727 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fcd94058c90 con 0x7fcda40d0ce0 2024-03-17T23:48:51.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.060+0000 7fcdaaa12640 1 -- 172.21.15.150:0/3063171727 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fcda40d19c0 con 0x7fcda40d0ce0 2024-03-17T23:48:51.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.061+0000 7fcda0ff9640 1 -- 172.21.15.150:0/3063171727 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v45) ==== 104+0+11 (secure 0 0 0) 0x7fcd9405cd30 con 0x7fcda40d0ce0 2024-03-17T23:48:51.061 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:48:51.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.063+0000 7fcdaaa12640 1 -- 172.21.15.150:0/3063171727 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcd9006bbd0 msgr2=0x7fcd9006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:51.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.063+0000 7fcdaaa12640 1 --2- 172.21.15.150:0/3063171727 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcd9006bbd0 0x7fcd9006e090 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fcd980079a0 tx=0x7fcd98008040 comp rx=0 tx=0).stop 2024-03-17T23:48:51.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.063+0000 7fcdaaa12640 1 -- 172.21.15.150:0/3063171727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcda40d0ce0 msgr2=0x7fcda40cadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:51.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.063+0000 7fcdaaa12640 1 --2- 172.21.15.150:0/3063171727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcda40d0ce0 0x7fcda40cadf0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fcd9400c430 tx=0x7fcd9400c8f0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.063+0000 7fcdaaa12640 1 -- 172.21.15.150:0/3063171727 shutdown_connections 2024-03-17T23:48:51.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.063+0000 7fcdaaa12640 1 --2- 172.21.15.150:0/3063171727 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcd9006bbd0 0x7fcd9006e090 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.063+0000 7fcdaaa12640 1 --2- 172.21.15.150:0/3063171727 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcda4161520 0x7fcda40cb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.063+0000 7fcdaaa12640 1 --2- 172.21.15.150:0/3063171727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcda40d0ce0 0x7fcda40cadf0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.063+0000 7fcdaaa12640 1 --2- 172.21.15.150:0/3063171727 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcda40b2bf0 0x7fcda41605d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.063+0000 7fcdaaa12640 1 -- 172.21.15.150:0/3063171727 >> 172.21.15.150:0/3063171727 conn(0x7fcda406ca00 msgr2=0x7fcda40c9fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.064+0000 7fcdaaa12640 1 -- 172.21.15.150:0/3063171727 shutdown_connections 2024-03-17T23:48:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.064+0000 7fcdaaa12640 1 -- 172.21.15.150:0/3063171727 wait complete. 2024-03-17T23:48:51.075 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:48:51.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.179+0000 7f47b8f10640 1 Processor -- start 2024-03-17T23:48:51.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.179+0000 7f47b8f10640 1 -- start start 2024-03-17T23:48:51.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.180+0000 7f47b8f10640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f47b40b2bf0 0x7f47b40b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:51.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.180+0000 7f47b8f10640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47b4161d30 con 0x7f47b40d0ce0 2024-03-17T23:48:51.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.180+0000 7f47b8f10640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47b4162010 con 0x7f47b4161520 2024-03-17T23:48:51.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.180+0000 7f47b8f10640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f47b40b9bf0 con 0x7f47b40b2bf0 2024-03-17T23:48:51.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.180+0000 7f47abfff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f47b40d0ce0 0x7f47b40d1fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49554/0 (socket says 172.21.15.150:49554) 2024-03-17T23:48:51.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.180+0000 7f47abfff640 1 -- 172.21.15.150:0/2932013415 learned_addr learned my addr 172.21.15.150:0/2932013415 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:51.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.180+0000 7f47b2d9d640 1 --2- 172.21.15.150:0/2932013415 >> v2:172.21.15.157:3300/0 conn(0x7f47b40b2bf0 0x7f47b40b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:51.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.180+0000 7f47b1d9b640 1 -- 172.21.15.150:0/2932013415 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1024691416 0 0) 0x7f47b4161d30 con 0x7f47b40d0ce0 2024-03-17T23:48:51.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.181+0000 7f47b1d9b640 1 -- 172.21.15.150:0/2932013415 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47a0003520 con 0x7f47b40d0ce0 2024-03-17T23:48:51.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.181+0000 7f47b2d9d640 1 -- 172.21.15.150:0/2932013415 >> v1:172.21.15.157:6789/0 conn(0x7f47b4161520 legacy=0x7f47b40d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:51.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.181+0000 7f47b2d9d640 1 -- 172.21.15.150:0/2932013415 >> v1:172.21.15.150:6789/0 conn(0x7f47b40d0ce0 legacy=0x7f47b40d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:51.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.181+0000 7f47b2d9d640 1 -- 172.21.15.150:0/2932013415 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47b41622f0 con 0x7f47b40b2bf0 2024-03-17T23:48:51.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.182+0000 7f47b2d9d640 1 --2- 172.21.15.150:0/2932013415 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47b40b2bf0 0x7f47b40b96b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f47ac009d90 tx=0x7f47ac02be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1c689ab3bd9f5be server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:51.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.182+0000 7f47b1d9b640 1 -- 172.21.15.150:0/2932013415 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47ac039070 con 0x7f47b40b2bf0 2024-03-17T23:48:51.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.182+0000 7f47b1d9b640 1 -- 172.21.15.150:0/2932013415 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47ac02cbd0 con 0x7f47b40b2bf0 2024-03-17T23:48:51.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.182+0000 7f47b1d9b640 1 -- 172.21.15.150:0/2932013415 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47ac034440 con 0x7f47b40b2bf0 2024-03-17T23:48:51.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.183+0000 7f47b8f10640 1 -- 172.21.15.150:0/2932013415 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47b40b2bf0 msgr2=0x7f47b40b96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:51.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.183+0000 7f47b8f10640 1 --2- 172.21.15.150:0/2932013415 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47b40b2bf0 0x7f47b40b96b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f47ac009d90 tx=0x7f47ac02be20 comp rx=0 tx=0).stop 2024-03-17T23:48:51.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.183+0000 7f47b8f10640 1 -- 172.21.15.150:0/2932013415 shutdown_connections 2024-03-17T23:48:51.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.183+0000 7f47b8f10640 1 --2- 172.21.15.150:0/2932013415 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47b40b2bf0 0x7f47b40b96b0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.183+0000 7f47b8f10640 1 -- 172.21.15.150:0/2932013415 >> 172.21.15.150:0/2932013415 conn(0x7f47b406ca00 msgr2=0x7f47b406ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:51.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.183+0000 7f47b8f10640 1 -- 172.21.15.150:0/2932013415 shutdown_connections 2024-03-17T23:48:51.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.183+0000 7f47b8f10640 1 -- 172.21.15.150:0/2932013415 wait complete. 2024-03-17T23:48:51.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.184+0000 7f47b8f10640 1 Processor -- start 2024-03-17T23:48:51.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.184+0000 7f47b8f10640 1 -- start start 2024-03-17T23:48:51.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.185+0000 7f47b8f10640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47b40b2bf0 0x7f47b40b1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:51.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.185+0000 7f47b8f10640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47b40d0ce0 0x7f47b40ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:51.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.185+0000 7f47b8f10640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47b4161520 0x7f47b40cd500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:51.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.185+0000 7f47b8f10640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f47b4161900 con 0x7f47b40b2bf0 2024-03-17T23:48:51.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.185+0000 7f47b8f10640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f47b4162940 con 0x7f47b4161520 2024-03-17T23:48:51.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.185+0000 7f47b8f10640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f47b40d26b0 con 0x7f47b40d0ce0 2024-03-17T23:48:51.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.185+0000 7f47b259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47b40b2bf0 0x7f47b40b1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:51.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.185+0000 7f47b259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47b40b2bf0 0x7f47b40b1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36494/0 (socket says 172.21.15.150:36494) 2024-03-17T23:48:51.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.185+0000 7f47b259c640 1 -- 172.21.15.150:0/1690402147 learned_addr learned my addr 172.21.15.150:0/1690402147 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:51.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.185+0000 7f47b259c640 1 -- 172.21.15.150:0/1690402147 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47b40d0ce0 msgr2=0x7f47b40ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:51.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.185+0000 7f47b259c640 1 --2- 172.21.15.150:0/1690402147 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47b40d0ce0 0x7f47b40ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.185+0000 7f47b259c640 1 -- 172.21.15.150:0/1690402147 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47b4161520 msgr2=0x7f47b40cd500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:51.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.185+0000 7f47b259c640 1 --2- 172.21.15.150:0/1690402147 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47b4161520 0x7f47b40cd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.185+0000 7f47b259c640 1 -- 172.21.15.150:0/1690402147 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47b41f8490 con 0x7f47b40b2bf0 2024-03-17T23:48:51.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.185+0000 7f47b259c640 1 --2- 172.21.15.150:0/1690402147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47b40b2bf0 0x7f47b40b1f80 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f47a400c3c0 tx=0x7f47a400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:51.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.185+0000 7f47aaffd640 1 -- 172.21.15.150:0/1690402147 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a40052d0 con 0x7f47b40b2bf0 2024-03-17T23:48:51.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.185+0000 7f47aaffd640 1 -- 172.21.15.150:0/1690402147 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47a4005470 con 0x7f47b40b2bf0 2024-03-17T23:48:51.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.185+0000 7f47aaffd640 1 -- 172.21.15.150:0/1690402147 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a4007260 con 0x7f47b40b2bf0 2024-03-17T23:48:51.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.185+0000 7f47b8f10640 1 -- 172.21.15.150:0/1690402147 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47b41f76d0 con 0x7f47b40b2bf0 2024-03-17T23:48:51.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.185+0000 7f47b8f10640 1 -- 172.21.15.150:0/1690402147 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47b41f7c90 con 0x7f47b40b2bf0 2024-03-17T23:48:51.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.186+0000 7f47b8f10640 1 -- 172.21.15.150:0/1690402147 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47b40681a0 con 0x7f47b40b2bf0 2024-03-17T23:48:51.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.187+0000 7f47aaffd640 1 -- 172.21.15.150:0/1690402147 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f47a4007400 con 0x7f47b40b2bf0 2024-03-17T23:48:51.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.187+0000 7f47aaffd640 1 --2- 172.21.15.150:0/1690402147 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f479406bbd0 0x7f479406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:51.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.187+0000 7f47aaffd640 1 -- 172.21.15.150:0/1690402147 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7f47a408a220 con 0x7f47b40b2bf0 2024-03-17T23:48:51.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.187+0000 7f47abfff640 1 --2- 172.21.15.150:0/1690402147 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f479406bbd0 0x7f479406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:51.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.188+0000 7f47abfff640 1 --2- 172.21.15.150:0/1690402147 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f479406bbd0 0x7f479406e090 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f47b415d090 tx=0x7f479c007930 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:51.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.192+0000 7f47aaffd640 1 -- 172.21.15.150:0/1690402147 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f47a4058a70 con 0x7f47b40b2bf0 2024-03-17T23:48:51.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.466+0000 7f47b8f10640 1 -- 172.21.15.150:0/1690402147 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f47b4160a30 con 0x7f47b40b2bf0 2024-03-17T23:48:51.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.467+0000 7f47aaffd640 1 -- 172.21.15.150:0/1690402147 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v45) ==== 100+0+11 (secure 0 0 0) 0x7f47a4058370 con 0x7f47b40b2bf0 2024-03-17T23:48:51.467 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:48:51.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.469+0000 7f47b8f10640 1 -- 172.21.15.150:0/1690402147 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f479406bbd0 msgr2=0x7f479406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:51.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.469+0000 7f47b8f10640 1 --2- 172.21.15.150:0/1690402147 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f479406bbd0 0x7f479406e090 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f47b415d090 tx=0x7f479c007930 comp rx=0 tx=0).stop 2024-03-17T23:48:51.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.469+0000 7f47b8f10640 1 -- 172.21.15.150:0/1690402147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47b40b2bf0 msgr2=0x7f47b40b1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:51.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.469+0000 7f47b8f10640 1 --2- 172.21.15.150:0/1690402147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47b40b2bf0 0x7f47b40b1f80 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f47a400c3c0 tx=0x7f47a400c880 comp rx=0 tx=0).stop 2024-03-17T23:48:51.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.469+0000 7f47b8f10640 1 -- 172.21.15.150:0/1690402147 shutdown_connections 2024-03-17T23:48:51.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.469+0000 7f47b8f10640 1 --2- 172.21.15.150:0/1690402147 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f479406bbd0 0x7f479406e090 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.469+0000 7f47b8f10640 1 --2- 172.21.15.150:0/1690402147 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47b4161520 0x7f47b40cd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.469+0000 7f47b8f10640 1 --2- 172.21.15.150:0/1690402147 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47b40d0ce0 0x7f47b40ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.469+0000 7f47b8f10640 1 --2- 172.21.15.150:0/1690402147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47b40b2bf0 0x7f47b40b1f80 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.469+0000 7f47b8f10640 1 -- 172.21.15.150:0/1690402147 >> 172.21.15.150:0/1690402147 conn(0x7f47b406ca00 msgr2=0x7f47b40cf650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:51.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.469+0000 7f47b8f10640 1 -- 172.21.15.150:0/1690402147 shutdown_connections 2024-03-17T23:48:51.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.469+0000 7f47b8f10640 1 -- 172.21.15.150:0/1690402147 wait complete. 2024-03-17T23:48:51.480 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-17T23:48:51.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.583+0000 7f3cd919e640 1 Processor -- start 2024-03-17T23:48:51.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.583+0000 7f3cd919e640 1 -- start start 2024-03-17T23:48:51.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.584+0000 7f3cd919e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3cd415e170 0x7f3cd40cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:51.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.584+0000 7f3cd919e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cd4162950 con 0x7f3cd406b5f0 2024-03-17T23:48:51.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.584+0000 7f3cd919e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cd40d0850 con 0x7f3cd406b190 2024-03-17T23:48:51.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.584+0000 7f3cd919e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3cd40d0b30 con 0x7f3cd415e170 2024-03-17T23:48:51.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.584+0000 7f3cd359e640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f3cd406b5f0 0x7f3cd4162240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49566/0 (socket says 172.21.15.150:49566) 2024-03-17T23:48:51.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.584+0000 7f3cd359e640 1 -- 172.21.15.150:0/1768262374 learned_addr learned my addr 172.21.15.150:0/1768262374 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:51.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.584+0000 7f3cd2d9d640 1 --2- 172.21.15.150:0/1768262374 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3cd415e170 0x7f3cd40cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:51.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.584+0000 7f3cd1d9b640 1 -- 172.21.15.150:0/1768262374 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1200034145 0 0) 0x7f3cd4162950 con 0x7f3cd406b5f0 2024-03-17T23:48:51.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.584+0000 7f3cd1d9b640 1 -- 172.21.15.150:0/1768262374 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cbc003520 con 0x7f3cd406b5f0 2024-03-17T23:48:51.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.584+0000 7f3cd1d9b640 1 -- 172.21.15.150:0/1768262374 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2376118729 0 0) 0x7f3cd40d0850 con 0x7f3cd406b190 2024-03-17T23:48:51.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.584+0000 7f3cd1d9b640 1 -- 172.21.15.150:0/1768262374 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cd4162950 con 0x7f3cd406b190 2024-03-17T23:48:51.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.584+0000 7f3cd2d9d640 1 -- 172.21.15.150:0/1768262374 >> v1:172.21.15.157:6789/0 conn(0x7f3cd406b190 legacy=0x7f3cd4161b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:51.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.584+0000 7f3cd2d9d640 1 -- 172.21.15.150:0/1768262374 >> v1:172.21.15.150:6789/0 conn(0x7f3cd406b5f0 legacy=0x7f3cd4162240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:51.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.584+0000 7f3cd2d9d640 1 -- 172.21.15.150:0/1768262374 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cd40d0e10 con 0x7f3cd415e170 2024-03-17T23:48:51.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.585+0000 7f3cd2d9d640 1 --2- 172.21.15.150:0/1768262374 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3cd415e170 0x7f3cd40cd9b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc009dc0 tx=0x7f3ccc02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de0dcde1405b5bf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:51.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.585+0000 7f3cd1d9b640 1 -- 172.21.15.150:0/1768262374 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc039070 con 0x7f3cd415e170 2024-03-17T23:48:51.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.585+0000 7f3cd1d9b640 1 -- 172.21.15.150:0/1768262374 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ccc02cc40 con 0x7f3cd415e170 2024-03-17T23:48:51.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.585+0000 7f3cd1d9b640 1 -- 172.21.15.150:0/1768262374 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc034440 con 0x7f3cd415e170 2024-03-17T23:48:51.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.586+0000 7f3cd919e640 1 -- 172.21.15.150:0/1768262374 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3cd415e170 msgr2=0x7f3cd40cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:51.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.586+0000 7f3cd919e640 1 --2- 172.21.15.150:0/1768262374 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3cd415e170 0x7f3cd40cd9b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc009dc0 tx=0x7f3ccc02be90 comp rx=0 tx=0).stop 2024-03-17T23:48:51.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.586+0000 7f3cd919e640 1 -- 172.21.15.150:0/1768262374 shutdown_connections 2024-03-17T23:48:51.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.586+0000 7f3cd919e640 1 --2- 172.21.15.150:0/1768262374 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3cd415e170 0x7f3cd40cd9b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.586+0000 7f3cd919e640 1 -- 172.21.15.150:0/1768262374 >> 172.21.15.150:0/1768262374 conn(0x7f3cd4099a70 msgr2=0x7f3cd4099e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:51.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.586+0000 7f3cd919e640 1 -- 172.21.15.150:0/1768262374 shutdown_connections 2024-03-17T23:48:51.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.586+0000 7f3cd919e640 1 -- 172.21.15.150:0/1768262374 wait complete. 2024-03-17T23:48:51.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.587+0000 7f3cd919e640 1 Processor -- start 2024-03-17T23:48:51.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.587+0000 7f3cd919e640 1 -- start start 2024-03-17T23:48:51.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.588+0000 7f3cd919e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cd406b190 0x7f3cd41f3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:51.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.588+0000 7f3cd919e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3cd406b5f0 0x7f3cd40c67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:51.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.588+0000 7f3cd919e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3cd415e170 0x7f3cd40c6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:51.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.588+0000 7f3cd919e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd406ba80 con 0x7f3cd406b190 2024-03-17T23:48:51.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.588+0000 7f3cd919e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd40d13d0 con 0x7f3cd406b5f0 2024-03-17T23:48:51.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.588+0000 7f3cd919e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3cd40cdef0 con 0x7f3cd415e170 2024-03-17T23:48:51.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.588+0000 7f3cd2d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cd406b190 0x7f3cd41f3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:51.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.588+0000 7f3cd2d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cd406b190 0x7f3cd41f3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36508/0 (socket says 172.21.15.150:36508) 2024-03-17T23:48:51.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.588+0000 7f3cd2d9d640 1 -- 172.21.15.150:0/543635136 learned_addr learned my addr 172.21.15.150:0/543635136 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:51.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.588+0000 7f3cd359e640 1 --2- 172.21.15.150:0/543635136 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3cd415e170 0x7f3cd40c6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:51.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.588+0000 7f3cd259c640 1 --2- 172.21.15.150:0/543635136 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3cd406b5f0 0x7f3cd40c67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:51.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.589+0000 7f3cd2d9d640 1 -- 172.21.15.150:0/543635136 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3cd415e170 msgr2=0x7f3cd40c6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:51.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.589+0000 7f3cd2d9d640 1 --2- 172.21.15.150:0/543635136 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3cd415e170 0x7f3cd40c6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.589+0000 7f3cd2d9d640 1 -- 172.21.15.150:0/543635136 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3cd406b5f0 msgr2=0x7f3cd40c67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:51.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.589+0000 7f3cd2d9d640 1 --2- 172.21.15.150:0/543635136 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3cd406b5f0 0x7f3cd40c67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.589+0000 7f3cd2d9d640 1 -- 172.21.15.150:0/543635136 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cd41f36d0 con 0x7f3cd406b190 2024-03-17T23:48:51.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.589+0000 7f3cd359e640 1 --2- 172.21.15.150:0/543635136 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3cd415e170 0x7f3cd40c6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:51.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.589+0000 7f3cd2d9d640 1 --2- 172.21.15.150:0/543635136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cd406b190 0x7f3cd41f3190 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc009ee0 tx=0x7f3ccc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:51.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.589+0000 7f3cbb7fe640 1 -- 172.21.15.150:0/543635136 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc044020 con 0x7f3cd406b190 2024-03-17T23:48:51.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.589+0000 7f3cd919e640 1 -- 172.21.15.150:0/543635136 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cd415cf20 con 0x7f3cd406b190 2024-03-17T23:48:51.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.589+0000 7f3cbb7fe640 1 -- 172.21.15.150:0/543635136 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ccc034d30 con 0x7f3cd406b190 2024-03-17T23:48:51.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.589+0000 7f3cbb7fe640 1 -- 172.21.15.150:0/543635136 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc039040 con 0x7f3cd406b190 2024-03-17T23:48:51.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.589+0000 7f3cd919e640 1 -- 172.21.15.150:0/543635136 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cd415c440 con 0x7f3cd406b190 2024-03-17T23:48:51.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.590+0000 7f3cd919e640 1 -- 172.21.15.150:0/543635136 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cd40688b0 con 0x7f3cd406b190 2024-03-17T23:48:51.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.591+0000 7f3cbb7fe640 1 -- 172.21.15.150:0/543635136 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3ccc047020 con 0x7f3cd406b190 2024-03-17T23:48:51.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.591+0000 7f3cbb7fe640 1 --2- 172.21.15.150:0/543635136 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3cc006bbd0 0x7f3cc006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:51.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.591+0000 7f3cd259c640 1 --2- 172.21.15.150:0/543635136 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3cc006bbd0 0x7f3cc006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:51.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.591+0000 7f3cbb7fe640 1 -- 172.21.15.150:0/543635136 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7f3ccc0ad640 con 0x7f3cd406b190 2024-03-17T23:48:51.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.592+0000 7f3cd259c640 1 --2- 172.21.15.150:0/543635136 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3cc006bbd0 0x7f3cc006e090 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3cd40b2e30 tx=0x7f3cc8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:51.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.596+0000 7f3cbb7fe640 1 -- 172.21.15.150:0/543635136 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3ccc04e030 con 0x7f3cd406b190 2024-03-17T23:48:51.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.772+0000 7f3cd919e640 1 -- 172.21.15.150:0/543635136 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7f3cd40cd560 con 0x7f3cd406b190 2024-03-17T23:48:51.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.772+0000 7f3cbb7fe640 1 -- 172.21.15.150:0/543635136 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v45) ==== 97+0+11 (secure 0 0 0) 0x7f3ccc07bd80 con 0x7f3cd406b190 2024-03-17T23:48:51.775 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:48:51.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.774+0000 7f3cd919e640 1 -- 172.21.15.150:0/543635136 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3cc006bbd0 msgr2=0x7f3cc006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:51.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.774+0000 7f3cd919e640 1 --2- 172.21.15.150:0/543635136 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3cc006bbd0 0x7f3cc006e090 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3cd40b2e30 tx=0x7f3cc8009040 comp rx=0 tx=0).stop 2024-03-17T23:48:51.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.774+0000 7f3cd919e640 1 -- 172.21.15.150:0/543635136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cd406b190 msgr2=0x7f3cd41f3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:51.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.774+0000 7f3cd919e640 1 --2- 172.21.15.150:0/543635136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cd406b190 0x7f3cd41f3190 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc009ee0 tx=0x7f3ccc004a40 comp rx=0 tx=0).stop 2024-03-17T23:48:51.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.774+0000 7f3cd919e640 1 -- 172.21.15.150:0/543635136 shutdown_connections 2024-03-17T23:48:51.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.774+0000 7f3cd919e640 1 --2- 172.21.15.150:0/543635136 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3cd415e170 0x7f3cd40c6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.774+0000 7f3cd919e640 1 --2- 172.21.15.150:0/543635136 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3cc006bbd0 0x7f3cc006e090 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.774+0000 7f3cd919e640 1 --2- 172.21.15.150:0/543635136 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3cd406b5f0 0x7f3cd40c67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.774+0000 7f3cd919e640 1 --2- 172.21.15.150:0/543635136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3cd406b190 0x7f3cd41f3190 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.774+0000 7f3cd919e640 1 -- 172.21.15.150:0/543635136 >> 172.21.15.150:0/543635136 conn(0x7f3cd4099a70 msgr2=0x7f3cd41591a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:51.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.774+0000 7f3cd919e640 1 -- 172.21.15.150:0/543635136 shutdown_connections 2024-03-17T23:48:51.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.774+0000 7f3cd919e640 1 -- 172.21.15.150:0/543635136 wait complete. 2024-03-17T23:48:51.791 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-17T23:48:51.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.900+0000 7ff4be397640 1 Processor -- start 2024-03-17T23:48:51.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.900+0000 7ff4be397640 1 -- start start 2024-03-17T23:48:51.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.900+0000 7ff4be397640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4b80b97f0 0x7ff4b8162160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:51.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.900+0000 7ff4be397640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff4b81626a0 0x7ff4b80d2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:51.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.900+0000 7ff4be397640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4b80b9c70 con 0x7ff4b815df40 2024-03-17T23:48:51.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.900+0000 7ff4be397640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff4b80b6d40 con 0x7ff4b81626a0 2024-03-17T23:48:51.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.900+0000 7ff4be397640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff4b815ea30 con 0x7ff4b80b97f0 2024-03-17T23:48:51.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.901+0000 7ff4b7fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7ff4b815df40 0x7ff4b815e320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49572/0 (socket says 172.21.15.150:49572) 2024-03-17T23:48:51.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.901+0000 7ff4b7fff640 1 -- 172.21.15.150:0/2502480353 learned_addr learned my addr 172.21.15.150:0/2502480353 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:51.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.901+0000 7ff4b77fe640 1 --2- 172.21.15.150:0/2502480353 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4b80b97f0 0x7ff4b8162160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:51.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.901+0000 7ff4bc934640 1 --2- 172.21.15.150:0/2502480353 >> v2:172.21.15.157:3300/0 conn(0x7ff4b81626a0 0x7ff4b80d2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:51.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.901+0000 7ff4b6ffd640 1 -- 172.21.15.150:0/2502480353 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 632084403 0 0) 0x7ff4b80b9c70 con 0x7ff4b815df40 2024-03-17T23:48:51.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.901+0000 7ff4b6ffd640 1 -- 172.21.15.150:0/2502480353 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4a0003520 con 0x7ff4b815df40 2024-03-17T23:48:51.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.901+0000 7ff4b6ffd640 1 -- 172.21.15.150:0/2502480353 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2925514684 0 0) 0x7ff4a0003520 con 0x7ff4b815df40 2024-03-17T23:48:51.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.901+0000 7ff4b6ffd640 1 -- 172.21.15.150:0/2502480353 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4b80b97f0 msgr2=0x7ff4b8162160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:51.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.901+0000 7ff4b6ffd640 1 --2- 172.21.15.150:0/2502480353 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4b80b97f0 0x7ff4b8162160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.901+0000 7ff4b6ffd640 1 -- 172.21.15.150:0/2502480353 >> v2:172.21.15.157:3300/0 conn(0x7ff4b81626a0 msgr2=0x7ff4b80d2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:51.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.901+0000 7ff4b6ffd640 1 --2- 172.21.15.150:0/2502480353 >> v2:172.21.15.157:3300/0 conn(0x7ff4b81626a0 0x7ff4b80d2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.901+0000 7ff4b6ffd640 1 -- 172.21.15.150:0/2502480353 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b815ec90 con 0x7ff4b815df40 2024-03-17T23:48:51.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.901+0000 7ff4b6ffd640 1 -- 172.21.15.150:0/2502480353 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7ff4a8001670 con 0x7ff4b815df40 2024-03-17T23:48:51.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.901+0000 7ff4b6ffd640 1 -- 172.21.15.150:0/2502480353 >> v1:172.21.15.150:6789/0 conn(0x7ff4b815df40 legacy=0x7ff4b815e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:51.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.901+0000 7ff4b6ffd640 1 --2- 172.21.15.150:0/2502480353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a00039e0 0x7ff4a0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:51.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.901+0000 7ff4b6ffd640 1 --2- 172.21.15.150:0/2502480353 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4a0004800 0x7ff4a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:51.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.901+0000 7ff4b6ffd640 1 --2- 172.21.15.150:0/2502480353 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4b80b97f0 0x7ff4a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:51.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.901+0000 7ff4b6ffd640 1 -- 172.21.15.150:0/2502480353 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b815ea30 con 0x7ff4a00039e0 2024-03-17T23:48:51.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.901+0000 7ff4b6ffd640 1 -- 172.21.15.150:0/2502480353 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a000b9c0 con 0x7ff4a0004800 2024-03-17T23:48:51.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.901+0000 7ff4b6ffd640 1 -- 172.21.15.150:0/2502480353 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff4a000bb70 con 0x7ff4b80b97f0 2024-03-17T23:48:51.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.901+0000 7ff4b7fff640 1 --2- 172.21.15.150:0/2502480353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a00039e0 0x7ff4a0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:51.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.901+0000 7ff4b77fe640 1 --2- 172.21.15.150:0/2502480353 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4a0004800 0x7ff4a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:51.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.902+0000 7ff4bc934640 1 --2- 172.21.15.150:0/2502480353 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4b80b97f0 0x7ff4a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:51.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.902+0000 7ff4b7fff640 1 -- 172.21.15.150:0/2502480353 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4b80b97f0 msgr2=0x7ff4a00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:51.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.902+0000 7ff4b7fff640 1 --2- 172.21.15.150:0/2502480353 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4b80b97f0 0x7ff4a00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.902+0000 7ff4b7fff640 1 -- 172.21.15.150:0/2502480353 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4a0004800 msgr2=0x7ff4a0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:51.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.902+0000 7ff4b7fff640 1 --2- 172.21.15.150:0/2502480353 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4a0004800 0x7ff4a0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.902+0000 7ff4b7fff640 1 -- 172.21.15.150:0/2502480353 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a0003520 con 0x7ff4a00039e0 2024-03-17T23:48:51.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.903+0000 7ff4b7fff640 1 --2- 172.21.15.150:0/2502480353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a00039e0 0x7ff4a0003e00 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7ff4a800c210 tx=0x7ff4a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d525d217f775f80 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:51.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.903+0000 7ff4b6ffd640 1 -- 172.21.15.150:0/2502480353 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8007d30 con 0x7ff4a00039e0 2024-03-17T23:48:51.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.903+0000 7ff4b6ffd640 1 -- 172.21.15.150:0/2502480353 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4a8030070 con 0x7ff4a00039e0 2024-03-17T23:48:51.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.903+0000 7ff4b6ffd640 1 -- 172.21.15.150:0/2502480353 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a802cd60 con 0x7ff4a00039e0 2024-03-17T23:48:51.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.903+0000 7ff4be397640 1 -- 172.21.15.150:0/2502480353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a00039e0 msgr2=0x7ff4a0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:51.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.903+0000 7ff4be397640 1 --2- 172.21.15.150:0/2502480353 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a00039e0 0x7ff4a0003e00 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7ff4a800c210 tx=0x7ff4a0005830 comp rx=0 tx=0).stop 2024-03-17T23:48:51.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.903+0000 7ff4bc934640 1 -- 172.21.15.150:0/2502480353 reap_dead start 2024-03-17T23:48:51.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.903+0000 7ff4be397640 1 -- 172.21.15.150:0/2502480353 shutdown_connections 2024-03-17T23:48:51.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.903+0000 7ff4be397640 1 -- 172.21.15.150:0/2502480353 >> 172.21.15.150:0/2502480353 conn(0x7ff4b8072ef0 msgr2=0x7ff4b80d3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:51.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.903+0000 7ff4be397640 1 -- 172.21.15.150:0/2502480353 shutdown_connections 2024-03-17T23:48:51.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.903+0000 7ff4be397640 1 -- 172.21.15.150:0/2502480353 wait complete. 2024-03-17T23:48:51.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.905+0000 7ff4be397640 1 Processor -- start 2024-03-17T23:48:51.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.905+0000 7ff4be397640 1 -- start start 2024-03-17T23:48:51.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.906+0000 7ff4be397640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a00039e0 0x7ff4b80c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:51.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.906+0000 7ff4be397640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4b80cad20 0x7ff4b80cb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:51.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.906+0000 7ff4be397640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4b80b55d0 0x7ff4b80b5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:51.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.906+0000 7ff4be397640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b8161c40 con 0x7ff4a00039e0 2024-03-17T23:48:51.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.906+0000 7ff4be397640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b815f250 con 0x7ff4b80b55d0 2024-03-17T23:48:51.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.906+0000 7ff4be397640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff4b8162160 con 0x7ff4b80cad20 2024-03-17T23:48:51.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.906+0000 7ff4b7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a00039e0 0x7ff4b80c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:51.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.906+0000 7ff4b7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a00039e0 0x7ff4b80c98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36532/0 (socket says 172.21.15.150:36532) 2024-03-17T23:48:51.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.906+0000 7ff4b7fff640 1 -- 172.21.15.150:0/3992713839 learned_addr learned my addr 172.21.15.150:0/3992713839 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:51.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.906+0000 7ff4b77fe640 1 --2- 172.21.15.150:0/3992713839 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4b80cad20 0x7ff4b80cb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:51.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.906+0000 7ff4bc934640 1 --2- 172.21.15.150:0/3992713839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4b80b55d0 0x7ff4b80b5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:51.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.906+0000 7ff4b7fff640 1 -- 172.21.15.150:0/3992713839 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4b80cad20 msgr2=0x7ff4b80cb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:51.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.906+0000 7ff4b7fff640 1 --2- 172.21.15.150:0/3992713839 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4b80cad20 0x7ff4b80cb180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.906+0000 7ff4b7fff640 1 -- 172.21.15.150:0/3992713839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4b80b55d0 msgr2=0x7ff4b80b5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:51.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.906+0000 7ff4b7fff640 1 --2- 172.21.15.150:0/3992713839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4b80b55d0 0x7ff4b80b5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:51.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.906+0000 7ff4b7fff640 1 -- 172.21.15.150:0/3992713839 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b80cb7a0 con 0x7ff4a00039e0 2024-03-17T23:48:51.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.906+0000 7ff4b7fff640 1 --2- 172.21.15.150:0/3992713839 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a00039e0 0x7ff4b80c98b0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7ff4a800c300 tx=0x7ff4a8001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:51.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.906+0000 7ff4b4ff9640 1 -- 172.21.15.150:0/3992713839 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8004bf0 con 0x7ff4a00039e0 2024-03-17T23:48:51.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.906+0000 7ff4b4ff9640 1 -- 172.21.15.150:0/3992713839 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4a8004d90 con 0x7ff4a00039e0 2024-03-17T23:48:51.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.906+0000 7ff4be397640 1 -- 172.21.15.150:0/3992713839 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4b80cba80 con 0x7ff4a00039e0 2024-03-17T23:48:51.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.906+0000 7ff4b4ff9640 1 -- 172.21.15.150:0/3992713839 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8007270 con 0x7ff4a00039e0 2024-03-17T23:48:51.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.906+0000 7ff4be397640 1 -- 172.21.15.150:0/3992713839 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4b80cc040 con 0x7ff4a00039e0 2024-03-17T23:48:51.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.907+0000 7ff4be397640 1 -- 172.21.15.150:0/3992713839 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4b80681a0 con 0x7ff4a00039e0 2024-03-17T23:48:51.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.908+0000 7ff4b4ff9640 1 -- 172.21.15.150:0/3992713839 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff4a8047020 con 0x7ff4a00039e0 2024-03-17T23:48:51.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.908+0000 7ff4b4ff9640 1 --2- 172.21.15.150:0/3992713839 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff4a406bbd0 0x7ff4a406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:51.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.908+0000 7ff4b4ff9640 1 -- 172.21.15.150:0/3992713839 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7ff4a80ac040 con 0x7ff4a00039e0 2024-03-17T23:48:51.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.908+0000 7ff4b77fe640 1 --2- 172.21.15.150:0/3992713839 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff4a406bbd0 0x7ff4a406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:51.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.909+0000 7ff4b77fe640 1 --2- 172.21.15.150:0/3992713839 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff4a406bbd0 0x7ff4a406e090 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff4b81f7210 tx=0x7ff4ac008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:51.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:51.912+0000 7ff4b4ff9640 1 -- 172.21.15.150:0/3992713839 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff4a807a800 con 0x7ff4a00039e0 2024-03-17T23:48:52.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.102+0000 7ff4be397640 1 -- 172.21.15.150:0/3992713839 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff4b80d19d0 con 0x7ff4a00039e0 2024-03-17T23:48:52.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.103+0000 7ff4b4ff9640 1 -- 172.21.15.150:0/3992713839 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v45) ==== 80+0+0 (secure 0 0 0) 0x7ff4a807a100 con 0x7ff4a00039e0 2024-03-17T23:48:52.104 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-03-17T23:48:52.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.105+0000 7ff4be397640 1 -- 172.21.15.150:0/3992713839 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff4a406bbd0 msgr2=0x7ff4a406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:52.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.105+0000 7ff4be397640 1 --2- 172.21.15.150:0/3992713839 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff4a406bbd0 0x7ff4a406e090 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff4b81f7210 tx=0x7ff4ac008040 comp rx=0 tx=0).stop 2024-03-17T23:48:52.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.105+0000 7ff4be397640 1 -- 172.21.15.150:0/3992713839 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a00039e0 msgr2=0x7ff4b80c98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:52.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.105+0000 7ff4be397640 1 --2- 172.21.15.150:0/3992713839 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a00039e0 0x7ff4b80c98b0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7ff4a800c300 tx=0x7ff4a8001a40 comp rx=0 tx=0).stop 2024-03-17T23:48:52.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.105+0000 7ff4be397640 1 -- 172.21.15.150:0/3992713839 shutdown_connections 2024-03-17T23:48:52.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.105+0000 7ff4be397640 1 --2- 172.21.15.150:0/3992713839 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff4a406bbd0 0x7ff4a406e090 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:52.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.105+0000 7ff4be397640 1 --2- 172.21.15.150:0/3992713839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4b80b55d0 0x7ff4b80b5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:52.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.105+0000 7ff4be397640 1 --2- 172.21.15.150:0/3992713839 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff4b80cad20 0x7ff4b80cb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:52.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.105+0000 7ff4be397640 1 --2- 172.21.15.150:0/3992713839 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff4a00039e0 0x7ff4b80c98b0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:52.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.105+0000 7ff4be397640 1 -- 172.21.15.150:0/3992713839 >> 172.21.15.150:0/3992713839 conn(0x7ff4b8072ef0 msgr2=0x7ff4b81608f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:52.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.105+0000 7ff4be397640 1 -- 172.21.15.150:0/3992713839 shutdown_connections 2024-03-17T23:48:52.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.105+0000 7ff4be397640 1 -- 172.21.15.150:0/3992713839 wait complete. 2024-03-17T23:48:52.116 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:48:52.116 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:48:52.133 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:48:52.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.274+0000 7f8f67836640 1 Processor -- start 2024-03-17T23:48:52.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.274+0000 7f8f67836640 1 -- start start 2024-03-17T23:48:52.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.275+0000 7f8f67836640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8f601619a0 0x7f8f600b9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:52.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.275+0000 7f8f67836640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f60161dd0 con 0x7f8f600cf120 2024-03-17T23:48:52.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.275+0000 7f8f67836640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f601620a0 con 0x7f8f6015de30 2024-03-17T23:48:52.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.275+0000 7f8f67836640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f8f60162940 con 0x7f8f601619a0 2024-03-17T23:48:52.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.275+0000 7f8f65dd3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8f601619a0 0x7f8f600b9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:52.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.275+0000 7f8f64dd1640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f8f600cf120 0x7f8f600d2690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49578/0 (socket says 172.21.15.150:49578) 2024-03-17T23:48:52.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.275+0000 7f8f64dd1640 1 -- 172.21.15.150:0/1121802417 learned_addr learned my addr 172.21.15.150:0/1121802417 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:52.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.275+0000 7f8f65dd3640 1 -- 172.21.15.150:0/1121802417 >> v1:172.21.15.157:6789/0 conn(0x7f8f6015de30 legacy=0x7f8f600cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:52.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.275+0000 7f8f65dd3640 1 -- 172.21.15.150:0/1121802417 >> v1:172.21.15.150:6789/0 conn(0x7f8f600cf120 legacy=0x7f8f600d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:52.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.275+0000 7f8f65dd3640 1 -- 172.21.15.150:0/1121802417 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f60162380 con 0x7f8f601619a0 2024-03-17T23:48:52.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.276+0000 7f8f65dd3640 1 --2- 172.21.15.150:0/1121802417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f601619a0 0x7f8f600b9860 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f8f58009d90 tx=0x7f8f5802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f71d099e4a54b2ee server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:52.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.276+0000 7f8f57fff640 1 -- 172.21.15.150:0/1121802417 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f58039070 con 0x7f8f601619a0 2024-03-17T23:48:52.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.276+0000 7f8f57fff640 1 -- 172.21.15.150:0/1121802417 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f5802cbd0 con 0x7f8f601619a0 2024-03-17T23:48:52.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.276+0000 7f8f57fff640 1 -- 172.21.15.150:0/1121802417 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f58034440 con 0x7f8f601619a0 2024-03-17T23:48:52.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.277+0000 7f8f67836640 1 -- 172.21.15.150:0/1121802417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f601619a0 msgr2=0x7f8f600b9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:52.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.277+0000 7f8f67836640 1 --2- 172.21.15.150:0/1121802417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f601619a0 0x7f8f600b9860 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f8f58009d90 tx=0x7f8f5802be20 comp rx=0 tx=0).stop 2024-03-17T23:48:52.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.277+0000 7f8f67836640 1 -- 172.21.15.150:0/1121802417 shutdown_connections 2024-03-17T23:48:52.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.277+0000 7f8f67836640 1 --2- 172.21.15.150:0/1121802417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f601619a0 0x7f8f600b9860 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:52.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.277+0000 7f8f67836640 1 -- 172.21.15.150:0/1121802417 >> 172.21.15.150:0/1121802417 conn(0x7f8f600730b0 msgr2=0x7f8f600d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:52.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.277+0000 7f8f67836640 1 -- 172.21.15.150:0/1121802417 shutdown_connections 2024-03-17T23:48:52.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.277+0000 7f8f67836640 1 -- 172.21.15.150:0/1121802417 wait complete. 2024-03-17T23:48:52.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.278+0000 7f8f67836640 1 Processor -- start 2024-03-17T23:48:52.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.278+0000 7f8f67836640 1 -- start start 2024-03-17T23:48:52.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.279+0000 7f8f67836640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f600cf120 0x7f8f6015cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:52.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.279+0000 7f8f67836640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f6015de30 0x7f8f601f7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:52.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.279+0000 7f8f67836640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f601619a0 0x7f8f600cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:52.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.279+0000 7f8f67836640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6015e210 con 0x7f8f600cf120 2024-03-17T23:48:52.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.279+0000 7f8f67836640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8f600d2170 con 0x7f8f6015de30 2024-03-17T23:48:52.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.279+0000 7f8f67836640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8f600d2da0 con 0x7f8f601619a0 2024-03-17T23:48:52.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.279+0000 7f8f655d2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f600cf120 0x7f8f6015cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:52.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.279+0000 7f8f655d2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f600cf120 0x7f8f6015cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36558/0 (socket says 172.21.15.150:36558) 2024-03-17T23:48:52.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.279+0000 7f8f655d2640 1 -- 172.21.15.150:0/1928469517 learned_addr learned my addr 172.21.15.150:0/1928469517 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:52.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.279+0000 7f8f655d2640 1 -- 172.21.15.150:0/1928469517 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f601619a0 msgr2=0x7f8f600cd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:52.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.279+0000 7f8f655d2640 1 --2- 172.21.15.150:0/1928469517 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f601619a0 0x7f8f600cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:52.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.279+0000 7f8f655d2640 1 -- 172.21.15.150:0/1928469517 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f6015de30 msgr2=0x7f8f601f7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:52.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.279+0000 7f8f655d2640 1 --2- 172.21.15.150:0/1928469517 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f6015de30 0x7f8f601f7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:52.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.279+0000 7f8f655d2640 1 -- 172.21.15.150:0/1928469517 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f601f8490 con 0x7f8f600cf120 2024-03-17T23:48:52.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.279+0000 7f8f655d2640 1 --2- 172.21.15.150:0/1928469517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f600cf120 0x7f8f6015cf20 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c00c440 tx=0x7f8f5c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:52.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.279+0000 7f8f55ffb640 1 -- 172.21.15.150:0/1928469517 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c017070 con 0x7f8f600cf120 2024-03-17T23:48:52.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.279+0000 7f8f55ffb640 1 -- 172.21.15.150:0/1928469517 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f5c0035f0 con 0x7f8f600cf120 2024-03-17T23:48:52.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.279+0000 7f8f67836640 1 -- 172.21.15.150:0/1928469517 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f600caf40 con 0x7f8f600cf120 2024-03-17T23:48:52.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.279+0000 7f8f55ffb640 1 -- 172.21.15.150:0/1928469517 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c0039e0 con 0x7f8f600cf120 2024-03-17T23:48:52.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.280+0000 7f8f67836640 1 -- 172.21.15.150:0/1928469517 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f600cb4c0 con 0x7f8f600cf120 2024-03-17T23:48:52.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.281+0000 7f8f67836640 1 -- 172.21.15.150:0/1928469517 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f600681a0 con 0x7f8f600cf120 2024-03-17T23:48:52.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.281+0000 7f8f55ffb640 1 -- 172.21.15.150:0/1928469517 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8f5c01a070 con 0x7f8f600cf120 2024-03-17T23:48:52.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.281+0000 7f8f55ffb640 1 --2- 172.21.15.150:0/1928469517 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8f4406bbd0 0x7f8f4406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:52.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.281+0000 7f8f55ffb640 1 -- 172.21.15.150:0/1928469517 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7f8f5c08a0b0 con 0x7f8f600cf120 2024-03-17T23:48:52.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.281+0000 7f8f64dd1640 1 --2- 172.21.15.150:0/1928469517 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8f4406bbd0 0x7f8f4406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:52.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.282+0000 7f8f64dd1640 1 --2- 172.21.15.150:0/1928469517 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8f4406bbd0 0x7f8f4406e090 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8f50000f60 tx=0x7f8f50007a80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:52.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.286+0000 7f8f55ffb640 1 -- 172.21.15.150:0/1928469517 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8f601f8490 con 0x7f8f600cf120 2024-03-17T23:48:52.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.466+0000 7f8f67836640 1 -- 172.21.15.150:0/1928469517 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8f600d1910 con 0x7f8f600cf120 2024-03-17T23:48:52.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.467+0000 7f8f55ffb640 1 -- 172.21.15.150:0/1928469517 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+18839 (secure 0 0 0) 0x7f8f5c058900 con 0x7f8f600cf120 2024-03-17T23:48:52.468 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:52.468 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":45,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:48:37.677776+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:48:08.957879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-17T23:48:09.698808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-17T23:48:22.562269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.3d","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:48:52.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.469+0000 7f8f67836640 1 -- 172.21.15.150:0/1928469517 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8f4406bbd0 msgr2=0x7f8f4406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:52.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.469+0000 7f8f67836640 1 --2- 172.21.15.150:0/1928469517 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8f4406bbd0 0x7f8f4406e090 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8f50000f60 tx=0x7f8f50007a80 comp rx=0 tx=0).stop 2024-03-17T23:48:52.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.469+0000 7f8f67836640 1 -- 172.21.15.150:0/1928469517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f600cf120 msgr2=0x7f8f6015cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:52.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.469+0000 7f8f67836640 1 --2- 172.21.15.150:0/1928469517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f600cf120 0x7f8f6015cf20 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c00c440 tx=0x7f8f5c00c900 comp rx=0 tx=0).stop 2024-03-17T23:48:52.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.469+0000 7f8f67836640 1 -- 172.21.15.150:0/1928469517 shutdown_connections 2024-03-17T23:48:52.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.469+0000 7f8f67836640 1 --2- 172.21.15.150:0/1928469517 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8f601619a0 0x7f8f600cd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:52.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.469+0000 7f8f67836640 1 --2- 172.21.15.150:0/1928469517 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8f4406bbd0 0x7f8f4406e090 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:52.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.469+0000 7f8f67836640 1 --2- 172.21.15.150:0/1928469517 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f6015de30 0x7f8f601f7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:52.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.469+0000 7f8f67836640 1 --2- 172.21.15.150:0/1928469517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f600cf120 0x7f8f6015cf20 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:52.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.469+0000 7f8f67836640 1 -- 172.21.15.150:0/1928469517 >> 172.21.15.150:0/1928469517 conn(0x7f8f600730b0 msgr2=0x7f8f601e9810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:52.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.469+0000 7f8f67836640 1 -- 172.21.15.150:0/1928469517 shutdown_connections 2024-03-17T23:48:52.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.469+0000 7f8f67836640 1 -- 172.21.15.150:0/1928469517 wait complete. 2024-03-17T23:48:52.482 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:48:08.957879+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:48:09.698808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-17T23:48:22.562269+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-17T23:48:52.482 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:48:52.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.585+0000 7f9acb67e640 1 Processor -- start 2024-03-17T23:48:52.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.586+0000 7f9acb67e640 1 -- start start 2024-03-17T23:48:52.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.586+0000 7f9acb67e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9ac415f3d0 0x7f9ac40cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:52.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.586+0000 7f9acb67e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9ac40d0880 0x7f9ac41603c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:52.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.586+0000 7f9acb67e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ac40ccc50 con 0x7f9ac40cc770 2024-03-17T23:48:52.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.586+0000 7f9acb67e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f9ac415f7b0 con 0x7f9ac415f3d0 2024-03-17T23:48:52.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.586+0000 7f9acb67e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9ac4160900 con 0x7f9ac40d0880 2024-03-17T23:48:52.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.587+0000 7f9ac9c1b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9ac40d0880 0x7f9ac41603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:52.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.587+0000 7f9ac941a640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f9ac415f3d0 0x7f9ac40cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:52.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.587+0000 7f9ac8c19640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f9ac40cc770 0x7f9ac40d0170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:54750/0 (socket says 172.21.15.150:54750) 2024-03-17T23:48:52.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.587+0000 7f9ac8c19640 1 -- 172.21.15.150:0/1331281558 learned_addr learned my addr 172.21.15.150:0/1331281558 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:52.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.587+0000 7f9abbfff640 1 -- 172.21.15.150:0/1331281558 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1617421714 0 0) 0x7f9ac40ccc50 con 0x7f9ac40cc770 2024-03-17T23:48:52.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.587+0000 7f9abbfff640 1 -- 172.21.15.150:0/1331281558 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ab0003520 con 0x7f9ac40cc770 2024-03-17T23:48:52.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.587+0000 7f9ac9c1b640 1 -- 172.21.15.150:0/1331281558 >> v2:172.21.15.157:3300/0 conn(0x7f9ac415f3d0 msgr2=0x7f9ac40cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:52.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.587+0000 7f9ac9c1b640 1 --2- 172.21.15.150:0/1331281558 >> v2:172.21.15.157:3300/0 conn(0x7f9ac415f3d0 0x7f9ac40cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:52.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.587+0000 7f9ac9c1b640 1 -- 172.21.15.150:0/1331281558 >> v1:172.21.15.157:6789/0 conn(0x7f9ac40cc770 legacy=0x7f9ac40d0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:52.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.587+0000 7f9ac9c1b640 1 -- 172.21.15.150:0/1331281558 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac40ccf30 con 0x7f9ac40d0880 2024-03-17T23:48:52.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.587+0000 7f9ac941a640 1 --2- 172.21.15.150:0/1331281558 >> v2:172.21.15.157:3300/0 conn(0x7f9ac415f3d0 0x7f9ac40cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:48:52.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.588+0000 7f9ac9c1b640 1 --2- 172.21.15.150:0/1331281558 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9ac40d0880 0x7f9ac41603c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0009d90 tx=0x7f9ac002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5762e78bdf14f2b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:52.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.588+0000 7f9abbfff640 1 -- 172.21.15.150:0/1331281558 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac0039070 con 0x7f9ac40d0880 2024-03-17T23:48:52.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.588+0000 7f9abbfff640 1 -- 172.21.15.150:0/1331281558 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ac002cc50 con 0x7f9ac40d0880 2024-03-17T23:48:52.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.588+0000 7f9abbfff640 1 -- 172.21.15.150:0/1331281558 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac0034440 con 0x7f9ac40d0880 2024-03-17T23:48:52.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.589+0000 7f9acb67e640 1 -- 172.21.15.150:0/1331281558 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9ac40d0880 msgr2=0x7f9ac41603c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:52.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.589+0000 7f9acb67e640 1 --2- 172.21.15.150:0/1331281558 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9ac40d0880 0x7f9ac41603c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9ac0009d90 tx=0x7f9ac002bea0 comp rx=0 tx=0).stop 2024-03-17T23:48:52.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.589+0000 7f9acb67e640 1 -- 172.21.15.150:0/1331281558 shutdown_connections 2024-03-17T23:48:52.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.589+0000 7f9acb67e640 1 --2- 172.21.15.150:0/1331281558 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9ac40d0880 0x7f9ac41603c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:52.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.589+0000 7f9acb67e640 1 --2- 172.21.15.150:0/1331281558 >> v2:172.21.15.157:3300/0 conn(0x7f9ac415f3d0 0x7f9ac40cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:52.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.589+0000 7f9acb67e640 1 -- 172.21.15.150:0/1331281558 >> 172.21.15.150:0/1331281558 conn(0x7f9ac40b36c0 msgr2=0x7f9ac40b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:52.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.589+0000 7f9acb67e640 1 -- 172.21.15.150:0/1331281558 shutdown_connections 2024-03-17T23:48:52.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.589+0000 7f9acb67e640 1 -- 172.21.15.150:0/1331281558 wait complete. 2024-03-17T23:48:52.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.591+0000 7f9acb67e640 1 Processor -- start 2024-03-17T23:48:52.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.591+0000 7f9acb67e640 1 -- start start 2024-03-17T23:48:52.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.592+0000 7f9acb67e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ac40cc770 0x7f9ac414bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:52.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.592+0000 7f9acb67e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9ac40d0880 0x7f9ac40b6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:52.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.592+0000 7f9acb67e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9ac415f3d0 0x7f9ac415da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:52.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.592+0000 7f9acb67e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac40cd7d0 con 0x7f9ac40cc770 2024-03-17T23:48:52.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.592+0000 7f9acb67e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac40cd4f0 con 0x7f9ac40d0880 2024-03-17T23:48:52.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.592+0000 7f9acb67e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f9ac40cfa30 con 0x7f9ac415f3d0 2024-03-17T23:48:52.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.592+0000 7f9ac941a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ac40cc770 0x7f9ac414bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:52.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.592+0000 7f9ac941a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ac40cc770 0x7f9ac414bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36570/0 (socket says 172.21.15.150:36570) 2024-03-17T23:48:52.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.592+0000 7f9ac941a640 1 -- 172.21.15.150:0/1593490056 learned_addr learned my addr 172.21.15.150:0/1593490056 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:52.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.592+0000 7f9ac9c1b640 1 --2- 172.21.15.150:0/1593490056 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9ac415f3d0 0x7f9ac415da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:52.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.592+0000 7f9ac8c19640 1 --2- 172.21.15.150:0/1593490056 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9ac40d0880 0x7f9ac40b6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:52.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.592+0000 7f9ac941a640 1 -- 172.21.15.150:0/1593490056 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9ac415f3d0 msgr2=0x7f9ac415da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:52.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.592+0000 7f9ac941a640 1 --2- 172.21.15.150:0/1593490056 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9ac415f3d0 0x7f9ac415da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:52.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.592+0000 7f9ac941a640 1 -- 172.21.15.150:0/1593490056 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9ac40d0880 msgr2=0x7f9ac40b6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:52.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.592+0000 7f9ac941a640 1 --2- 172.21.15.150:0/1593490056 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9ac40d0880 0x7f9ac40b6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:52.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.592+0000 7f9ac941a640 1 -- 172.21.15.150:0/1593490056 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac40b5e20 con 0x7f9ac40cc770 2024-03-17T23:48:52.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.592+0000 7f9ac8c19640 1 --2- 172.21.15.150:0/1593490056 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9ac40d0880 0x7f9ac40b6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:52.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.593+0000 7f9ac941a640 1 --2- 172.21.15.150:0/1593490056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ac40cc770 0x7f9ac414bee0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f9abc00ebe0 tx=0x7f9abc00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:52.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.593+0000 7f9ab9ffb640 1 -- 172.21.15.150:0/1593490056 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9abc017070 con 0x7f9ac40cc770 2024-03-17T23:48:52.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.593+0000 7f9ab9ffb640 1 -- 172.21.15.150:0/1593490056 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9abc00cc30 con 0x7f9ac40cc770 2024-03-17T23:48:52.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.593+0000 7f9acb67e640 1 -- 172.21.15.150:0/1593490056 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ac40b6100 con 0x7f9ac40cc770 2024-03-17T23:48:52.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.593+0000 7f9ab9ffb640 1 -- 172.21.15.150:0/1593490056 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9abc013470 con 0x7f9ac40cc770 2024-03-17T23:48:52.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.593+0000 7f9acb67e640 1 -- 172.21.15.150:0/1593490056 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ac40b66c0 con 0x7f9ac40cc770 2024-03-17T23:48:52.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.594+0000 7f9acb67e640 1 -- 172.21.15.150:0/1593490056 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ac40688b0 con 0x7f9ac40cc770 2024-03-17T23:48:52.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.594+0000 7f9ab9ffb640 1 -- 172.21.15.150:0/1593490056 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f9abc01a070 con 0x7f9ac40cc770 2024-03-17T23:48:52.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.595+0000 7f9ab9ffb640 1 --2- 172.21.15.150:0/1593490056 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f9aa806bc00 0x7f9aa806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:52.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.595+0000 7f9ab9ffb640 1 -- 172.21.15.150:0/1593490056 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7f9abc08b110 con 0x7f9ac40cc770 2024-03-17T23:48:52.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.595+0000 7f9ac8c19640 1 --2- 172.21.15.150:0/1593490056 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f9aa806bc00 0x7f9aa806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:52.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.595+0000 7f9ac8c19640 1 --2- 172.21.15.150:0/1593490056 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f9aa806bc00 0x7f9aa806e0c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9ac409fbe0 tx=0x7f9ab4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:52.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.603+0000 7f9ab9ffb640 1 -- 172.21.15.150:0/1593490056 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f9abc059840 con 0x7f9ac40cc770 2024-03-17T23:48:52.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.777+0000 7f9acb67e640 1 -- 172.21.15.150:0/1593490056 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9ac4074300 con 0x7f9ac40cc770 2024-03-17T23:48:52.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.778+0000 7f9ab9ffb640 1 -- 172.21.15.150:0/1593490056 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v45) ==== 93+0+10 (secure 0 0 0) 0x7f9abc059140 con 0x7f9ac40cc770 2024-03-17T23:48:52.779 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:48:52.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.781+0000 7f9acb67e640 1 -- 172.21.15.150:0/1593490056 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f9aa806bc00 msgr2=0x7f9aa806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:52.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.781+0000 7f9acb67e640 1 --2- 172.21.15.150:0/1593490056 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f9aa806bc00 0x7f9aa806e0c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9ac409fbe0 tx=0x7f9ab4009040 comp rx=0 tx=0).stop 2024-03-17T23:48:52.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.781+0000 7f9acb67e640 1 -- 172.21.15.150:0/1593490056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ac40cc770 msgr2=0x7f9ac414bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:52.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.781+0000 7f9acb67e640 1 --2- 172.21.15.150:0/1593490056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ac40cc770 0x7f9ac414bee0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f9abc00ebe0 tx=0x7f9abc00c520 comp rx=0 tx=0).stop 2024-03-17T23:48:52.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.781+0000 7f9acb67e640 1 -- 172.21.15.150:0/1593490056 shutdown_connections 2024-03-17T23:48:52.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.781+0000 7f9acb67e640 1 --2- 172.21.15.150:0/1593490056 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9ac415f3d0 0x7f9ac415da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:52.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.781+0000 7f9acb67e640 1 --2- 172.21.15.150:0/1593490056 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f9aa806bc00 0x7f9aa806e0c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:52.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.781+0000 7f9acb67e640 1 --2- 172.21.15.150:0/1593490056 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9ac40d0880 0x7f9ac40b6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:52.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.781+0000 7f9acb67e640 1 --2- 172.21.15.150:0/1593490056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ac40cc770 0x7f9ac414bee0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:52.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.781+0000 7f9acb67e640 1 -- 172.21.15.150:0/1593490056 >> 172.21.15.150:0/1593490056 conn(0x7f9ac40b36c0 msgr2=0x7f9ac4071130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:52.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.781+0000 7f9acb67e640 1 -- 172.21.15.150:0/1593490056 shutdown_connections 2024-03-17T23:48:52.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.781+0000 7f9acb67e640 1 -- 172.21.15.150:0/1593490056 wait complete. 2024-03-17T23:48:52.794 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:48:52.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.899+0000 7f25b5df4640 1 Processor -- start 2024-03-17T23:48:52.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.900+0000 7f25b5df4640 1 -- start start 2024-03-17T23:48:52.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.901+0000 7f25b5df4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f25b009a180 0x7f25b006b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:52.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.901+0000 7f25b5df4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25b009a610 con 0x7f25b01457d0 2024-03-17T23:48:52.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.901+0000 7f25b5df4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25b00d1800 con 0x7f25b006b6d0 2024-03-17T23:48:52.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.901+0000 7f25b5df4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f25b00d0980 con 0x7f25b009a180 2024-03-17T23:48:52.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.901+0000 7f25af7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f25b01457d0 0x7f25b00d10f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49584/0 (socket says 172.21.15.150:49584) 2024-03-17T23:48:52.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.901+0000 7f25af7fe640 1 -- 172.21.15.150:0/1364731961 learned_addr learned my addr 172.21.15.150:0/1364731961 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:52.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.901+0000 7f25a6ffd640 1 --2- 172.21.15.150:0/1364731961 >> v2:172.21.15.157:3300/0 conn(0x7f25b009a180 0x7f25b006b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:52.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.901+0000 7f25aeffd640 1 -- 172.21.15.150:0/1364731961 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2037140226 0 0) 0x7f25b009a610 con 0x7f25b01457d0 2024-03-17T23:48:52.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.901+0000 7f25aeffd640 1 -- 172.21.15.150:0/1364731961 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f259c003520 con 0x7f25b01457d0 2024-03-17T23:48:52.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.901+0000 7f25a6ffd640 1 -- 172.21.15.150:0/1364731961 >> v1:172.21.15.157:6789/0 conn(0x7f25b006b6d0 legacy=0x7f25b015cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:52.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.902+0000 7f25aeffd640 1 -- 172.21.15.150:0/1364731961 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3185065717 0 0) 0x7f259c003520 con 0x7f25b01457d0 2024-03-17T23:48:52.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.902+0000 7f25a6ffd640 1 -- 172.21.15.150:0/1364731961 >> v1:172.21.15.150:6789/0 conn(0x7f25b01457d0 legacy=0x7f25b00d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:52.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.902+0000 7f25a6ffd640 1 -- 172.21.15.150:0/1364731961 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b00b1e20 con 0x7f25b009a180 2024-03-17T23:48:52.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.902+0000 7f25a6ffd640 1 --2- 172.21.15.150:0/1364731961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25b009a180 0x7f25b006b190 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f25980097a0 tx=0x7f259802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abcaef84013c8590 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:52.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.903+0000 7f25aeffd640 1 -- 172.21.15.150:0/1364731961 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2598038070 con 0x7f25b009a180 2024-03-17T23:48:52.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.903+0000 7f25aeffd640 1 -- 172.21.15.150:0/1364731961 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2598031500 con 0x7f25b009a180 2024-03-17T23:48:52.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.903+0000 7f25aeffd640 1 -- 172.21.15.150:0/1364731961 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a0001670 con 0x7f25b009a180 2024-03-17T23:48:52.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.903+0000 7f25b5df4640 1 -- 172.21.15.150:0/1364731961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25b009a180 msgr2=0x7f25b006b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:52.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.903+0000 7f25b5df4640 1 --2- 172.21.15.150:0/1364731961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25b009a180 0x7f25b006b190 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f25980097a0 tx=0x7f259802b820 comp rx=0 tx=0).stop 2024-03-17T23:48:52.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.903+0000 7f25b5df4640 1 -- 172.21.15.150:0/1364731961 shutdown_connections 2024-03-17T23:48:52.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.903+0000 7f25b5df4640 1 --2- 172.21.15.150:0/1364731961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25b009a180 0x7f25b006b190 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:52.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.903+0000 7f25b5df4640 1 -- 172.21.15.150:0/1364731961 >> 172.21.15.150:0/1364731961 conn(0x7f25b00b6170 msgr2=0x7f25b0099a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:52.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.903+0000 7f25b5df4640 1 -- 172.21.15.150:0/1364731961 shutdown_connections 2024-03-17T23:48:52.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.903+0000 7f25b5df4640 1 -- 172.21.15.150:0/1364731961 wait complete. 2024-03-17T23:48:52.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.904+0000 7f25b5df4640 1 Processor -- start 2024-03-17T23:48:52.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.904+0000 7f25b5df4640 1 -- start start 2024-03-17T23:48:52.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.905+0000 7f25b5df4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25b006b6d0 0x7f25b013f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:52.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.905+0000 7f25b5df4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f25b009a180 0x7f25b00cf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:52.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.905+0000 7f25b5df4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25b01457d0 0x7f25b00cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:52.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.905+0000 7f25b5df4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f25b00b23e0 con 0x7f25b01457d0 2024-03-17T23:48:52.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.905+0000 7f25b5df4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f25b00d0c60 con 0x7f25b006b6d0 2024-03-17T23:48:52.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.905+0000 7f25b5df4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f25b015d4a0 con 0x7f25b009a180 2024-03-17T23:48:52.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.905+0000 7f25affff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25b01457d0 0x7f25b00cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:52.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.905+0000 7f25affff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25b01457d0 0x7f25b00cfb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36582/0 (socket says 172.21.15.150:36582) 2024-03-17T23:48:52.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.905+0000 7f25affff640 1 -- 172.21.15.150:0/3270166651 learned_addr learned my addr 172.21.15.150:0/3270166651 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:52.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.905+0000 7f25a6ffd640 1 --2- 172.21.15.150:0/3270166651 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f25b009a180 0x7f25b00cf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:52.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.905+0000 7f25af7fe640 1 --2- 172.21.15.150:0/3270166651 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25b006b6d0 0x7f25b013f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:52.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.906+0000 7f25affff640 1 -- 172.21.15.150:0/3270166651 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f25b009a180 msgr2=0x7f25b00cf5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:52.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.906+0000 7f25affff640 1 --2- 172.21.15.150:0/3270166651 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f25b009a180 0x7f25b00cf5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:52.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.906+0000 7f25affff640 1 -- 172.21.15.150:0/3270166651 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25b006b6d0 msgr2=0x7f25b013f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:52.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.906+0000 7f25affff640 1 --2- 172.21.15.150:0/3270166651 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25b006b6d0 0x7f25b013f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:52.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.906+0000 7f25affff640 1 -- 172.21.15.150:0/3270166651 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b00b5aa0 con 0x7f25b01457d0 2024-03-17T23:48:52.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.906+0000 7f25affff640 1 --2- 172.21.15.150:0/3270166651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25b01457d0 0x7f25b00cfb10 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f25a800c7a0 tx=0x7f25a800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:52.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.906+0000 7f25acff9640 1 -- 172.21.15.150:0/3270166651 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a8003b70 con 0x7f25b01457d0 2024-03-17T23:48:52.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.906+0000 7f25acff9640 1 -- 172.21.15.150:0/3270166651 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25a8003d30 con 0x7f25b01457d0 2024-03-17T23:48:52.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.906+0000 7f25acff9640 1 -- 172.21.15.150:0/3270166651 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a800aa30 con 0x7f25b01457d0 2024-03-17T23:48:52.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.906+0000 7f25b5df4640 1 -- 172.21.15.150:0/3270166651 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25b00c9e50 con 0x7f25b01457d0 2024-03-17T23:48:52.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.906+0000 7f25b5df4640 1 -- 172.21.15.150:0/3270166651 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25b00ca410 con 0x7f25b01457d0 2024-03-17T23:48:52.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.907+0000 7f25b5df4640 1 -- 172.21.15.150:0/3270166651 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25b00681a0 con 0x7f25b01457d0 2024-03-17T23:48:52.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.908+0000 7f25acff9640 1 -- 172.21.15.150:0/3270166651 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f25a800abd0 con 0x7f25b01457d0 2024-03-17T23:48:52.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.908+0000 7f25acff9640 1 --2- 172.21.15.150:0/3270166651 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f259006bbd0 0x7f259006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:52.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.908+0000 7f25acff9640 1 -- 172.21.15.150:0/3270166651 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7f25a808ab80 con 0x7f25b01457d0 2024-03-17T23:48:52.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.908+0000 7f25af7fe640 1 --2- 172.21.15.150:0/3270166651 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f259006bbd0 0x7f259006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:52.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.908+0000 7f25af7fe640 1 --2- 172.21.15.150:0/3270166651 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f259006bbd0 0x7f259006e090 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f25a0005880 tx=0x7f25a00057f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:52.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:52.912+0000 7f25acff9640 1 -- 172.21.15.150:0/3270166651 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f25a8058c60 con 0x7f25b01457d0 2024-03-17T23:48:53.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.101+0000 7f25b5df4640 1 -- 172.21.15.150:0/3270166651 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f25b006bf20 con 0x7f25b01457d0 2024-03-17T23:48:53.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.101+0000 7f25acff9640 1 -- 172.21.15.150:0/3270166651 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v45) ==== 104+0+11 (secure 0 0 0) 0x7f25a805cd00 con 0x7f25b01457d0 2024-03-17T23:48:53.102 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:48:53.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.103+0000 7f25b5df4640 1 -- 172.21.15.150:0/3270166651 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f259006bbd0 msgr2=0x7f259006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:53.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.103+0000 7f25b5df4640 1 --2- 172.21.15.150:0/3270166651 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f259006bbd0 0x7f259006e090 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f25a0005880 tx=0x7f25a00057f0 comp rx=0 tx=0).stop 2024-03-17T23:48:53.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.103+0000 7f25b5df4640 1 -- 172.21.15.150:0/3270166651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25b01457d0 msgr2=0x7f25b00cfb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:53.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.103+0000 7f25b5df4640 1 --2- 172.21.15.150:0/3270166651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25b01457d0 0x7f25b00cfb10 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f25a800c7a0 tx=0x7f25a800cc60 comp rx=0 tx=0).stop 2024-03-17T23:48:53.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.103+0000 7f25b5df4640 1 -- 172.21.15.150:0/3270166651 shutdown_connections 2024-03-17T23:48:53.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.103+0000 7f25b5df4640 1 --2- 172.21.15.150:0/3270166651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f25b01457d0 0x7f25b00cfb10 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:53.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.103+0000 7f25b5df4640 1 --2- 172.21.15.150:0/3270166651 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f25b009a180 0x7f25b00cf5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:53.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.103+0000 7f25b5df4640 1 --2- 172.21.15.150:0/3270166651 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f259006bbd0 0x7f259006e090 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:53.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.103+0000 7f25b5df4640 1 --2- 172.21.15.150:0/3270166651 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25b006b6d0 0x7f25b013f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:53.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.103+0000 7f25b5df4640 1 -- 172.21.15.150:0/3270166651 >> 172.21.15.150:0/3270166651 conn(0x7f25b00b6170 msgr2=0x7f25b015d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:53.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.104+0000 7f25b5df4640 1 -- 172.21.15.150:0/3270166651 shutdown_connections 2024-03-17T23:48:53.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.104+0000 7f25b5df4640 1 -- 172.21.15.150:0/3270166651 wait complete. 2024-03-17T23:48:53.119 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:48:53.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.223+0000 7f00ae8ac640 1 Processor -- start 2024-03-17T23:48:53.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.223+0000 7f00ae8ac640 1 -- start start 2024-03-17T23:48:53.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.224+0000 7f00ae8ac640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f00a80d3910 0x7f00a80d0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:53.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.224+0000 7f00ae8ac640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00a80b9660 0x7f00a80d0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:53.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.224+0000 7f00ae8ac640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00a80d2870 con 0x7f00a81615d0 2024-03-17T23:48:53.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.224+0000 7f00ae8ac640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f00a81619b0 con 0x7f00a80d3910 2024-03-17T23:48:53.225 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.224+0000 7f00ae8ac640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f00a80b7e80 con 0x7f00a80b9660 2024-03-17T23:48:53.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.224+0000 7f00a7fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f00a81615d0 0x7f00a80d2020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49590/0 (socket says 172.21.15.150:49590) 2024-03-17T23:48:53.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.224+0000 7f00a7fff640 1 -- 172.21.15.150:0/1704147243 learned_addr learned my addr 172.21.15.150:0/1704147243 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:53.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.224+0000 7f00ace49640 1 --2- 172.21.15.150:0/1704147243 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00a80b9660 0x7f00a80d0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:53.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.224+0000 7f00a77fe640 1 --2- 172.21.15.150:0/1704147243 >> v2:172.21.15.157:3300/0 conn(0x7f00a80d3910 0x7f00a80d0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:53.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.224+0000 7f00a6ffd640 1 -- 172.21.15.150:0/1704147243 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 908153894 0 0) 0x7f00a80d2870 con 0x7f00a81615d0 2024-03-17T23:48:53.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.224+0000 7f00a6ffd640 1 -- 172.21.15.150:0/1704147243 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0090003520 con 0x7f00a81615d0 2024-03-17T23:48:53.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.224+0000 7f00ace49640 1 -- 172.21.15.150:0/1704147243 >> v2:172.21.15.157:3300/0 conn(0x7f00a80d3910 msgr2=0x7f00a80d0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:53.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.224+0000 7f00ace49640 1 --2- 172.21.15.150:0/1704147243 >> v2:172.21.15.157:3300/0 conn(0x7f00a80d3910 0x7f00a80d0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:53.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.224+0000 7f00ace49640 1 -- 172.21.15.150:0/1704147243 >> v1:172.21.15.150:6789/0 conn(0x7f00a81615d0 legacy=0x7f00a80d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:53.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.224+0000 7f00ace49640 1 -- 172.21.15.150:0/1704147243 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00a80d2b50 con 0x7f00a80b9660 2024-03-17T23:48:53.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.226+0000 7f00ace49640 1 --2- 172.21.15.150:0/1704147243 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00a80b9660 0x7f00a80d0c60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f00a0009d90 tx=0x7f00a002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c74f7717c6ecd039 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:53.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.226+0000 7f00a6ffd640 1 -- 172.21.15.150:0/1704147243 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a0039070 con 0x7f00a80b9660 2024-03-17T23:48:53.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.226+0000 7f00a6ffd640 1 -- 172.21.15.150:0/1704147243 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00a002cbc0 con 0x7f00a80b9660 2024-03-17T23:48:53.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.226+0000 7f00a6ffd640 1 -- 172.21.15.150:0/1704147243 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a0034440 con 0x7f00a80b9660 2024-03-17T23:48:53.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.226+0000 7f00ae8ac640 1 -- 172.21.15.150:0/1704147243 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00a80b9660 msgr2=0x7f00a80d0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:53.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.226+0000 7f00ae8ac640 1 --2- 172.21.15.150:0/1704147243 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00a80b9660 0x7f00a80d0c60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f00a0009d90 tx=0x7f00a002be10 comp rx=0 tx=0).stop 2024-03-17T23:48:53.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.226+0000 7f00ae8ac640 1 -- 172.21.15.150:0/1704147243 shutdown_connections 2024-03-17T23:48:53.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.226+0000 7f00ae8ac640 1 --2- 172.21.15.150:0/1704147243 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00a80b9660 0x7f00a80d0c60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:53.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.226+0000 7f00ae8ac640 1 --2- 172.21.15.150:0/1704147243 >> v2:172.21.15.157:3300/0 conn(0x7f00a80d3910 0x7f00a80d0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:53.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.226+0000 7f00ae8ac640 1 -- 172.21.15.150:0/1704147243 >> 172.21.15.150:0/1704147243 conn(0x7f00a80730b0 msgr2=0x7f00a80b5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:53.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.226+0000 7f00ae8ac640 1 -- 172.21.15.150:0/1704147243 shutdown_connections 2024-03-17T23:48:53.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.226+0000 7f00ae8ac640 1 -- 172.21.15.150:0/1704147243 wait complete. 2024-03-17T23:48:53.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.227+0000 7f00ae8ac640 1 Processor -- start 2024-03-17T23:48:53.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.228+0000 7f00ae8ac640 1 -- start start 2024-03-17T23:48:53.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.228+0000 7f00ae8ac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00a80b9660 0x7f00a8158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:53.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.228+0000 7f00ae8ac640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00a80d3910 0x7f00a80b4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:53.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.228+0000 7f00ae8ac640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00a81615d0 0x7f00a80cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:53.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.228+0000 7f00ae8ac640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f00a80d33f0 con 0x7f00a80b9660 2024-03-17T23:48:53.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.228+0000 7f00ae8ac640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f00a80d3110 con 0x7f00a81615d0 2024-03-17T23:48:53.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.228+0000 7f00ae8ac640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f00a80b6e70 con 0x7f00a80d3910 2024-03-17T23:48:53.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.228+0000 7f00a7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00a80b9660 0x7f00a8158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:53.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.229+0000 7f00a7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00a80b9660 0x7f00a8158130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36594/0 (socket says 172.21.15.150:36594) 2024-03-17T23:48:53.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.229+0000 7f00a7fff640 1 -- 172.21.15.150:0/77992474 learned_addr learned my addr 172.21.15.150:0/77992474 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:53.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.229+0000 7f00a77fe640 1 --2- 172.21.15.150:0/77992474 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00a80d3910 0x7f00a80b4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:53.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.229+0000 7f00ace49640 1 --2- 172.21.15.150:0/77992474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00a81615d0 0x7f00a80cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:53.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.229+0000 7f00a7fff640 1 -- 172.21.15.150:0/77992474 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00a80d3910 msgr2=0x7f00a80b4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:53.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.229+0000 7f00a7fff640 1 --2- 172.21.15.150:0/77992474 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00a80d3910 0x7f00a80b4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:53.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.229+0000 7f00a7fff640 1 -- 172.21.15.150:0/77992474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00a81615d0 msgr2=0x7f00a80cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:53.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.229+0000 7f00a7fff640 1 --2- 172.21.15.150:0/77992474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00a81615d0 0x7f00a80cc6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:53.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.229+0000 7f00a7fff640 1 -- 172.21.15.150:0/77992474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00a8158670 con 0x7f00a80b9660 2024-03-17T23:48:53.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.229+0000 7f00a77fe640 1 --2- 172.21.15.150:0/77992474 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00a80d3910 0x7f00a80b4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:53.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.229+0000 7f00ace49640 1 --2- 172.21.15.150:0/77992474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00a81615d0 0x7f00a80cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:53.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.229+0000 7f00a7fff640 1 --2- 172.21.15.150:0/77992474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00a80b9660 0x7f00a8158130 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f009800c3c0 tx=0x7f009800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:53.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.229+0000 7f00a4ff9640 1 -- 172.21.15.150:0/77992474 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0098017070 con 0x7f00a80b9660 2024-03-17T23:48:53.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.229+0000 7f00a4ff9640 1 -- 172.21.15.150:0/77992474 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0098005ce0 con 0x7f00a80b9660 2024-03-17T23:48:53.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.229+0000 7f00ae8ac640 1 -- 172.21.15.150:0/77992474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00a8154d40 con 0x7f00a80b9660 2024-03-17T23:48:53.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.229+0000 7f00a4ff9640 1 -- 172.21.15.150:0/77992474 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00980072f0 con 0x7f00a80b9660 2024-03-17T23:48:53.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.229+0000 7f00ae8ac640 1 -- 172.21.15.150:0/77992474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00a8155580 con 0x7f00a80b9660 2024-03-17T23:48:53.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.230+0000 7f00ae8ac640 1 -- 172.21.15.150:0/77992474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00a80d1fa0 con 0x7f00a80b9660 2024-03-17T23:48:53.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.231+0000 7f00a4ff9640 1 -- 172.21.15.150:0/77992474 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f009801a070 con 0x7f00a80b9660 2024-03-17T23:48:53.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.231+0000 7f00a4ff9640 1 --2- 172.21.15.150:0/77992474 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f009406bbd0 0x7f009406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:53.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.231+0000 7f00a4ff9640 1 -- 172.21.15.150:0/77992474 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7f009808a0d0 con 0x7f00a80b9660 2024-03-17T23:48:53.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.231+0000 7f00a77fe640 1 --2- 172.21.15.150:0/77992474 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f009406bbd0 0x7f009406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:53.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.232+0000 7f00a77fe640 1 --2- 172.21.15.150:0/77992474 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f009406bbd0 0x7f009406e090 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f00a80b5ab0 tx=0x7f009c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:53.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.240+0000 7f00a4ff9640 1 -- 172.21.15.150:0/77992474 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f00a8154d40 con 0x7f00a80b9660 2024-03-17T23:48:53.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.576+0000 7f00ae8ac640 1 -- 172.21.15.150:0/77992474 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f00a80688b0 con 0x7f00a80b9660 2024-03-17T23:48:53.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.576+0000 7f00a4ff9640 1 -- 172.21.15.150:0/77992474 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v45) ==== 100+0+11 (secure 0 0 0) 0x7f0098058920 con 0x7f00a80b9660 2024-03-17T23:48:53.577 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:48:53.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.579+0000 7f00ae8ac640 1 -- 172.21.15.150:0/77992474 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f009406bbd0 msgr2=0x7f009406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:53.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.579+0000 7f00ae8ac640 1 --2- 172.21.15.150:0/77992474 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f009406bbd0 0x7f009406e090 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f00a80b5ab0 tx=0x7f009c008040 comp rx=0 tx=0).stop 2024-03-17T23:48:53.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.579+0000 7f00ae8ac640 1 -- 172.21.15.150:0/77992474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00a80b9660 msgr2=0x7f00a8158130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:53.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.579+0000 7f00ae8ac640 1 --2- 172.21.15.150:0/77992474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00a80b9660 0x7f00a8158130 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f009800c3c0 tx=0x7f009800c880 comp rx=0 tx=0).stop 2024-03-17T23:48:53.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.579+0000 7f00ae8ac640 1 -- 172.21.15.150:0/77992474 shutdown_connections 2024-03-17T23:48:53.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.579+0000 7f00ae8ac640 1 --2- 172.21.15.150:0/77992474 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f009406bbd0 0x7f009406e090 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:53.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.579+0000 7f00ae8ac640 1 --2- 172.21.15.150:0/77992474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f00a81615d0 0x7f00a80cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:53.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.579+0000 7f00ae8ac640 1 --2- 172.21.15.150:0/77992474 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f00a80d3910 0x7f00a80b4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:53.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.579+0000 7f00ae8ac640 1 --2- 172.21.15.150:0/77992474 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00a80b9660 0x7f00a8158130 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:53.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.579+0000 7f00ae8ac640 1 -- 172.21.15.150:0/77992474 >> 172.21.15.150:0/77992474 conn(0x7f00a80730b0 msgr2=0x7f00a815fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:53.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.579+0000 7f00ae8ac640 1 -- 172.21.15.150:0/77992474 shutdown_connections 2024-03-17T23:48:53.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.579+0000 7f00ae8ac640 1 -- 172.21.15.150:0/77992474 wait complete. 2024-03-17T23:48:53.591 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-17T23:48:53.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.695+0000 7f675b1f3640 1 Processor -- start 2024-03-17T23:48:53.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.695+0000 7f675b1f3640 1 -- start start 2024-03-17T23:48:53.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.696+0000 7f675b1f3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67540b9970 0x7f67540b5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:53.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.696+0000 7f675b1f3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f67540b5cd0 0x7f67540d26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:53.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.696+0000 7f675b1f3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f675406cec0 con 0x7f675415de00 2024-03-17T23:48:53.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.696+0000 7f675b1f3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f67540b6cd0 con 0x7f67540b5cd0 2024-03-17T23:48:53.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.696+0000 7f675b1f3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f67540b6120 con 0x7f67540b9970 2024-03-17T23:48:53.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.696+0000 7f6759790640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f67540b5cd0 0x7f67540d26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:53.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.696+0000 7f6758f8f640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f675415de00 0x7f675406c690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49598/0 (socket says 172.21.15.150:49598) 2024-03-17T23:48:53.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.696+0000 7f6758f8f640 1 -- 172.21.15.150:0/623516149 learned_addr learned my addr 172.21.15.150:0/623516149 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:53.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.697+0000 7f6753fff640 1 --2- 172.21.15.150:0/623516149 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67540b9970 0x7f67540b5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:53.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.697+0000 7f6759790640 1 -- 172.21.15.150:0/623516149 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67540b9970 msgr2=0x7f67540b5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:53.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.697+0000 7f6759790640 1 --2- 172.21.15.150:0/623516149 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67540b9970 0x7f67540b5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:53.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.697+0000 7f6759790640 1 -- 172.21.15.150:0/623516149 >> v1:172.21.15.150:6789/0 conn(0x7f675415de00 legacy=0x7f675406c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:53.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.697+0000 7f67537fe640 1 -- 172.21.15.150:0/623516149 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2908013243 0 0) 0x7f675406cec0 con 0x7f675415de00 2024-03-17T23:48:53.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.697+0000 7f6759790640 1 -- 172.21.15.150:0/623516149 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f675406d1a0 con 0x7f67540b5cd0 2024-03-17T23:48:53.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.697+0000 7f6759790640 1 --2- 172.21.15.150:0/623516149 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67540b5cd0 0x7f67540d26b0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f674c009d90 tx=0x7f674c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f5731eb6dc0b4d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:53.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.698+0000 7f67537fe640 1 -- 172.21.15.150:0/623516149 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c039070 con 0x7f67540b5cd0 2024-03-17T23:48:53.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.698+0000 7f67537fe640 1 -- 172.21.15.150:0/623516149 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f674c02cc50 con 0x7f67540b5cd0 2024-03-17T23:48:53.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.698+0000 7f67537fe640 1 -- 172.21.15.150:0/623516149 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674c034440 con 0x7f67540b5cd0 2024-03-17T23:48:53.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.698+0000 7f675b1f3640 1 -- 172.21.15.150:0/623516149 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67540b5cd0 msgr2=0x7f67540d26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:53.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.698+0000 7f675b1f3640 1 --2- 172.21.15.150:0/623516149 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67540b5cd0 0x7f67540d26b0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f674c009d90 tx=0x7f674c02bea0 comp rx=0 tx=0).stop 2024-03-17T23:48:53.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.698+0000 7f675b1f3640 1 -- 172.21.15.150:0/623516149 shutdown_connections 2024-03-17T23:48:53.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.698+0000 7f675b1f3640 1 --2- 172.21.15.150:0/623516149 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67540b9970 0x7f67540b5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:53.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.698+0000 7f675b1f3640 1 --2- 172.21.15.150:0/623516149 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67540b5cd0 0x7f67540d26b0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:53.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.698+0000 7f675b1f3640 1 -- 172.21.15.150:0/623516149 >> 172.21.15.150:0/623516149 conn(0x7f67540d0790 msgr2=0x7f67540d0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:53.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.698+0000 7f675b1f3640 1 -- 172.21.15.150:0/623516149 shutdown_connections 2024-03-17T23:48:53.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.698+0000 7f675b1f3640 1 -- 172.21.15.150:0/623516149 wait complete. 2024-03-17T23:48:53.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.699+0000 7f675b1f3640 1 Processor -- start 2024-03-17T23:48:53.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.699+0000 7f675b1f3640 1 -- start start 2024-03-17T23:48:53.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.700+0000 7f675b1f3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67540b5cd0 0x7f675414f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:53.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.700+0000 7f675b1f3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67540b9970 0x7f67540b7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:53.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.700+0000 7f675b1f3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f675415de00 0x7f67540b8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:53.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.700+0000 7f675b1f3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f675415e1e0 con 0x7f67540b5cd0 2024-03-17T23:48:53.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.700+0000 7f675b1f3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f67540d2190 con 0x7f675415de00 2024-03-17T23:48:53.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.700+0000 7f675b1f3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f67540d2bf0 con 0x7f67540b9970 2024-03-17T23:48:53.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.700+0000 7f6758f8f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67540b5cd0 0x7f675414f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:53.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.700+0000 7f6758f8f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67540b5cd0 0x7f675414f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36604/0 (socket says 172.21.15.150:36604) 2024-03-17T23:48:53.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.700+0000 7f6758f8f640 1 -- 172.21.15.150:0/220972038 learned_addr learned my addr 172.21.15.150:0/220972038 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:53.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.701+0000 7f6753fff640 1 --2- 172.21.15.150:0/220972038 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67540b9970 0x7f67540b7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:53.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.701+0000 7f6759790640 1 --2- 172.21.15.150:0/220972038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f675415de00 0x7f67540b8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:53.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.701+0000 7f6758f8f640 1 -- 172.21.15.150:0/220972038 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67540b9970 msgr2=0x7f67540b7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:53.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.701+0000 7f6758f8f640 1 --2- 172.21.15.150:0/220972038 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67540b9970 0x7f67540b7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:53.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.701+0000 7f6758f8f640 1 -- 172.21.15.150:0/220972038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f675415de00 msgr2=0x7f67540b8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:53.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.701+0000 7f6758f8f640 1 --2- 172.21.15.150:0/220972038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f675415de00 0x7f67540b8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:53.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.701+0000 7f6758f8f640 1 -- 172.21.15.150:0/220972038 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6754159940 con 0x7f67540b5cd0 2024-03-17T23:48:53.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.701+0000 7f6753fff640 1 --2- 172.21.15.150:0/220972038 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67540b9970 0x7f67540b7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:53.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.701+0000 7f6758f8f640 1 --2- 172.21.15.150:0/220972038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67540b5cd0 0x7f675414f0d0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f674800c430 tx=0x7f674800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:53.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.701+0000 7f67517fa640 1 -- 172.21.15.150:0/220972038 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67480074e0 con 0x7f67540b5cd0 2024-03-17T23:48:53.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.701+0000 7f67517fa640 1 -- 172.21.15.150:0/220972038 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67480047e0 con 0x7f67540b5cd0 2024-03-17T23:48:53.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.701+0000 7f675b1f3640 1 -- 172.21.15.150:0/220972038 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6754159c20 con 0x7f67540b5cd0 2024-03-17T23:48:53.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.701+0000 7f67517fa640 1 -- 172.21.15.150:0/220972038 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6748004bd0 con 0x7f67540b5cd0 2024-03-17T23:48:53.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.701+0000 7f675b1f3640 1 -- 172.21.15.150:0/220972038 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f675415a1e0 con 0x7f67540b5cd0 2024-03-17T23:48:53.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.702+0000 7f675b1f3640 1 -- 172.21.15.150:0/220972038 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67540681a0 con 0x7f67540b5cd0 2024-03-17T23:48:53.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.703+0000 7f67517fa640 1 -- 172.21.15.150:0/220972038 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6748011070 con 0x7f67540b5cd0 2024-03-17T23:48:53.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.703+0000 7f67517fa640 1 --2- 172.21.15.150:0/220972038 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f673806bbd0 0x7f673806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:53.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.703+0000 7f6753fff640 1 --2- 172.21.15.150:0/220972038 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f673806bbd0 0x7f673806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:53.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.703+0000 7f67517fa640 1 -- 172.21.15.150:0/220972038 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7f674808a990 con 0x7f67540b5cd0 2024-03-17T23:48:53.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.704+0000 7f6753fff640 1 --2- 172.21.15.150:0/220972038 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f673806bbd0 0x7f673806e090 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f67540d11a0 tx=0x7f6740008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:53.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.707+0000 7f67517fa640 1 -- 172.21.15.150:0/220972038 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f67480591e0 con 0x7f67540b5cd0 2024-03-17T23:48:53.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.882+0000 7f675b1f3640 1 -- 172.21.15.150:0/220972038 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7f67540d18e0 con 0x7f67540b5cd0 2024-03-17T23:48:53.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.883+0000 7f67517fa640 1 -- 172.21.15.150:0/220972038 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v45) ==== 97+0+11 (secure 0 0 0) 0x7f6748058ae0 con 0x7f67540b5cd0 2024-03-17T23:48:53.884 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:48:53.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.885+0000 7f675b1f3640 1 -- 172.21.15.150:0/220972038 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f673806bbd0 msgr2=0x7f673806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:53.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.885+0000 7f675b1f3640 1 --2- 172.21.15.150:0/220972038 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f673806bbd0 0x7f673806e090 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f67540d11a0 tx=0x7f6740008040 comp rx=0 tx=0).stop 2024-03-17T23:48:53.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.885+0000 7f675b1f3640 1 -- 172.21.15.150:0/220972038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67540b5cd0 msgr2=0x7f675414f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:53.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.885+0000 7f675b1f3640 1 --2- 172.21.15.150:0/220972038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67540b5cd0 0x7f675414f0d0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f674800c430 tx=0x7f674800c8f0 comp rx=0 tx=0).stop 2024-03-17T23:48:53.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.885+0000 7f675b1f3640 1 -- 172.21.15.150:0/220972038 shutdown_connections 2024-03-17T23:48:53.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.885+0000 7f675b1f3640 1 --2- 172.21.15.150:0/220972038 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f673806bbd0 0x7f673806e090 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:53.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.885+0000 7f675b1f3640 1 --2- 172.21.15.150:0/220972038 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f675415de00 0x7f67540b8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:53.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.885+0000 7f675b1f3640 1 --2- 172.21.15.150:0/220972038 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f67540b9970 0x7f67540b7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:53.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.885+0000 7f675b1f3640 1 --2- 172.21.15.150:0/220972038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67540b5cd0 0x7f675414f0d0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:53.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.885+0000 7f675b1f3640 1 -- 172.21.15.150:0/220972038 >> 172.21.15.150:0/220972038 conn(0x7f67540d0790 msgr2=0x7f675406c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:53.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.885+0000 7f675b1f3640 1 -- 172.21.15.150:0/220972038 shutdown_connections 2024-03-17T23:48:53.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:53.885+0000 7f675b1f3640 1 -- 172.21.15.150:0/220972038 wait complete. 2024-03-17T23:48:53.897 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-17T23:48:54.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.013+0000 7fd9a1137640 1 Processor -- start 2024-03-17T23:48:54.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.013+0000 7fd9a1137640 1 -- start start 2024-03-17T23:48:54.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.013+0000 7fd9a1137640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd99c15be30 0x7fd99c0cde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:54.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.013+0000 7fd9a1137640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd99c0cc6c0 0x7fd99c0b0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:54.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.013+0000 7fd9a1137640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd99c0b14f0 0x7fd99c0b1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:54.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.013+0000 7fd9a1137640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fd99c0ccb00 con 0x7fd99c0b14f0 2024-03-17T23:48:54.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.013+0000 7fd9a1137640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd99c0ccda0 con 0x7fd99c0cc6c0 2024-03-17T23:48:54.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.013+0000 7fd9a1137640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd99c0cd080 con 0x7fd99c15be30 2024-03-17T23:48:54.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.014+0000 7fd99b59e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd99c0b14f0 0x7fd99c0b1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:54.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.014+0000 7fd99b59e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd99c0b14f0 0x7fd99c0b1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36612/0 (socket says 172.21.15.150:36612) 2024-03-17T23:48:54.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.014+0000 7fd99b59e640 1 -- 172.21.15.150:0/3463190108 learned_addr learned my addr 172.21.15.150:0/3463190108 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:54.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.014+0000 7fd99ad9d640 1 --2- 172.21.15.150:0/3463190108 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd99c15be30 0x7fd99c0cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:54.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.014+0000 7fd99b59e640 1 -- 172.21.15.150:0/3463190108 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd99c15be30 msgr2=0x7fd99c0cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:54.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.014+0000 7fd99b59e640 1 --2- 172.21.15.150:0/3463190108 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd99c15be30 0x7fd99c0cde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:54.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.014+0000 7fd99b59e640 1 -- 172.21.15.150:0/3463190108 >> v2:172.21.15.157:3300/0 conn(0x7fd99c0cc6c0 msgr2=0x7fd99c0b0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:54.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.014+0000 7fd993fff640 1 --2- 172.21.15.150:0/3463190108 >> v2:172.21.15.157:3300/0 conn(0x7fd99c0cc6c0 0x7fd99c0b0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:54.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.014+0000 7fd99b59e640 1 --2- 172.21.15.150:0/3463190108 >> v2:172.21.15.157:3300/0 conn(0x7fd99c0cc6c0 0x7fd99c0b0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:54.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.014+0000 7fd99b59e640 1 -- 172.21.15.150:0/3463190108 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd99c0cd360 con 0x7fd99c0b14f0 2024-03-17T23:48:54.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.014+0000 7fd99ad9d640 1 --2- 172.21.15.150:0/3463190108 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd99c15be30 0x7fd99c0cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:54.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.014+0000 7fd993fff640 1 --2- 172.21.15.150:0/3463190108 >> v2:172.21.15.157:3300/0 conn(0x7fd99c0cc6c0 0x7fd99c0b0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:54.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.015+0000 7fd99b59e640 1 --2- 172.21.15.150:0/3463190108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd99c0b14f0 0x7fd99c0b1940 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fd99c0733c0 tx=0x7fd99402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8826e185c0a6d84a server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:54.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.015+0000 7fd99a59c640 1 -- 172.21.15.150:0/3463190108 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99402ca70 con 0x7fd99c0b14f0 2024-03-17T23:48:54.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.015+0000 7fd99a59c640 1 -- 172.21.15.150:0/3463190108 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd99402cc10 con 0x7fd99c0b14f0 2024-03-17T23:48:54.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.015+0000 7fd99a59c640 1 -- 172.21.15.150:0/3463190108 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9940343f0 con 0x7fd99c0b14f0 2024-03-17T23:48:54.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.015+0000 7fd9a1137640 1 -- 172.21.15.150:0/3463190108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd99c0b14f0 msgr2=0x7fd99c0b1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:54.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.015+0000 7fd9a1137640 1 --2- 172.21.15.150:0/3463190108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd99c0b14f0 0x7fd99c0b1940 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fd99c0733c0 tx=0x7fd99402bed0 comp rx=0 tx=0).stop 2024-03-17T23:48:54.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.015+0000 7fd9a1137640 1 -- 172.21.15.150:0/3463190108 shutdown_connections 2024-03-17T23:48:54.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.015+0000 7fd9a1137640 1 --2- 172.21.15.150:0/3463190108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd99c0b14f0 0x7fd99c0b1940 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:54.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.015+0000 7fd9a1137640 1 --2- 172.21.15.150:0/3463190108 >> v2:172.21.15.157:3300/0 conn(0x7fd99c0cc6c0 0x7fd99c0b0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:54.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.015+0000 7fd9a1137640 1 --2- 172.21.15.150:0/3463190108 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd99c15be30 0x7fd99c0cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:54.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.015+0000 7fd9a1137640 1 -- 172.21.15.150:0/3463190108 >> 172.21.15.150:0/3463190108 conn(0x7fd99c0b36a0 msgr2=0x7fd99c0b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:54.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.015+0000 7fd9a1137640 1 -- 172.21.15.150:0/3463190108 shutdown_connections 2024-03-17T23:48:54.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.015+0000 7fd9a1137640 1 -- 172.21.15.150:0/3463190108 wait complete. 2024-03-17T23:48:54.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.017+0000 7fd9a1137640 1 Processor -- start 2024-03-17T23:48:54.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.017+0000 7fd9a1137640 1 -- start start 2024-03-17T23:48:54.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.019+0000 7fd9a1137640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd99c0b14f0 0x7fd99c0b7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:54.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.019+0000 7fd9a1137640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd99c0cc6c0 0x7fd99c0b5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:54.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.019+0000 7fd9a1137640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd99c15be30 0x7fd99c1579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:54.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.019+0000 7fd9a1137640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c15c210 con 0x7fd99c15be30 2024-03-17T23:48:54.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.019+0000 7fd9a1137640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c0cd920 con 0x7fd99c0cc6c0 2024-03-17T23:48:54.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.019+0000 7fd9a1137640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd99c15c390 con 0x7fd99c0b14f0 2024-03-17T23:48:54.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.019+0000 7fd99b59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd99c15be30 0x7fd99c1579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:54.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.019+0000 7fd99b59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd99c15be30 0x7fd99c1579d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36624/0 (socket says 172.21.15.150:36624) 2024-03-17T23:48:54.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.019+0000 7fd99b59e640 1 -- 172.21.15.150:0/1765099921 learned_addr learned my addr 172.21.15.150:0/1765099921 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:54.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.019+0000 7fd99ad9d640 1 --2- 172.21.15.150:0/1765099921 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd99c0b14f0 0x7fd99c0b7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:54.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.019+0000 7fd993fff640 1 --2- 172.21.15.150:0/1765099921 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd99c0cc6c0 0x7fd99c0b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:54.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.019+0000 7fd99b59e640 1 -- 172.21.15.150:0/1765099921 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd99c0b14f0 msgr2=0x7fd99c0b7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:54.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.019+0000 7fd99b59e640 1 --2- 172.21.15.150:0/1765099921 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd99c0b14f0 0x7fd99c0b7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:54.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.019+0000 7fd99b59e640 1 -- 172.21.15.150:0/1765099921 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd99c0cc6c0 msgr2=0x7fd99c0b5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:54.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.019+0000 7fd99b59e640 1 --2- 172.21.15.150:0/1765099921 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd99c0cc6c0 0x7fd99c0b5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:54.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.019+0000 7fd99b59e640 1 -- 172.21.15.150:0/1765099921 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd99c15e8f0 con 0x7fd99c15be30 2024-03-17T23:48:54.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.019+0000 7fd993fff640 1 --2- 172.21.15.150:0/1765099921 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd99c0cc6c0 0x7fd99c0b5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:54.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.019+0000 7fd99b59e640 1 --2- 172.21.15.150:0/1765099921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd99c15be30 0x7fd99c1579d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fd994009f40 tx=0x7fd994004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:54.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.019+0000 7fd9937fe640 1 -- 172.21.15.150:0/1765099921 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd994043020 con 0x7fd99c15be30 2024-03-17T23:48:54.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.019+0000 7fd9937fe640 1 -- 172.21.15.150:0/1765099921 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd99403e450 con 0x7fd99c15be30 2024-03-17T23:48:54.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.019+0000 7fd9937fe640 1 -- 172.21.15.150:0/1765099921 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd994009510 con 0x7fd99c15be30 2024-03-17T23:48:54.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.019+0000 7fd9a1137640 1 -- 172.21.15.150:0/1765099921 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd99c15ebd0 con 0x7fd99c15be30 2024-03-17T23:48:54.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.020+0000 7fd9a1137640 1 -- 172.21.15.150:0/1765099921 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd99c15f190 con 0x7fd99c15be30 2024-03-17T23:48:54.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.020+0000 7fd9a1137640 1 -- 172.21.15.150:0/1765099921 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd99c0688b0 con 0x7fd99c15be30 2024-03-17T23:48:54.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.021+0000 7fd9937fe640 1 -- 172.21.15.150:0/1765099921 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd994046020 con 0x7fd99c15be30 2024-03-17T23:48:54.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.021+0000 7fd9937fe640 1 --2- 172.21.15.150:0/1765099921 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd97c06bbd0 0x7fd97c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:54.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.022+0000 7fd99ad9d640 1 --2- 172.21.15.150:0/1765099921 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd97c06bbd0 0x7fd97c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:54.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.022+0000 7fd9937fe640 1 -- 172.21.15.150:0/1765099921 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7fd9940ad550 con 0x7fd99c15be30 2024-03-17T23:48:54.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.022+0000 7fd99ad9d640 1 --2- 172.21.15.150:0/1765099921 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd97c06bbd0 0x7fd97c06e090 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd98c00aa40 tx=0x7fd98c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:54.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.026+0000 7fd9937fe640 1 -- 172.21.15.150:0/1765099921 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd99404d030 con 0x7fd99c15be30 2024-03-17T23:48:54.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.217+0000 7fd9a1137640 1 -- 172.21.15.150:0/1765099921 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd99c09b1d0 con 0x7fd99c15be30 2024-03-17T23:48:54.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.218+0000 7fd9937fe640 1 -- 172.21.15.150:0/1765099921 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v45) ==== 80+0+0 (secure 0 0 0) 0x7fd99407bda0 con 0x7fd99c15be30 2024-03-17T23:48:54.218 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-03-17T23:48:54.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.220+0000 7fd9a1137640 1 -- 172.21.15.150:0/1765099921 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd97c06bbd0 msgr2=0x7fd97c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:54.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.220+0000 7fd9a1137640 1 --2- 172.21.15.150:0/1765099921 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd97c06bbd0 0x7fd97c06e090 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd98c00aa40 tx=0x7fd98c008040 comp rx=0 tx=0).stop 2024-03-17T23:48:54.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.220+0000 7fd9a1137640 1 -- 172.21.15.150:0/1765099921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd99c15be30 msgr2=0x7fd99c1579d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:54.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.220+0000 7fd9a1137640 1 --2- 172.21.15.150:0/1765099921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd99c15be30 0x7fd99c1579d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fd994009f40 tx=0x7fd994004a40 comp rx=0 tx=0).stop 2024-03-17T23:48:54.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.220+0000 7fd9a1137640 1 -- 172.21.15.150:0/1765099921 shutdown_connections 2024-03-17T23:48:54.221 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.220+0000 7fd9a1137640 1 --2- 172.21.15.150:0/1765099921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd99c15be30 0x7fd99c1579d0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:54.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.220+0000 7fd9a1137640 1 --2- 172.21.15.150:0/1765099921 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd97c06bbd0 0x7fd97c06e090 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:54.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.220+0000 7fd9a1137640 1 --2- 172.21.15.150:0/1765099921 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd99c0cc6c0 0x7fd99c0b5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:54.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.220+0000 7fd9a1137640 1 --2- 172.21.15.150:0/1765099921 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd99c0b14f0 0x7fd99c0b7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:54.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.220+0000 7fd9a1137640 1 -- 172.21.15.150:0/1765099921 >> 172.21.15.150:0/1765099921 conn(0x7fd99c0b36a0 msgr2=0x7fd99c0b2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:54.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.220+0000 7fd9a1137640 1 -- 172.21.15.150:0/1765099921 shutdown_connections 2024-03-17T23:48:54.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.220+0000 7fd9a1137640 1 -- 172.21.15.150:0/1765099921 wait complete. 2024-03-17T23:48:54.234 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi150.front.sepia.ceph.com... 2024-03-17T23:48:54.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:54.234 DEBUG:teuthology.orchestra.run.smithi150:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-17T23:48:54.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:ceph-fuse[57019]: fuse finished with error 0 and tester_r 0 2024-03-17T23:48:54.262 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:48:54.337 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi150.front.sepia.ceph.com 2024-03-17T23:48:54.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:54.338 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-17T23:48:54.353 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2024-03-17T23:48:54.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.507+0000 7f4462459640 1 Processor -- start 2024-03-17T23:48:54.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.507+0000 7f4462459640 1 -- start start 2024-03-17T23:48:54.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.507+0000 7f4462459640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f445c1615d0 0x7f445c0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:54.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.507+0000 7f4462459640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f445c0d3910 0x7f445c0d0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:54.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.507+0000 7f4462459640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f445c0b9660 0x7f445c0d0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:54.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.507+0000 7f4462459640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f445c1619b0 con 0x7f445c0b9660 2024-03-17T23:48:54.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.507+0000 7f4462459640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f445c0d2870 con 0x7f445c1615d0 2024-03-17T23:48:54.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.507+0000 7f4462459640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f445c0d2b50 con 0x7f445c0d3910 2024-03-17T23:48:54.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.508+0000 7f44609f6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f445c0b9660 0x7f445c0d0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:54.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.508+0000 7f44609f6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f445c0b9660 0x7f445c0d0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36640/0 (socket says 172.21.15.150:36640) 2024-03-17T23:48:54.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.508+0000 7f44609f6640 1 -- 172.21.15.150:0/2779980521 learned_addr learned my addr 172.21.15.150:0/2779980521 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:54.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.508+0000 7f445b7fe640 1 --2- 172.21.15.150:0/2779980521 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f445c0d3910 0x7f445c0d0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:54.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.508+0000 7f445bfff640 1 --2- 172.21.15.150:0/2779980521 >> v2:172.21.15.157:3300/0 conn(0x7f445c1615d0 0x7f445c0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:54.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.508+0000 7f44609f6640 1 -- 172.21.15.150:0/2779980521 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f445c0d3910 msgr2=0x7f445c0d0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:54.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.508+0000 7f44609f6640 1 --2- 172.21.15.150:0/2779980521 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f445c0d3910 0x7f445c0d0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:54.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.508+0000 7f44609f6640 1 -- 172.21.15.150:0/2779980521 >> v2:172.21.15.157:3300/0 conn(0x7f445c1615d0 msgr2=0x7f445c0d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:54.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.508+0000 7f44609f6640 1 --2- 172.21.15.150:0/2779980521 >> v2:172.21.15.157:3300/0 conn(0x7f445c1615d0 0x7f445c0d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:54.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.508+0000 7f44609f6640 1 -- 172.21.15.150:0/2779980521 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f445c0d2e30 con 0x7f445c0b9660 2024-03-17T23:48:54.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.509+0000 7f44609f6640 1 --2- 172.21.15.150:0/2779980521 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f445c0b9660 0x7f445c0d0c60 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f445c06d7c0 tx=0x7f445402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f1ef18fd48febcb server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:54.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.509+0000 7f445affd640 1 -- 172.21.15.150:0/2779980521 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f445402ca70 con 0x7f445c0b9660 2024-03-17T23:48:54.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.509+0000 7f445affd640 1 -- 172.21.15.150:0/2779980521 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f445402cc10 con 0x7f445c0b9660 2024-03-17T23:48:54.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.509+0000 7f445affd640 1 -- 172.21.15.150:0/2779980521 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44540343f0 con 0x7f445c0b9660 2024-03-17T23:48:54.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.509+0000 7f4462459640 1 -- 172.21.15.150:0/2779980521 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f445c0b9660 msgr2=0x7f445c0d0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:54.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.509+0000 7f4462459640 1 --2- 172.21.15.150:0/2779980521 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f445c0b9660 0x7f445c0d0c60 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f445c06d7c0 tx=0x7f445402bed0 comp rx=0 tx=0).stop 2024-03-17T23:48:54.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.510+0000 7f4462459640 1 -- 172.21.15.150:0/2779980521 shutdown_connections 2024-03-17T23:48:54.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.510+0000 7f4462459640 1 --2- 172.21.15.150:0/2779980521 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f445c0b9660 0x7f445c0d0c60 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:54.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.510+0000 7f4462459640 1 --2- 172.21.15.150:0/2779980521 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f445c0d3910 0x7f445c0d0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:54.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.510+0000 7f4462459640 1 --2- 172.21.15.150:0/2779980521 >> v2:172.21.15.157:3300/0 conn(0x7f445c1615d0 0x7f445c0d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:54.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.510+0000 7f4462459640 1 -- 172.21.15.150:0/2779980521 >> 172.21.15.150:0/2779980521 conn(0x7f445c0730b0 msgr2=0x7f445c0b5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:54.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.510+0000 7f4462459640 1 -- 172.21.15.150:0/2779980521 shutdown_connections 2024-03-17T23:48:54.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.510+0000 7f4462459640 1 -- 172.21.15.150:0/2779980521 wait complete. 2024-03-17T23:48:54.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.511+0000 7f4462459640 1 Processor -- start 2024-03-17T23:48:54.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.511+0000 7f4462459640 1 -- start start 2024-03-17T23:48:54.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.512+0000 7f4462459640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f445c0b9660 0x7f445c154250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:54.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.512+0000 7f4462459640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f445c0d3910 0x7f445c0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:54.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.512+0000 7f4462459640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f445c1615d0 0x7f445c155530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:54.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.512+0000 7f4462459640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f445c0d2560 con 0x7f445c0b9660 2024-03-17T23:48:54.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.512+0000 7f4462459640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f445c0d33f0 con 0x7f445c0d3910 2024-03-17T23:48:54.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.512+0000 7f4462459640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f445c06c790 con 0x7f445c1615d0 2024-03-17T23:48:54.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.512+0000 7f445bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f445c0b9660 0x7f445c154250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:54.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.512+0000 7f445bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f445c0b9660 0x7f445c154250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36654/0 (socket says 172.21.15.150:36654) 2024-03-17T23:48:54.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.512+0000 7f445bfff640 1 -- 172.21.15.150:0/4245326247 learned_addr learned my addr 172.21.15.150:0/4245326247 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:54.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.512+0000 7f445b7fe640 1 --2- 172.21.15.150:0/4245326247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f445c0d3910 0x7f445c0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:54.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.512+0000 7f44609f6640 1 --2- 172.21.15.150:0/4245326247 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f445c1615d0 0x7f445c155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:54.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.512+0000 7f445bfff640 1 -- 172.21.15.150:0/4245326247 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f445c1615d0 msgr2=0x7f445c155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:54.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.512+0000 7f445bfff640 1 --2- 172.21.15.150:0/4245326247 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f445c1615d0 0x7f445c155530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:54.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.512+0000 7f445bfff640 1 -- 172.21.15.150:0/4245326247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f445c0d3910 msgr2=0x7f445c0cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:54.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.512+0000 7f445bfff640 1 --2- 172.21.15.150:0/4245326247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f445c0d3910 0x7f445c0cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:54.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.512+0000 7f445bfff640 1 -- 172.21.15.150:0/4245326247 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f445c0ccb80 con 0x7f445c0b9660 2024-03-17T23:48:54.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.512+0000 7f445bfff640 1 --2- 172.21.15.150:0/4245326247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f445c0b9660 0x7f445c154250 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f444c00e870 tx=0x7f444c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:54.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.512+0000 7f4458ff9640 1 -- 172.21.15.150:0/4245326247 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444c017070 con 0x7f445c0b9660 2024-03-17T23:48:54.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.512+0000 7f4458ff9640 1 -- 172.21.15.150:0/4245326247 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f444c00c940 con 0x7f445c0b9660 2024-03-17T23:48:54.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.512+0000 7f4458ff9640 1 -- 172.21.15.150:0/4245326247 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444c00cd30 con 0x7f445c0b9660 2024-03-17T23:48:54.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.512+0000 7f4462459640 1 -- 172.21.15.150:0/4245326247 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f445c0cce60 con 0x7f445c0b9660 2024-03-17T23:48:54.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.512+0000 7f4462459640 1 -- 172.21.15.150:0/4245326247 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f445c0cd420 con 0x7f445c0b9660 2024-03-17T23:48:54.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.513+0000 7f4462459640 1 -- 172.21.15.150:0/4245326247 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f445c0688b0 con 0x7f445c0b9660 2024-03-17T23:48:54.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.514+0000 7f4458ff9640 1 -- 172.21.15.150:0/4245326247 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f444c01a070 con 0x7f445c0b9660 2024-03-17T23:48:54.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.514+0000 7f4458ff9640 1 --2- 172.21.15.150:0/4245326247 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f444806bbd0 0x7f444806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:54.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.514+0000 7f4458ff9640 1 -- 172.21.15.150:0/4245326247 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7f444c08af90 con 0x7f445c0b9660 2024-03-17T23:48:54.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.514+0000 7f445b7fe640 1 --2- 172.21.15.150:0/4245326247 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f444806bbd0 0x7f444806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:54.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.515+0000 7f445b7fe640 1 --2- 172.21.15.150:0/4245326247 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f444806bbd0 0x7f444806e090 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f445c0cc2a0 tx=0x7f4450009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:54.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.519+0000 7f4458ff9640 1 -- 172.21.15.150:0/4245326247 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f444c059100 con 0x7f445c0b9660 2024-03-17T23:48:54.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.777+0000 7f4462459640 1 -- 172.21.15.150:0/4245326247 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) -- 0x7f445c0b90a0 con 0x7f445c0b9660 2024-03-17T23:48:54.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.779+0000 7f4458ff9640 1 -- 172.21.15.150:0/4245326247 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v16) ==== 184+0+141 (secure 0 0 0) 0x7f444c05d1a0 con 0x7f445c0b9660 2024-03-17T23:48:54.780 INFO:teuthology.orchestra.run.smithi150.stdout:[client.0] 2024-03-17T23:48:54.780 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQC3gPdlvlPJNRAAkUfetPJK4hZQG5rNiSDozw== 2024-03-17T23:48:54.780 INFO:teuthology.orchestra.run.smithi150.stdout: caps mds = "allow" 2024-03-17T23:48:54.780 INFO:teuthology.orchestra.run.smithi150.stdout: caps mon = "allow r" 2024-03-17T23:48:54.780 INFO:teuthology.orchestra.run.smithi150.stdout: caps osd = "allow rw pool=cephfs_data" 2024-03-17T23:48:54.780 INFO:teuthology.orchestra.run.smithi150.stderr:updated caps for client.0 2024-03-17T23:48:54.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.781+0000 7f4462459640 1 -- 172.21.15.150:0/4245326247 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f444806bbd0 msgr2=0x7f444806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:54.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.781+0000 7f4462459640 1 --2- 172.21.15.150:0/4245326247 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f444806bbd0 0x7f444806e090 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f445c0cc2a0 tx=0x7f4450009190 comp rx=0 tx=0).stop 2024-03-17T23:48:54.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.781+0000 7f4462459640 1 -- 172.21.15.150:0/4245326247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f445c0b9660 msgr2=0x7f445c154250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:54.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.781+0000 7f4462459640 1 --2- 172.21.15.150:0/4245326247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f445c0b9660 0x7f445c154250 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f444c00e870 tx=0x7f444c00ed30 comp rx=0 tx=0).stop 2024-03-17T23:48:54.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.781+0000 7f4462459640 1 -- 172.21.15.150:0/4245326247 shutdown_connections 2024-03-17T23:48:54.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.781+0000 7f4462459640 1 --2- 172.21.15.150:0/4245326247 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f445c1615d0 0x7f445c155530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:54.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.781+0000 7f4462459640 1 --2- 172.21.15.150:0/4245326247 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f444806bbd0 0x7f444806e090 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:54.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.781+0000 7f4462459640 1 --2- 172.21.15.150:0/4245326247 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f445c0d3910 0x7f445c0cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:54.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.781+0000 7f4462459640 1 --2- 172.21.15.150:0/4245326247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f445c0b9660 0x7f445c154250 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:54.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.781+0000 7f4462459640 1 -- 172.21.15.150:0/4245326247 >> 172.21.15.150:0/4245326247 conn(0x7f445c0730b0 msgr2=0x7f445c15ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:54.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.782+0000 7f4462459640 1 -- 172.21.15.150:0/4245326247 shutdown_connections 2024-03-17T23:48:54.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:54.782+0000 7f4462459640 1 -- 172.21.15.150:0/4245326247 wait complete. 2024-03-17T23:48:54.793 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-17T23:48:54.793 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-17T23:48:54.832 DEBUG:teuthology.orchestra.run.smithi157:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-17T23:48:54.887 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-17T23:48:54.887 INFO:tasks.cephfs.mount:Terminating background process 2024-03-17T23:48:54.888 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:48:54.888 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:48:54.903 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19488 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-17T23:48:54.903 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi150.front.sepia.ceph.com 2024-03-17T23:48:54.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:54.903 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-17T23:48:54.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-17T23:48:54.958 INFO:teuthology.orchestra.run.smithi150.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-17T23:48:54.959 INFO:tasks.cephfs.mount:Terminating background process 2024-03-17T23:48:54.959 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:48:54.959 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:48:54.977 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=626 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-17T23:48:54.977 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi157.front.sepia.ceph.com 2024-03-17T23:48:54.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:54.977 DEBUG:teuthology.orchestra.run.smithi157:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-17T23:48:55.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-17T23:48:55.035 INFO:teuthology.orchestra.run.smithi157.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-17T23:48:55.035 INFO:tasks.cephfs.mount:Terminating background process 2024-03-17T23:48:55.035 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:48:55.035 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:48:55.050 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19488 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-17T23:48:55.050 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi150.front.sepia.ceph.com 2024-03-17T23:48:55.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:55.051 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-17T23:48:55.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-17T23:48:55.104 INFO:teuthology.orchestra.run.smithi150.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-17T23:48:55.105 INFO:tasks.cephfs.mount:Terminating background process 2024-03-17T23:48:55.105 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:48:55.105 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:48:55.122 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=626 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-17T23:48:55.122 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi157.front.sepia.ceph.com 2024-03-17T23:48:55.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:48:55.122 DEBUG:teuthology.orchestra.run.smithi157:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-17T23:48:55.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-17T23:48:55.179 INFO:teuthology.orchestra.run.smithi157.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-17T23:48:55.180 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:48:55.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.280+0000 7fcf20e63640 1 Processor -- start 2024-03-17T23:48:55.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.280+0000 7fcf20e63640 1 -- start start 2024-03-17T23:48:55.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.281+0000 7fcf20e63640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcf1c15dc10 0x7fcf1c0d2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:55.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.281+0000 7fcf20e63640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fcf1c0d3860 0x7fcf1c0cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:55.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.281+0000 7fcf20e63640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf1c0d2540 con 0x7fcf1c0b9950 2024-03-17T23:48:55.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.281+0000 7fcf20e63640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fcf1c0d27c0 con 0x7fcf1c0d3860 2024-03-17T23:48:55.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.281+0000 7fcf20e63640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fcf1c0d2aa0 con 0x7fcf1c15dc10 2024-03-17T23:48:55.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.281+0000 7fcf1ad9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fcf1c0b9950 0x7fcf1c06cbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49612/0 (socket says 172.21.15.150:49612) 2024-03-17T23:48:55.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.281+0000 7fcf1ad9d640 1 -- 172.21.15.150:0/882341618 learned_addr learned my addr 172.21.15.150:0/882341618 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:55.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.282+0000 7fcf1959a640 1 -- 172.21.15.150:0/882341618 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 315667624 0 0) 0x7fcf14001490 con 0x7fcf1c0b9950 2024-03-17T23:48:55.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.282+0000 7fcf19d9b640 1 --2- 172.21.15.150:0/882341618 >> v2:172.21.15.157:3300/0 conn(0x7fcf1c0d3860 0x7fcf1c0cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:55.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.282+0000 7fcf1959a640 1 -- 172.21.15.150:0/882341618 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf04003520 con 0x7fcf1c0b9950 2024-03-17T23:48:55.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.282+0000 7fcf1a59c640 1 --2- 172.21.15.150:0/882341618 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcf1c15dc10 0x7fcf1c0d2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:55.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.282+0000 7fcf1959a640 1 -- 172.21.15.150:0/882341618 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2289160323 0 0) 0x7fcf04003520 con 0x7fcf1c0b9950 2024-03-17T23:48:55.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.282+0000 7fcf1959a640 1 -- 172.21.15.150:0/882341618 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcf1c15dc10 msgr2=0x7fcf1c0d2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:55.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.282+0000 7fcf1959a640 1 --2- 172.21.15.150:0/882341618 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcf1c15dc10 0x7fcf1c0d2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:55.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.282+0000 7fcf1959a640 1 -- 172.21.15.150:0/882341618 >> v2:172.21.15.157:3300/0 conn(0x7fcf1c0d3860 msgr2=0x7fcf1c0cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:55.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.282+0000 7fcf1959a640 1 --2- 172.21.15.150:0/882341618 >> v2:172.21.15.157:3300/0 conn(0x7fcf1c0d3860 0x7fcf1c0cea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:55.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.282+0000 7fcf1959a640 1 -- 172.21.15.150:0/882341618 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf1c0d2d80 con 0x7fcf1c0b9950 2024-03-17T23:48:55.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.282+0000 7fcf1959a640 1 -- 172.21.15.150:0/882341618 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7fcf14001bc0 con 0x7fcf1c0b9950 2024-03-17T23:48:55.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.282+0000 7fcf1959a640 1 -- 172.21.15.150:0/882341618 >> v1:172.21.15.150:6789/0 conn(0x7fcf1c0b9950 legacy=0x7fcf1c06cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:55.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.282+0000 7fcf1959a640 1 --2- 172.21.15.150:0/882341618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf040039b0 0x7fcf04003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:55.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.282+0000 7fcf1959a640 1 --2- 172.21.15.150:0/882341618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcf04004800 0x7fcf04004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:55.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.282+0000 7fcf1959a640 1 --2- 172.21.15.150:0/882341618 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcf1c15dc10 0x7fcf040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:55.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.282+0000 7fcf1959a640 1 -- 172.21.15.150:0/882341618 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1c0d27c0 con 0x7fcf040039b0 2024-03-17T23:48:55.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.282+0000 7fcf1959a640 1 -- 172.21.15.150:0/882341618 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1c0d2aa0 con 0x7fcf04004800 2024-03-17T23:48:55.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.282+0000 7fcf1959a640 1 -- 172.21.15.150:0/882341618 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fcf0400b9c0 con 0x7fcf1c15dc10 2024-03-17T23:48:55.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.282+0000 7fcf1a59c640 1 --2- 172.21.15.150:0/882341618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf040039b0 0x7fcf04003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:55.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.282+0000 7fcf19d9b640 1 --2- 172.21.15.150:0/882341618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcf04004800 0x7fcf04004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:55.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.282+0000 7fcf1ad9d640 1 --2- 172.21.15.150:0/882341618 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcf1c15dc10 0x7fcf040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:55.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.282+0000 7fcf1a59c640 1 -- 172.21.15.150:0/882341618 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcf1c15dc10 msgr2=0x7fcf040051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:55.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.282+0000 7fcf1a59c640 1 --2- 172.21.15.150:0/882341618 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcf1c15dc10 0x7fcf040051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:55.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.282+0000 7fcf1a59c640 1 -- 172.21.15.150:0/882341618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcf04004800 msgr2=0x7fcf04004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:55.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.282+0000 7fcf1a59c640 1 --2- 172.21.15.150:0/882341618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcf04004800 0x7fcf04004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:55.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.282+0000 7fcf1a59c640 1 -- 172.21.15.150:0/882341618 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf04003520 con 0x7fcf040039b0 2024-03-17T23:48:55.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.283+0000 7fcf1a59c640 1 --2- 172.21.15.150:0/882341618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf040039b0 0x7fcf04003dd0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fcf10009ac0 tx=0x7fcf04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f301b1c01c766ce6 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:55.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.283+0000 7fcf1959a640 1 -- 172.21.15.150:0/882341618 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf10039070 con 0x7fcf040039b0 2024-03-17T23:48:55.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.283+0000 7fcf1959a640 1 -- 172.21.15.150:0/882341618 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf1002c6b0 con 0x7fcf040039b0 2024-03-17T23:48:55.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.283+0000 7fcf1959a640 1 -- 172.21.15.150:0/882341618 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf10009690 con 0x7fcf040039b0 2024-03-17T23:48:55.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.284+0000 7fcf20e63640 1 -- 172.21.15.150:0/882341618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf040039b0 msgr2=0x7fcf04003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:55.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.284+0000 7fcf20e63640 1 --2- 172.21.15.150:0/882341618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf040039b0 0x7fcf04003dd0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fcf10009ac0 tx=0x7fcf04005830 comp rx=0 tx=0).stop 2024-03-17T23:48:55.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.284+0000 7fcf1ad9d640 1 -- 172.21.15.150:0/882341618 reap_dead start 2024-03-17T23:48:55.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.284+0000 7fcf20e63640 1 -- 172.21.15.150:0/882341618 shutdown_connections 2024-03-17T23:48:55.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.284+0000 7fcf20e63640 1 -- 172.21.15.150:0/882341618 >> 172.21.15.150:0/882341618 conn(0x7fcf1c072ce0 msgr2=0x7fcf1c0730f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:55.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.284+0000 7fcf20e63640 1 -- 172.21.15.150:0/882341618 shutdown_connections 2024-03-17T23:48:55.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.284+0000 7fcf20e63640 1 -- 172.21.15.150:0/882341618 wait complete. 2024-03-17T23:48:55.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.285+0000 7fcf20e63640 1 Processor -- start 2024-03-17T23:48:55.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.285+0000 7fcf20e63640 1 -- start start 2024-03-17T23:48:55.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.286+0000 7fcf20e63640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcf040039b0 0x7fcf1c152d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:55.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.286+0000 7fcf20e63640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcf1c157f90 0x7fcf1c159c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:55.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.286+0000 7fcf20e63640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf1c15a160 0x7fcf1c15f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:55.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.286+0000 7fcf20e63640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1c15dff0 con 0x7fcf1c15a160 2024-03-17T23:48:55.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.286+0000 7fcf20e63640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1c0d3340 con 0x7fcf1c157f90 2024-03-17T23:48:55.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.286+0000 7fcf20e63640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fcf1c15e170 con 0x7fcf040039b0 2024-03-17T23:48:55.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.286+0000 7fcf1ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf1c15a160 0x7fcf1c15f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:55.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.286+0000 7fcf1ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf1c15a160 0x7fcf1c15f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36668/0 (socket says 172.21.15.150:36668) 2024-03-17T23:48:55.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.286+0000 7fcf1ad9d640 1 -- 172.21.15.150:0/2097022982 learned_addr learned my addr 172.21.15.150:0/2097022982 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:55.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.286+0000 7fcf1a59c640 1 --2- 172.21.15.150:0/2097022982 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcf040039b0 0x7fcf1c152d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:55.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.286+0000 7fcf19d9b640 1 --2- 172.21.15.150:0/2097022982 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcf1c157f90 0x7fcf1c159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:55.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.286+0000 7fcf1ad9d640 1 -- 172.21.15.150:0/2097022982 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcf040039b0 msgr2=0x7fcf1c152d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:55.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.286+0000 7fcf1ad9d640 1 --2- 172.21.15.150:0/2097022982 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcf040039b0 0x7fcf1c152d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:55.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.286+0000 7fcf1ad9d640 1 -- 172.21.15.150:0/2097022982 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcf1c157f90 msgr2=0x7fcf1c159c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:55.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.286+0000 7fcf1ad9d640 1 --2- 172.21.15.150:0/2097022982 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcf1c157f90 0x7fcf1c159c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:55.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.286+0000 7fcf1ad9d640 1 -- 172.21.15.150:0/2097022982 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf1c1587c0 con 0x7fcf1c15a160 2024-03-17T23:48:55.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.286+0000 7fcf1a59c640 1 --2- 172.21.15.150:0/2097022982 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcf040039b0 0x7fcf1c152d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:55.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.287+0000 7fcf1ad9d640 1 --2- 172.21.15.150:0/2097022982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf1c15a160 0x7fcf1c15f8a0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0d3ca0 tx=0x7fcf14004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:55.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.287+0000 7fcf0affd640 1 -- 172.21.15.150:0/2097022982 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf14007900 con 0x7fcf1c15a160 2024-03-17T23:48:55.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.287+0000 7fcf0affd640 1 -- 172.21.15.150:0/2097022982 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf14018070 con 0x7fcf1c15a160 2024-03-17T23:48:55.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.287+0000 7fcf20e63640 1 -- 172.21.15.150:0/2097022982 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf1c1584e0 con 0x7fcf1c15a160 2024-03-17T23:48:55.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.287+0000 7fcf0affd640 1 -- 172.21.15.150:0/2097022982 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf14013630 con 0x7fcf1c15a160 2024-03-17T23:48:55.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.287+0000 7fcf20e63640 1 -- 172.21.15.150:0/2097022982 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf1c0cd9e0 con 0x7fcf1c15a160 2024-03-17T23:48:55.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.288+0000 7fcf20e63640 1 -- 172.21.15.150:0/2097022982 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf1c0cdd10 con 0x7fcf1c15a160 2024-03-17T23:48:55.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.288+0000 7fcf0affd640 1 -- 172.21.15.150:0/2097022982 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fcf140137d0 con 0x7fcf1c15a160 2024-03-17T23:48:55.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.288+0000 7fcf0affd640 1 --2- 172.21.15.150:0/2097022982 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcefc06bbd0 0x7fcefc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:55.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.289+0000 7fcf0affd640 1 -- 172.21.15.150:0/2097022982 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7fcf1408b540 con 0x7fcf1c15a160 2024-03-17T23:48:55.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.289+0000 7fcf1a59c640 1 --2- 172.21.15.150:0/2097022982 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcefc06bbd0 0x7fcefc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:55.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.289+0000 7fcf1a59c640 1 --2- 172.21.15.150:0/2097022982 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcefc06bbd0 0x7fcefc06e090 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fcf10036000 tx=0x7fcf10030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:55.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.293+0000 7fcf0affd640 1 -- 172.21.15.150:0/2097022982 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fcf1c1587c0 con 0x7fcf1c15a160 2024-03-17T23:48:55.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.550+0000 7fcf20e63640 1 -- 172.21.15.150:0/2097022982 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf1c0681a0 con 0x7fcf1c15a160 2024-03-17T23:48:55.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.551+0000 7fcf0affd640 1 -- 172.21.15.150:0/2097022982 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5195 (secure 0 0 0) 0x7fcf1c0d2540 con 0x7fcf1c15a160 2024-03-17T23:48:55.552 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:55.552 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5177,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6834/3126446677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":3126446677},{"type":"v1","addr":"172.21.15.157:6834","nonce":3126446677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1992779366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1992779366},{"type":"v1","addr":"172.21.15.150:6836","nonce":1992779366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5487,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/3615467865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6833","nonce":3615467865},{"type":"v1","addr":"172.21.15.157:6835","nonce":3615467865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-17T23:48:10.730994+0000","modified":"2024-03-17T23:48:23.827674+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5515":{"gid":5515,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6837/1413722826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1413722826},{"type":"v1","addr":"172.21.15.150:6837","nonce":1413722826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5515,"qdb_cluster":[5515]},"id":2}]} 2024-03-17T23:48:55.552 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 16 2024-03-17T23:48:55.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.553+0000 7fcf20e63640 1 -- 172.21.15.150:0/2097022982 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcefc06bbd0 msgr2=0x7fcefc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:55.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.553+0000 7fcf20e63640 1 --2- 172.21.15.150:0/2097022982 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcefc06bbd0 0x7fcefc06e090 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fcf10036000 tx=0x7fcf10030000 comp rx=0 tx=0).stop 2024-03-17T23:48:55.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.553+0000 7fcf20e63640 1 -- 172.21.15.150:0/2097022982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf1c15a160 msgr2=0x7fcf1c15f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:55.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.553+0000 7fcf20e63640 1 --2- 172.21.15.150:0/2097022982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf1c15a160 0x7fcf1c15f8a0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0d3ca0 tx=0x7fcf14004f80 comp rx=0 tx=0).stop 2024-03-17T23:48:55.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.553+0000 7fcf20e63640 1 -- 172.21.15.150:0/2097022982 shutdown_connections 2024-03-17T23:48:55.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.553+0000 7fcf20e63640 1 --2- 172.21.15.150:0/2097022982 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf1c15a160 0x7fcf1c15f8a0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:55.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.553+0000 7fcf20e63640 1 --2- 172.21.15.150:0/2097022982 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcefc06bbd0 0x7fcefc06e090 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:55.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.553+0000 7fcf20e63640 1 --2- 172.21.15.150:0/2097022982 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcf1c157f90 0x7fcf1c159c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:55.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.553+0000 7fcf20e63640 1 --2- 172.21.15.150:0/2097022982 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcf040039b0 0x7fcf1c152d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:55.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.553+0000 7fcf20e63640 1 -- 172.21.15.150:0/2097022982 >> 172.21.15.150:0/2097022982 conn(0x7fcf1c072ce0 msgr2=0x7fcf1c0b4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:55.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.553+0000 7fcf20e63640 1 -- 172.21.15.150:0/2097022982 shutdown_connections 2024-03-17T23:48:55.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.553+0000 7fcf20e63640 1 -- 172.21.15.150:0/2097022982 wait complete. 2024-03-17T23:48:55.565 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:48:55.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.666+0000 7f53a44dc640 1 Processor -- start 2024-03-17T23:48:55.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.666+0000 7f53a44dc640 1 -- start start 2024-03-17T23:48:55.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.667+0000 7f53a44dc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f539c15de30 0x7f539c1619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:55.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.667+0000 7f53a44dc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f539c0b9860 0x7f539c0cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:55.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.667+0000 7f53a44dc640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f539c0b9cc0 con 0x7f539c0cef50 2024-03-17T23:48:55.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.667+0000 7f53a44dc640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f539c0cf3c0 con 0x7f539c15de30 2024-03-17T23:48:55.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.667+0000 7f53a44dc640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f539c162940 con 0x7f539c0b9860 2024-03-17T23:48:55.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.667+0000 7f53a2278640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f539c15de30 0x7f539c1619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:55.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.667+0000 7f53a2278640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f539c15de30 0x7f539c1619a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42032/0 (socket says 172.21.15.150:42032) 2024-03-17T23:48:55.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.667+0000 7f53a2278640 1 -- 172.21.15.150:0/100229744 learned_addr learned my addr 172.21.15.150:0/100229744 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:55.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.667+0000 7f53a1a77640 1 --2- 172.21.15.150:0/100229744 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f539c0b9860 0x7f539c0cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:55.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.667+0000 7f53a2278640 1 -- 172.21.15.150:0/100229744 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f539c0b9860 msgr2=0x7f539c0cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:55.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.667+0000 7f53a2278640 1 --2- 172.21.15.150:0/100229744 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f539c0b9860 0x7f539c0cea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:55.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.667+0000 7f53a2278640 1 -- 172.21.15.150:0/100229744 >> v1:172.21.15.157:6789/0 conn(0x7f539c0cef50 legacy=0x7f539c06c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:55.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.667+0000 7f53a2278640 1 -- 172.21.15.150:0/100229744 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f539c1620a0 con 0x7f539c15de30 2024-03-17T23:48:55.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.667+0000 7f53a1a77640 1 --2- 172.21.15.150:0/100229744 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f539c0b9860 0x7f539c0cea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:55.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.668+0000 7f53a2278640 1 --2- 172.21.15.150:0/100229744 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f539c15de30 0x7f539c1619a0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5398009870 tx=0x7f539802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e381fee1491be19 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:55.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.668+0000 7f53a1276640 1 -- 172.21.15.150:0/100229744 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5398035020 con 0x7f539c15de30 2024-03-17T23:48:55.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.668+0000 7f53a1276640 1 -- 172.21.15.150:0/100229744 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5398031640 con 0x7f539c15de30 2024-03-17T23:48:55.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.668+0000 7f53a1276640 1 -- 172.21.15.150:0/100229744 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539802d040 con 0x7f539c15de30 2024-03-17T23:48:55.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.669+0000 7f53a44dc640 1 -- 172.21.15.150:0/100229744 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f539c15de30 msgr2=0x7f539c1619a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:55.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.669+0000 7f53a44dc640 1 --2- 172.21.15.150:0/100229744 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f539c15de30 0x7f539c1619a0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5398009870 tx=0x7f539802b8f0 comp rx=0 tx=0).stop 2024-03-17T23:48:55.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.669+0000 7f53a44dc640 1 -- 172.21.15.150:0/100229744 shutdown_connections 2024-03-17T23:48:55.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.669+0000 7f53a44dc640 1 --2- 172.21.15.150:0/100229744 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f539c0b9860 0x7f539c0cea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:55.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.669+0000 7f53a44dc640 1 --2- 172.21.15.150:0/100229744 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f539c15de30 0x7f539c1619a0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:55.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.669+0000 7f53a44dc640 1 -- 172.21.15.150:0/100229744 >> 172.21.15.150:0/100229744 conn(0x7f539c0730b0 msgr2=0x7f539c0d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:55.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.669+0000 7f53a44dc640 1 -- 172.21.15.150:0/100229744 shutdown_connections 2024-03-17T23:48:55.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.669+0000 7f53a44dc640 1 -- 172.21.15.150:0/100229744 wait complete. 2024-03-17T23:48:55.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.671+0000 7f53a44dc640 1 Processor -- start 2024-03-17T23:48:55.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.671+0000 7f53a44dc640 1 -- start start 2024-03-17T23:48:55.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.672+0000 7f53a44dc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f539c0b9860 0x7f539c15f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:55.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.672+0000 7f53a44dc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f539c0cef50 0x7f539c14c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:55.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.672+0000 7f53a44dc640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f539c15de30 0x7f539c0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:55.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.672+0000 7f53a44dc640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f539c0d2170 con 0x7f539c0b9860 2024-03-17T23:48:55.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.672+0000 7f53a44dc640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f539c162660 con 0x7f539c15de30 2024-03-17T23:48:55.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.672+0000 7f53a44dc640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f539c15e210 con 0x7f539c0cef50 2024-03-17T23:48:55.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.672+0000 7f53a2278640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f539c0b9860 0x7f539c15f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:55.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.672+0000 7f53a2278640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f539c0b9860 0x7f539c15f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42042/0 (socket says 172.21.15.150:42042) 2024-03-17T23:48:55.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.672+0000 7f53a2278640 1 -- 172.21.15.150:0/3167567942 learned_addr learned my addr 172.21.15.150:0/3167567942 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:55.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.673+0000 7f53a1a77640 1 --2- 172.21.15.150:0/3167567942 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f539c0cef50 0x7f539c14c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:55.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.673+0000 7f53a2278640 1 -- 172.21.15.150:0/3167567942 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f539c0cef50 msgr2=0x7f539c14c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:55.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.673+0000 7f53a2278640 1 --2- 172.21.15.150:0/3167567942 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f539c0cef50 0x7f539c14c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:55.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.673+0000 7f53a2278640 1 -- 172.21.15.150:0/3167567942 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f539c15de30 msgr2=0x7f539c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:55.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.673+0000 7f53a2a79640 1 --2- 172.21.15.150:0/3167567942 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f539c15de30 0x7f539c0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:55.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.673+0000 7f53a2278640 1 --2- 172.21.15.150:0/3167567942 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f539c15de30 0x7f539c0ccfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:55.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.673+0000 7f53a2278640 1 -- 172.21.15.150:0/3167567942 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f539c15fdc0 con 0x7f539c0b9860 2024-03-17T23:48:55.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.673+0000 7f53a2278640 1 --2- 172.21.15.150:0/3167567942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f539c0b9860 0x7f539c15f880 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f5398000c00 tx=0x7f53980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:55.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.673+0000 7f5392ffd640 1 -- 172.21.15.150:0/3167567942 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5398040070 con 0x7f539c0b9860 2024-03-17T23:48:55.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.673+0000 7f5392ffd640 1 -- 172.21.15.150:0/3167567942 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53980399f0 con 0x7f539c0b9860 2024-03-17T23:48:55.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.673+0000 7f53a44dc640 1 -- 172.21.15.150:0/3167567942 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f539c15e7e0 con 0x7f539c0b9860 2024-03-17T23:48:55.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.673+0000 7f5392ffd640 1 -- 172.21.15.150:0/3167567942 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5398035040 con 0x7f539c0b9860 2024-03-17T23:48:55.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.673+0000 7f53a44dc640 1 -- 172.21.15.150:0/3167567942 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f539c15f080 con 0x7f539c0b9860 2024-03-17T23:48:55.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.674+0000 7f53a44dc640 1 -- 172.21.15.150:0/3167567942 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f539c15ebd0 con 0x7f539c0b9860 2024-03-17T23:48:55.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.675+0000 7f5392ffd640 1 -- 172.21.15.150:0/3167567942 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f539804b060 con 0x7f539c0b9860 2024-03-17T23:48:55.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.675+0000 7f5392ffd640 1 --2- 172.21.15.150:0/3167567942 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f538006bbd0 0x7f538006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:55.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.675+0000 7f5392ffd640 1 -- 172.21.15.150:0/3167567942 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7f53980ac7a0 con 0x7f539c0b9860 2024-03-17T23:48:55.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.675+0000 7f53a1a77640 1 --2- 172.21.15.150:0/3167567942 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f538006bbd0 0x7f538006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:55.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.675+0000 7f53a1a77640 1 --2- 172.21.15.150:0/3167567942 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f538006bbd0 0x7f538006e090 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f539c0b2b20 tx=0x7f538c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:55.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.680+0000 7f5392ffd640 1 -- 172.21.15.150:0/3167567942 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f539807aed0 con 0x7f539c0b9860 2024-03-17T23:48:55.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.944+0000 7f53a44dc640 1 -- 172.21.15.150:0/3167567942 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f539c0d1910 con 0x7f539c0b9860 2024-03-17T23:48:55.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.945+0000 7f5392ffd640 1 -- 172.21.15.150:0/3167567942 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+18839 (secure 0 0 0) 0x7f539807a7d0 con 0x7f539c0b9860 2024-03-17T23:48:55.947 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:55.947 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":45,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:48:37.677776+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:48:08.957879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-17T23:48:09.698808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-17T23:48:22.562269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.3d","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:48:55.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.948+0000 7f53a44dc640 1 -- 172.21.15.150:0/3167567942 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f538006bbd0 msgr2=0x7f538006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:55.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.948+0000 7f53a44dc640 1 --2- 172.21.15.150:0/3167567942 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f538006bbd0 0x7f538006e090 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f539c0b2b20 tx=0x7f538c008040 comp rx=0 tx=0).stop 2024-03-17T23:48:55.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.948+0000 7f53a44dc640 1 -- 172.21.15.150:0/3167567942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f539c0b9860 msgr2=0x7f539c15f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:55.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.948+0000 7f53a44dc640 1 --2- 172.21.15.150:0/3167567942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f539c0b9860 0x7f539c15f880 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f5398000c00 tx=0x7f53980046c0 comp rx=0 tx=0).stop 2024-03-17T23:48:55.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.948+0000 7f53a44dc640 1 -- 172.21.15.150:0/3167567942 shutdown_connections 2024-03-17T23:48:55.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.948+0000 7f53a44dc640 1 --2- 172.21.15.150:0/3167567942 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f538006bbd0 0x7f538006e090 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:55.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.948+0000 7f53a44dc640 1 --2- 172.21.15.150:0/3167567942 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f539c15de30 0x7f539c0ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:55.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.948+0000 7f53a44dc640 1 --2- 172.21.15.150:0/3167567942 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f539c0cef50 0x7f539c14c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:55.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.948+0000 7f53a44dc640 1 --2- 172.21.15.150:0/3167567942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f539c0b9860 0x7f539c15f880 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:55.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.948+0000 7f53a44dc640 1 -- 172.21.15.150:0/3167567942 >> 172.21.15.150:0/3167567942 conn(0x7f539c0730b0 msgr2=0x7f539c0b6e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:55.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.948+0000 7f53a44dc640 1 -- 172.21.15.150:0/3167567942 shutdown_connections 2024-03-17T23:48:55.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:55.948+0000 7f53a44dc640 1 -- 172.21.15.150:0/3167567942 wait complete. 2024-03-17T23:48:55.960 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:48:08.957879+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:48:09.698808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2024-03-17T23:48:22.562269+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-17T23:48:55.960 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:48:56.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.063+0000 7f5db3ca1640 1 Processor -- start 2024-03-17T23:48:56.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.063+0000 7f5db3ca1640 1 -- start start 2024-03-17T23:48:56.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.064+0000 7f5db3ca1640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f5dac161520 0x7f5dac0b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:56.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.064+0000 7f5db3ca1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5dac0b2bf0 0x7f5dac0d05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:56.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.064+0000 7f5db3ca1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5dac161d30 con 0x7f5dac0d0b10 2024-03-17T23:48:56.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.064+0000 7f5db3ca1640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f5dac0b9bf0 con 0x7f5dac161520 2024-03-17T23:48:56.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.064+0000 7f5db3ca1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5dac162940 con 0x7f5dac0b2bf0 2024-03-17T23:48:56.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.064+0000 7f5db223e640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f5dac0d0b10 0x7f5dac0b57a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:47980/0 (socket says 172.21.15.150:47980) 2024-03-17T23:48:56.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.064+0000 7f5db223e640 1 -- 172.21.15.150:0/64481168 learned_addr learned my addr 172.21.15.150:0/64481168 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:56.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.065+0000 7f5db0a3b640 1 -- 172.21.15.150:0/64481168 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2526373068 0 0) 0x7f5dac161d30 con 0x7f5dac0d0b10 2024-03-17T23:48:56.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.065+0000 7f5db0a3b640 1 -- 172.21.15.150:0/64481168 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d98003520 con 0x7f5dac0d0b10 2024-03-17T23:48:56.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.065+0000 7f5db123c640 1 --2- 172.21.15.150:0/64481168 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5dac0b2bf0 0x7f5dac0d05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:56.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.065+0000 7f5db1a3d640 1 --2- 172.21.15.150:0/64481168 >> v2:172.21.15.157:3300/0 conn(0x7f5dac161520 0x7f5dac0b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:56.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.065+0000 7f5db0a3b640 1 -- 172.21.15.150:0/64481168 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1582875611 0 0) 0x7f5d98003520 con 0x7f5dac0d0b10 2024-03-17T23:48:56.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.065+0000 7f5db0a3b640 1 -- 172.21.15.150:0/64481168 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5dac0b2bf0 msgr2=0x7f5dac0d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:56.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.065+0000 7f5db0a3b640 1 --2- 172.21.15.150:0/64481168 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5dac0b2bf0 0x7f5dac0d05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.065+0000 7f5db0a3b640 1 -- 172.21.15.150:0/64481168 >> v2:172.21.15.157:3300/0 conn(0x7f5dac161520 msgr2=0x7f5dac0b96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:56.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.065+0000 7f5db0a3b640 1 --2- 172.21.15.150:0/64481168 >> v2:172.21.15.157:3300/0 conn(0x7f5dac161520 0x7f5dac0b96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.065+0000 7f5db0a3b640 1 -- 172.21.15.150:0/64481168 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dac162010 con 0x7f5dac0d0b10 2024-03-17T23:48:56.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.065+0000 7f5db0a3b640 1 -- 172.21.15.150:0/64481168 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f5da4001670 con 0x7f5dac0d0b10 2024-03-17T23:48:56.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.065+0000 7f5db0a3b640 1 -- 172.21.15.150:0/64481168 >> v1:172.21.15.150:6789/0 conn(0x7f5dac0d0b10 legacy=0x7f5dac0b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:56.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.065+0000 7f5db0a3b640 1 --2- 172.21.15.150:0/64481168 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d980039b0 0x7f5d98003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:56.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.065+0000 7f5db0a3b640 1 --2- 172.21.15.150:0/64481168 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d98004800 0x7f5d98004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:56.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.065+0000 7f5db0a3b640 1 --2- 172.21.15.150:0/64481168 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5dac0b2bf0 0x7f5d980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:56.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.065+0000 7f5db0a3b640 1 -- 172.21.15.150:0/64481168 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5dac0b9bf0 con 0x7f5d980039b0 2024-03-17T23:48:56.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.065+0000 7f5db0a3b640 1 -- 172.21.15.150:0/64481168 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5dac162940 con 0x7f5d98004800 2024-03-17T23:48:56.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.065+0000 7f5db0a3b640 1 -- 172.21.15.150:0/64481168 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5d9800ba70 con 0x7f5dac0b2bf0 2024-03-17T23:48:56.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.065+0000 7f5db1a3d640 1 --2- 172.21.15.150:0/64481168 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d980039b0 0x7f5d98003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:56.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.065+0000 7f5db123c640 1 --2- 172.21.15.150:0/64481168 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d98004800 0x7f5d98004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:56.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.065+0000 7f5db223e640 1 --2- 172.21.15.150:0/64481168 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5dac0b2bf0 0x7f5d980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:56.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.065+0000 7f5db1a3d640 1 -- 172.21.15.150:0/64481168 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5dac0b2bf0 msgr2=0x7f5d980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:56.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.065+0000 7f5db1a3d640 1 --2- 172.21.15.150:0/64481168 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5dac0b2bf0 0x7f5d980051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.065+0000 7f5db1a3d640 1 -- 172.21.15.150:0/64481168 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d98004800 msgr2=0x7f5d98004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:56.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.065+0000 7f5db1a3d640 1 --2- 172.21.15.150:0/64481168 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5d98004800 0x7f5d98004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.065+0000 7f5db1a3d640 1 -- 172.21.15.150:0/64481168 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d98003520 con 0x7f5d980039b0 2024-03-17T23:48:56.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.066+0000 7f5db1a3d640 1 --2- 172.21.15.150:0/64481168 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d980039b0 0x7f5d98003dd0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f5da8009fc0 tx=0x7f5d98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a7510c4ad9499a server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:56.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.066+0000 7f5db0a3b640 1 -- 172.21.15.150:0/64481168 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da8039070 con 0x7f5d980039b0 2024-03-17T23:48:56.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.066+0000 7f5db0a3b640 1 -- 172.21.15.150:0/64481168 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5da802cc60 con 0x7f5d980039b0 2024-03-17T23:48:56.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.066+0000 7f5db0a3b640 1 -- 172.21.15.150:0/64481168 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da8009c10 con 0x7f5d980039b0 2024-03-17T23:48:56.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.067+0000 7f5db3ca1640 1 -- 172.21.15.150:0/64481168 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d980039b0 msgr2=0x7f5d98003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:56.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.067+0000 7f5db3ca1640 1 --2- 172.21.15.150:0/64481168 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d980039b0 0x7f5d98003dd0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f5da8009fc0 tx=0x7f5d98005830 comp rx=0 tx=0).stop 2024-03-17T23:48:56.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.067+0000 7f5db223e640 1 -- 172.21.15.150:0/64481168 reap_dead start 2024-03-17T23:48:56.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.067+0000 7f5db3ca1640 1 -- 172.21.15.150:0/64481168 shutdown_connections 2024-03-17T23:48:56.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.067+0000 7f5db3ca1640 1 -- 172.21.15.150:0/64481168 >> 172.21.15.150:0/64481168 conn(0x7f5dac06ca00 msgr2=0x7f5dac06ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:56.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.067+0000 7f5db3ca1640 1 -- 172.21.15.150:0/64481168 shutdown_connections 2024-03-17T23:48:56.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.067+0000 7f5db3ca1640 1 -- 172.21.15.150:0/64481168 wait complete. 2024-03-17T23:48:56.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.069+0000 7f5db3ca1640 1 Processor -- start 2024-03-17T23:48:56.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.069+0000 7f5db3ca1640 1 -- start start 2024-03-17T23:48:56.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.069+0000 7f5db3ca1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5d980039b0 0x7f5dac0ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:56.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.069+0000 7f5db3ca1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dac1f86a0 0x7f5dac0b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:56.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.069+0000 7f5db3ca1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5dac1f7a20 0x7f5dac1f7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:56.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.070+0000 7f5db3ca1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5dac1625d0 con 0x7f5dac1f86a0 2024-03-17T23:48:56.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.070+0000 7f5db3ca1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5dac0d0f60 con 0x7f5dac1f7a20 2024-03-17T23:48:56.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.070+0000 7f5db3ca1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5dac161900 con 0x7f5d980039b0 2024-03-17T23:48:56.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.070+0000 7f5db123c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dac1f86a0 0x7f5dac0b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:56.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.070+0000 7f5db123c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dac1f86a0 0x7f5dac0b4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42070/0 (socket says 172.21.15.150:42070) 2024-03-17T23:48:56.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.070+0000 7f5db123c640 1 -- 172.21.15.150:0/164569806 learned_addr learned my addr 172.21.15.150:0/164569806 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:56.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.070+0000 7f5db1a3d640 1 --2- 172.21.15.150:0/164569806 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5d980039b0 0x7f5dac0ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:56.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.070+0000 7f5db223e640 1 --2- 172.21.15.150:0/164569806 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5dac1f7a20 0x7f5dac1f7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:56.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.070+0000 7f5db123c640 1 -- 172.21.15.150:0/164569806 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5d980039b0 msgr2=0x7f5dac0ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:56.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.070+0000 7f5db123c640 1 --2- 172.21.15.150:0/164569806 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5d980039b0 0x7f5dac0ccf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.070+0000 7f5db123c640 1 -- 172.21.15.150:0/164569806 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5dac1f7a20 msgr2=0x7f5dac1f7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:56.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.070+0000 7f5db123c640 1 --2- 172.21.15.150:0/164569806 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5dac1f7a20 0x7f5dac1f7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.070+0000 7f5db123c640 1 -- 172.21.15.150:0/164569806 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dac0cdad0 con 0x7f5dac1f86a0 2024-03-17T23:48:56.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.070+0000 7f5db223e640 1 --2- 172.21.15.150:0/164569806 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5dac1f7a20 0x7f5dac1f7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:56.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.070+0000 7f5db123c640 1 --2- 172.21.15.150:0/164569806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dac1f86a0 0x7f5dac0b4f20 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c0119d0 tx=0x7f5d9c011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:56.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.070+0000 7f5da27fc640 1 -- 172.21.15.150:0/164569806 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c00fa90 con 0x7f5dac1f86a0 2024-03-17T23:48:56.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.070+0000 7f5da27fc640 1 -- 172.21.15.150:0/164569806 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d9c00fc30 con 0x7f5dac1f86a0 2024-03-17T23:48:56.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.070+0000 7f5da27fc640 1 -- 172.21.15.150:0/164569806 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c014440 con 0x7f5dac1f86a0 2024-03-17T23:48:56.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.070+0000 7f5db3ca1640 1 -- 172.21.15.150:0/164569806 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dac0cddb0 con 0x7f5dac1f86a0 2024-03-17T23:48:56.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.070+0000 7f5db3ca1640 1 -- 172.21.15.150:0/164569806 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dac159730 con 0x7f5dac1f86a0 2024-03-17T23:48:56.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.072+0000 7f5db3ca1640 1 -- 172.21.15.150:0/164569806 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dac0681a0 con 0x7f5dac1f86a0 2024-03-17T23:48:56.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.072+0000 7f5da27fc640 1 -- 172.21.15.150:0/164569806 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5d9c012040 con 0x7f5dac1f86a0 2024-03-17T23:48:56.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.072+0000 7f5da27fc640 1 --2- 172.21.15.150:0/164569806 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5d9006bc00 0x7f5d9006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:56.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.072+0000 7f5db1a3d640 1 --2- 172.21.15.150:0/164569806 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5d9006bc00 0x7f5d9006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:56.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.072+0000 7f5da27fc640 1 -- 172.21.15.150:0/164569806 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7f5d9c08b2b0 con 0x7f5dac1f86a0 2024-03-17T23:48:56.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.073+0000 7f5db1a3d640 1 --2- 172.21.15.150:0/164569806 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5d9006bc00 0x7f5d9006e0c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5da8036000 tx=0x7f5da802e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:56.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.077+0000 7f5da27fc640 1 -- 172.21.15.150:0/164569806 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5d9c019030 con 0x7f5dac1f86a0 2024-03-17T23:48:56.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.267+0000 7f5db3ca1640 1 -- 172.21.15.150:0/164569806 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5dac0ca5c0 con 0x7f5dac1f86a0 2024-03-17T23:48:56.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.267+0000 7f5da27fc640 1 -- 172.21.15.150:0/164569806 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v45) ==== 93+0+10 (secure 0 0 0) 0x7f5d9c059a70 con 0x7f5dac1f86a0 2024-03-17T23:48:56.268 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:48:56.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.269+0000 7f5db3ca1640 1 -- 172.21.15.150:0/164569806 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5d9006bc00 msgr2=0x7f5d9006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:56.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.269+0000 7f5db3ca1640 1 --2- 172.21.15.150:0/164569806 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5d9006bc00 0x7f5d9006e0c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5da8036000 tx=0x7f5da802e000 comp rx=0 tx=0).stop 2024-03-17T23:48:56.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.269+0000 7f5db3ca1640 1 -- 172.21.15.150:0/164569806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dac1f86a0 msgr2=0x7f5dac0b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:56.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.269+0000 7f5db3ca1640 1 --2- 172.21.15.150:0/164569806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dac1f86a0 0x7f5dac0b4f20 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c0119d0 tx=0x7f5d9c011e90 comp rx=0 tx=0).stop 2024-03-17T23:48:56.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.269+0000 7f5db3ca1640 1 -- 172.21.15.150:0/164569806 shutdown_connections 2024-03-17T23:48:56.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.269+0000 7f5db3ca1640 1 --2- 172.21.15.150:0/164569806 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5d9006bc00 0x7f5d9006e0c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.269+0000 7f5db3ca1640 1 --2- 172.21.15.150:0/164569806 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5dac1f7a20 0x7f5dac1f7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.269+0000 7f5db3ca1640 1 --2- 172.21.15.150:0/164569806 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dac1f86a0 0x7f5dac0b4f20 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.269+0000 7f5db3ca1640 1 --2- 172.21.15.150:0/164569806 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5d980039b0 0x7f5dac0ccf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.269+0000 7f5db3ca1640 1 -- 172.21.15.150:0/164569806 >> 172.21.15.150:0/164569806 conn(0x7f5dac06ca00 msgr2=0x7f5dac0b6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:56.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.269+0000 7f5db3ca1640 1 -- 172.21.15.150:0/164569806 shutdown_connections 2024-03-17T23:48:56.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.269+0000 7f5db3ca1640 1 -- 172.21.15.150:0/164569806 wait complete. 2024-03-17T23:48:56.281 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:48:56.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.385+0000 7f7ebfbd3640 1 Processor -- start 2024-03-17T23:48:56.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.385+0000 7f7ebfbd3640 1 -- start start 2024-03-17T23:48:56.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.386+0000 7f7ebfbd3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7eb815ab90 0x7f7eb815bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:56.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.386+0000 7f7ebfbd3640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7eb806ecd0 0x7f7eb806f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:56.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.386+0000 7f7ebfbd3640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7eb80ce790 con 0x7f7eb81514f0 2024-03-17T23:48:56.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.386+0000 7f7ebfbd3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f7eb806fba0 con 0x7f7eb815ab90 2024-03-17T23:48:56.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.386+0000 7f7ebfbd3640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f7eb815b050 con 0x7f7eb806ecd0 2024-03-17T23:48:56.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.386+0000 7f7ebd16e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7eb815ab90 0x7f7eb815bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:56.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.386+0000 7f7ebd16e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7eb815ab90 0x7f7eb815bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42074/0 (socket says 172.21.15.150:42074) 2024-03-17T23:48:56.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.386+0000 7f7ebd16e640 1 -- 172.21.15.150:0/1826880762 learned_addr learned my addr 172.21.15.150:0/1826880762 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:56.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.386+0000 7f7ebe170640 1 --2- 172.21.15.150:0/1826880762 >> v2:172.21.15.157:3300/0 conn(0x7f7eb806ecd0 0x7f7eb806f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:56.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.386+0000 7f7ebd16e640 1 -- 172.21.15.150:0/1826880762 >> v2:172.21.15.157:3300/0 conn(0x7f7eb806ecd0 msgr2=0x7f7eb806f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:56.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.386+0000 7f7ebd16e640 1 --2- 172.21.15.150:0/1826880762 >> v2:172.21.15.157:3300/0 conn(0x7f7eb806ecd0 0x7f7eb806f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.386+0000 7f7ebd16e640 1 -- 172.21.15.150:0/1826880762 >> v1:172.21.15.157:6789/0 conn(0x7f7eb81514f0 legacy=0x7f7eb806e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:56.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.386+0000 7f7ebd16e640 1 -- 172.21.15.150:0/1826880762 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eb815b330 con 0x7f7eb815ab90 2024-03-17T23:48:56.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.386+0000 7f7ebe170640 1 --2- 172.21.15.150:0/1826880762 >> v2:172.21.15.157:3300/0 conn(0x7f7eb806ecd0 0x7f7eb806f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:56.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.387+0000 7f7ebd16e640 1 --2- 172.21.15.150:0/1826880762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb815ab90 0x7f7eb815bd10 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f7eb806bf60 tx=0x7f7eac02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14214d5adc36439b server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:56.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.387+0000 7f7ebc96d640 1 -- 172.21.15.150:0/1826880762 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eac02cb20 con 0x7f7eb815ab90 2024-03-17T23:48:56.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.387+0000 7f7ebc96d640 1 -- 172.21.15.150:0/1826880762 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7eac02ccc0 con 0x7f7eb815ab90 2024-03-17T23:48:56.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.387+0000 7f7ebc96d640 1 -- 172.21.15.150:0/1826880762 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eac034610 con 0x7f7eb815ab90 2024-03-17T23:48:56.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.388+0000 7f7ebfbd3640 1 -- 172.21.15.150:0/1826880762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb815ab90 msgr2=0x7f7eb815bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:56.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.388+0000 7f7ebfbd3640 1 --2- 172.21.15.150:0/1826880762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb815ab90 0x7f7eb815bd10 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f7eb806bf60 tx=0x7f7eac02bf80 comp rx=0 tx=0).stop 2024-03-17T23:48:56.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.388+0000 7f7ebfbd3640 1 -- 172.21.15.150:0/1826880762 shutdown_connections 2024-03-17T23:48:56.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.388+0000 7f7ebfbd3640 1 --2- 172.21.15.150:0/1826880762 >> v2:172.21.15.157:3300/0 conn(0x7f7eb806ecd0 0x7f7eb806f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.388+0000 7f7ebfbd3640 1 --2- 172.21.15.150:0/1826880762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb815ab90 0x7f7eb815bd10 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.388+0000 7f7ebfbd3640 1 -- 172.21.15.150:0/1826880762 >> 172.21.15.150:0/1826880762 conn(0x7f7eb806b190 msgr2=0x7f7eb806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:56.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.388+0000 7f7ebfbd3640 1 -- 172.21.15.150:0/1826880762 shutdown_connections 2024-03-17T23:48:56.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.388+0000 7f7ebfbd3640 1 -- 172.21.15.150:0/1826880762 wait complete. 2024-03-17T23:48:56.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.389+0000 7f7ebfbd3640 1 Processor -- start 2024-03-17T23:48:56.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.389+0000 7f7ebfbd3640 1 -- start start 2024-03-17T23:48:56.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.390+0000 7f7ebfbd3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb806ecd0 0x7f7eb80b4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:56.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.390+0000 7f7ebfbd3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7eb81514f0 0x7f7eb81548b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:56.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.390+0000 7f7ebfbd3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7eb815ab90 0x7f7eb8151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:56.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.390+0000 7f7ebfbd3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7eb80af510 con 0x7f7eb806ecd0 2024-03-17T23:48:56.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.390+0000 7f7ebfbd3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7eb815b8f0 con 0x7f7eb81514f0 2024-03-17T23:48:56.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.390+0000 7f7ebfbd3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7eb81518d0 con 0x7f7eb815ab90 2024-03-17T23:48:56.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.390+0000 7f7ebd96f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb806ecd0 0x7f7eb80b4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:56.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.390+0000 7f7ebd96f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb806ecd0 0x7f7eb80b4be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42086/0 (socket says 172.21.15.150:42086) 2024-03-17T23:48:56.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.390+0000 7f7ebd96f640 1 -- 172.21.15.150:0/1060170933 learned_addr learned my addr 172.21.15.150:0/1060170933 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:56.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.390+0000 7f7ebd16e640 1 --2- 172.21.15.150:0/1060170933 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7eb81514f0 0x7f7eb81548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:56.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.390+0000 7f7ebe170640 1 --2- 172.21.15.150:0/1060170933 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7eb815ab90 0x7f7eb8151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:56.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.390+0000 7f7ebd96f640 1 -- 172.21.15.150:0/1060170933 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7eb815ab90 msgr2=0x7f7eb8151bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:56.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.390+0000 7f7ebd96f640 1 --2- 172.21.15.150:0/1060170933 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7eb815ab90 0x7f7eb8151bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.390+0000 7f7ebd96f640 1 -- 172.21.15.150:0/1060170933 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7eb81514f0 msgr2=0x7f7eb81548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:56.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.390+0000 7f7ebd96f640 1 --2- 172.21.15.150:0/1060170933 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7eb81514f0 0x7f7eb81548b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.390+0000 7f7ebd96f640 1 -- 172.21.15.150:0/1060170933 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eb80b3b40 con 0x7f7eb806ecd0 2024-03-17T23:48:56.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.390+0000 7f7ebd96f640 1 --2- 172.21.15.150:0/1060170933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb806ecd0 0x7f7eb80b4be0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f7eb000c400 tx=0x7f7eb000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:56.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.391+0000 7f7eb67fc640 1 -- 172.21.15.150:0/1060170933 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eb00052d0 con 0x7f7eb806ecd0 2024-03-17T23:48:56.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.391+0000 7f7eb67fc640 1 -- 172.21.15.150:0/1060170933 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7eb0005490 con 0x7f7eb806ecd0 2024-03-17T23:48:56.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.391+0000 7f7eb67fc640 1 -- 172.21.15.150:0/1060170933 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eb0007260 con 0x7f7eb806ecd0 2024-03-17T23:48:56.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.391+0000 7f7ebfbd3640 1 -- 172.21.15.150:0/1060170933 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7eb80b3e20 con 0x7f7eb806ecd0 2024-03-17T23:48:56.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.391+0000 7f7ebfbd3640 1 -- 172.21.15.150:0/1060170933 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7eb80b43e0 con 0x7f7eb806ecd0 2024-03-17T23:48:56.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.392+0000 7f7ebfbd3640 1 -- 172.21.15.150:0/1060170933 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7eb80681a0 con 0x7f7eb806ecd0 2024-03-17T23:48:56.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.392+0000 7f7eb67fc640 1 -- 172.21.15.150:0/1060170933 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7eb0007400 con 0x7f7eb806ecd0 2024-03-17T23:48:56.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.393+0000 7f7eb67fc640 1 --2- 172.21.15.150:0/1060170933 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7ea406bbd0 0x7f7ea406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:56.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.393+0000 7f7eb67fc640 1 -- 172.21.15.150:0/1060170933 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7f7eb008a310 con 0x7f7eb806ecd0 2024-03-17T23:48:56.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.393+0000 7f7ebd16e640 1 --2- 172.21.15.150:0/1060170933 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7ea406bbd0 0x7f7ea406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:56.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.393+0000 7f7ebd16e640 1 --2- 172.21.15.150:0/1060170933 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7ea406bbd0 0x7f7ea406e090 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7eac035000 tx=0x7f7eac030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:56.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.397+0000 7f7eb67fc640 1 -- 172.21.15.150:0/1060170933 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7eb80ce790 con 0x7f7eb806ecd0 2024-03-17T23:48:56.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.577+0000 7f7ebfbd3640 1 -- 172.21.15.150:0/1060170933 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7eb806bf90 con 0x7f7eb806ecd0 2024-03-17T23:48:56.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.578+0000 7f7eb67fc640 1 -- 172.21.15.150:0/1060170933 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v45) ==== 104+0+11 (secure 0 0 0) 0x7f7eb0058ad0 con 0x7f7eb806ecd0 2024-03-17T23:48:56.578 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:48:56.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.579+0000 7f7ebfbd3640 1 -- 172.21.15.150:0/1060170933 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7ea406bbd0 msgr2=0x7f7ea406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:56.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.579+0000 7f7ebfbd3640 1 --2- 172.21.15.150:0/1060170933 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7ea406bbd0 0x7f7ea406e090 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7eac035000 tx=0x7f7eac030000 comp rx=0 tx=0).stop 2024-03-17T23:48:56.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.579+0000 7f7ebfbd3640 1 -- 172.21.15.150:0/1060170933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb806ecd0 msgr2=0x7f7eb80b4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:56.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.579+0000 7f7ebfbd3640 1 --2- 172.21.15.150:0/1060170933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb806ecd0 0x7f7eb80b4be0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f7eb000c400 tx=0x7f7eb000c8c0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.580+0000 7f7ebfbd3640 1 -- 172.21.15.150:0/1060170933 shutdown_connections 2024-03-17T23:48:56.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.580+0000 7f7ebfbd3640 1 --2- 172.21.15.150:0/1060170933 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7eb815ab90 0x7f7eb8151bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.580+0000 7f7ebfbd3640 1 --2- 172.21.15.150:0/1060170933 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7ea406bbd0 0x7f7ea406e090 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.580+0000 7f7ebfbd3640 1 --2- 172.21.15.150:0/1060170933 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7eb81514f0 0x7f7eb81548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.580+0000 7f7ebfbd3640 1 --2- 172.21.15.150:0/1060170933 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7eb806ecd0 0x7f7eb80b4be0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.580+0000 7f7ebfbd3640 1 -- 172.21.15.150:0/1060170933 >> 172.21.15.150:0/1060170933 conn(0x7f7eb806b190 msgr2=0x7f7eb806e280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:56.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.580+0000 7f7ebfbd3640 1 -- 172.21.15.150:0/1060170933 shutdown_connections 2024-03-17T23:48:56.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.580+0000 7f7ebfbd3640 1 -- 172.21.15.150:0/1060170933 wait complete. 2024-03-17T23:48:56.591 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:48:56.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.694+0000 7f119ebb9640 1 Processor -- start 2024-03-17T23:48:56.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.695+0000 7f119ebb9640 1 -- start start 2024-03-17T23:48:56.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.695+0000 7f119ebb9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1198161520 0x7f11980b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:56.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.695+0000 7f119ebb9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1198161d30 con 0x7f11980b2bf0 2024-03-17T23:48:56.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.695+0000 7f119ebb9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1198162010 con 0x7f11980d0ce0 2024-03-17T23:48:56.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.695+0000 7f119ebb9640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f11980b9bf0 con 0x7f1198161520 2024-03-17T23:48:56.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.695+0000 7f1197fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f11980b2bf0 0x7f11980d05d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:47996/0 (socket says 172.21.15.150:47996) 2024-03-17T23:48:56.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.695+0000 7f1197fff640 1 -- 172.21.15.150:0/380614889 learned_addr learned my addr 172.21.15.150:0/380614889 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:56.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.696+0000 7f119c955640 1 --2- 172.21.15.150:0/380614889 >> v2:172.21.15.157:3300/0 conn(0x7f1198161520 0x7f11980b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:56.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.696+0000 7f11977fe640 1 -- 172.21.15.150:0/380614889 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3444936423 0 0) 0x7f1198161d30 con 0x7f11980b2bf0 2024-03-17T23:48:56.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.696+0000 7f11977fe640 1 -- 172.21.15.150:0/380614889 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1188003520 con 0x7f11980b2bf0 2024-03-17T23:48:56.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.696+0000 7f11977fe640 1 -- 172.21.15.150:0/380614889 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1182918560 0 0) 0x7f1188003520 con 0x7f11980b2bf0 2024-03-17T23:48:56.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.696+0000 7f11977fe640 1 -- 172.21.15.150:0/380614889 >> v2:172.21.15.157:3300/0 conn(0x7f1198161520 msgr2=0x7f11980b96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:56.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.696+0000 7f11977fe640 1 --2- 172.21.15.150:0/380614889 >> v2:172.21.15.157:3300/0 conn(0x7f1198161520 0x7f11980b96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.696+0000 7f11977fe640 1 -- 172.21.15.150:0/380614889 >> v1:172.21.15.157:6789/0 conn(0x7f11980d0ce0 legacy=0x7f11980d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:56.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.696+0000 7f11977fe640 1 -- 172.21.15.150:0/380614889 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11981622f0 con 0x7f11980b2bf0 2024-03-17T23:48:56.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.696+0000 7f11977fe640 1 -- 172.21.15.150:0/380614889 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f1184001670 con 0x7f11980b2bf0 2024-03-17T23:48:56.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.696+0000 7f11977fe640 1 -- 172.21.15.150:0/380614889 >> v1:172.21.15.150:6789/0 conn(0x7f11980b2bf0 legacy=0x7f11980d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:56.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.696+0000 7f11977fe640 1 --2- 172.21.15.150:0/380614889 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11880039b0 0x7f1188003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:56.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.696+0000 7f11977fe640 1 --2- 172.21.15.150:0/380614889 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1188004800 0x7f1188004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:56.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.696+0000 7f11977fe640 1 --2- 172.21.15.150:0/380614889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11880052e0 0x7f118800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:56.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.696+0000 7f11977fe640 1 -- 172.21.15.150:0/380614889 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f11980b9bf0 con 0x7f11880052e0 2024-03-17T23:48:56.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.696+0000 7f11977fe640 1 -- 172.21.15.150:0/380614889 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f118800c1b0 con 0x7f11880039b0 2024-03-17T23:48:56.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.696+0000 7f11977fe640 1 -- 172.21.15.150:0/380614889 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f118800c3b0 con 0x7f1188004800 2024-03-17T23:48:56.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.696+0000 7f119d156640 1 --2- 172.21.15.150:0/380614889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11880052e0 0x7f118800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:56.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.696+0000 7f119c955640 1 --2- 172.21.15.150:0/380614889 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11880039b0 0x7f1188003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:56.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.696+0000 7f1197fff640 1 --2- 172.21.15.150:0/380614889 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1188004800 0x7f1188004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:56.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.696+0000 7f119d156640 1 -- 172.21.15.150:0/380614889 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1188004800 msgr2=0x7f1188004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:56.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.697+0000 7f119d156640 1 --2- 172.21.15.150:0/380614889 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1188004800 0x7f1188004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.697+0000 7f119d156640 1 -- 172.21.15.150:0/380614889 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11880039b0 msgr2=0x7f1188003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:56.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.697+0000 7f119d156640 1 --2- 172.21.15.150:0/380614889 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11880039b0 0x7f1188003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.697+0000 7f119d156640 1 -- 172.21.15.150:0/380614889 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1188003520 con 0x7f11880052e0 2024-03-17T23:48:56.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.697+0000 7f119d156640 1 -- 172.21.15.150:0/380614889 reap_dead start 2024-03-17T23:48:56.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.697+0000 7f119d156640 1 --2- 172.21.15.150:0/380614889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11880052e0 0x7f118800b9c0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f1190001e20 tx=0x7f1188005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47842c79cef85793 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:56.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.697+0000 7f11977fe640 1 -- 172.21.15.150:0/380614889 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1190001a70 con 0x7f11880052e0 2024-03-17T23:48:56.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.697+0000 7f11977fe640 1 -- 172.21.15.150:0/380614889 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1190003b70 con 0x7f11880052e0 2024-03-17T23:48:56.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.697+0000 7f11977fe640 1 -- 172.21.15.150:0/380614889 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11900050d0 con 0x7f11880052e0 2024-03-17T23:48:56.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.698+0000 7f119ebb9640 1 -- 172.21.15.150:0/380614889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11880052e0 msgr2=0x7f118800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:56.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.698+0000 7f119ebb9640 1 --2- 172.21.15.150:0/380614889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11880052e0 0x7f118800b9c0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f1190001e20 tx=0x7f1188005830 comp rx=0 tx=0).stop 2024-03-17T23:48:56.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.698+0000 7f119ebb9640 1 -- 172.21.15.150:0/380614889 shutdown_connections 2024-03-17T23:48:56.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.698+0000 7f119ebb9640 1 --2- 172.21.15.150:0/380614889 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11880052e0 0x7f118800b9c0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.698+0000 7f119ebb9640 1 -- 172.21.15.150:0/380614889 >> 172.21.15.150:0/380614889 conn(0x7f119806ca00 msgr2=0x7f119806ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:56.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.698+0000 7f119ebb9640 1 -- 172.21.15.150:0/380614889 shutdown_connections 2024-03-17T23:48:56.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.698+0000 7f119ebb9640 1 -- 172.21.15.150:0/380614889 wait complete. 2024-03-17T23:48:56.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.700+0000 7f119ebb9640 1 Processor -- start 2024-03-17T23:48:56.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.700+0000 7f119ebb9640 1 -- start start 2024-03-17T23:48:56.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.701+0000 7f119ebb9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11880052e0 0x7f119815b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:56.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.701+0000 7f119ebb9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f119815e050 0x7f11980b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:56.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.701+0000 7f119ebb9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11980b55e0 0x7f11980cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:56.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.701+0000 7f119ebb9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1198161900 con 0x7f11880052e0 2024-03-17T23:48:56.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.701+0000 7f119ebb9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1198162940 con 0x7f11980b55e0 2024-03-17T23:48:56.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.701+0000 7f119ebb9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f11980b96b0 con 0x7f119815e050 2024-03-17T23:48:56.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.701+0000 7f119c955640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11880052e0 0x7f119815b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:56.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.701+0000 7f119c955640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11880052e0 0x7f119815b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42104/0 (socket says 172.21.15.150:42104) 2024-03-17T23:48:56.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.701+0000 7f119c955640 1 -- 172.21.15.150:0/2794820856 learned_addr learned my addr 172.21.15.150:0/2794820856 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:56.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.702+0000 7f119c955640 1 -- 172.21.15.150:0/2794820856 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f119815e050 msgr2=0x7f11980b50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:56.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.702+0000 7f119c955640 1 --2- 172.21.15.150:0/2794820856 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f119815e050 0x7f11980b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.702+0000 7f119c955640 1 -- 172.21.15.150:0/2794820856 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11980b55e0 msgr2=0x7f11980cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:56.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.702+0000 7f119c955640 1 --2- 172.21.15.150:0/2794820856 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11980b55e0 0x7f11980cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.702+0000 7f119c955640 1 -- 172.21.15.150:0/2794820856 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f119815d290 con 0x7f11880052e0 2024-03-17T23:48:56.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.702+0000 7f119c955640 1 --2- 172.21.15.150:0/2794820856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11880052e0 0x7f119815b100 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f118c011910 tx=0x7f118c011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:56.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.702+0000 7f11957fa640 1 -- 172.21.15.150:0/2794820856 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f118c00faf0 con 0x7f11880052e0 2024-03-17T23:48:56.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.702+0000 7f11957fa640 1 -- 172.21.15.150:0/2794820856 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f118c00fc90 con 0x7f11880052e0 2024-03-17T23:48:56.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.702+0000 7f119ebb9640 1 -- 172.21.15.150:0/2794820856 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f119815d570 con 0x7f11880052e0 2024-03-17T23:48:56.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.702+0000 7f11957fa640 1 -- 172.21.15.150:0/2794820856 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f118c014440 con 0x7f11880052e0 2024-03-17T23:48:56.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.702+0000 7f119ebb9640 1 -- 172.21.15.150:0/2794820856 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f119815db30 con 0x7f11880052e0 2024-03-17T23:48:56.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.703+0000 7f119ebb9640 1 -- 172.21.15.150:0/2794820856 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11980b5a00 con 0x7f11880052e0 2024-03-17T23:48:56.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.704+0000 7f11957fa640 1 -- 172.21.15.150:0/2794820856 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f118c012040 con 0x7f11880052e0 2024-03-17T23:48:56.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.704+0000 7f11957fa640 1 --2- 172.21.15.150:0/2794820856 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f117c06bc00 0x7f117c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:56.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.704+0000 7f11957fa640 1 -- 172.21.15.150:0/2794820856 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7f118c08b400 con 0x7f11880052e0 2024-03-17T23:48:56.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.704+0000 7f1197fff640 1 --2- 172.21.15.150:0/2794820856 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f117c06bc00 0x7f117c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:56.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.704+0000 7f1197fff640 1 --2- 172.21.15.150:0/2794820856 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f117c06bc00 0x7f117c06e0c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1184001cd0 tx=0x7f1184001b60 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:56.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.713+0000 7f11957fa640 1 -- 172.21.15.150:0/2794820856 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f118c019030 con 0x7f11880052e0 2024-03-17T23:48:56.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.890+0000 7f119ebb9640 1 -- 172.21.15.150:0/2794820856 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f11980681a0 con 0x7f11880052e0 2024-03-17T23:48:56.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.891+0000 7f11957fa640 1 -- 172.21.15.150:0/2794820856 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v45) ==== 100+0+11 (secure 0 0 0) 0x7f118c059bc0 con 0x7f11880052e0 2024-03-17T23:48:56.891 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:48:56.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.893+0000 7f119ebb9640 1 -- 172.21.15.150:0/2794820856 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f117c06bc00 msgr2=0x7f117c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:56.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.893+0000 7f119ebb9640 1 --2- 172.21.15.150:0/2794820856 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f117c06bc00 0x7f117c06e0c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1184001cd0 tx=0x7f1184001b60 comp rx=0 tx=0).stop 2024-03-17T23:48:56.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.893+0000 7f119ebb9640 1 -- 172.21.15.150:0/2794820856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11880052e0 msgr2=0x7f119815b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:56.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.893+0000 7f119ebb9640 1 --2- 172.21.15.150:0/2794820856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11880052e0 0x7f119815b100 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f118c011910 tx=0x7f118c011dd0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.893+0000 7f119ebb9640 1 -- 172.21.15.150:0/2794820856 shutdown_connections 2024-03-17T23:48:56.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.893+0000 7f119ebb9640 1 --2- 172.21.15.150:0/2794820856 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f117c06bc00 0x7f117c06e0c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.893+0000 7f119ebb9640 1 --2- 172.21.15.150:0/2794820856 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11980b55e0 0x7f11980cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.893+0000 7f119ebb9640 1 --2- 172.21.15.150:0/2794820856 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f119815e050 0x7f11980b50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.893+0000 7f119ebb9640 1 --2- 172.21.15.150:0/2794820856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f11880052e0 0x7f119815b100 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:56.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.893+0000 7f119ebb9640 1 -- 172.21.15.150:0/2794820856 >> 172.21.15.150:0/2794820856 conn(0x7f119806ca00 msgr2=0x7f11980b84f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:56.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.893+0000 7f119ebb9640 1 -- 172.21.15.150:0/2794820856 shutdown_connections 2024-03-17T23:48:56.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:56.893+0000 7f119ebb9640 1 -- 172.21.15.150:0/2794820856 wait complete. 2024-03-17T23:48:56.906 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-17T23:48:57.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.014+0000 7fd21bbc4640 1 Processor -- start 2024-03-17T23:48:57.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.014+0000 7fd21bbc4640 1 -- start start 2024-03-17T23:48:57.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.015+0000 7fd21bbc4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd2140cc770 0x7fd2140cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:57.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.015+0000 7fd21bbc4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2140ccc50 con 0x7fd2140d0880 2024-03-17T23:48:57.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.015+0000 7fd21bbc4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2140ccf30 con 0x7fd21415f3d0 2024-03-17T23:48:57.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.015+0000 7fd21bbc4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fd21415f7b0 con 0x7fd2140cc770 2024-03-17T23:48:57.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.015+0000 7fd21915f640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd2140d0880 0x7fd2140c5720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:48004/0 (socket says 172.21.15.150:48004) 2024-03-17T23:48:57.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.015+0000 7fd21915f640 1 -- 172.21.15.150:0/3090977759 learned_addr learned my addr 172.21.15.150:0/3090977759 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:57.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.015+0000 7fd21a161640 1 --2- 172.21.15.150:0/3090977759 >> v2:172.21.15.150:3300/0 conn(0x7fd2140cc770 0x7fd2140cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:57.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.016+0000 7fd21a161640 1 -- 172.21.15.150:0/3090977759 >> v1:172.21.15.157:6789/0 conn(0x7fd21415f3d0 legacy=0x7fd2140d0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:57.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.016+0000 7fd21a161640 1 -- 172.21.15.150:0/3090977759 >> v1:172.21.15.150:6789/0 conn(0x7fd2140d0880 legacy=0x7fd2140c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:57.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.016+0000 7fd21a161640 1 -- 172.21.15.150:0/3090977759 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2140cd210 con 0x7fd2140cc770 2024-03-17T23:48:57.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.016+0000 7fd21a161640 1 --2- 172.21.15.150:0/3090977759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2140cc770 0x7fd2140cdcf0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fd20c009d90 tx=0x7fd20c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f79f32838ae1b4a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:57.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.016+0000 7fd21895e640 1 -- 172.21.15.150:0/3090977759 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd20c039070 con 0x7fd2140cc770 2024-03-17T23:48:57.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.016+0000 7fd21895e640 1 -- 172.21.15.150:0/3090977759 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd20c02cc60 con 0x7fd2140cc770 2024-03-17T23:48:57.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.016+0000 7fd21895e640 1 -- 172.21.15.150:0/3090977759 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd20c034660 con 0x7fd2140cc770 2024-03-17T23:48:57.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.017+0000 7fd21bbc4640 1 -- 172.21.15.150:0/3090977759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2140cc770 msgr2=0x7fd2140cdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:57.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.017+0000 7fd21bbc4640 1 --2- 172.21.15.150:0/3090977759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2140cc770 0x7fd2140cdcf0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fd20c009d90 tx=0x7fd20c02beb0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.017+0000 7fd21bbc4640 1 -- 172.21.15.150:0/3090977759 shutdown_connections 2024-03-17T23:48:57.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.017+0000 7fd21bbc4640 1 --2- 172.21.15.150:0/3090977759 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2140cc770 0x7fd2140cdcf0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.017+0000 7fd21bbc4640 1 -- 172.21.15.150:0/3090977759 >> 172.21.15.150:0/3090977759 conn(0x7fd2140b36c0 msgr2=0x7fd2140b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:57.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.017+0000 7fd21bbc4640 1 -- 172.21.15.150:0/3090977759 shutdown_connections 2024-03-17T23:48:57.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.017+0000 7fd21bbc4640 1 -- 172.21.15.150:0/3090977759 wait complete. 2024-03-17T23:48:57.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.019+0000 7fd21bbc4640 1 Processor -- start 2024-03-17T23:48:57.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.019+0000 7fd21bbc4640 1 -- start start 2024-03-17T23:48:57.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.019+0000 7fd21bbc4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2140cc770 0x7fd21415c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:57.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.019+0000 7fd21bbc4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2140d0880 0x7fd21409ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:57.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.019+0000 7fd21bbc4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd21415f3d0 0x7fd21409f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:57.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.019+0000 7fd21bbc4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd2140cfa30 con 0x7fd2140cc770 2024-03-17T23:48:57.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.019+0000 7fd21bbc4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd2140cd7d0 con 0x7fd2140d0880 2024-03-17T23:48:57.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.019+0000 7fd21bbc4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd2141603c0 con 0x7fd21415f3d0 2024-03-17T23:48:57.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.019+0000 7fd219960640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2140cc770 0x7fd21415c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:57.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.019+0000 7fd219960640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2140cc770 0x7fd21415c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42114/0 (socket says 172.21.15.150:42114) 2024-03-17T23:48:57.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.019+0000 7fd219960640 1 -- 172.21.15.150:0/4098262641 learned_addr learned my addr 172.21.15.150:0/4098262641 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:57.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.020+0000 7fd21915f640 1 --2- 172.21.15.150:0/4098262641 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2140d0880 0x7fd21409ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:57.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.020+0000 7fd21a161640 1 --2- 172.21.15.150:0/4098262641 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd21415f3d0 0x7fd21409f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:57.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.020+0000 7fd219960640 1 -- 172.21.15.150:0/4098262641 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd21415f3d0 msgr2=0x7fd21409f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:57.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.020+0000 7fd219960640 1 --2- 172.21.15.150:0/4098262641 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd21415f3d0 0x7fd21409f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.020+0000 7fd219960640 1 -- 172.21.15.150:0/4098262641 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2140d0880 msgr2=0x7fd21409ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:57.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.020+0000 7fd219960640 1 --2- 172.21.15.150:0/4098262641 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2140d0880 0x7fd21409ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.020+0000 7fd219960640 1 -- 172.21.15.150:0/4098262641 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2140c8ca0 con 0x7fd2140cc770 2024-03-17T23:48:57.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.020+0000 7fd219960640 1 --2- 172.21.15.150:0/4098262641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2140cc770 0x7fd21415c060 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd20800b3f0 tx=0x7fd20800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:57.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.020+0000 7fd2067fc640 1 -- 172.21.15.150:0/4098262641 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd208015020 con 0x7fd2140cc770 2024-03-17T23:48:57.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.020+0000 7fd2067fc640 1 -- 172.21.15.150:0/4098262641 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd208011830 con 0x7fd2140cc770 2024-03-17T23:48:57.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.020+0000 7fd2067fc640 1 -- 172.21.15.150:0/4098262641 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd208011c20 con 0x7fd2140cc770 2024-03-17T23:48:57.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.020+0000 7fd21bbc4640 1 -- 172.21.15.150:0/4098262641 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2140c8fe0 con 0x7fd2140cc770 2024-03-17T23:48:57.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.020+0000 7fd21bbc4640 1 -- 172.21.15.150:0/4098262641 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2140c9820 con 0x7fd2140cc770 2024-03-17T23:48:57.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.021+0000 7fd21bbc4640 1 -- 172.21.15.150:0/4098262641 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2140688b0 con 0x7fd2140cc770 2024-03-17T23:48:57.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.022+0000 7fd2067fc640 1 -- 172.21.15.150:0/4098262641 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd208018020 con 0x7fd2140cc770 2024-03-17T23:48:57.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.022+0000 7fd2067fc640 1 --2- 172.21.15.150:0/4098262641 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd1f806bc00 0x7fd1f806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:57.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.022+0000 7fd21915f640 1 --2- 172.21.15.150:0/4098262641 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd1f806bc00 0x7fd1f806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:57.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.022+0000 7fd2067fc640 1 -- 172.21.15.150:0/4098262641 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7fd20808a9e0 con 0x7fd2140cc770 2024-03-17T23:48:57.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.023+0000 7fd21915f640 1 --2- 172.21.15.150:0/4098262641 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd1f806bc00 0x7fd1f806e0c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd2140c5140 tx=0x7fd210009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:57.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.026+0000 7fd2067fc640 1 -- 172.21.15.150:0/4098262641 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd20801f030 con 0x7fd2140cc770 2024-03-17T23:48:57.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.205+0000 7fd21bbc4640 1 -- 172.21.15.150:0/4098262641 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7fd214074300 con 0x7fd2140cc770 2024-03-17T23:48:57.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.205+0000 7fd2067fc640 1 -- 172.21.15.150:0/4098262641 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v45) ==== 97+0+11 (secure 0 0 0) 0x7fd2140c8fe0 con 0x7fd2140cc770 2024-03-17T23:48:57.206 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:48:57.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.207+0000 7fd21bbc4640 1 -- 172.21.15.150:0/4098262641 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd1f806bc00 msgr2=0x7fd1f806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:57.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.207+0000 7fd21bbc4640 1 --2- 172.21.15.150:0/4098262641 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd1f806bc00 0x7fd1f806e0c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd2140c5140 tx=0x7fd210009040 comp rx=0 tx=0).stop 2024-03-17T23:48:57.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.207+0000 7fd21bbc4640 1 -- 172.21.15.150:0/4098262641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2140cc770 msgr2=0x7fd21415c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:57.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.207+0000 7fd21bbc4640 1 --2- 172.21.15.150:0/4098262641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2140cc770 0x7fd21415c060 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd20800b3f0 tx=0x7fd20800b8b0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.207+0000 7fd21bbc4640 1 -- 172.21.15.150:0/4098262641 shutdown_connections 2024-03-17T23:48:57.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.207+0000 7fd21bbc4640 1 --2- 172.21.15.150:0/4098262641 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd21415f3d0 0x7fd21409f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.207+0000 7fd21bbc4640 1 --2- 172.21.15.150:0/4098262641 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd1f806bc00 0x7fd1f806e0c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.207+0000 7fd21bbc4640 1 --2- 172.21.15.150:0/4098262641 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2140d0880 0x7fd21409ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.207+0000 7fd21bbc4640 1 --2- 172.21.15.150:0/4098262641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2140cc770 0x7fd21415c060 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.207+0000 7fd21bbc4640 1 -- 172.21.15.150:0/4098262641 >> 172.21.15.150:0/4098262641 conn(0x7fd2140b36c0 msgr2=0x7fd2140c5700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:57.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.208+0000 7fd21bbc4640 1 -- 172.21.15.150:0/4098262641 shutdown_connections 2024-03-17T23:48:57.209 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.208+0000 7fd21bbc4640 1 -- 172.21.15.150:0/4098262641 wait complete. 2024-03-17T23:48:57.219 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:48:57.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.320+0000 7fc999fd4640 1 Processor -- start 2024-03-17T23:48:57.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.320+0000 7fc999fd4640 1 -- start start 2024-03-17T23:48:57.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.321+0000 7fc999fd4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc9941616b0 0x7fc994099e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:57.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.321+0000 7fc999fd4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc99409b320 0x7fc99409a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:57.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.321+0000 7fc999fd4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9940d0770 con 0x7fc9940b0ab0 2024-03-17T23:48:57.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.321+0000 7fc999fd4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fc9940cc220 con 0x7fc99409b320 2024-03-17T23:48:57.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.321+0000 7fc999fd4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc9940d0a10 con 0x7fc9941616b0 2024-03-17T23:48:57.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.321+0000 7fc98affd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc99409b320 0x7fc99409a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:57.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.321+0000 7fc993fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fc9940b0ab0 0x7fc99406c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:48010/0 (socket says 172.21.15.150:48010) 2024-03-17T23:48:57.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.321+0000 7fc993fff640 1 -- 172.21.15.150:0/1939744526 learned_addr learned my addr 172.21.15.150:0/1939744526 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:57.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.321+0000 7fc9937fe640 1 --2- 172.21.15.150:0/1939744526 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc9941616b0 0x7fc994099e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:57.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.321+0000 7fc98affd640 1 -- 172.21.15.150:0/1939744526 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc9941616b0 msgr2=0x7fc994099e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:57.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.321+0000 7fc98affd640 1 --2- 172.21.15.150:0/1939744526 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc9941616b0 0x7fc994099e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.321+0000 7fc98affd640 1 -- 172.21.15.150:0/1939744526 >> v1:172.21.15.150:6789/0 conn(0x7fc9940b0ab0 legacy=0x7fc99406c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:57.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.321+0000 7fc98affd640 1 -- 172.21.15.150:0/1939744526 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc99406d520 con 0x7fc99409b320 2024-03-17T23:48:57.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.321+0000 7fc992ffd640 1 -- 172.21.15.150:0/1939744526 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1559999063 0 0) 0x7fc9940d0770 con 0x7fc9940b0ab0 2024-03-17T23:48:57.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.322+0000 7fc98affd640 1 --2- 172.21.15.150:0/1939744526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99409b320 0x7fc99409a360 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fc97c0097a0 tx=0x7fc97c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0b03ffa43b27057 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:57.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.322+0000 7fc992ffd640 1 -- 172.21.15.150:0/1939744526 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97c0315a0 con 0x7fc99409b320 2024-03-17T23:48:57.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.322+0000 7fc992ffd640 1 -- 172.21.15.150:0/1939744526 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc97c037070 con 0x7fc99409b320 2024-03-17T23:48:57.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.322+0000 7fc992ffd640 1 -- 172.21.15.150:0/1939744526 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97c031960 con 0x7fc99409b320 2024-03-17T23:48:57.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.323+0000 7fc999fd4640 1 -- 172.21.15.150:0/1939744526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99409b320 msgr2=0x7fc99409a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:57.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.323+0000 7fc999fd4640 1 --2- 172.21.15.150:0/1939744526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99409b320 0x7fc99409a360 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fc97c0097a0 tx=0x7fc97c02b840 comp rx=0 tx=0).stop 2024-03-17T23:48:57.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.323+0000 7fc999fd4640 1 -- 172.21.15.150:0/1939744526 shutdown_connections 2024-03-17T23:48:57.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.323+0000 7fc999fd4640 1 --2- 172.21.15.150:0/1939744526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99409b320 0x7fc99409a360 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.323+0000 7fc999fd4640 1 --2- 172.21.15.150:0/1939744526 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc9941616b0 0x7fc994099e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.323+0000 7fc999fd4640 1 -- 172.21.15.150:0/1939744526 >> 172.21.15.150:0/1939744526 conn(0x7fc9940b3630 msgr2=0x7fc9940b3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:57.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.323+0000 7fc999fd4640 1 -- 172.21.15.150:0/1939744526 shutdown_connections 2024-03-17T23:48:57.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.323+0000 7fc999fd4640 1 -- 172.21.15.150:0/1939744526 wait complete. 2024-03-17T23:48:57.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.324+0000 7fc999fd4640 1 Processor -- start 2024-03-17T23:48:57.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.324+0000 7fc999fd4640 1 -- start start 2024-03-17T23:48:57.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.325+0000 7fc999fd4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99409b320 0x7fc994151f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:57.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.325+0000 7fc999fd4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc9940b0ab0 0x7fc99415cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:57.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.325+0000 7fc999fd4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9941616b0 0x7fc99415d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:57.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.325+0000 7fc999fd4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc9940d1290 con 0x7fc99409b320 2024-03-17T23:48:57.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.325+0000 7fc999fd4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc9940d0fb0 con 0x7fc9941616b0 2024-03-17T23:48:57.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.325+0000 7fc999fd4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc9940cd6c0 con 0x7fc9940b0ab0 2024-03-17T23:48:57.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.325+0000 7fc9937fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99409b320 0x7fc994151f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:57.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.325+0000 7fc9937fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99409b320 0x7fc994151f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42132/0 (socket says 172.21.15.150:42132) 2024-03-17T23:48:57.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.325+0000 7fc9937fe640 1 -- 172.21.15.150:0/2122205165 learned_addr learned my addr 172.21.15.150:0/2122205165 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:57.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.325+0000 7fc98affd640 1 --2- 172.21.15.150:0/2122205165 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc9940b0ab0 0x7fc99415cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:57.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.325+0000 7fc993fff640 1 --2- 172.21.15.150:0/2122205165 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9941616b0 0x7fc99415d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:57.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.325+0000 7fc9937fe640 1 -- 172.21.15.150:0/2122205165 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc9940b0ab0 msgr2=0x7fc99415cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:57.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.325+0000 7fc9937fe640 1 --2- 172.21.15.150:0/2122205165 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc9940b0ab0 0x7fc99415cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.325+0000 7fc9937fe640 1 -- 172.21.15.150:0/2122205165 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9941616b0 msgr2=0x7fc99415d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:57.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.325+0000 7fc9937fe640 1 --2- 172.21.15.150:0/2122205165 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9941616b0 0x7fc99415d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.325+0000 7fc9937fe640 1 -- 172.21.15.150:0/2122205165 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9940b58c0 con 0x7fc99409b320 2024-03-17T23:48:57.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.325+0000 7fc993fff640 1 --2- 172.21.15.150:0/2122205165 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9941616b0 0x7fc99415d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:57.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.325+0000 7fc9937fe640 1 --2- 172.21.15.150:0/2122205165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99409b320 0x7fc994151f50 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fc98400e870 tx=0x7fc98400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:57.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.326+0000 7fc990ff9640 1 -- 172.21.15.150:0/2122205165 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc984017070 con 0x7fc99409b320 2024-03-17T23:48:57.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.326+0000 7fc990ff9640 1 -- 172.21.15.150:0/2122205165 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc98400c940 con 0x7fc99409b320 2024-03-17T23:48:57.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.326+0000 7fc990ff9640 1 -- 172.21.15.150:0/2122205165 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98400cd30 con 0x7fc99409b320 2024-03-17T23:48:57.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.326+0000 7fc999fd4640 1 -- 172.21.15.150:0/2122205165 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9940b5ba0 con 0x7fc99409b320 2024-03-17T23:48:57.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.326+0000 7fc999fd4640 1 -- 172.21.15.150:0/2122205165 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9940b6440 con 0x7fc99409b320 2024-03-17T23:48:57.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.327+0000 7fc999fd4640 1 -- 172.21.15.150:0/2122205165 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9940b5d70 con 0x7fc99409b320 2024-03-17T23:48:57.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.327+0000 7fc990ff9640 1 -- 172.21.15.150:0/2122205165 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc98401a070 con 0x7fc99409b320 2024-03-17T23:48:57.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.327+0000 7fc990ff9640 1 --2- 172.21.15.150:0/2122205165 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc97406bbd0 0x7fc97406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:57.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.328+0000 7fc990ff9640 1 -- 172.21.15.150:0/2122205165 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7fc98408af80 con 0x7fc99409b320 2024-03-17T23:48:57.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.328+0000 7fc98affd640 1 --2- 172.21.15.150:0/2122205165 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc97406bbd0 0x7fc97406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:57.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.328+0000 7fc98affd640 1 --2- 172.21.15.150:0/2122205165 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc97406bbd0 0x7fc97406e090 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc97c036000 tx=0x7fc97c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:57.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.332+0000 7fc990ff9640 1 -- 172.21.15.150:0/2122205165 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc984059060 con 0x7fc99409b320 2024-03-17T23:48:57.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.611+0000 7fc999fd4640 1 -- 172.21.15.150:0/2122205165 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9940681a0 con 0x7fc99409b320 2024-03-17T23:48:57.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.612+0000 7fc990ff9640 1 -- 172.21.15.150:0/2122205165 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5195 (secure 0 0 0) 0x7fc98405d100 con 0x7fc99409b320 2024-03-17T23:48:57.612 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:57.612 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5177,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6834/3126446677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":3126446677},{"type":"v1","addr":"172.21.15.157:6834","nonce":3126446677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1992779366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1992779366},{"type":"v1","addr":"172.21.15.150:6836","nonce":1992779366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5487,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/3615467865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6833","nonce":3615467865},{"type":"v1","addr":"172.21.15.157:6835","nonce":3615467865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-17T23:48:10.730994+0000","modified":"2024-03-17T23:48:23.827674+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5515":{"gid":5515,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6837/1413722826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1413722826},{"type":"v1","addr":"172.21.15.150:6837","nonce":1413722826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5515,"qdb_cluster":[5515]},"id":2}]} 2024-03-17T23:48:57.613 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 16 2024-03-17T23:48:57.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.614+0000 7fc999fd4640 1 -- 172.21.15.150:0/2122205165 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc97406bbd0 msgr2=0x7fc97406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:57.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.614+0000 7fc999fd4640 1 --2- 172.21.15.150:0/2122205165 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc97406bbd0 0x7fc97406e090 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc97c036000 tx=0x7fc97c008030 comp rx=0 tx=0).stop 2024-03-17T23:48:57.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.614+0000 7fc999fd4640 1 -- 172.21.15.150:0/2122205165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99409b320 msgr2=0x7fc994151f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:57.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.614+0000 7fc999fd4640 1 --2- 172.21.15.150:0/2122205165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99409b320 0x7fc994151f50 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fc98400e870 tx=0x7fc98400ed30 comp rx=0 tx=0).stop 2024-03-17T23:48:57.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.614+0000 7fc999fd4640 1 -- 172.21.15.150:0/2122205165 shutdown_connections 2024-03-17T23:48:57.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.614+0000 7fc999fd4640 1 --2- 172.21.15.150:0/2122205165 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc97406bbd0 0x7fc97406e090 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.614+0000 7fc999fd4640 1 --2- 172.21.15.150:0/2122205165 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9941616b0 0x7fc99415d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.614+0000 7fc999fd4640 1 --2- 172.21.15.150:0/2122205165 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc9940b0ab0 0x7fc99415cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.614+0000 7fc999fd4640 1 --2- 172.21.15.150:0/2122205165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99409b320 0x7fc994151f50 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.614+0000 7fc999fd4640 1 -- 172.21.15.150:0/2122205165 >> 172.21.15.150:0/2122205165 conn(0x7fc9940b3630 msgr2=0x7fc9940cec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:57.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.614+0000 7fc999fd4640 1 -- 172.21.15.150:0/2122205165 shutdown_connections 2024-03-17T23:48:57.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.615+0000 7fc999fd4640 1 -- 172.21.15.150:0/2122205165 wait complete. 2024-03-17T23:48:57.626 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:48:57.705 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-17T23:48:57.709 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-17T23:48:57.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.732+0000 7f99542ac640 1 Processor -- start 2024-03-17T23:48:57.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.732+0000 7f99542ac640 1 -- start start 2024-03-17T23:48:57.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.733+0000 7f99542ac640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f994c0c8210 0x7f994c0cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:57.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.733+0000 7f99542ac640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f994c0ccb80 0x7f994c0cd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:57.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.733+0000 7f99542ac640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f994c0cd550 con 0x7f994c15be40 2024-03-17T23:48:57.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.733+0000 7f99542ac640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f994c0c5d70 con 0x7f994c0ccb80 2024-03-17T23:48:57.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.733+0000 7f99542ac640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f994c09b260 con 0x7f994c0c8210 2024-03-17T23:48:57.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.733+0000 7f9952849640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f994c0ccb80 0x7f994c0cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:57.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.733+0000 7f9952048640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f994c15be40 0x7f994c09a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:48014/0 (socket says 172.21.15.150:48014) 2024-03-17T23:48:57.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.733+0000 7f9952048640 1 -- 172.21.15.150:0/618177139 learned_addr learned my addr 172.21.15.150:0/618177139 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:57.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.733+0000 7f9951847640 1 --2- 172.21.15.150:0/618177139 >> v2:172.21.15.157:3300/0 conn(0x7f994c0c8210 0x7f994c0cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:57.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.733+0000 7f9952849640 1 -- 172.21.15.150:0/618177139 >> v2:172.21.15.157:3300/0 conn(0x7f994c0c8210 msgr2=0x7f994c0cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:57.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.733+0000 7f9952849640 1 --2- 172.21.15.150:0/618177139 >> v2:172.21.15.157:3300/0 conn(0x7f994c0c8210 0x7f994c0cc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.733+0000 7f9952849640 1 -- 172.21.15.150:0/618177139 >> v1:172.21.15.150:6789/0 conn(0x7f994c15be40 legacy=0x7f994c09a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:57.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.733+0000 7f9951046640 1 -- 172.21.15.150:0/618177139 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 920151292 0 0) 0x7f994c0cd550 con 0x7f994c15be40 2024-03-17T23:48:57.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.733+0000 7f9952849640 1 -- 172.21.15.150:0/618177139 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f994c09af80 con 0x7f994c0ccb80 2024-03-17T23:48:57.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.734+0000 7f9952849640 1 --2- 172.21.15.150:0/618177139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f994c0ccb80 0x7f994c0cd010 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f994c0b4720 tx=0x7f994402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af8b8dec27f98b79 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:57.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.734+0000 7f9951046640 1 -- 172.21.15.150:0/618177139 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9944039070 con 0x7f994c0ccb80 2024-03-17T23:48:57.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.734+0000 7f9951046640 1 -- 172.21.15.150:0/618177139 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f994402cc80 con 0x7f994c0ccb80 2024-03-17T23:48:57.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.734+0000 7f9951046640 1 -- 172.21.15.150:0/618177139 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9944034660 con 0x7f994c0ccb80 2024-03-17T23:48:57.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.735+0000 7f99542ac640 1 -- 172.21.15.150:0/618177139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f994c0ccb80 msgr2=0x7f994c0cd010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:57.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.735+0000 7f99542ac640 1 --2- 172.21.15.150:0/618177139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f994c0ccb80 0x7f994c0cd010 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f994c0b4720 tx=0x7f994402bed0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.735+0000 7f99542ac640 1 -- 172.21.15.150:0/618177139 shutdown_connections 2024-03-17T23:48:57.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.735+0000 7f99542ac640 1 --2- 172.21.15.150:0/618177139 >> v2:172.21.15.157:3300/0 conn(0x7f994c0c8210 0x7f994c0cc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.735+0000 7f99542ac640 1 --2- 172.21.15.150:0/618177139 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f994c0ccb80 0x7f994c0cd010 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.735+0000 7f99542ac640 1 -- 172.21.15.150:0/618177139 >> 172.21.15.150:0/618177139 conn(0x7f994c0b3620 msgr2=0x7f994c0b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:57.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.735+0000 7f99542ac640 1 -- 172.21.15.150:0/618177139 shutdown_connections 2024-03-17T23:48:57.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.735+0000 7f99542ac640 1 -- 172.21.15.150:0/618177139 wait complete. 2024-03-17T23:48:57.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.737+0000 7f99542ac640 1 Processor -- start 2024-03-17T23:48:57.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.737+0000 7f99542ac640 1 -- start start 2024-03-17T23:48:57.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.738+0000 7f99542ac640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f994c0c8210 0x7f994c15cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:57.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.738+0000 7f99542ac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f994c0ccb80 0x7f994c0c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:57.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.738+0000 7f99542ac640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f994c15be40 0x7f994c0cb100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:57.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.738+0000 7f99542ac640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f994c15c220 con 0x7f994c0ccb80 2024-03-17T23:48:57.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.738+0000 7f99542ac640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f994c0c5a00 con 0x7f994c15be40 2024-03-17T23:48:57.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.738+0000 7f99542ac640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f994c1659e0 con 0x7f994c0c8210 2024-03-17T23:48:57.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.738+0000 7f9951847640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f994c0ccb80 0x7f994c0c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:57.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.738+0000 7f9951847640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f994c0ccb80 0x7f994c0c8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42150/0 (socket says 172.21.15.150:42150) 2024-03-17T23:48:57.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.738+0000 7f9951847640 1 -- 172.21.15.150:0/2597981544 learned_addr learned my addr 172.21.15.150:0/2597981544 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:57.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.738+0000 7f9952048640 1 --2- 172.21.15.150:0/2597981544 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f994c0c8210 0x7f994c15cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:57.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.738+0000 7f9952849640 1 --2- 172.21.15.150:0/2597981544 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f994c15be40 0x7f994c0cb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:57.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.738+0000 7f9951847640 1 -- 172.21.15.150:0/2597981544 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f994c0c8210 msgr2=0x7f994c15cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:57.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.738+0000 7f9951847640 1 --2- 172.21.15.150:0/2597981544 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f994c0c8210 0x7f994c15cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.738+0000 7f9951847640 1 -- 172.21.15.150:0/2597981544 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f994c15be40 msgr2=0x7f994c0cb100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:57.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.738+0000 7f9951847640 1 --2- 172.21.15.150:0/2597981544 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f994c15be40 0x7f994c0cb100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.738+0000 7f9951847640 1 -- 172.21.15.150:0/2597981544 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f994c0cb640 con 0x7f994c0ccb80 2024-03-17T23:48:57.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.739+0000 7f9951847640 1 --2- 172.21.15.150:0/2597981544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f994c0ccb80 0x7f994c0c8ce0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f993c00e870 tx=0x7f993c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:57.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.739+0000 7f9942ffd640 1 -- 172.21.15.150:0/2597981544 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993c017070 con 0x7f994c0ccb80 2024-03-17T23:48:57.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.739+0000 7f9942ffd640 1 -- 172.21.15.150:0/2597981544 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f993c00c940 con 0x7f994c0ccb80 2024-03-17T23:48:57.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.739+0000 7f99542ac640 1 -- 172.21.15.150:0/2597981544 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f994c0cbbc0 con 0x7f994c0ccb80 2024-03-17T23:48:57.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.739+0000 7f9942ffd640 1 -- 172.21.15.150:0/2597981544 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993c00cd30 con 0x7f994c0ccb80 2024-03-17T23:48:57.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.739+0000 7f99542ac640 1 -- 172.21.15.150:0/2597981544 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f994c156010 con 0x7f994c0ccb80 2024-03-17T23:48:57.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.740+0000 7f99542ac640 1 -- 172.21.15.150:0/2597981544 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f994c0688b0 con 0x7f994c0ccb80 2024-03-17T23:48:57.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.740+0000 7f9942ffd640 1 -- 172.21.15.150:0/2597981544 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f993c01a070 con 0x7f994c0ccb80 2024-03-17T23:48:57.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.741+0000 7f9942ffd640 1 --2- 172.21.15.150:0/2597981544 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f993006bc00 0x7f993006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:57.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.741+0000 7f9942ffd640 1 -- 172.21.15.150:0/2597981544 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7f993c08aba0 con 0x7f994c0ccb80 2024-03-17T23:48:57.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.741+0000 7f9952048640 1 --2- 172.21.15.150:0/2597981544 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f993006bc00 0x7f993006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:57.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.741+0000 7f9952048640 1 --2- 172.21.15.150:0/2597981544 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f993006bc00 0x7f993006e0c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9948000f60 tx=0x7f9948007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:57.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.745+0000 7f9942ffd640 1 -- 172.21.15.150:0/2597981544 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f993c058d10 con 0x7f994c0ccb80 2024-03-17T23:48:57.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.922+0000 7f99542ac640 1 -- 172.21.15.150:0/2597981544 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f994c0cdbb0 con 0x7f994c0ccb80 2024-03-17T23:48:57.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.923+0000 7f9942ffd640 1 -- 172.21.15.150:0/2597981544 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+18839 (secure 0 0 0) 0x7f993c05cdb0 con 0x7f994c0ccb80 2024-03-17T23:48:57.924 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:57.924 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":45,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:48:37.677776+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:48:08.957879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-17T23:48:09.698808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-17T23:48:22.562269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.3d","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:48:57.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.925+0000 7f99542ac640 1 -- 172.21.15.150:0/2597981544 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f993006bc00 msgr2=0x7f993006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:57.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.925+0000 7f99542ac640 1 --2- 172.21.15.150:0/2597981544 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f993006bc00 0x7f993006e0c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f9948000f60 tx=0x7f9948007d50 comp rx=0 tx=0).stop 2024-03-17T23:48:57.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.925+0000 7f99542ac640 1 -- 172.21.15.150:0/2597981544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f994c0ccb80 msgr2=0x7f994c0c8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:57.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.925+0000 7f99542ac640 1 --2- 172.21.15.150:0/2597981544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f994c0ccb80 0x7f994c0c8ce0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f993c00e870 tx=0x7f993c00ed30 comp rx=0 tx=0).stop 2024-03-17T23:48:57.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.925+0000 7f99542ac640 1 -- 172.21.15.150:0/2597981544 shutdown_connections 2024-03-17T23:48:57.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.926+0000 7f99542ac640 1 --2- 172.21.15.150:0/2597981544 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f993006bc00 0x7f993006e0c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.926+0000 7f99542ac640 1 --2- 172.21.15.150:0/2597981544 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f994c15be40 0x7f994c0cb100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.926+0000 7f99542ac640 1 --2- 172.21.15.150:0/2597981544 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f994c0ccb80 0x7f994c0c8ce0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.926+0000 7f99542ac640 1 --2- 172.21.15.150:0/2597981544 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f994c0c8210 0x7f994c15cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:57.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.926+0000 7f99542ac640 1 -- 172.21.15.150:0/2597981544 >> 172.21.15.150:0/2597981544 conn(0x7f994c0b3620 msgr2=0x7f994c0ccf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:57.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.926+0000 7f99542ac640 1 -- 172.21.15.150:0/2597981544 shutdown_connections 2024-03-17T23:48:57.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:57.926+0000 7f99542ac640 1 -- 172.21.15.150:0/2597981544 wait complete. 2024-03-17T23:48:57.937 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-17T23:48:57.937 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:48:58.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.038+0000 7fc363feb640 1 Processor -- start 2024-03-17T23:48:58.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.038+0000 7fc363feb640 1 -- start start 2024-03-17T23:48:58.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.039+0000 7fc363feb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc35c0cf120 0x7fc35c1619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:58.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.039+0000 7fc363feb640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc35c0b9860 0x7fc35c0d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:58.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.040+0000 7fc363feb640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc35c0b9cb0 con 0x7fc35c15de30 2024-03-17T23:48:58.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.040+0000 7fc363feb640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fc35c162940 con 0x7fc35c0b9860 2024-03-17T23:48:58.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.040+0000 7fc363feb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc35c1620a0 con 0x7fc35c0cf120 2024-03-17T23:48:58.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.040+0000 7fc362588640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc35c0b9860 0x7fc35c0d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:58.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.040+0000 7fc361d87640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fc35c15de30 0x7fc35c0cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:48026/0 (socket says 172.21.15.150:48026) 2024-03-17T23:48:58.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.040+0000 7fc361d87640 1 -- 172.21.15.150:0/1979686565 learned_addr learned my addr 172.21.15.150:0/1979686565 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:58.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.040+0000 7fc361586640 1 --2- 172.21.15.150:0/1979686565 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc35c0cf120 0x7fc35c1619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:58.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.040+0000 7fc362588640 1 -- 172.21.15.150:0/1979686565 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc35c0cf120 msgr2=0x7fc35c1619a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:58.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.040+0000 7fc362588640 1 --2- 172.21.15.150:0/1979686565 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc35c0cf120 0x7fc35c1619a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:58.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.040+0000 7fc362588640 1 -- 172.21.15.150:0/1979686565 >> v1:172.21.15.150:6789/0 conn(0x7fc35c15de30 legacy=0x7fc35c0cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:58.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.040+0000 7fc362588640 1 -- 172.21.15.150:0/1979686565 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc35c162380 con 0x7fc35c0b9860 2024-03-17T23:48:58.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.040+0000 7fc360d85640 1 -- 172.21.15.150:0/1979686565 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4094409623 0 0) 0x7fc35c0b9cb0 con 0x7fc35c15de30 2024-03-17T23:48:58.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.040+0000 7fc361586640 1 --2- 172.21.15.150:0/1979686565 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc35c0cf120 0x7fc35c1619a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:58.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.041+0000 7fc362588640 1 --2- 172.21.15.150:0/1979686565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc35c0b9860 0x7fc35c0d2690 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc354009d90 tx=0x7fc35402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cf4ddde0c53184d server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:58.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.041+0000 7fc360d85640 1 -- 172.21.15.150:0/1979686565 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc354039070 con 0x7fc35c0b9860 2024-03-17T23:48:58.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.041+0000 7fc360d85640 1 -- 172.21.15.150:0/1979686565 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc35402cbd0 con 0x7fc35c0b9860 2024-03-17T23:48:58.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.041+0000 7fc360d85640 1 -- 172.21.15.150:0/1979686565 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc354034440 con 0x7fc35c0b9860 2024-03-17T23:48:58.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.042+0000 7fc363feb640 1 -- 172.21.15.150:0/1979686565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc35c0b9860 msgr2=0x7fc35c0d2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:58.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.042+0000 7fc363feb640 1 --2- 172.21.15.150:0/1979686565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc35c0b9860 0x7fc35c0d2690 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc354009d90 tx=0x7fc35402be20 comp rx=0 tx=0).stop 2024-03-17T23:48:58.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.042+0000 7fc363feb640 1 -- 172.21.15.150:0/1979686565 shutdown_connections 2024-03-17T23:48:58.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.042+0000 7fc363feb640 1 --2- 172.21.15.150:0/1979686565 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc35c0cf120 0x7fc35c1619a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:58.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.042+0000 7fc363feb640 1 --2- 172.21.15.150:0/1979686565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc35c0b9860 0x7fc35c0d2690 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:58.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.042+0000 7fc363feb640 1 -- 172.21.15.150:0/1979686565 >> 172.21.15.150:0/1979686565 conn(0x7fc35c0730b0 msgr2=0x7fc35c0d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:58.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.042+0000 7fc363feb640 1 -- 172.21.15.150:0/1979686565 shutdown_connections 2024-03-17T23:48:58.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.042+0000 7fc363feb640 1 -- 172.21.15.150:0/1979686565 wait complete. 2024-03-17T23:48:58.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.043+0000 7fc363feb640 1 Processor -- start 2024-03-17T23:48:58.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.043+0000 7fc363feb640 1 -- start start 2024-03-17T23:48:58.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.044+0000 7fc363feb640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc35c0b9860 0x7fc35c0cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:58.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.044+0000 7fc363feb640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc35c0cf120 0x7fc35c14eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:58.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.044+0000 7fc363feb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc35c15de30 0x7fc35c0acb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:58.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.044+0000 7fc363feb640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc35c15e210 con 0x7fc35c15de30 2024-03-17T23:48:58.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.044+0000 7fc363feb640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fc35c0d2170 con 0x7fc35c0cf120 2024-03-17T23:48:58.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.044+0000 7fc363feb640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fc35c0d2bd0 con 0x7fc35c0b9860 2024-03-17T23:48:58.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.044+0000 7fc362588640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc35c15de30 0x7fc35c0acb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:58.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.044+0000 7fc362588640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc35c15de30 0x7fc35c0acb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42174/0 (socket says 172.21.15.150:42174) 2024-03-17T23:48:58.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.044+0000 7fc362588640 1 -- 172.21.15.150:0/3179016788 learned_addr learned my addr 172.21.15.150:0/3179016788 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:58.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.044+0000 7fc361d87640 1 --2- 172.21.15.150:0/3179016788 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc35c0b9860 0x7fc35c0cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:58.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.044+0000 7fc361586640 1 --2- 172.21.15.150:0/3179016788 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc35c0cf120 0x7fc35c14eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:58.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.044+0000 7fc362588640 1 -- 172.21.15.150:0/3179016788 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc35c0b9860 msgr2=0x7fc35c0cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:58.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.044+0000 7fc362588640 1 --2- 172.21.15.150:0/3179016788 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc35c0b9860 0x7fc35c0cdd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:58.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.044+0000 7fc362588640 1 -- 172.21.15.150:0/3179016788 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc35c0cf120 msgr2=0x7fc35c14eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:58.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.044+0000 7fc362588640 1 --2- 172.21.15.150:0/3179016788 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc35c0cf120 0x7fc35c14eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:58.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.044+0000 7fc362588640 1 -- 172.21.15.150:0/3179016788 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc35c0cb250 con 0x7fc35c15de30 2024-03-17T23:48:58.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.044+0000 7fc361d87640 1 --2- 172.21.15.150:0/3179016788 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc35c0b9860 0x7fc35c0cdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:48:58.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.044+0000 7fc362588640 1 --2- 172.21.15.150:0/3179016788 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc35c15de30 0x7fc35c0acb40 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fc354009d90 tx=0x7fc354004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:58.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.044+0000 7fc3527fc640 1 -- 172.21.15.150:0/3179016788 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc354039050 con 0x7fc35c15de30 2024-03-17T23:48:58.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.044+0000 7fc3527fc640 1 -- 172.21.15.150:0/3179016788 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc354045070 con 0x7fc35c15de30 2024-03-17T23:48:58.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.044+0000 7fc363feb640 1 -- 172.21.15.150:0/3179016788 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc35c1f33e0 con 0x7fc35c15de30 2024-03-17T23:48:58.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.044+0000 7fc3527fc640 1 -- 172.21.15.150:0/3179016788 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc35403e830 con 0x7fc35c15de30 2024-03-17T23:48:58.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.045+0000 7fc363feb640 1 -- 172.21.15.150:0/3179016788 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc35c0cb4f0 con 0x7fc35c15de30 2024-03-17T23:48:58.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.046+0000 7fc363feb640 1 -- 172.21.15.150:0/3179016788 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc35c0681a0 con 0x7fc35c15de30 2024-03-17T23:48:58.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.046+0000 7fc3527fc640 1 -- 172.21.15.150:0/3179016788 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fc35403e9d0 con 0x7fc35c15de30 2024-03-17T23:48:58.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.046+0000 7fc3527fc640 1 --2- 172.21.15.150:0/3179016788 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc34006bbd0 0x7fc34006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:58.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.046+0000 7fc3527fc640 1 -- 172.21.15.150:0/3179016788 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7fc3540addc0 con 0x7fc35c15de30 2024-03-17T23:48:58.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.046+0000 7fc361d87640 1 --2- 172.21.15.150:0/3179016788 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc34006bbd0 0x7fc34006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:58.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.047+0000 7fc361d87640 1 --2- 172.21.15.150:0/3179016788 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc34006bbd0 0x7fc34006e090 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc358000f60 tx=0x7fc358009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:58.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.051+0000 7fc3527fc640 1 -- 172.21.15.150:0/3179016788 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fc35407c6a0 con 0x7fc35c15de30 2024-03-17T23:48:58.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.286+0000 7fc363feb640 1 -- 172.21.15.150:0/3179016788 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc35c0d1910 con 0x7fc35c15de30 2024-03-17T23:48:58.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.287+0000 7fc3527fc640 1 -- 172.21.15.150:0/3179016788 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5195 (secure 0 0 0) 0x7fc35407bfa0 con 0x7fc35c15de30 2024-03-17T23:48:58.288 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:58.288 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5177,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6834/3126446677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":3126446677},{"type":"v1","addr":"172.21.15.157:6834","nonce":3126446677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1992779366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1992779366},{"type":"v1","addr":"172.21.15.150:6836","nonce":1992779366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5487,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/3615467865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6833","nonce":3615467865},{"type":"v1","addr":"172.21.15.157:6835","nonce":3615467865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-17T23:48:10.730994+0000","modified":"2024-03-17T23:48:23.827674+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5515":{"gid":5515,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6837/1413722826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1413722826},{"type":"v1","addr":"172.21.15.150:6837","nonce":1413722826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5515,"qdb_cluster":[5515]},"id":2}]} 2024-03-17T23:48:58.288 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 16 2024-03-17T23:48:58.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.289+0000 7fc363feb640 1 -- 172.21.15.150:0/3179016788 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc34006bbd0 msgr2=0x7fc34006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:58.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.289+0000 7fc363feb640 1 --2- 172.21.15.150:0/3179016788 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc34006bbd0 0x7fc34006e090 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc358000f60 tx=0x7fc358009040 comp rx=0 tx=0).stop 2024-03-17T23:48:58.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.289+0000 7fc363feb640 1 -- 172.21.15.150:0/3179016788 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc35c15de30 msgr2=0x7fc35c0acb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:58.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.289+0000 7fc363feb640 1 --2- 172.21.15.150:0/3179016788 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc35c15de30 0x7fc35c0acb40 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fc354009d90 tx=0x7fc354004a40 comp rx=0 tx=0).stop 2024-03-17T23:48:58.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.289+0000 7fc363feb640 1 -- 172.21.15.150:0/3179016788 shutdown_connections 2024-03-17T23:48:58.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.289+0000 7fc363feb640 1 --2- 172.21.15.150:0/3179016788 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc35c15de30 0x7fc35c0acb40 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:58.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.289+0000 7fc363feb640 1 --2- 172.21.15.150:0/3179016788 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fc34006bbd0 0x7fc34006e090 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:58.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.289+0000 7fc363feb640 1 --2- 172.21.15.150:0/3179016788 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc35c0cf120 0x7fc35c14eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:58.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.289+0000 7fc363feb640 1 --2- 172.21.15.150:0/3179016788 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fc35c0b9860 0x7fc35c0cdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:58.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.289+0000 7fc363feb640 1 -- 172.21.15.150:0/3179016788 >> 172.21.15.150:0/3179016788 conn(0x7fc35c0730b0 msgr2=0x7fc35c0b4e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:58.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.289+0000 7fc363feb640 1 -- 172.21.15.150:0/3179016788 shutdown_connections 2024-03-17T23:48:58.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.289+0000 7fc363feb640 1 -- 172.21.15.150:0/3179016788 wait complete. 2024-03-17T23:48:58.301 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:48:58.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.403+0000 7f5326c97640 1 Processor -- start 2024-03-17T23:48:58.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.403+0000 7f5326c97640 1 -- start start 2024-03-17T23:48:58.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.404+0000 7f5326c97640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f53201619a0 0x7f53200b9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:58.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.404+0000 7f5326c97640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5320161dd0 con 0x7f532015de30 2024-03-17T23:48:58.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.404+0000 7f5326c97640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53201620a0 con 0x7f53200cf120 2024-03-17T23:48:58.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.404+0000 7f5326c97640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f5320162940 con 0x7f53201619a0 2024-03-17T23:48:58.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.404+0000 7f5324a33640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f532015de30 0x7f53200cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:48042/0 (socket says 172.21.15.150:48042) 2024-03-17T23:48:58.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.404+0000 7f5324a33640 1 -- 172.21.15.150:0/1817489361 learned_addr learned my addr 172.21.15.150:0/1817489361 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:58.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.404+0000 7f531f7fe640 1 -- 172.21.15.150:0/1817489361 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1794253755 0 0) 0x7f5320161dd0 con 0x7f532015de30 2024-03-17T23:48:58.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.404+0000 7f531f7fe640 1 -- 172.21.15.150:0/1817489361 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5310003520 con 0x7f532015de30 2024-03-17T23:48:58.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.404+0000 7f5325234640 1 --2- 172.21.15.150:0/1817489361 >> v2:172.21.15.157:3300/0 conn(0x7f53201619a0 0x7f53200b9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:58.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.404+0000 7f531f7fe640 1 -- 172.21.15.150:0/1817489361 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4287826438 0 0) 0x7f5310003520 con 0x7f532015de30 2024-03-17T23:48:58.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.404+0000 7f531f7fe640 1 -- 172.21.15.150:0/1817489361 >> v2:172.21.15.157:3300/0 conn(0x7f53201619a0 msgr2=0x7f53200b9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:58.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.404+0000 7f531f7fe640 1 --2- 172.21.15.150:0/1817489361 >> v2:172.21.15.157:3300/0 conn(0x7f53201619a0 0x7f53200b9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:58.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.404+0000 7f531f7fe640 1 -- 172.21.15.150:0/1817489361 >> v1:172.21.15.157:6789/0 conn(0x7f53200cf120 legacy=0x7f53200d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:58.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.404+0000 7f531f7fe640 1 -- 172.21.15.150:0/1817489361 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5320162380 con 0x7f532015de30 2024-03-17T23:48:58.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.404+0000 7f531f7fe640 1 -- 172.21.15.150:0/1817489361 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f5314001670 con 0x7f532015de30 2024-03-17T23:48:58.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.404+0000 7f531f7fe640 1 -- 172.21.15.150:0/1817489361 >> v1:172.21.15.150:6789/0 conn(0x7f532015de30 legacy=0x7f53200cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:58.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.404+0000 7f531f7fe640 1 --2- 172.21.15.150:0/1817489361 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f53100039b0 0x7f5310003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:58.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.404+0000 7f531f7fe640 1 --2- 172.21.15.150:0/1817489361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5310004800 0x7f5310004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:58.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.404+0000 7f531f7fe640 1 --2- 172.21.15.150:0/1817489361 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f53100051c0 0x7f531000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:58.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.404+0000 7f531f7fe640 1 -- 172.21.15.150:0/1817489361 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5320162940 con 0x7f5310004800 2024-03-17T23:48:58.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.404+0000 7f531f7fe640 1 -- 172.21.15.150:0/1817489361 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f531000c160 con 0x7f53100051c0 2024-03-17T23:48:58.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.404+0000 7f531f7fe640 1 -- 172.21.15.150:0/1817489361 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f531000c360 con 0x7f53100039b0 2024-03-17T23:48:58.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.404+0000 7f531ffff640 1 --2- 172.21.15.150:0/1817489361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5310004800 0x7f5310004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:58.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.405+0000 7f5324a33640 1 --2- 172.21.15.150:0/1817489361 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f53100039b0 0x7f5310003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:58.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.405+0000 7f5325234640 1 --2- 172.21.15.150:0/1817489361 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f53100051c0 0x7f531000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:58.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.405+0000 7f531ffff640 1 -- 172.21.15.150:0/1817489361 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f53100039b0 msgr2=0x7f5310003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:58.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.405+0000 7f531ffff640 1 --2- 172.21.15.150:0/1817489361 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f53100039b0 0x7f5310003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:58.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.405+0000 7f531ffff640 1 -- 172.21.15.150:0/1817489361 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f53100051c0 msgr2=0x7f531000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:58.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.405+0000 7f531ffff640 1 --2- 172.21.15.150:0/1817489361 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f53100051c0 0x7f531000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:58.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.405+0000 7f5325234640 1 -- 172.21.15.150:0/1817489361 reap_dead start 2024-03-17T23:48:58.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.405+0000 7f531ffff640 1 -- 172.21.15.150:0/1817489361 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5310003520 con 0x7f5310004800 2024-03-17T23:48:58.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.405+0000 7f531ffff640 1 --2- 172.21.15.150:0/1817489361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5310004800 0x7f5310004c80 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f530c001750 tx=0x7f5310005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf35e5b2da21d011 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:58.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.406+0000 7f531f7fe640 1 -- 172.21.15.150:0/1817489361 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530c001370 con 0x7f5310004800 2024-03-17T23:48:58.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.406+0000 7f531f7fe640 1 -- 172.21.15.150:0/1817489361 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f530c0052d0 con 0x7f5310004800 2024-03-17T23:48:58.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.406+0000 7f531f7fe640 1 -- 172.21.15.150:0/1817489361 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530c003830 con 0x7f5310004800 2024-03-17T23:48:58.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.406+0000 7f5326c97640 1 -- 172.21.15.150:0/1817489361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5310004800 msgr2=0x7f5310004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:58.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.406+0000 7f5326c97640 1 --2- 172.21.15.150:0/1817489361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5310004800 0x7f5310004c80 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f530c001750 tx=0x7f5310005830 comp rx=0 tx=0).stop 2024-03-17T23:48:58.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.406+0000 7f5326c97640 1 -- 172.21.15.150:0/1817489361 shutdown_connections 2024-03-17T23:48:58.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.406+0000 7f5326c97640 1 --2- 172.21.15.150:0/1817489361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5310004800 0x7f5310004c80 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:58.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.406+0000 7f5326c97640 1 -- 172.21.15.150:0/1817489361 >> 172.21.15.150:0/1817489361 conn(0x7f53200730b0 msgr2=0x7f53200d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:58.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.406+0000 7f5326c97640 1 -- 172.21.15.150:0/1817489361 shutdown_connections 2024-03-17T23:48:58.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.406+0000 7f5326c97640 1 -- 172.21.15.150:0/1817489361 wait complete. 2024-03-17T23:48:58.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.407+0000 7f5326c97640 1 Processor -- start 2024-03-17T23:48:58.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.407+0000 7f5326c97640 1 -- start start 2024-03-17T23:48:58.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.408+0000 7f5326c97640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5310004800 0x7f53200cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:58.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.408+0000 7f5326c97640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5320160ac0 0x7f53200c9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:58.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.408+0000 7f5326c97640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5320157120 0x7f532015abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:58.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.408+0000 7f5326c97640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f532015e210 con 0x7f5320160ac0 2024-03-17T23:48:58.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.408+0000 7f5326c97640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f53200d2170 con 0x7f5320157120 2024-03-17T23:48:58.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.408+0000 7f5326c97640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f53200b9860 con 0x7f5310004800 2024-03-17T23:48:58.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.408+0000 7f531ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5320160ac0 0x7f53200c9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:58.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.408+0000 7f531ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5320160ac0 0x7f53200c9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42182/0 (socket says 172.21.15.150:42182) 2024-03-17T23:48:58.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.408+0000 7f531ffff640 1 -- 172.21.15.150:0/1965404532 learned_addr learned my addr 172.21.15.150:0/1965404532 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:58.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.409+0000 7f5324a33640 1 --2- 172.21.15.150:0/1965404532 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5310004800 0x7f53200cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:58.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.409+0000 7f5325234640 1 --2- 172.21.15.150:0/1965404532 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5320157120 0x7f532015abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:58.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.409+0000 7f531ffff640 1 -- 172.21.15.150:0/1965404532 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5310004800 msgr2=0x7f53200cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:58.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.409+0000 7f531ffff640 1 --2- 172.21.15.150:0/1965404532 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5310004800 0x7f53200cbb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:58.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.409+0000 7f531ffff640 1 -- 172.21.15.150:0/1965404532 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5320157120 msgr2=0x7f532015abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:58.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.409+0000 7f531ffff640 1 --2- 172.21.15.150:0/1965404532 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5320157120 0x7f532015abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:58.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.409+0000 7f531ffff640 1 -- 172.21.15.150:0/1965404532 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5320157900 con 0x7f5320160ac0 2024-03-17T23:48:58.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.409+0000 7f531ffff640 1 --2- 172.21.15.150:0/1965404532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5320160ac0 0x7f53200c9860 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f530c005040 tx=0x7f530c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:58.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.409+0000 7f531d7fa640 1 -- 172.21.15.150:0/1965404532 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530c041020 con 0x7f5320160ac0 2024-03-17T23:48:58.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.409+0000 7f531d7fa640 1 -- 172.21.15.150:0/1965404532 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f530c005480 con 0x7f5320160ac0 2024-03-17T23:48:58.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.409+0000 7f5326c97640 1 -- 172.21.15.150:0/1965404532 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5320157be0 con 0x7f5320160ac0 2024-03-17T23:48:58.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.409+0000 7f531d7fa640 1 -- 172.21.15.150:0/1965404532 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530c007a30 con 0x7f5320160ac0 2024-03-17T23:48:58.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.409+0000 7f5326c97640 1 -- 172.21.15.150:0/1965404532 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f532015e4c0 con 0x7f5320160ac0 2024-03-17T23:48:58.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.410+0000 7f5326c97640 1 -- 172.21.15.150:0/1965404532 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53200681a0 con 0x7f5320160ac0 2024-03-17T23:48:58.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.411+0000 7f531d7fa640 1 -- 172.21.15.150:0/1965404532 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f530c044020 con 0x7f5320160ac0 2024-03-17T23:48:58.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.411+0000 7f531d7fa640 1 --2- 172.21.15.150:0/1965404532 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f530406bc00 0x7f530406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:58.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.411+0000 7f531d7fa640 1 -- 172.21.15.150:0/1965404532 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7f530c0ac5b0 con 0x7f5320160ac0 2024-03-17T23:48:58.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.411+0000 7f5324a33640 1 --2- 172.21.15.150:0/1965404532 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f530406bc00 0x7f530406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:58.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.412+0000 7f5324a33640 1 --2- 172.21.15.150:0/1965404532 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f530406bc00 0x7f530406e0c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f532015bf00 tx=0x7f5314009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:58.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.415+0000 7f531d7fa640 1 -- 172.21.15.150:0/1965404532 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f530c04b030 con 0x7f5320160ac0 2024-03-17T23:48:58.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.633+0000 7f5326c97640 1 -- 172.21.15.150:0/1965404532 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53200d1910 con 0x7f5320160ac0 2024-03-17T23:48:58.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.634+0000 7f531d7fa640 1 -- 172.21.15.150:0/1965404532 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5195 (secure 0 0 0) 0x7f530c006e20 con 0x7f5320160ac0 2024-03-17T23:48:58.635 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:58.635 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5177,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6834/3126446677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":3126446677},{"type":"v1","addr":"172.21.15.157:6834","nonce":3126446677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1992779366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1992779366},{"type":"v1","addr":"172.21.15.150:6836","nonce":1992779366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5487,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/3615467865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6833","nonce":3615467865},{"type":"v1","addr":"172.21.15.157:6835","nonce":3615467865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-17T23:48:10.730994+0000","modified":"2024-03-17T23:48:23.827674+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5515":{"gid":5515,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6837/1413722826","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":1413722826},{"type":"v1","addr":"172.21.15.150:6837","nonce":1413722826}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5515,"qdb_cluster":[5515]},"id":2}]} 2024-03-17T23:48:58.635 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 16 2024-03-17T23:48:58.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.636+0000 7f5326c97640 1 -- 172.21.15.150:0/1965404532 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f530406bc00 msgr2=0x7f530406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:58.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.636+0000 7f5326c97640 1 --2- 172.21.15.150:0/1965404532 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f530406bc00 0x7f530406e0c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f532015bf00 tx=0x7f5314009040 comp rx=0 tx=0).stop 2024-03-17T23:48:58.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.636+0000 7f5326c97640 1 -- 172.21.15.150:0/1965404532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5320160ac0 msgr2=0x7f53200c9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:58.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.636+0000 7f5326c97640 1 --2- 172.21.15.150:0/1965404532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5320160ac0 0x7f53200c9860 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f530c005040 tx=0x7f530c001120 comp rx=0 tx=0).stop 2024-03-17T23:48:58.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.637+0000 7f5326c97640 1 -- 172.21.15.150:0/1965404532 shutdown_connections 2024-03-17T23:48:58.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.637+0000 7f5326c97640 1 --2- 172.21.15.150:0/1965404532 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f530406bc00 0x7f530406e0c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:58.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.637+0000 7f5326c97640 1 --2- 172.21.15.150:0/1965404532 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5320157120 0x7f532015abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:58.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.637+0000 7f5326c97640 1 --2- 172.21.15.150:0/1965404532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5320160ac0 0x7f53200c9860 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:58.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.637+0000 7f5326c97640 1 --2- 172.21.15.150:0/1965404532 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5310004800 0x7f53200cbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:58.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.637+0000 7f5326c97640 1 -- 172.21.15.150:0/1965404532 >> 172.21.15.150:0/1965404532 conn(0x7f53200730b0 msgr2=0x7f53200d2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:58.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.637+0000 7f5326c97640 1 -- 172.21.15.150:0/1965404532 shutdown_connections 2024-03-17T23:48:58.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.637+0000 7f5326c97640 1 -- 172.21.15.150:0/1965404532 wait complete. 2024-03-17T23:48:58.652 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:48:58.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.753+0000 7fd3d37c4640 1 Processor -- start 2024-03-17T23:48:58.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.753+0000 7fd3d37c4640 1 -- start start 2024-03-17T23:48:58.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.754+0000 7fd3d37c4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd3cc15be40 0x7fd3cc09a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:58.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.754+0000 7fd3d37c4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3cc09ad00 con 0x7fd3cc0d0240 2024-03-17T23:48:58.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.754+0000 7fd3d37c4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3cc09b260 con 0x7fd3cc0c8210 2024-03-17T23:48:58.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.754+0000 7fd3d37c4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd3cc0c5d70 con 0x7fd3cc15be40 2024-03-17T23:48:58.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.754+0000 7fd3d1d61640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd3cc0d0240 0x7fd3cc1659e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:48044/0 (socket says 172.21.15.150:48044) 2024-03-17T23:48:58.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.754+0000 7fd3d1d61640 1 -- 172.21.15.150:0/3274101232 learned_addr learned my addr 172.21.15.150:0/3274101232 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:58.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.754+0000 7fd3d1560640 1 --2- 172.21.15.150:0/3274101232 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd3cc15be40 0x7fd3cc09a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:58.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.754+0000 7fd3c3fff640 1 -- 172.21.15.150:0/3274101232 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4272858192 0 0) 0x7fd3c4001490 con 0x7fd3cc0d0240 2024-03-17T23:48:58.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.754+0000 7fd3c3fff640 1 -- 172.21.15.150:0/3274101232 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3b8003520 con 0x7fd3cc0d0240 2024-03-17T23:48:58.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.754+0000 7fd3c3fff640 1 -- 172.21.15.150:0/3274101232 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 771073871 0 0) 0x7fd3cc09b260 con 0x7fd3cc0c8210 2024-03-17T23:48:58.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.754+0000 7fd3c3fff640 1 -- 172.21.15.150:0/3274101232 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3c4001490 con 0x7fd3cc0c8210 2024-03-17T23:48:58.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.754+0000 7fd3d1560640 1 -- 172.21.15.150:0/3274101232 >> v1:172.21.15.157:6789/0 conn(0x7fd3cc0c8210 legacy=0x7fd3cc0cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:58.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.754+0000 7fd3c3fff640 1 -- 172.21.15.150:0/3274101232 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1588389543 0 0) 0x7fd3b8003520 con 0x7fd3cc0d0240 2024-03-17T23:48:58.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.754+0000 7fd3d1560640 1 -- 172.21.15.150:0/3274101232 >> v1:172.21.15.150:6789/0 conn(0x7fd3cc0d0240 legacy=0x7fd3cc1659e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:58.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.754+0000 7fd3d1560640 1 -- 172.21.15.150:0/3274101232 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3cc09af80 con 0x7fd3cc15be40 2024-03-17T23:48:58.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.755+0000 7fd3d1560640 1 --2- 172.21.15.150:0/3274101232 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd3cc15be40 0x7fd3cc09a7c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd3c80097d0 tx=0x7fd3c802b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e4b93e86e891b6f server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:58.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.755+0000 7fd3c3fff640 1 -- 172.21.15.150:0/3274101232 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8037070 con 0x7fd3cc15be40 2024-03-17T23:48:58.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.755+0000 7fd3c3fff640 1 -- 172.21.15.150:0/3274101232 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3c8030510 con 0x7fd3cc15be40 2024-03-17T23:48:58.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.755+0000 7fd3c3fff640 1 -- 172.21.15.150:0/3274101232 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c4001bc0 con 0x7fd3cc15be40 2024-03-17T23:48:58.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.756+0000 7fd3d37c4640 1 -- 172.21.15.150:0/3274101232 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd3cc15be40 msgr2=0x7fd3cc09a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:58.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.756+0000 7fd3d37c4640 1 --2- 172.21.15.150:0/3274101232 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd3cc15be40 0x7fd3cc09a7c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd3c80097d0 tx=0x7fd3c802b800 comp rx=0 tx=0).stop 2024-03-17T23:48:58.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.756+0000 7fd3d37c4640 1 -- 172.21.15.150:0/3274101232 shutdown_connections 2024-03-17T23:48:58.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.756+0000 7fd3d37c4640 1 --2- 172.21.15.150:0/3274101232 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd3cc15be40 0x7fd3cc09a7c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:58.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.756+0000 7fd3d37c4640 1 -- 172.21.15.150:0/3274101232 >> 172.21.15.150:0/3274101232 conn(0x7fd3cc0b3620 msgr2=0x7fd3cc0b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:58.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.756+0000 7fd3d37c4640 1 -- 172.21.15.150:0/3274101232 shutdown_connections 2024-03-17T23:48:58.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.756+0000 7fd3d37c4640 1 -- 172.21.15.150:0/3274101232 wait complete. 2024-03-17T23:48:58.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.757+0000 7fd3d37c4640 1 Processor -- start 2024-03-17T23:48:58.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.757+0000 7fd3d37c4640 1 -- start start 2024-03-17T23:48:58.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.758+0000 7fd3d37c4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd3cc0c8210 0x7fd3cc0c73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:58.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.758+0000 7fd3d37c4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3cc0d0240 0x7fd3cc0c7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:58.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.758+0000 7fd3d37c4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd3cc15be40 0x7fd3cc1572c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:58.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.758+0000 7fd3d37c4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd3cc15c220 con 0x7fd3cc0d0240 2024-03-17T23:48:58.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.758+0000 7fd3d37c4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd3cc0c5a00 con 0x7fd3cc0c8210 2024-03-17T23:48:58.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.758+0000 7fd3d37c4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd3cc0d06d0 con 0x7fd3cc15be40 2024-03-17T23:48:58.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.758+0000 7fd3d0d5f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3cc0d0240 0x7fd3cc0c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:58.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.758+0000 7fd3d0d5f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3cc0d0240 0x7fd3cc0c7930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42188/0 (socket says 172.21.15.150:42188) 2024-03-17T23:48:58.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.758+0000 7fd3d0d5f640 1 -- 172.21.15.150:0/2132143506 learned_addr learned my addr 172.21.15.150:0/2132143506 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:58.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.758+0000 7fd3d1560640 1 --2- 172.21.15.150:0/2132143506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd3cc0c8210 0x7fd3cc0c73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:58.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.758+0000 7fd3d1d61640 1 --2- 172.21.15.150:0/2132143506 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd3cc15be40 0x7fd3cc1572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:58.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.758+0000 7fd3d0d5f640 1 -- 172.21.15.150:0/2132143506 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd3cc15be40 msgr2=0x7fd3cc1572c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:58.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.758+0000 7fd3d0d5f640 1 --2- 172.21.15.150:0/2132143506 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd3cc15be40 0x7fd3cc1572c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:58.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.758+0000 7fd3d0d5f640 1 -- 172.21.15.150:0/2132143506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd3cc0c8210 msgr2=0x7fd3cc0c73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:58.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.758+0000 7fd3d0d5f640 1 --2- 172.21.15.150:0/2132143506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd3cc0c8210 0x7fd3cc0c73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:58.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.758+0000 7fd3d0d5f640 1 -- 172.21.15.150:0/2132143506 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3cc0c8da0 con 0x7fd3cc0d0240 2024-03-17T23:48:58.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.758+0000 7fd3d1d61640 1 --2- 172.21.15.150:0/2132143506 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd3cc15be40 0x7fd3cc1572c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:58.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.758+0000 7fd3d0d5f640 1 --2- 172.21.15.150:0/2132143506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3cc0d0240 0x7fd3cc0c7930 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc00c490 tx=0x7fd3bc00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:58.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.759+0000 7fd3c1ffb640 1 -- 172.21.15.150:0/2132143506 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc0052d0 con 0x7fd3cc0d0240 2024-03-17T23:48:58.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.759+0000 7fd3c1ffb640 1 -- 172.21.15.150:0/2132143506 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3bc005470 con 0x7fd3cc0d0240 2024-03-17T23:48:58.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.759+0000 7fd3d37c4640 1 -- 172.21.15.150:0/2132143506 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3cc0c9080 con 0x7fd3cc0d0240 2024-03-17T23:48:58.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.759+0000 7fd3c1ffb640 1 -- 172.21.15.150:0/2132143506 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc012850 con 0x7fd3cc0d0240 2024-03-17T23:48:58.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.759+0000 7fd3d37c4640 1 -- 172.21.15.150:0/2132143506 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3cc0c9640 con 0x7fd3cc0d0240 2024-03-17T23:48:58.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.760+0000 7fd3d37c4640 1 -- 172.21.15.150:0/2132143506 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3cc0688b0 con 0x7fd3cc0d0240 2024-03-17T23:48:58.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.760+0000 7fd3c1ffb640 1 -- 172.21.15.150:0/2132143506 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd3bc0129f0 con 0x7fd3cc0d0240 2024-03-17T23:48:58.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.760+0000 7fd3c1ffb640 1 --2- 172.21.15.150:0/2132143506 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd3b006bbd0 0x7fd3b006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:58.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.761+0000 7fd3c1ffb640 1 -- 172.21.15.150:0/2132143506 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7fd3bc08a9e0 con 0x7fd3cc0d0240 2024-03-17T23:48:58.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.761+0000 7fd3d1560640 1 --2- 172.21.15.150:0/2132143506 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd3b006bbd0 0x7fd3b006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:58.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.761+0000 7fd3d1560640 1 --2- 172.21.15.150:0/2132143506 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd3b006bbd0 0x7fd3b006e090 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8034000 tx=0x7fd3c8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:58.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.765+0000 7fd3c1ffb640 1 -- 172.21.15.150:0/2132143506 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd3bc0591a0 con 0x7fd3cc0d0240 2024-03-17T23:48:58.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.945+0000 7fd3d37c4640 1 -- 172.21.15.150:0/2132143506 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd3cc0b28d0 con 0x7fd3cc0d0240 2024-03-17T23:48:58.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.946+0000 7fd3c1ffb640 1 -- 172.21.15.150:0/2132143506 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+18839 (secure 0 0 0) 0x7fd3bc058aa0 con 0x7fd3cc0d0240 2024-03-17T23:48:58.947 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:48:58.947 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":45,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:48:37.677776+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:48:08.957879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-17T23:48:09.698808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"data_new","create_time":"2024-03-17T23:48:22.562269+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.3d","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:48:58.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.948+0000 7fd3d37c4640 1 -- 172.21.15.150:0/2132143506 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd3b006bbd0 msgr2=0x7fd3b006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:58.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.948+0000 7fd3d37c4640 1 --2- 172.21.15.150:0/2132143506 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd3b006bbd0 0x7fd3b006e090 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8034000 tx=0x7fd3c8008030 comp rx=0 tx=0).stop 2024-03-17T23:48:58.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.948+0000 7fd3d37c4640 1 -- 172.21.15.150:0/2132143506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3cc0d0240 msgr2=0x7fd3cc0c7930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:58.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.948+0000 7fd3d37c4640 1 --2- 172.21.15.150:0/2132143506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3cc0d0240 0x7fd3cc0c7930 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc00c490 tx=0x7fd3bc00c950 comp rx=0 tx=0).stop 2024-03-17T23:48:58.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.948+0000 7fd3d37c4640 1 -- 172.21.15.150:0/2132143506 shutdown_connections 2024-03-17T23:48:58.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.948+0000 7fd3d37c4640 1 --2- 172.21.15.150:0/2132143506 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd3cc15be40 0x7fd3cc1572c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:58.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.948+0000 7fd3d37c4640 1 --2- 172.21.15.150:0/2132143506 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd3cc0d0240 0x7fd3cc0c7930 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:58.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.948+0000 7fd3d37c4640 1 --2- 172.21.15.150:0/2132143506 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd3b006bbd0 0x7fd3b006e090 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:58.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.948+0000 7fd3d37c4640 1 --2- 172.21.15.150:0/2132143506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd3cc0c8210 0x7fd3cc0c73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:58.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.948+0000 7fd3d37c4640 1 -- 172.21.15.150:0/2132143506 >> 172.21.15.150:0/2132143506 conn(0x7fd3cc0b3620 msgr2=0x7fd3cc15e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:58.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.948+0000 7fd3d37c4640 1 -- 172.21.15.150:0/2132143506 shutdown_connections 2024-03-17T23:48:58.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:58.948+0000 7fd3d37c4640 1 -- 172.21.15.150:0/2132143506 wait complete. 2024-03-17T23:48:58.962 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-17T23:48:59.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.064+0000 7f23b16b5640 1 Processor -- start 2024-03-17T23:48:59.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.065+0000 7f23b16b5640 1 -- start start 2024-03-17T23:48:59.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.065+0000 7f23b16b5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f23ac15e1a0 0x7f23ac0afe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:59.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.065+0000 7f23b16b5640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f23ac0ae290 0x7f23ac0b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:59.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.065+0000 7f23b16b5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23ac15e580 con 0x7f23ac099a70 2024-03-17T23:48:59.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.065+0000 7f23b16b5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f23ac06bcd0 con 0x7f23ac15e1a0 2024-03-17T23:48:59.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.065+0000 7f23b16b5640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f23ac06bfb0 con 0x7f23ac0ae290 2024-03-17T23:48:59.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.065+0000 7f23aaffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f23ac15e1a0 0x7f23ac0afe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:59.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.065+0000 7f23aa7fc640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f23ac099a70 0x7f23ac0d01d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:48050/0 (socket says 172.21.15.150:48050) 2024-03-17T23:48:59.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.066+0000 7f23aa7fc640 1 -- 172.21.15.150:0/3994240799 learned_addr learned my addr 172.21.15.150:0/3994240799 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:59.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.066+0000 7f23aaffd640 1 -- 172.21.15.150:0/3994240799 >> v2:172.21.15.157:3300/0 conn(0x7f23ac0ae290 msgr2=0x7f23ac0b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:59.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.066+0000 7f23ab7fe640 1 --2- 172.21.15.150:0/3994240799 >> v2:172.21.15.157:3300/0 conn(0x7f23ac0ae290 0x7f23ac0b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:59.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.066+0000 7f23a9ffb640 1 -- 172.21.15.150:0/3994240799 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2847645474 0 0) 0x7f23ac15e580 con 0x7f23ac099a70 2024-03-17T23:48:59.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.066+0000 7f23aaffd640 1 --2- 172.21.15.150:0/3994240799 >> v2:172.21.15.157:3300/0 conn(0x7f23ac0ae290 0x7f23ac0b03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:59.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.066+0000 7f23aaffd640 1 -- 172.21.15.150:0/3994240799 >> v1:172.21.15.150:6789/0 conn(0x7f23ac099a70 legacy=0x7f23ac0d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:59.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.066+0000 7f23aaffd640 1 -- 172.21.15.150:0/3994240799 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23ac06c290 con 0x7f23ac15e1a0 2024-03-17T23:48:59.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.067+0000 7f23aaffd640 1 --2- 172.21.15.150:0/3994240799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23ac15e1a0 0x7f23ac0afe80 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f23ac067720 tx=0x7f239c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fc4d3a3c73371d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:59.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.067+0000 7f23a9ffb640 1 -- 172.21.15.150:0/3994240799 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f239c0315a0 con 0x7f23ac15e1a0 2024-03-17T23:48:59.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.067+0000 7f23a9ffb640 1 -- 172.21.15.150:0/3994240799 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f239c037070 con 0x7f23ac15e1a0 2024-03-17T23:48:59.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.067+0000 7f23a9ffb640 1 -- 172.21.15.150:0/3994240799 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f239c031960 con 0x7f23ac15e1a0 2024-03-17T23:48:59.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.067+0000 7f23b16b5640 1 -- 172.21.15.150:0/3994240799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23ac15e1a0 msgr2=0x7f23ac0afe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:59.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.067+0000 7f23b16b5640 1 --2- 172.21.15.150:0/3994240799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23ac15e1a0 0x7f23ac0afe80 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f23ac067720 tx=0x7f239c02b840 comp rx=0 tx=0).stop 2024-03-17T23:48:59.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.067+0000 7f23b16b5640 1 -- 172.21.15.150:0/3994240799 shutdown_connections 2024-03-17T23:48:59.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.068+0000 7f23b16b5640 1 --2- 172.21.15.150:0/3994240799 >> v2:172.21.15.157:3300/0 conn(0x7f23ac0ae290 0x7f23ac0b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:59.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.068+0000 7f23b16b5640 1 --2- 172.21.15.150:0/3994240799 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23ac15e1a0 0x7f23ac0afe80 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:59.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.068+0000 7f23b16b5640 1 -- 172.21.15.150:0/3994240799 >> 172.21.15.150:0/3994240799 conn(0x7f23ac072a60 msgr2=0x7f23ac0b5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:59.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.068+0000 7f23b16b5640 1 -- 172.21.15.150:0/3994240799 shutdown_connections 2024-03-17T23:48:59.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.068+0000 7f23b16b5640 1 -- 172.21.15.150:0/3994240799 wait complete. 2024-03-17T23:48:59.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.069+0000 7f23b16b5640 1 Processor -- start 2024-03-17T23:48:59.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.069+0000 7f23b16b5640 1 -- start start 2024-03-17T23:48:59.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.070+0000 7f23b16b5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23ac099a70 0x7f23ac150030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:59.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.070+0000 7f23b16b5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f23ac0ae290 0x7f23ac09b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:59.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.070+0000 7f23b16b5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f23ac15e1a0 0x7f23ac15b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:59.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.070+0000 7f23b16b5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f23ac0b4110 con 0x7f23ac099a70 2024-03-17T23:48:59.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.070+0000 7f23b16b5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f23ac06c850 con 0x7f23ac0ae290 2024-03-17T23:48:59.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.070+0000 7f23b16b5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f23ac0af980 con 0x7f23ac15e1a0 2024-03-17T23:48:59.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.070+0000 7f23aaffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23ac099a70 0x7f23ac150030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:59.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.070+0000 7f23aaffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23ac099a70 0x7f23ac150030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42202/0 (socket says 172.21.15.150:42202) 2024-03-17T23:48:59.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.070+0000 7f23aaffd640 1 -- 172.21.15.150:0/3406179630 learned_addr learned my addr 172.21.15.150:0/3406179630 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:59.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.070+0000 7f23aa7fc640 1 --2- 172.21.15.150:0/3406179630 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f23ac0ae290 0x7f23ac09b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:59.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.070+0000 7f23ab7fe640 1 --2- 172.21.15.150:0/3406179630 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f23ac15e1a0 0x7f23ac15b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:59.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.070+0000 7f23aaffd640 1 -- 172.21.15.150:0/3406179630 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f23ac15e1a0 msgr2=0x7f23ac15b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:59.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.070+0000 7f23aaffd640 1 --2- 172.21.15.150:0/3406179630 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f23ac15e1a0 0x7f23ac15b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:59.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.070+0000 7f23aaffd640 1 -- 172.21.15.150:0/3406179630 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f23ac0ae290 msgr2=0x7f23ac09b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:59.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.070+0000 7f23aaffd640 1 --2- 172.21.15.150:0/3406179630 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f23ac0ae290 0x7f23ac09b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:59.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.070+0000 7f23aaffd640 1 -- 172.21.15.150:0/3406179630 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23ac15c170 con 0x7f23ac099a70 2024-03-17T23:48:59.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.070+0000 7f23aa7fc640 1 --2- 172.21.15.150:0/3406179630 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f23ac0ae290 0x7f23ac09b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:59.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.070+0000 7f23ab7fe640 1 --2- 172.21.15.150:0/3406179630 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f23ac15e1a0 0x7f23ac15b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:48:59.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.070+0000 7f23aaffd640 1 --2- 172.21.15.150:0/3406179630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23ac099a70 0x7f23ac150030 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f239c009890 tx=0x7f239c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:59.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.070+0000 7f23937fe640 1 -- 172.21.15.150:0/3406179630 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f239c032840 con 0x7f23ac099a70 2024-03-17T23:48:59.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.070+0000 7f23937fe640 1 -- 172.21.15.150:0/3406179630 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f239c0329e0 con 0x7f23ac099a70 2024-03-17T23:48:59.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.070+0000 7f23937fe640 1 -- 172.21.15.150:0/3406179630 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f239c032dd0 con 0x7f23ac099a70 2024-03-17T23:48:59.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.070+0000 7f23b16b5640 1 -- 172.21.15.150:0/3406179630 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23ac15c450 con 0x7f23ac099a70 2024-03-17T23:48:59.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.070+0000 7f23b16b5640 1 -- 172.21.15.150:0/3406179630 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23ac1509e0 con 0x7f23ac099a70 2024-03-17T23:48:59.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.071+0000 7f23b16b5640 1 -- 172.21.15.150:0/3406179630 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23ac0681a0 con 0x7f23ac099a70 2024-03-17T23:48:59.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.072+0000 7f23937fe640 1 -- 172.21.15.150:0/3406179630 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f239c04d3f0 con 0x7f23ac099a70 2024-03-17T23:48:59.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.072+0000 7f23937fe640 1 --2- 172.21.15.150:0/3406179630 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f239806bbd0 0x7f239806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:59.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.072+0000 7f23937fe640 1 -- 172.21.15.150:0/3406179630 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6525+0+0 (secure 0 0 0) 0x7f239c02f030 con 0x7f23ac099a70 2024-03-17T23:48:59.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.072+0000 7f23aa7fc640 1 --2- 172.21.15.150:0/3406179630 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f239806bbd0 0x7f239806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:59.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.073+0000 7f23aa7fc640 1 --2- 172.21.15.150:0/3406179630 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f239806bbd0 0x7f239806e090 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f23ac151fe0 tx=0x7f23a0007c10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:59.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.077+0000 7f23937fe640 1 -- 172.21.15.150:0/3406179630 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f239c040070 con 0x7f23ac099a70 2024-03-17T23:48:59.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.294+0000 7f23b16b5640 1 -- 172.21.15.150:0/3406179630 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f23ac15cc30 con 0x7f23ac099a70 2024-03-17T23:48:59.296 INFO:tasks.ceph.mon.a.smithi150.stderr:2024-03-17T23:48:59.295+0000 7fd9a9062640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-17T23:48:59.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.297+0000 7f23937fe640 1 -- 172.21.15.150:0/3406179630 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v17) ==== 161+0+0 (secure 0 0 0) 0x7f239c07b180 con 0x7f23ac099a70 2024-03-17T23:48:59.297 INFO:teuthology.orchestra.run.smithi150.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-17T23:48:59.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.300+0000 7f23b16b5640 1 -- 172.21.15.150:0/3406179630 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f239806bbd0 msgr2=0x7f239806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:59.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.300+0000 7f23b16b5640 1 --2- 172.21.15.150:0/3406179630 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f239806bbd0 0x7f239806e090 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f23ac151fe0 tx=0x7f23a0007c10 comp rx=0 tx=0).stop 2024-03-17T23:48:59.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.300+0000 7f23b16b5640 1 -- 172.21.15.150:0/3406179630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23ac099a70 msgr2=0x7f23ac150030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:59.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.300+0000 7f23b16b5640 1 --2- 172.21.15.150:0/3406179630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23ac099a70 0x7f23ac150030 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f239c009890 tx=0x7f239c0046c0 comp rx=0 tx=0).stop 2024-03-17T23:48:59.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.300+0000 7f23b16b5640 1 -- 172.21.15.150:0/3406179630 shutdown_connections 2024-03-17T23:48:59.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.300+0000 7f23b16b5640 1 --2- 172.21.15.150:0/3406179630 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f23ac15e1a0 0x7f23ac15b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:59.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.300+0000 7f23b16b5640 1 --2- 172.21.15.150:0/3406179630 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f239806bbd0 0x7f239806e090 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:59.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.300+0000 7f23b16b5640 1 --2- 172.21.15.150:0/3406179630 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f23ac0ae290 0x7f23ac09b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:59.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.300+0000 7f23b16b5640 1 --2- 172.21.15.150:0/3406179630 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23ac099a70 0x7f23ac150030 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:59.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.300+0000 7f23b16b5640 1 -- 172.21.15.150:0/3406179630 >> 172.21.15.150:0/3406179630 conn(0x7f23ac072a60 msgr2=0x7f23ac0b5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:59.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.300+0000 7f23b16b5640 1 -- 172.21.15.150:0/3406179630 shutdown_connections 2024-03-17T23:48:59.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.300+0000 7f23b16b5640 1 -- 172.21.15.150:0/3406179630 wait complete. 2024-03-17T23:48:59.302 INFO:tasks.ceph.mds.c.smithi150.stderr: -3895> 2024-03-17T23:48:15.142+0000 7f6c778f1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-17T23:48:59.313 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-17T23:48:59.333 INFO:tasks.ceph.mds.c.smithi150.stdout:starting mds.c at 2024-03-17T23:48:59.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.419+0000 7f2871a7e640 1 Processor -- start 2024-03-17T23:48:59.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.419+0000 7f2871a7e640 1 -- start start 2024-03-17T23:48:59.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.420+0000 7f2871a7e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f286c1514f0 0x7f286c15bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:59.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.420+0000 7f2871a7e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f286c0ce790 con 0x7f286c06ecd0 2024-03-17T23:48:59.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.420+0000 7f2871a7e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f286c15b050 con 0x7f286c15ab90 2024-03-17T23:48:59.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.420+0000 7f2871a7e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f286c15b330 con 0x7f286c1514f0 2024-03-17T23:48:59.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.420+0000 7f286bfff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f286c06ecd0 0x7f286c0cec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:48064/0 (socket says 172.21.15.150:48064) 2024-03-17T23:48:59.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.420+0000 7f286bfff640 1 -- 172.21.15.150:0/3793219218 learned_addr learned my addr 172.21.15.150:0/3793219218 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:48:59.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.420+0000 7f286b7fe640 1 --2- 172.21.15.150:0/3793219218 >> v2:172.21.15.157:3300/0 conn(0x7f286c1514f0 0x7f286c15bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:59.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.420+0000 7f286affd640 1 -- 172.21.15.150:0/3793219218 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 194561087 0 0) 0x7f2864001490 con 0x7f286c06ecd0 2024-03-17T23:48:59.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.420+0000 7f286affd640 1 -- 172.21.15.150:0/3793219218 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2858003520 con 0x7f286c06ecd0 2024-03-17T23:48:59.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.420+0000 7f286affd640 1 -- 172.21.15.150:0/3793219218 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2128869918 0 0) 0x7f2858003520 con 0x7f286c06ecd0 2024-03-17T23:48:59.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.420+0000 7f286affd640 1 -- 172.21.15.150:0/3793219218 >> v2:172.21.15.157:3300/0 conn(0x7f286c1514f0 msgr2=0x7f286c15bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:59.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.420+0000 7f286affd640 1 --2- 172.21.15.150:0/3793219218 >> v2:172.21.15.157:3300/0 conn(0x7f286c1514f0 0x7f286c15bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:59.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.420+0000 7f286affd640 1 -- 172.21.15.150:0/3793219218 >> v1:172.21.15.157:6789/0 conn(0x7f286c15ab90 legacy=0x7f286c06e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:59.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.420+0000 7f286affd640 1 -- 172.21.15.150:0/3793219218 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f286c15b610 con 0x7f286c06ecd0 2024-03-17T23:48:59.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.420+0000 7f286affd640 1 -- 172.21.15.150:0/3793219218 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f2864001bc0 con 0x7f286c06ecd0 2024-03-17T23:48:59.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.421+0000 7f286affd640 1 -- 172.21.15.150:0/3793219218 >> v1:172.21.15.150:6789/0 conn(0x7f286c06ecd0 legacy=0x7f286c0cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:59.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.421+0000 7f286affd640 1 --2- 172.21.15.150:0/3793219218 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f28580039b0 0x7f2858003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:59.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.421+0000 7f286affd640 1 --2- 172.21.15.150:0/3793219218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2858004800 0x7f2858004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:59.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.421+0000 7f286affd640 1 --2- 172.21.15.150:0/3793219218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28580051c0 0x7f285800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:59.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.421+0000 7f286affd640 1 -- 172.21.15.150:0/3793219218 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f286c15b330 con 0x7f2858004800 2024-03-17T23:48:59.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.421+0000 7f286affd640 1 -- 172.21.15.150:0/3793219218 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f285800c060 con 0x7f28580051c0 2024-03-17T23:48:59.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.421+0000 7f286affd640 1 -- 172.21.15.150:0/3793219218 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f285800c260 con 0x7f28580039b0 2024-03-17T23:48:59.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.421+0000 7f286b7fe640 1 --2- 172.21.15.150:0/3793219218 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f28580039b0 0x7f2858003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:59.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.421+0000 7f2862ffd640 1 --2- 172.21.15.150:0/3793219218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2858004800 0x7f2858004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:59.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.421+0000 7f286bfff640 1 --2- 172.21.15.150:0/3793219218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28580051c0 0x7f285800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:59.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.421+0000 7f2862ffd640 1 -- 172.21.15.150:0/3793219218 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f28580039b0 msgr2=0x7f2858003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:59.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.421+0000 7f2862ffd640 1 --2- 172.21.15.150:0/3793219218 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f28580039b0 0x7f2858003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:59.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.421+0000 7f2862ffd640 1 -- 172.21.15.150:0/3793219218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28580051c0 msgr2=0x7f285800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:48:59.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.421+0000 7f2862ffd640 1 --2- 172.21.15.150:0/3793219218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28580051c0 0x7f285800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:59.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.421+0000 7f2862ffd640 1 -- 172.21.15.150:0/3793219218 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2858003520 con 0x7f2858004800 2024-03-17T23:48:59.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.421+0000 7f286bfff640 1 -- 172.21.15.150:0/3793219218 reap_dead start 2024-03-17T23:48:59.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.422+0000 7f2862ffd640 1 --2- 172.21.15.150:0/3793219218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2858004800 0x7f2858004c80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f2854001720 tx=0x7f2858005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7e2855391b8150a server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:59.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.422+0000 7f286affd640 1 -- 172.21.15.150:0/3793219218 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2854001370 con 0x7f2858004800 2024-03-17T23:48:59.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.422+0000 7f286affd640 1 -- 172.21.15.150:0/3793219218 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28540052d0 con 0x7f2858004800 2024-03-17T23:48:59.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.422+0000 7f286affd640 1 -- 172.21.15.150:0/3793219218 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2854003830 con 0x7f2858004800 2024-03-17T23:48:59.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.422+0000 7f2871a7e640 1 -- 172.21.15.150:0/3793219218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2858004800 msgr2=0x7f2858004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:59.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.422+0000 7f2871a7e640 1 --2- 172.21.15.150:0/3793219218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2858004800 0x7f2858004c80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f2854001720 tx=0x7f2858005830 comp rx=0 tx=0).stop 2024-03-17T23:48:59.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.422+0000 7f2871a7e640 1 -- 172.21.15.150:0/3793219218 shutdown_connections 2024-03-17T23:48:59.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.422+0000 7f2871a7e640 1 --2- 172.21.15.150:0/3793219218 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2858004800 0x7f2858004c80 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:59.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.422+0000 7f2871a7e640 1 -- 172.21.15.150:0/3793219218 >> 172.21.15.150:0/3793219218 conn(0x7f286c06b190 msgr2=0x7f286c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:48:59.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.422+0000 7f2871a7e640 1 -- 172.21.15.150:0/3793219218 shutdown_connections 2024-03-17T23:48:59.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.423+0000 7f2871a7e640 1 -- 172.21.15.150:0/3793219218 wait complete. 2024-03-17T23:48:59.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.424+0000 7f2871a7e640 1 Processor -- start 2024-03-17T23:48:59.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.424+0000 7f2871a7e640 1 -- start start 2024-03-17T23:48:59.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.424+0000 7f2871a7e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2858004800 0x7f286c14dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:59.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.424+0000 7f2871a7e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f286c156bb0 0x7f286c0ad930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:59.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.425+0000 7f2871a7e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f286c0ade70 0x7f286c154da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:59.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.425+0000 7f2871a7e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f286c1518d0 con 0x7f286c156bb0 2024-03-17T23:48:59.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.425+0000 7f2871a7e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f286c0af510 con 0x7f2858004800 2024-03-17T23:48:59.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.425+0000 7f2871a7e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f286c151a50 con 0x7f286c0ade70 2024-03-17T23:48:59.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.425+0000 7f2862ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f286c156bb0 0x7f286c0ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:59.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.425+0000 7f2862ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f286c156bb0 0x7f286c0ad930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42234/0 (socket says 172.21.15.150:42234) 2024-03-17T23:48:59.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.425+0000 7f2862ffd640 1 -- 172.21.15.150:0/2376117684 learned_addr learned my addr 172.21.15.150:0/2376117684 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:48:59.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.425+0000 7f286bfff640 1 --2- 172.21.15.150:0/2376117684 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f286c0ade70 0x7f286c154da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:59.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.425+0000 7f286b7fe640 1 --2- 172.21.15.150:0/2376117684 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2858004800 0x7f286c14dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:59.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.425+0000 7f2862ffd640 1 -- 172.21.15.150:0/2376117684 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f286c0ade70 msgr2=0x7f286c154da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:59.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.425+0000 7f2862ffd640 1 --2- 172.21.15.150:0/2376117684 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f286c0ade70 0x7f286c154da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:59.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.425+0000 7f2862ffd640 1 -- 172.21.15.150:0/2376117684 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2858004800 msgr2=0x7f286c14dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:48:59.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.425+0000 7f2862ffd640 1 --2- 172.21.15.150:0/2376117684 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2858004800 0x7f286c14dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:48:59.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.425+0000 7f2862ffd640 1 -- 172.21.15.150:0/2376117684 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f286c151bd0 con 0x7f286c156bb0 2024-03-17T23:48:59.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.425+0000 7f2862ffd640 1 --2- 172.21.15.150:0/2376117684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f286c156bb0 0x7f286c0ad930 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2854000c00 tx=0x7f2854001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:59.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.425+0000 7f2868ff9640 1 -- 172.21.15.150:0/2376117684 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2854041020 con 0x7f286c156bb0 2024-03-17T23:48:59.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.425+0000 7f2868ff9640 1 -- 172.21.15.150:0/2376117684 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2854005480 con 0x7f286c156bb0 2024-03-17T23:48:59.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.425+0000 7f2871a7e640 1 -- 172.21.15.150:0/2376117684 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f286c151eb0 con 0x7f286c156bb0 2024-03-17T23:48:59.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.425+0000 7f2868ff9640 1 -- 172.21.15.150:0/2376117684 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2854007a30 con 0x7f286c156bb0 2024-03-17T23:48:59.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.425+0000 7f2871a7e640 1 -- 172.21.15.150:0/2376117684 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f286c152470 con 0x7f286c156bb0 2024-03-17T23:48:59.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.426+0000 7f2871a7e640 1 -- 172.21.15.150:0/2376117684 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f286c0681a0 con 0x7f286c156bb0 2024-03-17T23:48:59.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.427+0000 7f2868ff9640 1 -- 172.21.15.150:0/2376117684 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2854044020 con 0x7f286c156bb0 2024-03-17T23:48:59.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.427+0000 7f2868ff9640 1 --2- 172.21.15.150:0/2376117684 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f284c06bbd0 0x7f284c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:48:59.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.427+0000 7f2868ff9640 1 -- 172.21.15.150:0/2376117684 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6611+0+0 (secure 0 0 0) 0x7f28540ac6c0 con 0x7f286c156bb0 2024-03-17T23:48:59.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.427+0000 7f286b7fe640 1 --2- 172.21.15.150:0/2376117684 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f284c06bbd0 0x7f284c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:48:59.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.428+0000 7f286b7fe640 1 --2- 172.21.15.150:0/2376117684 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f284c06bbd0 0x7f284c06e090 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f285c009c40 tx=0x7f285c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:48:59.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.431+0000 7f2868ff9640 1 -- 172.21.15.150:0/2376117684 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f285404b030 con 0x7f286c156bb0 2024-03-17T23:48:59.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:48:59.641+0000 7f2871a7e640 1 -- 172.21.15.150:0/2376117684 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f286c14fc10 con 0x7f286c156bb0 2024-03-17T23:49:00.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.305+0000 7f2868ff9640 1 -- 172.21.15.150:0/2376117684 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v18) ==== 104+0+0 (secure 0 0 0) 0x7f285407ae20 con 0x7f286c156bb0 2024-03-17T23:49:00.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.307+0000 7f2871a7e640 1 -- 172.21.15.150:0/2376117684 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f284c06bbd0 msgr2=0x7f284c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:00.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.307+0000 7f2871a7e640 1 --2- 172.21.15.150:0/2376117684 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f284c06bbd0 0x7f284c06e090 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f285c009c40 tx=0x7f285c009190 comp rx=0 tx=0).stop 2024-03-17T23:49:00.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.307+0000 7f2871a7e640 1 -- 172.21.15.150:0/2376117684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f286c156bb0 msgr2=0x7f286c0ad930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:00.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.307+0000 7f2871a7e640 1 --2- 172.21.15.150:0/2376117684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f286c156bb0 0x7f286c0ad930 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2854000c00 tx=0x7f2854001120 comp rx=0 tx=0).stop 2024-03-17T23:49:00.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.307+0000 7f2871a7e640 1 -- 172.21.15.150:0/2376117684 shutdown_connections 2024-03-17T23:49:00.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.307+0000 7f2871a7e640 1 --2- 172.21.15.150:0/2376117684 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f286c0ade70 0x7f286c154da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:00.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.307+0000 7f2871a7e640 1 --2- 172.21.15.150:0/2376117684 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f286c156bb0 0x7f286c0ad930 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:00.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.307+0000 7f2871a7e640 1 --2- 172.21.15.150:0/2376117684 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f284c06bbd0 0x7f284c06e090 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:00.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.307+0000 7f2871a7e640 1 --2- 172.21.15.150:0/2376117684 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2858004800 0x7f286c14dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:00.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.307+0000 7f2871a7e640 1 -- 172.21.15.150:0/2376117684 >> 172.21.15.150:0/2376117684 conn(0x7f286c06b190 msgr2=0x7f286c0afa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:00.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.308+0000 7f2871a7e640 1 -- 172.21.15.150:0/2376117684 shutdown_connections 2024-03-17T23:49:00.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.308+0000 7f2871a7e640 1 -- 172.21.15.150:0/2376117684 wait complete. 2024-03-17T23:49:00.321 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-17T23:49:00.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.470+0000 7fa7619a9640 1 Processor -- start 2024-03-17T23:49:00.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.470+0000 7fa7619a9640 1 -- start start 2024-03-17T23:49:00.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.471+0000 7fa7619a9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa75c0ce9f0 0x7fa75c0cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:00.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.471+0000 7fa7619a9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa75c0d2330 0x7fa75c0d27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:00.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.471+0000 7fa7619a9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa75c0d0560 con 0x7fa75c15e030 2024-03-17T23:49:00.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.471+0000 7fa7619a9640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fa75c0d0820 con 0x7fa75c0ce9f0 2024-03-17T23:49:00.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.471+0000 7fa7619a9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa75c0d0b00 con 0x7fa75c0d2330 2024-03-17T23:49:00.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.471+0000 7fa75b7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa75c0d2330 0x7fa75c0d27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:00.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.471+0000 7fa753fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa75c0ce9f0 0x7fa75c0cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:00.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.471+0000 7fa753fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa75c0ce9f0 0x7fa75c0cee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:35746/0 (socket says 172.21.15.150:35746) 2024-03-17T23:49:00.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.471+0000 7fa75affd640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fa75c15e030 0x7fa75c162420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:36906/0 (socket says 172.21.15.150:36906) 2024-03-17T23:49:00.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.471+0000 7fa75b7fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa75c0d2330 0x7fa75c0d27c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:43236/0 (socket says 172.21.15.150:43236) 2024-03-17T23:49:00.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.471+0000 7fa753fff640 1 -- 172.21.15.150:0/3777657895 learned_addr learned my addr 172.21.15.150:0/3777657895 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:00.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.471+0000 7fa75a7fc640 1 -- 172.21.15.150:0/3777657895 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1786924992 0 0) 0x7fa75c0d0560 con 0x7fa75c15e030 2024-03-17T23:49:00.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.471+0000 7fa75a7fc640 1 -- 172.21.15.150:0/3777657895 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa748003520 con 0x7fa75c15e030 2024-03-17T23:49:00.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.471+0000 7fa75b7fe640 1 -- 172.21.15.150:0/3777657895 >> v2:172.21.15.157:3300/0 conn(0x7fa75c0ce9f0 msgr2=0x7fa75c0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:00.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.471+0000 7fa75b7fe640 1 --2- 172.21.15.150:0/3777657895 >> v2:172.21.15.157:3300/0 conn(0x7fa75c0ce9f0 0x7fa75c0cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:00.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.471+0000 7fa75b7fe640 1 -- 172.21.15.150:0/3777657895 >> v1:172.21.15.157:6789/0 conn(0x7fa75c15e030 legacy=0x7fa75c162420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:00.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.471+0000 7fa75b7fe640 1 -- 172.21.15.150:0/3777657895 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa75c0d0de0 con 0x7fa75c0d2330 2024-03-17T23:49:00.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.471+0000 7fa753fff640 1 --2- 172.21.15.150:0/3777657895 >> v2:172.21.15.157:3300/0 conn(0x7fa75c0ce9f0 0x7fa75c0cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:00.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.472+0000 7fa75b7fe640 1 --2- 172.21.15.150:0/3777657895 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa75c0d2330 0x7fa75c0d27c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa754009d90 tx=0x7fa75402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7046acc71ed981ef server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:00.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.472+0000 7fa75a7fc640 1 -- 172.21.15.150:0/3777657895 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa754039070 con 0x7fa75c0d2330 2024-03-17T23:49:00.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.472+0000 7fa75a7fc640 1 -- 172.21.15.150:0/3777657895 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa75402cc50 con 0x7fa75c0d2330 2024-03-17T23:49:00.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.472+0000 7fa75a7fc640 1 -- 172.21.15.150:0/3777657895 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa754034440 con 0x7fa75c0d2330 2024-03-17T23:49:00.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.473+0000 7fa7619a9640 1 -- 172.21.15.150:0/3777657895 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa75c0d2330 msgr2=0x7fa75c0d27c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:00.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.473+0000 7fa7619a9640 1 --2- 172.21.15.150:0/3777657895 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa75c0d2330 0x7fa75c0d27c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa754009d90 tx=0x7fa75402bea0 comp rx=0 tx=0).stop 2024-03-17T23:49:00.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.473+0000 7fa7619a9640 1 -- 172.21.15.150:0/3777657895 shutdown_connections 2024-03-17T23:49:00.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.473+0000 7fa7619a9640 1 --2- 172.21.15.150:0/3777657895 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa75c0d2330 0x7fa75c0d27c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:00.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.473+0000 7fa7619a9640 1 --2- 172.21.15.150:0/3777657895 >> v2:172.21.15.157:3300/0 conn(0x7fa75c0ce9f0 0x7fa75c0cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:00.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.473+0000 7fa7619a9640 1 -- 172.21.15.150:0/3777657895 >> 172.21.15.150:0/3777657895 conn(0x7fa75c06ca10 msgr2=0x7fa75c06ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:00.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.473+0000 7fa7619a9640 1 -- 172.21.15.150:0/3777657895 shutdown_connections 2024-03-17T23:49:00.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.473+0000 7fa7619a9640 1 -- 172.21.15.150:0/3777657895 wait complete. 2024-03-17T23:49:00.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.474+0000 7fa7619a9640 1 Processor -- start 2024-03-17T23:49:00.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.474+0000 7fa7619a9640 1 -- start start 2024-03-17T23:49:00.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.475+0000 7fa7619a9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa75c0ce9f0 0x7fa75c14c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:00.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.475+0000 7fa7619a9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa75c0d2330 0x7fa75c0c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:00.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.475+0000 7fa7619a9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa75c15e030 0x7fa75c0c8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:00.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.475+0000 7fa7619a9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c0d38e0 con 0x7fa75c0d2330 2024-03-17T23:49:00.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.475+0000 7fa7619a9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa75c0d13a0 con 0x7fa75c0ce9f0 2024-03-17T23:49:00.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.475+0000 7fa7619a9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa75c0d3a60 con 0x7fa75c15e030 2024-03-17T23:49:00.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.475+0000 7fa753fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa75c0d2330 0x7fa75c0c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:00.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.475+0000 7fa753fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa75c0d2330 0x7fa75c0c88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42240/0 (socket says 172.21.15.150:42240) 2024-03-17T23:49:00.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.475+0000 7fa753fff640 1 -- 172.21.15.150:0/4019369372 learned_addr learned my addr 172.21.15.150:0/4019369372 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:00.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.475+0000 7fa75b7fe640 1 --2- 172.21.15.150:0/4019369372 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa75c15e030 0x7fa75c0c8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:00.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.475+0000 7fa75affd640 1 --2- 172.21.15.150:0/4019369372 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa75c0ce9f0 0x7fa75c14c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:00.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.475+0000 7fa753fff640 1 -- 172.21.15.150:0/4019369372 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa75c15e030 msgr2=0x7fa75c0c8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:00.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.475+0000 7fa753fff640 1 --2- 172.21.15.150:0/4019369372 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa75c15e030 0x7fa75c0c8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:00.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.475+0000 7fa753fff640 1 -- 172.21.15.150:0/4019369372 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa75c0ce9f0 msgr2=0x7fa75c14c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:00.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.475+0000 7fa753fff640 1 --2- 172.21.15.150:0/4019369372 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa75c0ce9f0 0x7fa75c14c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:00.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.475+0000 7fa753fff640 1 -- 172.21.15.150:0/4019369372 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa75c14db20 con 0x7fa75c0d2330 2024-03-17T23:49:00.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.476+0000 7fa753fff640 1 --2- 172.21.15.150:0/4019369372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa75c0d2330 0x7fa75c0c88b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fa74400e9b0 tx=0x7fa74400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:00.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.476+0000 7fa758ff9640 1 -- 172.21.15.150:0/4019369372 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa744017070 con 0x7fa75c0d2330 2024-03-17T23:49:00.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.476+0000 7fa758ff9640 1 -- 172.21.15.150:0/4019369372 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa74400c9a0 con 0x7fa75c0d2330 2024-03-17T23:49:00.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.476+0000 7fa758ff9640 1 -- 172.21.15.150:0/4019369372 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa74400cd50 con 0x7fa75c0d2330 2024-03-17T23:49:00.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.476+0000 7fa7619a9640 1 -- 172.21.15.150:0/4019369372 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa75c14d280 con 0x7fa75c0d2330 2024-03-17T23:49:00.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.476+0000 7fa7619a9640 1 -- 172.21.15.150:0/4019369372 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa75c14d840 con 0x7fa75c0d2330 2024-03-17T23:49:00.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.477+0000 7fa7619a9640 1 -- 172.21.15.150:0/4019369372 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa75c0681a0 con 0x7fa75c0d2330 2024-03-17T23:49:00.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.477+0000 7fa758ff9640 1 -- 172.21.15.150:0/4019369372 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa74401a070 con 0x7fa75c0d2330 2024-03-17T23:49:00.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.478+0000 7fa758ff9640 1 --2- 172.21.15.150:0/4019369372 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa73c06bbd0 0x7fa73c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:00.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.478+0000 7fa758ff9640 1 -- 172.21.15.150:0/4019369372 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6611+0+0 (secure 0 0 0) 0x7fa74408ac60 con 0x7fa75c0d2330 2024-03-17T23:49:00.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.478+0000 7fa75affd640 1 --2- 172.21.15.150:0/4019369372 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa73c06bbd0 0x7fa73c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:00.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.478+0000 7fa75affd640 1 --2- 172.21.15.150:0/4019369372 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa73c06bbd0 0x7fa73c06e090 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa74c000f60 tx=0x7fa74c007cf0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:00.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.482+0000 7fa758ff9640 1 -- 172.21.15.150:0/4019369372 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa744058d80 con 0x7fa75c0d2330 2024-03-17T23:49:00.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:00.654+0000 7fa7619a9640 1 -- 172.21.15.150:0/4019369372 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa75c0d19c0 con 0x7fa75c0d2330 2024-03-17T23:49:01.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.306+0000 7fa758ff9640 1 -- 172.21.15.150:0/4019369372 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v47) ==== 181+0+0 (secure 0 0 0) 0x7fa74405ce20 con 0x7fa75c0d2330 2024-03-17T23:49:01.307 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_metadata' removed 2024-03-17T23:49:01.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.310+0000 7fa7619a9640 1 -- 172.21.15.150:0/4019369372 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa73c06bbd0 msgr2=0x7fa73c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:01.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.310+0000 7fa7619a9640 1 --2- 172.21.15.150:0/4019369372 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa73c06bbd0 0x7fa73c06e090 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa74c000f60 tx=0x7fa74c007cf0 comp rx=0 tx=0).stop 2024-03-17T23:49:01.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.310+0000 7fa7619a9640 1 -- 172.21.15.150:0/4019369372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa75c0d2330 msgr2=0x7fa75c0c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:01.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.310+0000 7fa7619a9640 1 --2- 172.21.15.150:0/4019369372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa75c0d2330 0x7fa75c0c88b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fa74400e9b0 tx=0x7fa74400ee70 comp rx=0 tx=0).stop 2024-03-17T23:49:01.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.310+0000 7fa7619a9640 1 -- 172.21.15.150:0/4019369372 shutdown_connections 2024-03-17T23:49:01.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.310+0000 7fa7619a9640 1 --2- 172.21.15.150:0/4019369372 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa75c15e030 0x7fa75c0c8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:01.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.310+0000 7fa7619a9640 1 --2- 172.21.15.150:0/4019369372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa75c0d2330 0x7fa75c0c88b0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:01.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.310+0000 7fa7619a9640 1 --2- 172.21.15.150:0/4019369372 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa73c06bbd0 0x7fa73c06e090 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:01.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.310+0000 7fa7619a9640 1 --2- 172.21.15.150:0/4019369372 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa75c0ce9f0 0x7fa75c14c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:01.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.310+0000 7fa7619a9640 1 -- 172.21.15.150:0/4019369372 >> 172.21.15.150:0/4019369372 conn(0x7fa75c06ca10 msgr2=0x7fa75c0c9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:01.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.310+0000 7fa7619a9640 1 -- 172.21.15.150:0/4019369372 shutdown_connections 2024-03-17T23:49:01.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.310+0000 7fa7619a9640 1 -- 172.21.15.150:0/4019369372 wait complete. 2024-03-17T23:49:01.325 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-17T23:49:01.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.434+0000 7f6e59ec6640 1 Processor -- start 2024-03-17T23:49:01.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.434+0000 7f6e59ec6640 1 -- start start 2024-03-17T23:49:01.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.435+0000 7f6e59ec6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6e54099db0 0x7f6e5409a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:01.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.435+0000 7f6e59ec6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6e540b0c50 0x7f6e540cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:01.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.435+0000 7f6e59ec6640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6e540d0d40 0x7f6e540d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:01.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.435+0000 7f6e59ec6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f6e5409a6d0 con 0x7f6e54099db0 2024-03-17T23:49:01.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.435+0000 7f6e59ec6640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f6e540cff80 con 0x7f6e540d0d40 2024-03-17T23:49:01.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.435+0000 7f6e59ec6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6e540d0260 con 0x7f6e540b0c50 2024-03-17T23:49:01.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.435+0000 7f6e537fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6e54099db0 0x7f6e5409a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:01.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.435+0000 7f6e537fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6e54099db0 0x7f6e5409a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42248/0 (socket says 172.21.15.150:42248) 2024-03-17T23:49:01.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.435+0000 7f6e537fe640 1 -- 172.21.15.150:0/1457182567 learned_addr learned my addr 172.21.15.150:0/1457182567 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:01.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.436+0000 7f6e4affd640 1 --2- 172.21.15.150:0/1457182567 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6e540b0c50 0x7f6e540cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:01.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.436+0000 7f6e53fff640 1 --2- 172.21.15.150:0/1457182567 >> v2:172.21.15.157:3300/0 conn(0x7f6e540d0d40 0x7f6e540d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:01.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.436+0000 7f6e537fe640 1 -- 172.21.15.150:0/1457182567 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6e540b0c50 msgr2=0x7f6e540cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:01.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.436+0000 7f6e537fe640 1 --2- 172.21.15.150:0/1457182567 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6e540b0c50 0x7f6e540cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:01.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.436+0000 7f6e537fe640 1 -- 172.21.15.150:0/1457182567 >> v2:172.21.15.157:3300/0 conn(0x7f6e540d0d40 msgr2=0x7f6e540d11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:01.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.436+0000 7f6e537fe640 1 --2- 172.21.15.150:0/1457182567 >> v2:172.21.15.157:3300/0 conn(0x7f6e540d0d40 0x7f6e540d11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:01.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.436+0000 7f6e537fe640 1 -- 172.21.15.150:0/1457182567 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e540d0540 con 0x7f6e54099db0 2024-03-17T23:49:01.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.436+0000 7f6e537fe640 1 --2- 172.21.15.150:0/1457182567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e54099db0 0x7f6e5409a190 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f6e540b4220 tx=0x7f6e4402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8bcd56be7ceaf7e server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:01.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.437+0000 7f6e52ffd640 1 -- 172.21.15.150:0/1457182567 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e44035020 con 0x7f6e54099db0 2024-03-17T23:49:01.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.437+0000 7f6e52ffd640 1 -- 172.21.15.150:0/1457182567 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e44031870 con 0x7f6e54099db0 2024-03-17T23:49:01.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.437+0000 7f6e52ffd640 1 -- 172.21.15.150:0/1457182567 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e44031c40 con 0x7f6e54099db0 2024-03-17T23:49:01.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.437+0000 7f6e59ec6640 1 -- 172.21.15.150:0/1457182567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e54099db0 msgr2=0x7f6e5409a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:01.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.437+0000 7f6e59ec6640 1 --2- 172.21.15.150:0/1457182567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e54099db0 0x7f6e5409a190 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f6e540b4220 tx=0x7f6e4402b910 comp rx=0 tx=0).stop 2024-03-17T23:49:01.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.437+0000 7f6e59ec6640 1 -- 172.21.15.150:0/1457182567 shutdown_connections 2024-03-17T23:49:01.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.437+0000 7f6e59ec6640 1 --2- 172.21.15.150:0/1457182567 >> v2:172.21.15.157:3300/0 conn(0x7f6e540d0d40 0x7f6e540d11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:01.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.437+0000 7f6e59ec6640 1 --2- 172.21.15.150:0/1457182567 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6e540b0c50 0x7f6e540cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:01.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.437+0000 7f6e59ec6640 1 --2- 172.21.15.150:0/1457182567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e54099db0 0x7f6e5409a190 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:01.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.437+0000 7f6e59ec6640 1 -- 172.21.15.150:0/1457182567 >> 172.21.15.150:0/1457182567 conn(0x7f6e540b3600 msgr2=0x7f6e540b3a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:01.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.437+0000 7f6e59ec6640 1 -- 172.21.15.150:0/1457182567 shutdown_connections 2024-03-17T23:49:01.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.437+0000 7f6e59ec6640 1 -- 172.21.15.150:0/1457182567 wait complete. 2024-03-17T23:49:01.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.438+0000 7f6e59ec6640 1 Processor -- start 2024-03-17T23:49:01.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.439+0000 7f6e59ec6640 1 -- start start 2024-03-17T23:49:01.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.439+0000 7f6e59ec6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6e54099db0 0x7f6e540b5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:01.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.439+0000 7f6e59ec6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6e540b0c50 0x7f6e541592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:01.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.439+0000 7f6e59ec6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e540d0d40 0x7f6e541597e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:01.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.439+0000 7f6e59ec6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6e540d1710 con 0x7f6e540d0d40 2024-03-17T23:49:01.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.439+0000 7f6e59ec6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5406c920 con 0x7f6e540b0c50 2024-03-17T23:49:01.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.439+0000 7f6e59ec6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6e540cc6a0 con 0x7f6e54099db0 2024-03-17T23:49:01.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.439+0000 7f6e53fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e540d0d40 0x7f6e541597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:01.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.439+0000 7f6e53fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e540d0d40 0x7f6e541597e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42256/0 (socket says 172.21.15.150:42256) 2024-03-17T23:49:01.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.439+0000 7f6e53fff640 1 -- 172.21.15.150:0/1028574606 learned_addr learned my addr 172.21.15.150:0/1028574606 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:01.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.439+0000 7f6e537fe640 1 --2- 172.21.15.150:0/1028574606 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6e54099db0 0x7f6e540b5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:01.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.440+0000 7f6e4affd640 1 --2- 172.21.15.150:0/1028574606 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6e540b0c50 0x7f6e541592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:01.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.440+0000 7f6e53fff640 1 -- 172.21.15.150:0/1028574606 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6e54099db0 msgr2=0x7f6e540b5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:01.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.440+0000 7f6e53fff640 1 --2- 172.21.15.150:0/1028574606 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6e54099db0 0x7f6e540b5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:01.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.440+0000 7f6e53fff640 1 -- 172.21.15.150:0/1028574606 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6e540b0c50 msgr2=0x7f6e541592a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:01.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.440+0000 7f6e53fff640 1 --2- 172.21.15.150:0/1028574606 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6e540b0c50 0x7f6e541592a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:01.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.440+0000 7f6e53fff640 1 -- 172.21.15.150:0/1028574606 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e540b5800 con 0x7f6e540d0d40 2024-03-17T23:49:01.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.440+0000 7f6e53fff640 1 --2- 172.21.15.150:0/1028574606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e540d0d40 0x7f6e541597e0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c007e60 tx=0x7f6e4c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:01.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.440+0000 7f6e50ff9640 1 -- 172.21.15.150:0/1028574606 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c014420 con 0x7f6e540d0d40 2024-03-17T23:49:01.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.440+0000 7f6e50ff9640 1 -- 172.21.15.150:0/1028574606 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e4c019070 con 0x7f6e540d0d40 2024-03-17T23:49:01.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.440+0000 7f6e59ec6640 1 -- 172.21.15.150:0/1028574606 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e540b5ae0 con 0x7f6e540d0d40 2024-03-17T23:49:01.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.440+0000 7f6e50ff9640 1 -- 172.21.15.150:0/1028574606 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c014820 con 0x7f6e540d0d40 2024-03-17T23:49:01.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.440+0000 7f6e59ec6640 1 -- 172.21.15.150:0/1028574606 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e540cb9d0 con 0x7f6e540d0d40 2024-03-17T23:49:01.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.441+0000 7f6e59ec6640 1 -- 172.21.15.150:0/1028574606 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e540688b0 con 0x7f6e540d0d40 2024-03-17T23:49:01.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.441+0000 7f6e50ff9640 1 -- 172.21.15.150:0/1028574606 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6e4c0149c0 con 0x7f6e540d0d40 2024-03-17T23:49:01.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.442+0000 7f6e50ff9640 1 --2- 172.21.15.150:0/1028574606 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6e3406bbd0 0x7f6e3406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:01.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.442+0000 7f6e50ff9640 1 -- 172.21.15.150:0/1028574606 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 6173+0+0 (secure 0 0 0) 0x7f6e4c08b3b0 con 0x7f6e540d0d40 2024-03-17T23:49:01.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.442+0000 7f6e537fe640 1 --2- 172.21.15.150:0/1028574606 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6e3406bbd0 0x7f6e3406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:01.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.442+0000 7f6e537fe640 1 --2- 172.21.15.150:0/1028574606 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6e3406bbd0 0x7f6e3406e090 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6e44036000 tx=0x7f6e4403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:01.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.446+0000 7f6e50ff9640 1 -- 172.21.15.150:0/1028574606 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6e4c059d60 con 0x7f6e540d0d40 2024-03-17T23:49:01.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:01.618+0000 7f6e59ec6640 1 -- 172.21.15.150:0/1028574606 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6e540cc980 con 0x7f6e540d0d40 2024-03-17T23:49:02.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.311+0000 7f6e50ff9640 1 -- 172.21.15.150:0/1028574606 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v48) ==== 169+0+0 (secure 0 0 0) 0x7f6e4c059660 con 0x7f6e540d0d40 2024-03-17T23:49:02.311 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_data' removed 2024-03-17T23:49:02.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.314+0000 7f6e59ec6640 1 -- 172.21.15.150:0/1028574606 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6e3406bbd0 msgr2=0x7f6e3406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:02.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.314+0000 7f6e59ec6640 1 --2- 172.21.15.150:0/1028574606 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6e3406bbd0 0x7f6e3406e090 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6e44036000 tx=0x7f6e4403c040 comp rx=0 tx=0).stop 2024-03-17T23:49:02.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.314+0000 7f6e59ec6640 1 -- 172.21.15.150:0/1028574606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e540d0d40 msgr2=0x7f6e541597e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:02.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.314+0000 7f6e59ec6640 1 --2- 172.21.15.150:0/1028574606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e540d0d40 0x7f6e541597e0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c007e60 tx=0x7f6e4c00e4b0 comp rx=0 tx=0).stop 2024-03-17T23:49:02.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.314+0000 7f6e59ec6640 1 -- 172.21.15.150:0/1028574606 shutdown_connections 2024-03-17T23:49:02.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.314+0000 7f6e59ec6640 1 --2- 172.21.15.150:0/1028574606 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6e540d0d40 0x7f6e541597e0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:02.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.314+0000 7f6e59ec6640 1 --2- 172.21.15.150:0/1028574606 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6e3406bbd0 0x7f6e3406e090 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:02.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.314+0000 7f6e59ec6640 1 --2- 172.21.15.150:0/1028574606 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6e540b0c50 0x7f6e541592a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:02.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.314+0000 7f6e59ec6640 1 --2- 172.21.15.150:0/1028574606 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6e54099db0 0x7f6e540b5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:02.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.314+0000 7f6e59ec6640 1 -- 172.21.15.150:0/1028574606 >> 172.21.15.150:0/1028574606 conn(0x7f6e540b3600 msgr2=0x7f6e540cbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:02.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.315+0000 7f6e59ec6640 1 -- 172.21.15.150:0/1028574606 shutdown_connections 2024-03-17T23:49:02.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.315+0000 7f6e59ec6640 1 -- 172.21.15.150:0/1028574606 wait complete. 2024-03-17T23:49:02.329 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm data_new data_new --yes-i-really-really-mean-it 2024-03-17T23:49:02.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.437+0000 7fa6a6e27640 1 Processor -- start 2024-03-17T23:49:02.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.438+0000 7fa6a6e27640 1 -- start start 2024-03-17T23:49:02.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.438+0000 7fa6a6e27640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa6a0072780 0x7fa6a00cee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:02.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.438+0000 7fa6a6e27640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa6a00cf3c0 0x7fa6a00b3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:02.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.438+0000 7fa6a6e27640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6a015aff0 con 0x7fa6a00cd4d0 2024-03-17T23:49:02.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.438+0000 7fa6a6e27640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fa6a015b260 con 0x7fa6a0072780 2024-03-17T23:49:02.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.438+0000 7fa6a6e27640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fa6a015b540 con 0x7fa6a00cf3c0 2024-03-17T23:49:02.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.438+0000 7fa69ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa6a0072780 0x7fa6a00cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:02.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.438+0000 7fa6a4bc3640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa6a00cd4d0 0x7fa6a0072070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:48074/0 (socket says 172.21.15.150:48074) 2024-03-17T23:49:02.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.438+0000 7fa6a4bc3640 1 -- 172.21.15.150:0/1185420175 learned_addr learned my addr 172.21.15.150:0/1185420175 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:02.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.439+0000 7fa6a53c4640 1 --2- 172.21.15.150:0/1185420175 >> v2:172.21.15.157:3300/0 conn(0x7fa6a00cf3c0 0x7fa6a00b3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:02.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.439+0000 7fa69f7fe640 1 -- 172.21.15.150:0/1185420175 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2062246987 0 0) 0x7fa6a015aff0 con 0x7fa6a00cd4d0 2024-03-17T23:49:02.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.439+0000 7fa69f7fe640 1 -- 172.21.15.150:0/1185420175 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa690003520 con 0x7fa6a00cd4d0 2024-03-17T23:49:02.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.439+0000 7fa69ffff640 1 -- 172.21.15.150:0/1185420175 >> v2:172.21.15.157:3300/0 conn(0x7fa6a00cf3c0 msgr2=0x7fa6a00b3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:02.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.439+0000 7fa69ffff640 1 --2- 172.21.15.150:0/1185420175 >> v2:172.21.15.157:3300/0 conn(0x7fa6a00cf3c0 0x7fa6a00b3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:02.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.439+0000 7fa69ffff640 1 -- 172.21.15.150:0/1185420175 >> v1:172.21.15.150:6789/0 conn(0x7fa6a00cd4d0 legacy=0x7fa6a0072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:02.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.439+0000 7fa69ffff640 1 -- 172.21.15.150:0/1185420175 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6a00cf7f0 con 0x7fa6a0072780 2024-03-17T23:49:02.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.440+0000 7fa69ffff640 1 --2- 172.21.15.150:0/1185420175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6a0072780 0x7fa6a00cee80 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fa6a00cc2a0 tx=0x7fa68c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e17c6b1dd8ce5f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:02.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.440+0000 7fa69f7fe640 1 -- 172.21.15.150:0/1185420175 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa68c035020 con 0x7fa6a0072780 2024-03-17T23:49:02.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.440+0000 7fa69f7fe640 1 -- 172.21.15.150:0/1185420175 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa68c031830 con 0x7fa6a0072780 2024-03-17T23:49:02.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.440+0000 7fa69f7fe640 1 -- 172.21.15.150:0/1185420175 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa68c031c00 con 0x7fa6a0072780 2024-03-17T23:49:02.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.440+0000 7fa6a6e27640 1 -- 172.21.15.150:0/1185420175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6a0072780 msgr2=0x7fa6a00cee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:02.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.440+0000 7fa6a6e27640 1 --2- 172.21.15.150:0/1185420175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6a0072780 0x7fa6a00cee80 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fa6a00cc2a0 tx=0x7fa68c02b8d0 comp rx=0 tx=0).stop 2024-03-17T23:49:02.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.440+0000 7fa6a6e27640 1 -- 172.21.15.150:0/1185420175 shutdown_connections 2024-03-17T23:49:02.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.440+0000 7fa6a6e27640 1 --2- 172.21.15.150:0/1185420175 >> v2:172.21.15.157:3300/0 conn(0x7fa6a00cf3c0 0x7fa6a00b3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:02.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.440+0000 7fa6a6e27640 1 --2- 172.21.15.150:0/1185420175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6a0072780 0x7fa6a00cee80 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:02.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.440+0000 7fa6a6e27640 1 -- 172.21.15.150:0/1185420175 >> 172.21.15.150:0/1185420175 conn(0x7fa6a00b1be0 msgr2=0x7fa6a00b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:02.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.440+0000 7fa6a6e27640 1 -- 172.21.15.150:0/1185420175 shutdown_connections 2024-03-17T23:49:02.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.440+0000 7fa6a6e27640 1 -- 172.21.15.150:0/1185420175 wait complete. 2024-03-17T23:49:02.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.441+0000 7fa6a6e27640 1 Processor -- start 2024-03-17T23:49:02.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.442+0000 7fa6a6e27640 1 -- start start 2024-03-17T23:49:02.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.442+0000 7fa6a6e27640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6a0072780 0x7fa6a0149980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:02.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.442+0000 7fa6a6e27640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6a00cd4d0 0x7fa6a006d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:02.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.442+0000 7fa6a6e27640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6a00cf3c0 0x7fa6a006df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:02.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.442+0000 7fa6a6e27640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a015bde0 con 0x7fa6a00cf3c0 2024-03-17T23:49:02.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.442+0000 7fa6a6e27640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a015bb00 con 0x7fa6a0072780 2024-03-17T23:49:02.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.442+0000 7fa6a6e27640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa6a00cd8b0 con 0x7fa6a00cd4d0 2024-03-17T23:49:02.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.443+0000 7fa6a53c4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6a00cf3c0 0x7fa6a006df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:02.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.443+0000 7fa6a53c4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6a00cf3c0 0x7fa6a006df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42274/0 (socket says 172.21.15.150:42274) 2024-03-17T23:49:02.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.443+0000 7fa6a53c4640 1 -- 172.21.15.150:0/2148668436 learned_addr learned my addr 172.21.15.150:0/2148668436 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:02.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.443+0000 7fa6a4bc3640 1 --2- 172.21.15.150:0/2148668436 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6a0072780 0x7fa6a0149980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:02.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.443+0000 7fa69ffff640 1 --2- 172.21.15.150:0/2148668436 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6a00cd4d0 0x7fa6a006d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:02.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.443+0000 7fa6a53c4640 1 -- 172.21.15.150:0/2148668436 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6a00cd4d0 msgr2=0x7fa6a006d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:02.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.443+0000 7fa6a53c4640 1 --2- 172.21.15.150:0/2148668436 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6a00cd4d0 0x7fa6a006d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:02.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.443+0000 7fa6a53c4640 1 -- 172.21.15.150:0/2148668436 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6a0072780 msgr2=0x7fa6a0149980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:02.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.443+0000 7fa6a53c4640 1 --2- 172.21.15.150:0/2148668436 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6a0072780 0x7fa6a0149980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:02.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.443+0000 7fa6a53c4640 1 -- 172.21.15.150:0/2148668436 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6a0149ec0 con 0x7fa6a00cf3c0 2024-03-17T23:49:02.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.443+0000 7fa69ffff640 1 --2- 172.21.15.150:0/2148668436 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6a00cd4d0 0x7fa6a006d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:49:02.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.443+0000 7fa6a53c4640 1 --2- 172.21.15.150:0/2148668436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6a00cf3c0 0x7fa6a006df10 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fa69800efc0 tx=0x7fa69800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:02.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.443+0000 7fa69d7fa640 1 -- 172.21.15.150:0/2148668436 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa698018020 con 0x7fa6a00cf3c0 2024-03-17T23:49:02.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.443+0000 7fa69d7fa640 1 -- 172.21.15.150:0/2148668436 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa698014480 con 0x7fa6a00cf3c0 2024-03-17T23:49:02.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.443+0000 7fa6a6e27640 1 -- 172.21.15.150:0/2148668436 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6a01f7e00 con 0x7fa6a00cf3c0 2024-03-17T23:49:02.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.443+0000 7fa69d7fa640 1 -- 172.21.15.150:0/2148668436 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa698014870 con 0x7fa6a00cf3c0 2024-03-17T23:49:02.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.443+0000 7fa6a6e27640 1 -- 172.21.15.150:0/2148668436 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6a014a410 con 0x7fa6a00cf3c0 2024-03-17T23:49:02.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.444+0000 7fa6a6e27640 1 -- 172.21.15.150:0/2148668436 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6a00681a0 con 0x7fa6a00cf3c0 2024-03-17T23:49:02.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.445+0000 7fa69d7fa640 1 -- 172.21.15.150:0/2148668436 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa698005d10 con 0x7fa6a00cf3c0 2024-03-17T23:49:02.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.445+0000 7fa69d7fa640 1 --2- 172.21.15.150:0/2148668436 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa68406bc00 0x7fa68406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:02.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.445+0000 7fa69d7fa640 1 -- 172.21.15.150:0/2148668436 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5730+0+0 (secure 0 0 0) 0x7fa69808b430 con 0x7fa6a00cf3c0 2024-03-17T23:49:02.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.445+0000 7fa6a4bc3640 1 --2- 172.21.15.150:0/2148668436 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa68406bc00 0x7fa68406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:02.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.446+0000 7fa6a4bc3640 1 --2- 172.21.15.150:0/2148668436 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa68406bc00 0x7fa68406e0c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa694005880 tx=0x7fa694009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:02.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.450+0000 7fa69d7fa640 1 -- 172.21.15.150:0/2148668436 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa698022070 con 0x7fa6a00cf3c0 2024-03-17T23:49:02.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:02.623+0000 7fa6a6e27640 1 -- 172.21.15.150:0/2148668436 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "data_new", "pool2": "data_new", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa6a00cc090 con 0x7fa6a00cf3c0 2024-03-17T23:49:03.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.312+0000 7fa69d7fa640 1 -- 172.21.15.150:0/2148668436 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "data_new", "pool2": "data_new", "yes_i_really_really_mean_it": true}]=0 pool 'data_new' removed v49) ==== 160+0+0 (secure 0 0 0) 0x7fa698010040 con 0x7fa6a00cf3c0 2024-03-17T23:49:03.313 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'data_new' removed 2024-03-17T23:49:03.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.315+0000 7fa6a6e27640 1 -- 172.21.15.150:0/2148668436 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa68406bc00 msgr2=0x7fa68406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:03.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.315+0000 7fa6a6e27640 1 --2- 172.21.15.150:0/2148668436 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa68406bc00 0x7fa68406e0c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa694005880 tx=0x7fa694009040 comp rx=0 tx=0).stop 2024-03-17T23:49:03.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.315+0000 7fa6a6e27640 1 -- 172.21.15.150:0/2148668436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6a00cf3c0 msgr2=0x7fa6a006df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:03.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.315+0000 7fa6a6e27640 1 --2- 172.21.15.150:0/2148668436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6a00cf3c0 0x7fa6a006df10 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fa69800efc0 tx=0x7fa69800c490 comp rx=0 tx=0).stop 2024-03-17T23:49:03.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.316+0000 7fa6a6e27640 1 -- 172.21.15.150:0/2148668436 shutdown_connections 2024-03-17T23:49:03.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.316+0000 7fa6a6e27640 1 --2- 172.21.15.150:0/2148668436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6a00cf3c0 0x7fa6a006df10 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:03.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.316+0000 7fa6a6e27640 1 --2- 172.21.15.150:0/2148668436 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa6a00cd4d0 0x7fa6a006d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:03.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.316+0000 7fa6a6e27640 1 --2- 172.21.15.150:0/2148668436 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa68406bc00 0x7fa68406e0c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:03.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.316+0000 7fa6a6e27640 1 --2- 172.21.15.150:0/2148668436 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6a0072780 0x7fa6a0149980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:03.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.316+0000 7fa6a6e27640 1 -- 172.21.15.150:0/2148668436 >> 172.21.15.150:0/2148668436 conn(0x7fa6a00b1be0 msgr2=0x7fa6a00b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:03.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.316+0000 7fa6a6e27640 1 -- 172.21.15.150:0/2148668436 shutdown_connections 2024-03-17T23:49:03.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.316+0000 7fa6a6e27640 1 -- 172.21.15.150:0/2148668436 wait complete. 2024-03-17T23:49:03.329 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-17T23:49:03.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.432+0000 7fce5d6ff640 1 Processor -- start 2024-03-17T23:49:03.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.432+0000 7fce5d6ff640 1 -- start start 2024-03-17T23:49:03.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.433+0000 7fce5d6ff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce5806f400 0x7fce580cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:03.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.433+0000 7fce5d6ff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fce580ce4d0 0x7fce58165a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:03.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.433+0000 7fce5d6ff640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce58145170 con 0x7fce58144bb0 2024-03-17T23:49:03.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.433+0000 7fce5d6ff640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fce580ce920 con 0x7fce580ce4d0 2024-03-17T23:49:03.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.433+0000 7fce5d6ff640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fce58145450 con 0x7fce5806f400 2024-03-17T23:49:03.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.433+0000 7fce577fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fce580ce4d0 0x7fce58165a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:03.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.433+0000 7fce56ffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fce58144bb0 0x7fce5815ce50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:48082/0 (socket says 172.21.15.150:48082) 2024-03-17T23:49:03.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.433+0000 7fce577fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fce580ce4d0 0x7fce58165a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42282/0 (socket says 172.21.15.150:42282) 2024-03-17T23:49:03.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.433+0000 7fce56ffd640 1 -- 172.21.15.150:0/3482847391 learned_addr learned my addr 172.21.15.150:0/3482847391 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:03.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.433+0000 7fce4ffff640 1 --2- 172.21.15.150:0/3482847391 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce5806f400 0x7fce580cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:03.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.433+0000 7fce577fe640 1 -- 172.21.15.150:0/3482847391 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce5806f400 msgr2=0x7fce580cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:03.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.433+0000 7fce577fe640 1 --2- 172.21.15.150:0/3482847391 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce5806f400 0x7fce580cdf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:03.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.433+0000 7fce567fc640 1 -- 172.21.15.150:0/3482847391 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1185987061 0 0) 0x7fce58145170 con 0x7fce58144bb0 2024-03-17T23:49:03.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.434+0000 7fce577fe640 1 -- 172.21.15.150:0/3482847391 >> v1:172.21.15.150:6789/0 conn(0x7fce58144bb0 legacy=0x7fce5815ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:03.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.434+0000 7fce577fe640 1 -- 172.21.15.150:0/3482847391 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce58145730 con 0x7fce580ce4d0 2024-03-17T23:49:03.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.434+0000 7fce577fe640 1 --2- 172.21.15.150:0/3482847391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce580ce4d0 0x7fce58165a30 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fce50009d90 tx=0x7fce5002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=512c378e173712e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:03.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.434+0000 7fce567fc640 1 -- 172.21.15.150:0/3482847391 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce50039070 con 0x7fce580ce4d0 2024-03-17T23:49:03.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.434+0000 7fce567fc640 1 -- 172.21.15.150:0/3482847391 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce5002cbc0 con 0x7fce580ce4d0 2024-03-17T23:49:03.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.434+0000 7fce567fc640 1 -- 172.21.15.150:0/3482847391 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce50034440 con 0x7fce580ce4d0 2024-03-17T23:49:03.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.435+0000 7fce5d6ff640 1 -- 172.21.15.150:0/3482847391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce580ce4d0 msgr2=0x7fce58165a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:03.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.435+0000 7fce5d6ff640 1 --2- 172.21.15.150:0/3482847391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce580ce4d0 0x7fce58165a30 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fce50009d90 tx=0x7fce5002be10 comp rx=0 tx=0).stop 2024-03-17T23:49:03.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.435+0000 7fce5d6ff640 1 -- 172.21.15.150:0/3482847391 shutdown_connections 2024-03-17T23:49:03.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.435+0000 7fce5d6ff640 1 --2- 172.21.15.150:0/3482847391 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce5806f400 0x7fce580cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:03.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.435+0000 7fce5d6ff640 1 --2- 172.21.15.150:0/3482847391 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce580ce4d0 0x7fce58165a30 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:03.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.435+0000 7fce5d6ff640 1 -- 172.21.15.150:0/3482847391 >> 172.21.15.150:0/3482847391 conn(0x7fce5806b190 msgr2=0x7fce5806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:03.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.435+0000 7fce5d6ff640 1 -- 172.21.15.150:0/3482847391 shutdown_connections 2024-03-17T23:49:03.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.435+0000 7fce5d6ff640 1 -- 172.21.15.150:0/3482847391 wait complete. 2024-03-17T23:49:03.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.436+0000 7fce5d6ff640 1 Processor -- start 2024-03-17T23:49:03.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.436+0000 7fce5d6ff640 1 -- start start 2024-03-17T23:49:03.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.437+0000 7fce5d6ff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5806f400 0x7fce58139400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:03.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.437+0000 7fce5d6ff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce580ce4d0 0x7fce58158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:03.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.437+0000 7fce5d6ff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce58144bb0 0x7fce58142b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:03.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.437+0000 7fce5d6ff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fce5806bee0 con 0x7fce5806f400 2024-03-17T23:49:03.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.437+0000 7fce5d6ff640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fce5806bd60 con 0x7fce580ce4d0 2024-03-17T23:49:03.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.437+0000 7fce5d6ff640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fce58165f70 con 0x7fce58144bb0 2024-03-17T23:49:03.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.437+0000 7fce56ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5806f400 0x7fce58139400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:03.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.437+0000 7fce56ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5806f400 0x7fce58139400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42294/0 (socket says 172.21.15.150:42294) 2024-03-17T23:49:03.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.437+0000 7fce56ffd640 1 -- 172.21.15.150:0/2025078614 learned_addr learned my addr 172.21.15.150:0/2025078614 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:03.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.437+0000 7fce577fe640 1 --2- 172.21.15.150:0/2025078614 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce58144bb0 0x7fce58142b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:03.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.437+0000 7fce4ffff640 1 --2- 172.21.15.150:0/2025078614 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce580ce4d0 0x7fce58158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:03.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.437+0000 7fce56ffd640 1 -- 172.21.15.150:0/2025078614 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce58144bb0 msgr2=0x7fce58142b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:03.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.437+0000 7fce56ffd640 1 --2- 172.21.15.150:0/2025078614 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce58144bb0 0x7fce58142b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:03.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.437+0000 7fce56ffd640 1 -- 172.21.15.150:0/2025078614 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce580ce4d0 msgr2=0x7fce58158ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:03.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.437+0000 7fce56ffd640 1 --2- 172.21.15.150:0/2025078614 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce580ce4d0 0x7fce58158ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:03.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.437+0000 7fce56ffd640 1 -- 172.21.15.150:0/2025078614 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce5813aff0 con 0x7fce5806f400 2024-03-17T23:49:03.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.437+0000 7fce577fe640 1 --2- 172.21.15.150:0/2025078614 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce58144bb0 0x7fce58142b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:49:03.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.437+0000 7fce4ffff640 1 --2- 172.21.15.150:0/2025078614 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce580ce4d0 0x7fce58158ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:49:03.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.437+0000 7fce56ffd640 1 --2- 172.21.15.150:0/2025078614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5806f400 0x7fce58139400 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fce4800c500 tx=0x7fce4800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:03.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.438+0000 7fce4f7fe640 1 -- 172.21.15.150:0/2025078614 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce48018070 con 0x7fce5806f400 2024-03-17T23:49:03.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.438+0000 7fce4f7fe640 1 -- 172.21.15.150:0/2025078614 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce48007580 con 0x7fce5806f400 2024-03-17T23:49:03.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.438+0000 7fce5d6ff640 1 -- 172.21.15.150:0/2025078614 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce5813b2d0 con 0x7fce5806f400 2024-03-17T23:49:03.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.438+0000 7fce4f7fe640 1 -- 172.21.15.150:0/2025078614 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce48004a20 con 0x7fce5806f400 2024-03-17T23:49:03.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.438+0000 7fce5d6ff640 1 -- 172.21.15.150:0/2025078614 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce5813b890 con 0x7fce5806f400 2024-03-17T23:49:03.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.439+0000 7fce5d6ff640 1 -- 172.21.15.150:0/2025078614 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce580681a0 con 0x7fce5806f400 2024-03-17T23:49:03.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.439+0000 7fce4f7fe640 1 -- 172.21.15.150:0/2025078614 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fce4801b070 con 0x7fce5806f400 2024-03-17T23:49:03.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.439+0000 7fce4f7fe640 1 --2- 172.21.15.150:0/2025078614 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fce3806bc00 0x7fce3806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:03.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.440+0000 7fce4f7fe640 1 -- 172.21.15.150:0/2025078614 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5335+0+0 (secure 0 0 0) 0x7fce48089b70 con 0x7fce5806f400 2024-03-17T23:49:03.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.440+0000 7fce4ffff640 1 --2- 172.21.15.150:0/2025078614 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fce3806bc00 0x7fce3806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:03.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.440+0000 7fce4ffff640 1 --2- 172.21.15.150:0/2025078614 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fce3806bc00 0x7fce3806e0c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fce58144910 tx=0x7fce40008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:03.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.444+0000 7fce4f7fe640 1 -- 172.21.15.150:0/2025078614 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fce48058860 con 0x7fce5806f400 2024-03-17T23:49:03.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.616+0000 7fce5d6ff640 1 -- 172.21.15.150:0/2025078614 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fce580cdd70 con 0x7fce5806f400 2024-03-17T23:49:03.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.620+0000 7fce4f7fe640 1 -- 172.21.15.150:0/2025078614 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7fce48058160 con 0x7fce5806f400 2024-03-17T23:49:03.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.622+0000 7fce5d6ff640 1 -- 172.21.15.150:0/2025078614 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fce3806bc00 msgr2=0x7fce3806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:03.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.622+0000 7fce5d6ff640 1 --2- 172.21.15.150:0/2025078614 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fce3806bc00 0x7fce3806e0c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fce58144910 tx=0x7fce40008040 comp rx=0 tx=0).stop 2024-03-17T23:49:03.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.622+0000 7fce5d6ff640 1 -- 172.21.15.150:0/2025078614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5806f400 msgr2=0x7fce58139400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:03.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.622+0000 7fce5d6ff640 1 --2- 172.21.15.150:0/2025078614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5806f400 0x7fce58139400 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fce4800c500 tx=0x7fce4800c9c0 comp rx=0 tx=0).stop 2024-03-17T23:49:03.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.622+0000 7fce5d6ff640 1 -- 172.21.15.150:0/2025078614 shutdown_connections 2024-03-17T23:49:03.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.622+0000 7fce5d6ff640 1 --2- 172.21.15.150:0/2025078614 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fce58144bb0 0x7fce58142b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:03.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.622+0000 7fce5d6ff640 1 --2- 172.21.15.150:0/2025078614 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fce3806bc00 0x7fce3806e0c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:03.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.622+0000 7fce5d6ff640 1 --2- 172.21.15.150:0/2025078614 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fce580ce4d0 0x7fce58158ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:03.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.622+0000 7fce5d6ff640 1 --2- 172.21.15.150:0/2025078614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce5806f400 0x7fce58139400 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:03.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.622+0000 7fce5d6ff640 1 -- 172.21.15.150:0/2025078614 >> 172.21.15.150:0/2025078614 conn(0x7fce5806b190 msgr2=0x7fce580b0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:03.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.622+0000 7fce5d6ff640 1 -- 172.21.15.150:0/2025078614 shutdown_connections 2024-03-17T23:49:03.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.622+0000 7fce5d6ff640 1 -- 172.21.15.150:0/2025078614 wait complete. 2024-03-17T23:49:03.637 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification' 2024-03-17T23:49:03.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.737+0000 7f708dec0640 1 Processor -- start 2024-03-17T23:49:03.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.737+0000 7f708dec0640 1 -- start start 2024-03-17T23:49:03.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.738+0000 7f708dec0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f708815a4b0 0x7f70880d0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:03.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.738+0000 7f708dec0640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f708806e580 0x7f70880d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:03.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.738+0000 7f708dec0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f708815a890 0x7f708806b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:03.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.738+0000 7f708dec0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f708806ea90 con 0x7f708815a4b0 2024-03-17T23:49:03.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.738+0000 7f708dec0640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f708806ed30 con 0x7f708806e580 2024-03-17T23:49:03.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.738+0000 7f708dec0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f708806c340 con 0x7f708815a890 2024-03-17T23:49:03.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.739+0000 7f70877fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f708815a4b0 0x7f70880d0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:03.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.739+0000 7f70877fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f708815a4b0 0x7f70880d0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42300/0 (socket says 172.21.15.150:42300) 2024-03-17T23:49:03.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.739+0000 7f70877fe640 1 -- 172.21.15.150:0/970065404 learned_addr learned my addr 172.21.15.150:0/970065404 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:03.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.739+0000 7f7087fff640 1 --2- 172.21.15.150:0/970065404 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f708815a890 0x7f708806b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:03.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.739+0000 7f7086ffd640 1 --2- 172.21.15.150:0/970065404 >> v2:172.21.15.157:3300/0 conn(0x7f708806e580 0x7f70880d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:03.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.740+0000 7f70877fe640 1 -- 172.21.15.150:0/970065404 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f708815a890 msgr2=0x7f708806b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:03.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.740+0000 7f70877fe640 1 --2- 172.21.15.150:0/970065404 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f708815a890 0x7f708806b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:03.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.740+0000 7f70877fe640 1 -- 172.21.15.150:0/970065404 >> v2:172.21.15.157:3300/0 conn(0x7f708806e580 msgr2=0x7f70880d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:03.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.740+0000 7f70877fe640 1 --2- 172.21.15.150:0/970065404 >> v2:172.21.15.157:3300/0 conn(0x7f708806e580 0x7f70880d0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:03.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.740+0000 7f70877fe640 1 -- 172.21.15.150:0/970065404 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f708806c620 con 0x7f708815a4b0 2024-03-17T23:49:03.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.741+0000 7f70877fe640 1 --2- 172.21.15.150:0/970065404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f708815a4b0 0x7f70880d0140 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f7088072990 tx=0x7f708002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ef5890e5853ae6 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:03.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.742+0000 7f70867fc640 1 -- 172.21.15.150:0/970065404 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708002cb00 con 0x7f708815a4b0 2024-03-17T23:49:03.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.742+0000 7f70867fc640 1 -- 172.21.15.150:0/970065404 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f708002cca0 con 0x7f708815a4b0 2024-03-17T23:49:03.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.742+0000 7f70867fc640 1 -- 172.21.15.150:0/970065404 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7080034610 con 0x7f708815a4b0 2024-03-17T23:49:03.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.742+0000 7f708dec0640 1 -- 172.21.15.150:0/970065404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f708815a4b0 msgr2=0x7f70880d0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:03.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.742+0000 7f708dec0640 1 --2- 172.21.15.150:0/970065404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f708815a4b0 0x7f70880d0140 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f7088072990 tx=0x7f708002bf60 comp rx=0 tx=0).stop 2024-03-17T23:49:03.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.743+0000 7f708dec0640 1 -- 172.21.15.150:0/970065404 shutdown_connections 2024-03-17T23:49:03.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.743+0000 7f708dec0640 1 --2- 172.21.15.150:0/970065404 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f708815a890 0x7f708806b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:03.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.743+0000 7f708dec0640 1 --2- 172.21.15.150:0/970065404 >> v2:172.21.15.157:3300/0 conn(0x7f708806e580 0x7f70880d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:03.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.743+0000 7f708dec0640 1 --2- 172.21.15.150:0/970065404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f708815a4b0 0x7f70880d0140 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:03.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.743+0000 7f708dec0640 1 -- 172.21.15.150:0/970065404 >> 172.21.15.150:0/970065404 conn(0x7f70880b3da0 msgr2=0x7f70880b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:03.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.743+0000 7f708dec0640 1 -- 172.21.15.150:0/970065404 shutdown_connections 2024-03-17T23:49:03.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.743+0000 7f708dec0640 1 -- 172.21.15.150:0/970065404 wait complete. 2024-03-17T23:49:03.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.744+0000 7f708dec0640 1 Processor -- start 2024-03-17T23:49:03.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.744+0000 7f708dec0640 1 -- start start 2024-03-17T23:49:03.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.745+0000 7f708dec0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f708806e580 0x7f70880b2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:03.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.745+0000 7f708dec0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f708815a4b0 0x7f708813c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:03.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.745+0000 7f708dec0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f708815a890 0x7f708813fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:03.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.745+0000 7f708dec0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7088072dc0 con 0x7f708806e580 2024-03-17T23:49:03.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.745+0000 7f708dec0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7088072c40 con 0x7f708815a4b0 2024-03-17T23:49:03.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.745+0000 7f708dec0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7088099ae0 con 0x7f708815a890 2024-03-17T23:49:03.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.745+0000 7f70877fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f708806e580 0x7f70880b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:03.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.745+0000 7f70877fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f708806e580 0x7f70880b2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42310/0 (socket says 172.21.15.150:42310) 2024-03-17T23:49:03.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.745+0000 7f70877fe640 1 -- 172.21.15.150:0/1205169036 learned_addr learned my addr 172.21.15.150:0/1205169036 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:03.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.745+0000 7f7087fff640 1 --2- 172.21.15.150:0/1205169036 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f708815a890 0x7f708813fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:03.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.745+0000 7f7086ffd640 1 --2- 172.21.15.150:0/1205169036 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f708815a4b0 0x7f708813c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:03.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.745+0000 7f70877fe640 1 -- 172.21.15.150:0/1205169036 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f708815a890 msgr2=0x7f708813fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:03.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.745+0000 7f70877fe640 1 --2- 172.21.15.150:0/1205169036 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f708815a890 0x7f708813fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:03.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.745+0000 7f70877fe640 1 -- 172.21.15.150:0/1205169036 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f708815a4b0 msgr2=0x7f708813c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:03.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.745+0000 7f70877fe640 1 --2- 172.21.15.150:0/1205169036 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f708815a4b0 0x7f708813c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:03.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.745+0000 7f70877fe640 1 -- 172.21.15.150:0/1205169036 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f708813d040 con 0x7f708806e580 2024-03-17T23:49:03.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.746+0000 7f70877fe640 1 --2- 172.21.15.150:0/1205169036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f708806e580 0x7f70880b2190 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f7080000f80 tx=0x7f7080004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:03.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.746+0000 7f707ffff640 1 -- 172.21.15.150:0/1205169036 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7080043020 con 0x7f708806e580 2024-03-17T23:49:03.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.746+0000 7f707ffff640 1 -- 172.21.15.150:0/1205169036 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f708002ce50 con 0x7f708806e580 2024-03-17T23:49:03.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.746+0000 7f708dec0640 1 -- 172.21.15.150:0/1205169036 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f708813d320 con 0x7f708806e580 2024-03-17T23:49:03.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.746+0000 7f707ffff640 1 -- 172.21.15.150:0/1205169036 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708003e690 con 0x7f708806e580 2024-03-17T23:49:03.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.746+0000 7f708dec0640 1 -- 172.21.15.150:0/1205169036 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f708813d8e0 con 0x7f708806e580 2024-03-17T23:49:03.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.747+0000 7f708dec0640 1 -- 172.21.15.150:0/1205169036 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70880681a0 con 0x7f708806e580 2024-03-17T23:49:03.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.747+0000 7f707ffff640 1 -- 172.21.15.150:0/1205169036 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7080046020 con 0x7f708806e580 2024-03-17T23:49:03.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.748+0000 7f707ffff640 1 --2- 172.21.15.150:0/1205169036 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f706806bc00 0x7f706806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:03.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.748+0000 7f707ffff640 1 -- 172.21.15.150:0/1205169036 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5335+0+0 (secure 0 0 0) 0x7f70800ace80 con 0x7f708806e580 2024-03-17T23:49:03.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.748+0000 7f7086ffd640 1 --2- 172.21.15.150:0/1205169036 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f706806bc00 0x7f706806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:03.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.748+0000 7f7086ffd640 1 --2- 172.21.15.150:0/1205169036 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f706806bc00 0x7f706806e0c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7088147570 tx=0x7f7070008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:03.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.752+0000 7f707ffff640 1 -- 172.21.15.150:0/1205169036 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f708004d030 con 0x7f708806e580 2024-03-17T23:49:03.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:03.995+0000 7f708dec0640 1 -- 172.21.15.150:0/1205169036 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]} v 0) -- 0x7f708809d6f0 con 0x7f708806e580 2024-03-17T23:49:04.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.320+0000 7f707ffff640 1 -- 172.21.15.150:0/1205169036 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]}]=0 v143) ==== 147+0+0 (secure 0 0 0) 0x7f708007b400 con 0x7f708806e580 2024-03-17T23:49:04.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.323+0000 7f708dec0640 1 -- 172.21.15.150:0/1205169036 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f706806bc00 msgr2=0x7f706806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:04.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.323+0000 7f708dec0640 1 --2- 172.21.15.150:0/1205169036 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f706806bc00 0x7f706806e0c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7088147570 tx=0x7f7070008040 comp rx=0 tx=0).stop 2024-03-17T23:49:04.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.323+0000 7f708dec0640 1 -- 172.21.15.150:0/1205169036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f708806e580 msgr2=0x7f70880b2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:04.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.323+0000 7f708dec0640 1 --2- 172.21.15.150:0/1205169036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f708806e580 0x7f70880b2190 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f7080000f80 tx=0x7f7080004a40 comp rx=0 tx=0).stop 2024-03-17T23:49:04.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.323+0000 7f708dec0640 1 -- 172.21.15.150:0/1205169036 shutdown_connections 2024-03-17T23:49:04.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.323+0000 7f708dec0640 1 --2- 172.21.15.150:0/1205169036 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f708815a890 0x7f708813fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.323+0000 7f708dec0640 1 --2- 172.21.15.150:0/1205169036 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f706806bc00 0x7f706806e0c0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.323+0000 7f708dec0640 1 --2- 172.21.15.150:0/1205169036 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f708815a4b0 0x7f708813c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.323+0000 7f708dec0640 1 --2- 172.21.15.150:0/1205169036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f708806e580 0x7f70880b2190 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.323+0000 7f708dec0640 1 -- 172.21.15.150:0/1205169036 >> 172.21.15.150:0/1205169036 conn(0x7f70880b3da0 msgr2=0x7f708806ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:04.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.323+0000 7f708dec0640 1 -- 172.21.15.150:0/1205169036 shutdown_connections 2024-03-17T23:49:04.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.323+0000 7f708dec0640 1 -- 172.21.15.150:0/1205169036 wait complete. 2024-03-17T23:49:04.335 INFO:tasks.cephfs_test_runner:That a client who does not have the capability for setting ... ok 2024-03-17T23:49:04.335 INFO:tasks.cephfs_test_runner:Starting test: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-17T23:49:04.336 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:49:04.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.435+0000 7f5460d1b640 1 Processor -- start 2024-03-17T23:49:04.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.436+0000 7f5460d1b640 1 -- start start 2024-03-17T23:49:04.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.437+0000 7f5460d1b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f545c144bb0 0x7f545c15ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:04.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.437+0000 7f5460d1b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f545c15d390 0x7f545c0cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:04.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.437+0000 7f5460d1b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f545c0ce4d0 0x7f545c06e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:04.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.437+0000 7f5460d1b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f545c0ce920 con 0x7f545c144bb0 2024-03-17T23:49:04.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.437+0000 7f5460d1b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f545c145170 con 0x7f545c0ce4d0 2024-03-17T23:49:04.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.437+0000 7f5460d1b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f545c145450 con 0x7f545c15d390 2024-03-17T23:49:04.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.437+0000 7f545a59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f545c144bb0 0x7f545c15ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:04.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.437+0000 7f545a59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f545c144bb0 0x7f545c15ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42318/0 (socket says 172.21.15.150:42318) 2024-03-17T23:49:04.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.437+0000 7f545a59c640 1 -- 172.21.15.150:0/1245829631 learned_addr learned my addr 172.21.15.150:0/1245829631 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:04.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.437+0000 7f545ad9d640 1 --2- 172.21.15.150:0/1245829631 >> v2:172.21.15.157:3300/0 conn(0x7f545c0ce4d0 0x7f545c06e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:04.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.437+0000 7f5459d9b640 1 --2- 172.21.15.150:0/1245829631 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f545c15d390 0x7f545c0cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:04.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.437+0000 7f545a59c640 1 -- 172.21.15.150:0/1245829631 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f545c15d390 msgr2=0x7f545c0cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:04.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.437+0000 7f545a59c640 1 --2- 172.21.15.150:0/1245829631 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f545c15d390 0x7f545c0cdf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.437+0000 7f545a59c640 1 -- 172.21.15.150:0/1245829631 >> v2:172.21.15.157:3300/0 conn(0x7f545c0ce4d0 msgr2=0x7f545c06e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:04.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.437+0000 7f545a59c640 1 --2- 172.21.15.150:0/1245829631 >> v2:172.21.15.157:3300/0 conn(0x7f545c0ce4d0 0x7f545c06e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.437+0000 7f545a59c640 1 -- 172.21.15.150:0/1245829631 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f545c145730 con 0x7f545c144bb0 2024-03-17T23:49:04.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.438+0000 7f545a59c640 1 --2- 172.21.15.150:0/1245829631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f545c144bb0 0x7f545c15ce50 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f545c071c60 tx=0x7f545002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=330802d359c3c348 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:04.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.438+0000 7f545959a640 1 -- 172.21.15.150:0/1245829631 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f545002e070 con 0x7f545c144bb0 2024-03-17T23:49:04.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.438+0000 7f545959a640 1 -- 172.21.15.150:0/1245829631 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5450037070 con 0x7f545c144bb0 2024-03-17T23:49:04.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.438+0000 7f545959a640 1 -- 172.21.15.150:0/1245829631 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5450031a00 con 0x7f545c144bb0 2024-03-17T23:49:04.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.438+0000 7f5460d1b640 1 -- 172.21.15.150:0/1245829631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f545c144bb0 msgr2=0x7f545c15ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:04.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.438+0000 7f5460d1b640 1 --2- 172.21.15.150:0/1245829631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f545c144bb0 0x7f545c15ce50 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f545c071c60 tx=0x7f545002b880 comp rx=0 tx=0).stop 2024-03-17T23:49:04.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.438+0000 7f5460d1b640 1 -- 172.21.15.150:0/1245829631 shutdown_connections 2024-03-17T23:49:04.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.438+0000 7f5460d1b640 1 --2- 172.21.15.150:0/1245829631 >> v2:172.21.15.157:3300/0 conn(0x7f545c0ce4d0 0x7f545c06e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.438+0000 7f5460d1b640 1 --2- 172.21.15.150:0/1245829631 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f545c15d390 0x7f545c0cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.438+0000 7f5460d1b640 1 --2- 172.21.15.150:0/1245829631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f545c144bb0 0x7f545c15ce50 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.438+0000 7f5460d1b640 1 -- 172.21.15.150:0/1245829631 >> 172.21.15.150:0/1245829631 conn(0x7f545c06b190 msgr2=0x7f545c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:04.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.439+0000 7f5460d1b640 1 -- 172.21.15.150:0/1245829631 shutdown_connections 2024-03-17T23:49:04.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.439+0000 7f5460d1b640 1 -- 172.21.15.150:0/1245829631 wait complete. 2024-03-17T23:49:04.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.440+0000 7f5460d1b640 1 Processor -- start 2024-03-17T23:49:04.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.440+0000 7f5460d1b640 1 -- start start 2024-03-17T23:49:04.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.440+0000 7f5460d1b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f545c0ce4d0 0x7f545c0aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:04.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.440+0000 7f5460d1b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f545c144bb0 0x7f545c15a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:04.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.440+0000 7f5460d1b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f545c15d390 0x7f545c13bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:04.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.441+0000 7f5460d1b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f545c06eab0 con 0x7f545c0ce4d0 2024-03-17T23:49:04.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.441+0000 7f5460d1b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f545c06e930 con 0x7f545c144bb0 2024-03-17T23:49:04.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.441+0000 7f5460d1b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f545c06ec30 con 0x7f545c15d390 2024-03-17T23:49:04.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.441+0000 7f545a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f545c0ce4d0 0x7f545c0aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:04.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.441+0000 7f545a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f545c0ce4d0 0x7f545c0aeba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42320/0 (socket says 172.21.15.150:42320) 2024-03-17T23:49:04.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.441+0000 7f545a59c640 1 -- 172.21.15.150:0/473871722 learned_addr learned my addr 172.21.15.150:0/473871722 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:04.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.441+0000 7f5459d9b640 1 --2- 172.21.15.150:0/473871722 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f545c144bb0 0x7f545c15a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:04.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.441+0000 7f545ad9d640 1 --2- 172.21.15.150:0/473871722 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f545c15d390 0x7f545c13bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:04.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.441+0000 7f545a59c640 1 -- 172.21.15.150:0/473871722 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f545c15d390 msgr2=0x7f545c13bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:04.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.441+0000 7f545a59c640 1 --2- 172.21.15.150:0/473871722 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f545c15d390 0x7f545c13bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.441+0000 7f545a59c640 1 -- 172.21.15.150:0/473871722 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f545c144bb0 msgr2=0x7f545c15a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:04.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.441+0000 7f545a59c640 1 --2- 172.21.15.150:0/473871722 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f545c144bb0 0x7f545c15a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.441+0000 7f545a59c640 1 -- 172.21.15.150:0/473871722 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f545c142eb0 con 0x7f545c0ce4d0 2024-03-17T23:49:04.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.441+0000 7f5459d9b640 1 --2- 172.21.15.150:0/473871722 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f545c144bb0 0x7f545c15a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:49:04.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.441+0000 7f545a59c640 1 --2- 172.21.15.150:0/473871722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f545c0ce4d0 0x7f545c0aeba0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f545002b8c0 tx=0x7f54500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:04.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.441+0000 7f544affd640 1 -- 172.21.15.150:0/473871722 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5450032880 con 0x7f545c0ce4d0 2024-03-17T23:49:04.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.441+0000 7f544affd640 1 -- 172.21.15.150:0/473871722 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5450032a20 con 0x7f545c0ce4d0 2024-03-17T23:49:04.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.441+0000 7f544affd640 1 -- 172.21.15.150:0/473871722 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5450032e10 con 0x7f545c0ce4d0 2024-03-17T23:49:04.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.441+0000 7f5460d1b640 1 -- 172.21.15.150:0/473871722 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f545c143150 con 0x7f545c0ce4d0 2024-03-17T23:49:04.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.441+0000 7f5460d1b640 1 -- 172.21.15.150:0/473871722 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f545c13a050 con 0x7f545c0ce4d0 2024-03-17T23:49:04.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.442+0000 7f5460d1b640 1 -- 172.21.15.150:0/473871722 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f545c0681a0 con 0x7f545c0ce4d0 2024-03-17T23:49:04.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.443+0000 7f544affd640 1 -- 172.21.15.150:0/473871722 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f545004c060 con 0x7f545c0ce4d0 2024-03-17T23:49:04.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.443+0000 7f544affd640 1 --2- 172.21.15.150:0/473871722 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f543c06bbd0 0x7f543c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:04.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.443+0000 7f544affd640 1 -- 172.21.15.150:0/473871722 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5335+0+0 (secure 0 0 0) 0x7f54500ac510 con 0x7f545c0ce4d0 2024-03-17T23:49:04.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.443+0000 7f5459d9b640 1 --2- 172.21.15.150:0/473871722 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f543c06bbd0 0x7f543c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:04.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.443+0000 7f5459d9b640 1 --2- 172.21.15.150:0/473871722 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f543c06bbd0 0x7f543c06e090 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f544c006fd0 tx=0x7f544c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:04.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.447+0000 7f544affd640 1 -- 172.21.15.150:0/473871722 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5450040070 con 0x7f545c0ce4d0 2024-03-17T23:49:04.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.621+0000 7f5460d1b640 1 -- 172.21.15.150:0/473871722 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f545c15bdf0 con 0x7f545c0ce4d0 2024-03-17T23:49:04.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.622+0000 7f544affd640 1 -- 172.21.15.150:0/473871722 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) ==== 74+0+13331 (secure 0 0 0) 0x7f545004c350 con 0x7f545c0ce4d0 2024-03-17T23:49:04.623 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:04.623 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":49,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:49:03.311963+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:6837/1413722826":"2024-03-18T23:48:59.295980+0000","172.21.15.150:6835/1413722826":"2024-03-18T23:48:59.295980+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:49:04.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.624+0000 7f5460d1b640 1 -- 172.21.15.150:0/473871722 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f543c06bbd0 msgr2=0x7f543c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:04.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.624+0000 7f5460d1b640 1 --2- 172.21.15.150:0/473871722 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f543c06bbd0 0x7f543c06e090 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f544c006fd0 tx=0x7f544c008040 comp rx=0 tx=0).stop 2024-03-17T23:49:04.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.624+0000 7f5460d1b640 1 -- 172.21.15.150:0/473871722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f545c0ce4d0 msgr2=0x7f545c0aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:04.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.624+0000 7f5460d1b640 1 --2- 172.21.15.150:0/473871722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f545c0ce4d0 0x7f545c0aeba0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f545002b8c0 tx=0x7f54500046c0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.624+0000 7f5460d1b640 1 -- 172.21.15.150:0/473871722 shutdown_connections 2024-03-17T23:49:04.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.624+0000 7f5460d1b640 1 --2- 172.21.15.150:0/473871722 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f545c15d390 0x7f545c13bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.624+0000 7f5460d1b640 1 --2- 172.21.15.150:0/473871722 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f543c06bbd0 0x7f543c06e090 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.624+0000 7f5460d1b640 1 --2- 172.21.15.150:0/473871722 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f545c144bb0 0x7f545c15a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.624+0000 7f5460d1b640 1 --2- 172.21.15.150:0/473871722 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f545c0ce4d0 0x7f545c0aeba0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.624+0000 7f5460d1b640 1 -- 172.21.15.150:0/473871722 >> 172.21.15.150:0/473871722 conn(0x7f545c06b190 msgr2=0x7f545c0b0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:04.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.624+0000 7f5460d1b640 1 -- 172.21.15.150:0/473871722 shutdown_connections 2024-03-17T23:49:04.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.624+0000 7f5460d1b640 1 -- 172.21.15.150:0/473871722 wait complete. 2024-03-17T23:49:04.636 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-17T23:49:04.637 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:49:04.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.740+0000 7ffa276ae640 1 Processor -- start 2024-03-17T23:49:04.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.740+0000 7ffa276ae640 1 -- start start 2024-03-17T23:49:04.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.741+0000 7ffa276ae640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa20161520 0x7ffa200b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:04.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.741+0000 7ffa276ae640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ffa200b2bf0 0x7ffa200d05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:04.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.741+0000 7ffa276ae640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa20161d30 con 0x7ffa200d0b10 2024-03-17T23:49:04.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.741+0000 7ffa276ae640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7ffa200b9bf0 con 0x7ffa200b2bf0 2024-03-17T23:49:04.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.741+0000 7ffa276ae640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ffa20162940 con 0x7ffa20161520 2024-03-17T23:49:04.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.741+0000 7ffa24c49640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ffa200b2bf0 0x7ffa200d05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:04.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.741+0000 7ffa25c4b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7ffa200d0b10 0x7ffa200b57a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:48088/0 (socket says 172.21.15.150:48088) 2024-03-17T23:49:04.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.741+0000 7ffa25c4b640 1 -- 172.21.15.150:0/3640814658 learned_addr learned my addr 172.21.15.150:0/3640814658 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:04.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.742+0000 7ffa2544a640 1 --2- 172.21.15.150:0/3640814658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa20161520 0x7ffa200b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:04.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.742+0000 7ffa24c49640 1 -- 172.21.15.150:0/3640814658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa20161520 msgr2=0x7ffa200b96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:04.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.742+0000 7ffa24c49640 1 --2- 172.21.15.150:0/3640814658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa20161520 0x7ffa200b96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.742+0000 7ffa24c49640 1 -- 172.21.15.150:0/3640814658 >> v1:172.21.15.150:6789/0 conn(0x7ffa200d0b10 legacy=0x7ffa200b57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:04.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.742+0000 7ffa17fff640 1 -- 172.21.15.150:0/3640814658 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4063492460 0 0) 0x7ffa20161d30 con 0x7ffa200d0b10 2024-03-17T23:49:04.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.742+0000 7ffa24c49640 1 -- 172.21.15.150:0/3640814658 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa20162010 con 0x7ffa200b2bf0 2024-03-17T23:49:04.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.742+0000 7ffa2544a640 1 --2- 172.21.15.150:0/3640814658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa20161520 0x7ffa200b96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:49:04.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.742+0000 7ffa24c49640 1 --2- 172.21.15.150:0/3640814658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa200b2bf0 0x7ffa200d05d0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7ffa100097a0 tx=0x7ffa1002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf7a83522aa05103 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:04.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.743+0000 7ffa17fff640 1 -- 172.21.15.150:0/3640814658 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa10038070 con 0x7ffa200b2bf0 2024-03-17T23:49:04.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.743+0000 7ffa17fff640 1 -- 172.21.15.150:0/3640814658 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa10031700 con 0x7ffa200b2bf0 2024-03-17T23:49:04.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.743+0000 7ffa17fff640 1 -- 172.21.15.150:0/3640814658 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa10031ad0 con 0x7ffa200b2bf0 2024-03-17T23:49:04.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.743+0000 7ffa276ae640 1 -- 172.21.15.150:0/3640814658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa200b2bf0 msgr2=0x7ffa200d05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:04.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.743+0000 7ffa276ae640 1 --2- 172.21.15.150:0/3640814658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa200b2bf0 0x7ffa200d05d0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7ffa100097a0 tx=0x7ffa1002b7a0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.743+0000 7ffa276ae640 1 -- 172.21.15.150:0/3640814658 shutdown_connections 2024-03-17T23:49:04.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.743+0000 7ffa276ae640 1 --2- 172.21.15.150:0/3640814658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa200b2bf0 0x7ffa200d05d0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.743+0000 7ffa276ae640 1 --2- 172.21.15.150:0/3640814658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa20161520 0x7ffa200b96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.743+0000 7ffa276ae640 1 -- 172.21.15.150:0/3640814658 >> 172.21.15.150:0/3640814658 conn(0x7ffa2006ca00 msgr2=0x7ffa2006ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:04.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.743+0000 7ffa276ae640 1 -- 172.21.15.150:0/3640814658 shutdown_connections 2024-03-17T23:49:04.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.743+0000 7ffa276ae640 1 -- 172.21.15.150:0/3640814658 wait complete. 2024-03-17T23:49:04.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.744+0000 7ffa276ae640 1 Processor -- start 2024-03-17T23:49:04.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.744+0000 7ffa276ae640 1 -- start start 2024-03-17T23:49:04.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.745+0000 7ffa276ae640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa200b2bf0 0x7ffa20153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:04.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.745+0000 7ffa276ae640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa200d0b10 0x7ffa20153880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:04.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.745+0000 7ffa276ae640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa20161520 0x7ffa20153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:04.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.745+0000 7ffa276ae640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ffa201625d0 con 0x7ffa200d0b10 2024-03-17T23:49:04.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.745+0000 7ffa276ae640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ffa200d0f60 con 0x7ffa200b2bf0 2024-03-17T23:49:04.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.745+0000 7ffa276ae640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ffa20161900 con 0x7ffa20161520 2024-03-17T23:49:04.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.745+0000 7ffa24c49640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa200d0b10 0x7ffa20153880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:04.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.745+0000 7ffa24c49640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa200d0b10 0x7ffa20153880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42346/0 (socket says 172.21.15.150:42346) 2024-03-17T23:49:04.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.745+0000 7ffa24c49640 1 -- 172.21.15.150:0/3032144005 learned_addr learned my addr 172.21.15.150:0/3032144005 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:04.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.745+0000 7ffa2544a640 1 --2- 172.21.15.150:0/3032144005 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa200b2bf0 0x7ffa20153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:04.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.745+0000 7ffa25c4b640 1 --2- 172.21.15.150:0/3032144005 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa20161520 0x7ffa20153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:04.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.745+0000 7ffa24c49640 1 -- 172.21.15.150:0/3032144005 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa20161520 msgr2=0x7ffa20153dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:04.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.745+0000 7ffa24c49640 1 --2- 172.21.15.150:0/3032144005 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa20161520 0x7ffa20153dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.745+0000 7ffa24c49640 1 -- 172.21.15.150:0/3032144005 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa200b2bf0 msgr2=0x7ffa20153340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:04.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.745+0000 7ffa24c49640 1 --2- 172.21.15.150:0/3032144005 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa200b2bf0 0x7ffa20153340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.745+0000 7ffa24c49640 1 -- 172.21.15.150:0/3032144005 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa20154420 con 0x7ffa200d0b10 2024-03-17T23:49:04.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.745+0000 7ffa25c4b640 1 --2- 172.21.15.150:0/3032144005 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa20161520 0x7ffa20153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:49:04.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.746+0000 7ffa24c49640 1 --2- 172.21.15.150:0/3032144005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa200d0b10 0x7ffa20153880 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7ffa100097a0 tx=0x7ffa100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:04.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.746+0000 7ffa15ffb640 1 -- 172.21.15.150:0/3032144005 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa10038040 con 0x7ffa200d0b10 2024-03-17T23:49:04.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.746+0000 7ffa15ffb640 1 -- 172.21.15.150:0/3032144005 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa10004250 con 0x7ffa200d0b10 2024-03-17T23:49:04.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.746+0000 7ffa276ae640 1 -- 172.21.15.150:0/3032144005 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa201f50d0 con 0x7ffa200d0b10 2024-03-17T23:49:04.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.746+0000 7ffa15ffb640 1 -- 172.21.15.150:0/3032144005 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa10033820 con 0x7ffa200d0b10 2024-03-17T23:49:04.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.746+0000 7ffa276ae640 1 -- 172.21.15.150:0/3032144005 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa201f5690 con 0x7ffa200d0b10 2024-03-17T23:49:04.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.747+0000 7ffa276ae640 1 -- 172.21.15.150:0/3032144005 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa201f4ec0 con 0x7ffa200d0b10 2024-03-17T23:49:04.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.747+0000 7ffa15ffb640 1 -- 172.21.15.150:0/3032144005 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ffa100315c0 con 0x7ffa200d0b10 2024-03-17T23:49:04.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.748+0000 7ffa15ffb640 1 --2- 172.21.15.150:0/3032144005 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ffa0406bbd0 0x7ffa0406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:04.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.748+0000 7ffa15ffb640 1 -- 172.21.15.150:0/3032144005 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5335+0+0 (secure 0 0 0) 0x7ffa100acc30 con 0x7ffa200d0b10 2024-03-17T23:49:04.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.748+0000 7ffa2544a640 1 --2- 172.21.15.150:0/3032144005 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ffa0406bbd0 0x7ffa0406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:04.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.748+0000 7ffa2544a640 1 --2- 172.21.15.150:0/3032144005 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ffa0406bbd0 0x7ffa0406e090 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00ab60 tx=0x7ffa1c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:04.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.752+0000 7ffa15ffb640 1 -- 172.21.15.150:0/3032144005 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ffa10030070 con 0x7ffa200d0b10 2024-03-17T23:49:04.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.925+0000 7ffa276ae640 1 -- 172.21.15.150:0/3032144005 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ffa200681a0 con 0x7ffa200d0b10 2024-03-17T23:49:04.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.925+0000 7ffa15ffb640 1 -- 172.21.15.150:0/3032144005 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v49) ==== 93+0+10 (secure 0 0 0) 0x7ffa1007b890 con 0x7ffa200d0b10 2024-03-17T23:49:04.926 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:49:04.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.928+0000 7ffa276ae640 1 -- 172.21.15.150:0/3032144005 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ffa0406bbd0 msgr2=0x7ffa0406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:04.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.928+0000 7ffa276ae640 1 --2- 172.21.15.150:0/3032144005 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ffa0406bbd0 0x7ffa0406e090 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00ab60 tx=0x7ffa1c008040 comp rx=0 tx=0).stop 2024-03-17T23:49:04.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.928+0000 7ffa276ae640 1 -- 172.21.15.150:0/3032144005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa200d0b10 msgr2=0x7ffa20153880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:04.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.928+0000 7ffa276ae640 1 --2- 172.21.15.150:0/3032144005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa200d0b10 0x7ffa20153880 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7ffa100097a0 tx=0x7ffa100046c0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.928+0000 7ffa276ae640 1 -- 172.21.15.150:0/3032144005 shutdown_connections 2024-03-17T23:49:04.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.928+0000 7ffa276ae640 1 --2- 172.21.15.150:0/3032144005 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffa20161520 0x7ffa20153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.928+0000 7ffa276ae640 1 --2- 172.21.15.150:0/3032144005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa200d0b10 0x7ffa20153880 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.928+0000 7ffa276ae640 1 --2- 172.21.15.150:0/3032144005 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ffa0406bbd0 0x7ffa0406e090 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.928+0000 7ffa276ae640 1 --2- 172.21.15.150:0/3032144005 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffa200b2bf0 0x7ffa20153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:04.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.928+0000 7ffa276ae640 1 -- 172.21.15.150:0/3032144005 >> 172.21.15.150:0/3032144005 conn(0x7ffa2006ca00 msgr2=0x7ffa200b9230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:04.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.928+0000 7ffa276ae640 1 -- 172.21.15.150:0/3032144005 shutdown_connections 2024-03-17T23:49:04.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:04.928+0000 7ffa276ae640 1 -- 172.21.15.150:0/3032144005 wait complete. 2024-03-17T23:49:04.939 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm' 2024-03-17T23:49:05.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.039+0000 7f3bf89b9640 1 Processor -- start 2024-03-17T23:49:05.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.039+0000 7f3bf89b9640 1 -- start start 2024-03-17T23:49:05.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.040+0000 7f3bf89b9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3bf409a470 0x7f3bf409a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:05.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.040+0000 7f3bf89b9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3bf409ad90 0x7f3bf415f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:05.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.040+0000 7f3bf89b9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bf40cfad0 con 0x7f3bf415fcd0 2024-03-17T23:49:05.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.040+0000 7f3bf89b9640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3bf40cfdb0 con 0x7f3bf409a470 2024-03-17T23:49:05.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.040+0000 7f3bf89b9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3bf40d0090 con 0x7f3bf409ad90 2024-03-17T23:49:05.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.040+0000 7f3bf2d9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f3bf415fcd0 0x7f3bf415c150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:48104/0 (socket says 172.21.15.150:48104) 2024-03-17T23:49:05.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.040+0000 7f3bf2d9d640 1 -- 172.21.15.150:0/940921483 learned_addr learned my addr 172.21.15.150:0/940921483 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:05.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.040+0000 7f3bf259c640 1 --2- 172.21.15.150:0/940921483 >> v2:172.21.15.157:3300/0 conn(0x7f3bf409a470 0x7f3bf409a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:05.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.040+0000 7f3bf1d9b640 1 --2- 172.21.15.150:0/940921483 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3bf409ad90 0x7f3bf415f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:05.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.040+0000 7f3bf159a640 1 -- 172.21.15.150:0/940921483 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2323946870 0 0) 0x7f3bf40cfad0 con 0x7f3bf415fcd0 2024-03-17T23:49:05.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.040+0000 7f3bf159a640 1 -- 172.21.15.150:0/940921483 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bdc003520 con 0x7f3bf415fcd0 2024-03-17T23:49:05.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.041+0000 7f3bf159a640 1 -- 172.21.15.150:0/940921483 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3506221439 0 0) 0x7f3bdc003520 con 0x7f3bf415fcd0 2024-03-17T23:49:05.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.041+0000 7f3bf159a640 1 -- 172.21.15.150:0/940921483 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3bf409ad90 msgr2=0x7f3bf415f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:05.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.041+0000 7f3bf159a640 1 --2- 172.21.15.150:0/940921483 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3bf409ad90 0x7f3bf415f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:05.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.041+0000 7f3bf159a640 1 -- 172.21.15.150:0/940921483 >> v2:172.21.15.157:3300/0 conn(0x7f3bf409a470 msgr2=0x7f3bf409a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:05.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.041+0000 7f3bf159a640 1 --2- 172.21.15.150:0/940921483 >> v2:172.21.15.157:3300/0 conn(0x7f3bf409a470 0x7f3bf409a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:05.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.041+0000 7f3bf159a640 1 -- 172.21.15.150:0/940921483 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bf40d0370 con 0x7f3bf415fcd0 2024-03-17T23:49:05.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.041+0000 7f3bf159a640 1 -- 172.21.15.150:0/940921483 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f3bec001670 con 0x7f3bf415fcd0 2024-03-17T23:49:05.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.041+0000 7f3bf159a640 1 -- 172.21.15.150:0/940921483 >> v1:172.21.15.150:6789/0 conn(0x7f3bf415fcd0 legacy=0x7f3bf415c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:05.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.041+0000 7f3bf159a640 1 --2- 172.21.15.150:0/940921483 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bdc0039b0 0x7f3bdc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:05.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.041+0000 7f3bf159a640 1 --2- 172.21.15.150:0/940921483 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3bf409ad90 0x7f3bdc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:05.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.041+0000 7f3bf159a640 1 --2- 172.21.15.150:0/940921483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bdc004d40 0x7f3bdc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:05.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.041+0000 7f3bf159a640 1 -- 172.21.15.150:0/940921483 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf40cfdb0 con 0x7f3bdc004d40 2024-03-17T23:49:05.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.041+0000 7f3bf159a640 1 -- 172.21.15.150:0/940921483 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf40d0090 con 0x7f3bdc0039b0 2024-03-17T23:49:05.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.041+0000 7f3bf159a640 1 -- 172.21.15.150:0/940921483 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3bdc00b9c0 con 0x7f3bf409ad90 2024-03-17T23:49:05.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.041+0000 7f3bf2d9d640 1 --2- 172.21.15.150:0/940921483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bdc004d40 0x7f3bdc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:05.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.041+0000 7f3bf259c640 1 --2- 172.21.15.150:0/940921483 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bdc0039b0 0x7f3bdc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:05.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.041+0000 7f3bf1d9b640 1 --2- 172.21.15.150:0/940921483 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3bf409ad90 0x7f3bdc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:05.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.041+0000 7f3bf2d9d640 1 -- 172.21.15.150:0/940921483 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3bf409ad90 msgr2=0x7f3bdc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:05.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.041+0000 7f3bf2d9d640 1 --2- 172.21.15.150:0/940921483 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3bf409ad90 0x7f3bdc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:05.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.041+0000 7f3bf2d9d640 1 -- 172.21.15.150:0/940921483 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bdc0039b0 msgr2=0x7f3bdc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:05.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.041+0000 7f3bf2d9d640 1 --2- 172.21.15.150:0/940921483 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bdc0039b0 0x7f3bdc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:05.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.041+0000 7f3bf2d9d640 1 -- 172.21.15.150:0/940921483 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bdc003520 con 0x7f3bdc004d40 2024-03-17T23:49:05.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.042+0000 7f3bf2d9d640 1 --2- 172.21.15.150:0/940921483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bdc004d40 0x7f3bdc0051c0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f3bec00c210 tx=0x7f3bdc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb0ddc4489bc7c8b server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:05.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.042+0000 7f3bf159a640 1 -- 172.21.15.150:0/940921483 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bec007d70 con 0x7f3bdc004d40 2024-03-17T23:49:05.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.042+0000 7f3bf159a640 1 -- 172.21.15.150:0/940921483 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bec030070 con 0x7f3bdc004d40 2024-03-17T23:49:05.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.042+0000 7f3bf159a640 1 -- 172.21.15.150:0/940921483 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bec02cd60 con 0x7f3bdc004d40 2024-03-17T23:49:05.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.043+0000 7f3bf89b9640 1 -- 172.21.15.150:0/940921483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bdc004d40 msgr2=0x7f3bdc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:05.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.043+0000 7f3bf89b9640 1 --2- 172.21.15.150:0/940921483 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bdc004d40 0x7f3bdc0051c0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f3bec00c210 tx=0x7f3bdc005830 comp rx=0 tx=0).stop 2024-03-17T23:49:05.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.043+0000 7f3bf2d9d640 1 -- 172.21.15.150:0/940921483 reap_dead start 2024-03-17T23:49:05.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.043+0000 7f3bf89b9640 1 -- 172.21.15.150:0/940921483 shutdown_connections 2024-03-17T23:49:05.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.043+0000 7f3bf89b9640 1 -- 172.21.15.150:0/940921483 >> 172.21.15.150:0/940921483 conn(0x7f3bf40b3630 msgr2=0x7f3bf40b3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:05.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.043+0000 7f3bf89b9640 1 -- 172.21.15.150:0/940921483 shutdown_connections 2024-03-17T23:49:05.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.043+0000 7f3bf89b9640 1 -- 172.21.15.150:0/940921483 wait complete. 2024-03-17T23:49:05.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.044+0000 7f3bf89b9640 1 Processor -- start 2024-03-17T23:49:05.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.044+0000 7f3bf89b9640 1 -- start start 2024-03-17T23:49:05.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.045+0000 7f3bf89b9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bdc004d40 0x7f3bf4153f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:05.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.045+0000 7f3bf89b9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bf40cb320 0x7f3bf40cb780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:05.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.045+0000 7f3bf89b9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3bf415e140 0x7f3bf415e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:05.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.045+0000 7f3bf89b9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf40cf890 con 0x7f3bdc004d40 2024-03-17T23:49:05.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.045+0000 7f3bf89b9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf40cf710 con 0x7f3bf40cb320 2024-03-17T23:49:05.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.045+0000 7f3bf89b9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3bf40cd5f0 con 0x7f3bf415e140 2024-03-17T23:49:05.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.045+0000 7f3bf259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bdc004d40 0x7f3bf4153f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:05.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.045+0000 7f3bf259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bdc004d40 0x7f3bf4153f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42378/0 (socket says 172.21.15.150:42378) 2024-03-17T23:49:05.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.045+0000 7f3bf259c640 1 -- 172.21.15.150:0/1964016140 learned_addr learned my addr 172.21.15.150:0/1964016140 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:05.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.045+0000 7f3bf1d9b640 1 --2- 172.21.15.150:0/1964016140 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bf40cb320 0x7f3bf40cb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:05.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.045+0000 7f3bf2d9d640 1 --2- 172.21.15.150:0/1964016140 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3bf415e140 0x7f3bf415e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:05.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.045+0000 7f3bf259c640 1 -- 172.21.15.150:0/1964016140 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3bf415e140 msgr2=0x7f3bf415e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:05.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.045+0000 7f3bf259c640 1 --2- 172.21.15.150:0/1964016140 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3bf415e140 0x7f3bf415e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:05.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.045+0000 7f3bf259c640 1 -- 172.21.15.150:0/1964016140 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bf40cb320 msgr2=0x7f3bf40cb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:05.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.045+0000 7f3bf259c640 1 --2- 172.21.15.150:0/1964016140 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bf40cb320 0x7f3bf40cb780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:05.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.045+0000 7f3bf259c640 1 -- 172.21.15.150:0/1964016140 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bf41f70e0 con 0x7f3bdc004d40 2024-03-17T23:49:05.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.045+0000 7f3bf1d9b640 1 --2- 172.21.15.150:0/1964016140 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bf40cb320 0x7f3bf40cb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:49:05.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.045+0000 7f3bf259c640 1 --2- 172.21.15.150:0/1964016140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bdc004d40 0x7f3bf4153f70 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f3be4009f90 tx=0x7f3be400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:05.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.045+0000 7f3be2ffd640 1 -- 172.21.15.150:0/1964016140 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be4015430 con 0x7f3bdc004d40 2024-03-17T23:49:05.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.045+0000 7f3be2ffd640 1 -- 172.21.15.150:0/1964016140 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3be40155d0 con 0x7f3bdc004d40 2024-03-17T23:49:05.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.045+0000 7f3bf89b9640 1 -- 172.21.15.150:0/1964016140 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bf41f73c0 con 0x7f3bdc004d40 2024-03-17T23:49:05.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.045+0000 7f3be2ffd640 1 -- 172.21.15.150:0/1964016140 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be40159c0 con 0x7f3bdc004d40 2024-03-17T23:49:05.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.046+0000 7f3bf89b9640 1 -- 172.21.15.150:0/1964016140 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bf41f7980 con 0x7f3bdc004d40 2024-03-17T23:49:05.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.046+0000 7f3bf89b9640 1 -- 172.21.15.150:0/1964016140 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bf40681a0 con 0x7f3bdc004d40 2024-03-17T23:49:05.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.047+0000 7f3be2ffd640 1 -- 172.21.15.150:0/1964016140 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3be4015b60 con 0x7f3bdc004d40 2024-03-17T23:49:05.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.047+0000 7f3be2ffd640 1 --2- 172.21.15.150:0/1964016140 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3bd406bbd0 0x7f3bd406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:05.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.047+0000 7f3be2ffd640 1 -- 172.21.15.150:0/1964016140 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5335+0+0 (secure 0 0 0) 0x7f3be408bac0 con 0x7f3bdc004d40 2024-03-17T23:49:05.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.047+0000 7f3bf1d9b640 1 --2- 172.21.15.150:0/1964016140 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3bd406bbd0 0x7f3bd406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:05.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.048+0000 7f3bf1d9b640 1 --2- 172.21.15.150:0/1964016140 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3bd406bbd0 0x7f3bd406e090 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3bf41f6f30 tx=0x7f3be8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:05.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.052+0000 7f3be2ffd640 1 -- 172.21.15.150:0/1964016140 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3be4019030 con 0x7f3bdc004d40 2024-03-17T23:49:05.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.291+0000 7f3bf89b9640 1 -- 172.21.15.150:0/1964016140 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]} v 0) -- 0x7f3bf40aea80 con 0x7f3bdc004d40 2024-03-17T23:49:05.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.321+0000 7f3be2ffd640 1 -- 172.21.15.150:0/1964016140 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]}]=0 v144) ==== 137+0+0 (secure 0 0 0) 0x7f3be4010040 con 0x7f3bdc004d40 2024-03-17T23:49:05.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.323+0000 7f3bf89b9640 1 -- 172.21.15.150:0/1964016140 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3bd406bbd0 msgr2=0x7f3bd406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:05.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.323+0000 7f3bf89b9640 1 --2- 172.21.15.150:0/1964016140 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3bd406bbd0 0x7f3bd406e090 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3bf41f6f30 tx=0x7f3be8008040 comp rx=0 tx=0).stop 2024-03-17T23:49:05.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.323+0000 7f3bf89b9640 1 -- 172.21.15.150:0/1964016140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bdc004d40 msgr2=0x7f3bf4153f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:05.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.323+0000 7f3bf89b9640 1 --2- 172.21.15.150:0/1964016140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bdc004d40 0x7f3bf4153f70 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f3be4009f90 tx=0x7f3be400f580 comp rx=0 tx=0).stop 2024-03-17T23:49:05.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.323+0000 7f3bf89b9640 1 -- 172.21.15.150:0/1964016140 shutdown_connections 2024-03-17T23:49:05.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.323+0000 7f3bf89b9640 1 --2- 172.21.15.150:0/1964016140 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3bf415e140 0x7f3bf415e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:05.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.323+0000 7f3bf89b9640 1 --2- 172.21.15.150:0/1964016140 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3bd406bbd0 0x7f3bd406e090 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:05.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.323+0000 7f3bf89b9640 1 --2- 172.21.15.150:0/1964016140 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bf40cb320 0x7f3bf40cb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:05.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.323+0000 7f3bf89b9640 1 --2- 172.21.15.150:0/1964016140 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3bdc004d40 0x7f3bf4153f70 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:05.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.323+0000 7f3bf89b9640 1 -- 172.21.15.150:0/1964016140 >> 172.21.15.150:0/1964016140 conn(0x7f3bf40b3630 msgr2=0x7f3bf40aed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:05.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.323+0000 7f3bf89b9640 1 -- 172.21.15.150:0/1964016140 shutdown_connections 2024-03-17T23:49:05.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.323+0000 7f3bf89b9640 1 -- 172.21.15.150:0/1964016140 wait complete. 2024-03-17T23:49:05.335 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-17T23:49:05.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.437+0000 7f5878759640 1 Processor -- start 2024-03-17T23:49:05.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.437+0000 7f5878759640 1 -- start start 2024-03-17T23:49:05.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.438+0000 7f5878759640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f587015e0a0 0x7f587015e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:05.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.438+0000 7f5878759640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f587006c2a0 0x7f5870099db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:05.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.438+0000 7f5878759640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58700d0040 con 0x7f587009a2f0 2024-03-17T23:49:05.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.438+0000 7f5878759640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f587015edb0 con 0x7f587015e0a0 2024-03-17T23:49:05.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.438+0000 7f5878759640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f587015f090 con 0x7f587006c2a0 2024-03-17T23:49:05.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.438+0000 7f5876cf6640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f587009a2f0 0x7f5870161ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:48116/0 (socket says 172.21.15.150:48116) 2024-03-17T23:49:05.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.438+0000 7f5876cf6640 1 -- 172.21.15.150:0/4269940759 learned_addr learned my addr 172.21.15.150:0/4269940759 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:05.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.439+0000 7f5875cf4640 1 --2- 172.21.15.150:0/4269940759 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f587006c2a0 0x7f5870099db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:05.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.439+0000 7f58764f5640 1 --2- 172.21.15.150:0/4269940759 >> v2:172.21.15.157:3300/0 conn(0x7f587015e0a0 0x7f587015e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:05.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.439+0000 7f58754f3640 1 -- 172.21.15.150:0/4269940759 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3186891827 0 0) 0x7f58700d0040 con 0x7f587009a2f0 2024-03-17T23:49:05.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.439+0000 7f58754f3640 1 -- 172.21.15.150:0/4269940759 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f585c003520 con 0x7f587009a2f0 2024-03-17T23:49:05.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.439+0000 7f58764f5640 1 -- 172.21.15.150:0/4269940759 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f587006c2a0 msgr2=0x7f5870099db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:05.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.439+0000 7f58764f5640 1 --2- 172.21.15.150:0/4269940759 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f587006c2a0 0x7f5870099db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:05.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.439+0000 7f58764f5640 1 -- 172.21.15.150:0/4269940759 >> v1:172.21.15.150:6789/0 conn(0x7f587009a2f0 legacy=0x7f5870161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:05.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.439+0000 7f58764f5640 1 -- 172.21.15.150:0/4269940759 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f587015f370 con 0x7f587015e0a0 2024-03-17T23:49:05.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.440+0000 7f58764f5640 1 --2- 172.21.15.150:0/4269940759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f587015e0a0 0x7f587015e480 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f586c009d90 tx=0x7f586c02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6026a0d18f775bee server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:05.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.440+0000 7f58754f3640 1 -- 172.21.15.150:0/4269940759 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f586c039070 con 0x7f587015e0a0 2024-03-17T23:49:05.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.440+0000 7f58754f3640 1 -- 172.21.15.150:0/4269940759 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f586c02cc50 con 0x7f587015e0a0 2024-03-17T23:49:05.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.440+0000 7f58754f3640 1 -- 172.21.15.150:0/4269940759 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f586c034440 con 0x7f587015e0a0 2024-03-17T23:49:05.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.440+0000 7f5878759640 1 -- 172.21.15.150:0/4269940759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f587015e0a0 msgr2=0x7f587015e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:05.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.440+0000 7f5878759640 1 --2- 172.21.15.150:0/4269940759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f587015e0a0 0x7f587015e480 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f586c009d90 tx=0x7f586c02bea0 comp rx=0 tx=0).stop 2024-03-17T23:49:05.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.440+0000 7f5878759640 1 -- 172.21.15.150:0/4269940759 shutdown_connections 2024-03-17T23:49:05.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.440+0000 7f5878759640 1 --2- 172.21.15.150:0/4269940759 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f587006c2a0 0x7f5870099db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:05.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.440+0000 7f5878759640 1 --2- 172.21.15.150:0/4269940759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f587015e0a0 0x7f587015e480 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:05.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.440+0000 7f5878759640 1 -- 172.21.15.150:0/4269940759 >> 172.21.15.150:0/4269940759 conn(0x7f58700b36c0 msgr2=0x7f58700b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:05.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.440+0000 7f5878759640 1 -- 172.21.15.150:0/4269940759 shutdown_connections 2024-03-17T23:49:05.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.441+0000 7f5878759640 1 -- 172.21.15.150:0/4269940759 wait complete. 2024-03-17T23:49:05.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.442+0000 7f5878759640 1 Processor -- start 2024-03-17T23:49:05.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.442+0000 7f5878759640 1 -- start start 2024-03-17T23:49:05.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.442+0000 7f5878759640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f587006c2a0 0x7f587014c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:05.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.442+0000 7f5878759640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f587009a2f0 0x7f587009e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:05.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.442+0000 7f5878759640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f587015e0a0 0x7f587009e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:05.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.442+0000 7f5878759640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f587015eb40 con 0x7f587006c2a0 2024-03-17T23:49:05.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.442+0000 7f5878759640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f587015e9c0 con 0x7f587009a2f0 2024-03-17T23:49:05.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.442+0000 7f5878759640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f58700d1530 con 0x7f587015e0a0 2024-03-17T23:49:05.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.442+0000 7f58764f5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f587006c2a0 0x7f587014c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:05.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.442+0000 7f58764f5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f587006c2a0 0x7f587014c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:42394/0 (socket says 172.21.15.150:42394) 2024-03-17T23:49:05.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.442+0000 7f58764f5640 1 -- 172.21.15.150:0/2008924652 learned_addr learned my addr 172.21.15.150:0/2008924652 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:05.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.443+0000 7f58764f5640 1 -- 172.21.15.150:0/2008924652 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f587015e0a0 msgr2=0x7f587009e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:05.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.443+0000 7f5876cf6640 1 --2- 172.21.15.150:0/2008924652 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f587015e0a0 0x7f587009e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:05.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.443+0000 7f5875cf4640 1 --2- 172.21.15.150:0/2008924652 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f587009a2f0 0x7f587009e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:05.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.443+0000 7f58764f5640 1 --2- 172.21.15.150:0/2008924652 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f587015e0a0 0x7f587009e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:05.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.443+0000 7f58764f5640 1 -- 172.21.15.150:0/2008924652 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f587009a2f0 msgr2=0x7f587009e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:05.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.443+0000 7f58764f5640 1 --2- 172.21.15.150:0/2008924652 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f587009a2f0 0x7f587009e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:05.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.443+0000 7f58764f5640 1 -- 172.21.15.150:0/2008924652 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58700b7020 con 0x7f587006c2a0 2024-03-17T23:49:05.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.443+0000 7f5876cf6640 1 --2- 172.21.15.150:0/2008924652 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f587015e0a0 0x7f587009e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:49:05.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.443+0000 7f58764f5640 1 --2- 172.21.15.150:0/2008924652 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f587006c2a0 0x7f587014c730 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f586c037040 tx=0x7f586c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:05.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.443+0000 7f5866ffd640 1 -- 172.21.15.150:0/2008924652 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f586c039050 con 0x7f587006c2a0 2024-03-17T23:49:05.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.443+0000 7f5866ffd640 1 -- 172.21.15.150:0/2008924652 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f586c045070 con 0x7f587006c2a0 2024-03-17T23:49:05.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.443+0000 7f5878759640 1 -- 172.21.15.150:0/2008924652 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f587014d300 con 0x7f587006c2a0 2024-03-17T23:49:05.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.443+0000 7f5866ffd640 1 -- 172.21.15.150:0/2008924652 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f586c009450 con 0x7f587006c2a0 2024-03-17T23:49:05.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.443+0000 7f5878759640 1 -- 172.21.15.150:0/2008924652 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f587014dba0 con 0x7f587006c2a0 2024-03-17T23:49:05.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.444+0000 7f5878759640 1 -- 172.21.15.150:0/2008924652 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f587014d4d0 con 0x7f587006c2a0 2024-03-17T23:49:05.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.444+0000 7f5866ffd640 1 -- 172.21.15.150:0/2008924652 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f586c03e610 con 0x7f587006c2a0 2024-03-17T23:49:05.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.445+0000 7f5866ffd640 1 --2- 172.21.15.150:0/2008924652 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f585406bc00 0x7f585406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:05.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.445+0000 7f5866ffd640 1 -- 172.21.15.150:0/2008924652 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5335+0+0 (secure 0 0 0) 0x7f586c0ad320 con 0x7f587006c2a0 2024-03-17T23:49:05.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.445+0000 7f5875cf4640 1 --2- 172.21.15.150:0/2008924652 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f585406bc00 0x7f585406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:05.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.445+0000 7f5875cf4640 1 --2- 172.21.15.150:0/2008924652 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f585406bc00 0x7f585406e0c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5860009a10 tx=0x7f5860008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:05.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.449+0000 7f5866ffd640 1 -- 172.21.15.150:0/2008924652 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f586c07c0a0 con 0x7f587006c2a0 2024-03-17T23:49:05.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.619+0000 7f5878759640 1 -- 172.21.15.150:0/2008924652 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f58700688b0 con 0x7f587006c2a0 2024-03-17T23:49:05.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.622+0000 7f5866ffd640 1 -- 172.21.15.150:0/2008924652 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) ==== 122+0+0 (secure 0 0 0) 0x7f586c07b9a0 con 0x7f587006c2a0 2024-03-17T23:49:05.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.624+0000 7f5878759640 1 -- 172.21.15.150:0/2008924652 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f585406bc00 msgr2=0x7f585406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:05.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.624+0000 7f5878759640 1 --2- 172.21.15.150:0/2008924652 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f585406bc00 0x7f585406e0c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5860009a10 tx=0x7f5860008040 comp rx=0 tx=0).stop 2024-03-17T23:49:05.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.624+0000 7f5878759640 1 -- 172.21.15.150:0/2008924652 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f587006c2a0 msgr2=0x7f587014c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:05.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.624+0000 7f5878759640 1 --2- 172.21.15.150:0/2008924652 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f587006c2a0 0x7f587014c730 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f586c037040 tx=0x7f586c004a40 comp rx=0 tx=0).stop 2024-03-17T23:49:05.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.624+0000 7f5878759640 1 -- 172.21.15.150:0/2008924652 shutdown_connections 2024-03-17T23:49:05.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.624+0000 7f5878759640 1 --2- 172.21.15.150:0/2008924652 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f587015e0a0 0x7f587009e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:05.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.624+0000 7f5878759640 1 --2- 172.21.15.150:0/2008924652 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f585406bc00 0x7f585406e0c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:05.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.624+0000 7f5878759640 1 --2- 172.21.15.150:0/2008924652 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f587009a2f0 0x7f587009e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:05.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.624+0000 7f5878759640 1 --2- 172.21.15.150:0/2008924652 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f587006c2a0 0x7f587014c730 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:05.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.624+0000 7f5878759640 1 -- 172.21.15.150:0/2008924652 >> 172.21.15.150:0/2008924652 conn(0x7f58700b36c0 msgr2=0x7f5870073dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:05.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.624+0000 7f5878759640 1 -- 172.21.15.150:0/2008924652 shutdown_connections 2024-03-17T23:49:05.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.624+0000 7f5878759640 1 -- 172.21.15.150:0/2008924652 wait complete. 2024-03-17T23:49:05.636 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-17T23:49:05.636 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-17T23:49:05.682 DEBUG:teuthology.orchestra.run.smithi157:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-17T23:49:05.729 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-17T23:49:05.729 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:49:05.729 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:49:05.744 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19488 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-17T23:49:05.745 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:49:05.745 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:49:05.762 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=626 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-17T23:49:05.762 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:49:05.762 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:49:05.800 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19488 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-17T23:49:05.800 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:49:05.801 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:49:05.819 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=626 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-17T23:49:05.819 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-17T23:49:05.819 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:49:05.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.940+0000 7fde5a37e640 1 Processor -- start 2024-03-17T23:49:05.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.940+0000 7fde5a37e640 1 -- start start 2024-03-17T23:49:05.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.941+0000 7fde5a37e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde5415bd60 0x7fde540d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:05.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.941+0000 7fde5a37e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde540d10d0 con 0x7fde540ad890 2024-03-17T23:49:05.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.941+0000 7fde5a37e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde540679d0 con 0x7fde54099a70 2024-03-17T23:49:05.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.941+0000 7fde5a37e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fde5415c140 con 0x7fde5415bd60 2024-03-17T23:49:05.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.941+0000 7fde5891b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fde540ad890 0x7fde5406b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55560/0 (socket says 172.21.15.150:55560) 2024-03-17T23:49:05.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.941+0000 7fde5891b640 1 -- 172.21.15.150:0/1728248237 learned_addr learned my addr 172.21.15.150:0/1728248237 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:05.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.941+0000 7fde53fff640 1 --2- 172.21.15.150:0/1728248237 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde5415bd60 0x7fde540d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:05.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.941+0000 7fde52ffd640 1 -- 172.21.15.150:0/1728248237 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 123464995 0 0) 0x7fde540d10d0 con 0x7fde540ad890 2024-03-17T23:49:05.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.941+0000 7fde52ffd640 1 -- 172.21.15.150:0/1728248237 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde3c003520 con 0x7fde540ad890 2024-03-17T23:49:05.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.942+0000 7fde52ffd640 1 -- 172.21.15.150:0/1728248237 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3673712739 0 0) 0x7fde3c003520 con 0x7fde540ad890 2024-03-17T23:49:05.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.942+0000 7fde52ffd640 1 -- 172.21.15.150:0/1728248237 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde5415bd60 msgr2=0x7fde540d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:05.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.942+0000 7fde52ffd640 1 --2- 172.21.15.150:0/1728248237 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde5415bd60 0x7fde540d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:05.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.942+0000 7fde52ffd640 1 -- 172.21.15.150:0/1728248237 >> v1:172.21.15.157:6789/0 conn(0x7fde54099a70 legacy=0x7fde540ad180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:05.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.942+0000 7fde52ffd640 1 -- 172.21.15.150:0/1728248237 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde540cf920 con 0x7fde540ad890 2024-03-17T23:49:05.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.942+0000 7fde52ffd640 1 -- 172.21.15.150:0/1728248237 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7fde4c0019f0 con 0x7fde540ad890 2024-03-17T23:49:05.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.942+0000 7fde52ffd640 1 -- 172.21.15.150:0/1728248237 >> v1:172.21.15.150:6789/0 conn(0x7fde540ad890 legacy=0x7fde5406b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:05.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.942+0000 7fde52ffd640 1 --2- 172.21.15.150:0/1728248237 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde3c0039e0 0x7fde3c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:05.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.942+0000 7fde52ffd640 1 --2- 172.21.15.150:0/1728248237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde3c004800 0x7fde3c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:05.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.942+0000 7fde52ffd640 1 --2- 172.21.15.150:0/1728248237 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde5415bd60 0x7fde3c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:05.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.942+0000 7fde52ffd640 1 -- 172.21.15.150:0/1728248237 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fde5415c140 con 0x7fde3c004800 2024-03-17T23:49:05.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.942+0000 7fde52ffd640 1 -- 172.21.15.150:0/1728248237 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fde3c00ba70 con 0x7fde3c0039e0 2024-03-17T23:49:05.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.942+0000 7fde52ffd640 1 -- 172.21.15.150:0/1728248237 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fde3c00bc70 con 0x7fde5415bd60 2024-03-17T23:49:05.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.942+0000 7fde537fe640 1 --2- 172.21.15.150:0/1728248237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde3c004800 0x7fde3c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:05.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.942+0000 7fde53fff640 1 --2- 172.21.15.150:0/1728248237 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde3c0039e0 0x7fde3c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:05.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.942+0000 7fde5891b640 1 --2- 172.21.15.150:0/1728248237 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde5415bd60 0x7fde3c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:05.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.942+0000 7fde537fe640 1 -- 172.21.15.150:0/1728248237 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde5415bd60 msgr2=0x7fde3c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:05.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.942+0000 7fde537fe640 1 --2- 172.21.15.150:0/1728248237 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde5415bd60 0x7fde3c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:05.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.942+0000 7fde537fe640 1 -- 172.21.15.150:0/1728248237 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde3c0039e0 msgr2=0x7fde3c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:05.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.942+0000 7fde537fe640 1 --2- 172.21.15.150:0/1728248237 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde3c0039e0 0x7fde3c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:05.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.942+0000 7fde537fe640 1 -- 172.21.15.150:0/1728248237 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde3c003520 con 0x7fde3c004800 2024-03-17T23:49:05.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.943+0000 7fde537fe640 1 --2- 172.21.15.150:0/1728248237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde3c004800 0x7fde3c004c80 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fde48001720 tx=0x7fde3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b354177b5e55c156 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:05.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.943+0000 7fde52ffd640 1 -- 172.21.15.150:0/1728248237 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde48001370 con 0x7fde3c004800 2024-03-17T23:49:05.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.943+0000 7fde52ffd640 1 -- 172.21.15.150:0/1728248237 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde480052d0 con 0x7fde3c004800 2024-03-17T23:49:05.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.943+0000 7fde52ffd640 1 -- 172.21.15.150:0/1728248237 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde48003830 con 0x7fde3c004800 2024-03-17T23:49:05.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.944+0000 7fde5a37e640 1 -- 172.21.15.150:0/1728248237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde3c004800 msgr2=0x7fde3c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:05.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.944+0000 7fde5a37e640 1 --2- 172.21.15.150:0/1728248237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde3c004800 0x7fde3c004c80 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fde48001720 tx=0x7fde3c005830 comp rx=0 tx=0).stop 2024-03-17T23:49:05.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.944+0000 7fde5891b640 1 -- 172.21.15.150:0/1728248237 reap_dead start 2024-03-17T23:49:05.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.944+0000 7fde5a37e640 1 -- 172.21.15.150:0/1728248237 shutdown_connections 2024-03-17T23:49:05.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.944+0000 7fde5a37e640 1 -- 172.21.15.150:0/1728248237 >> 172.21.15.150:0/1728248237 conn(0x7fde540b3da0 msgr2=0x7fde540b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:05.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.944+0000 7fde5a37e640 1 -- 172.21.15.150:0/1728248237 shutdown_connections 2024-03-17T23:49:05.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.944+0000 7fde5a37e640 1 -- 172.21.15.150:0/1728248237 wait complete. 2024-03-17T23:49:05.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.945+0000 7fde5a37e640 1 Processor -- start 2024-03-17T23:49:05.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.945+0000 7fde5a37e640 1 -- start start 2024-03-17T23:49:05.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.946+0000 7fde5a37e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde3c004800 0x7fde54150130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:05.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.946+0000 7fde5a37e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde5414cd20 0x7fde5406f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:05.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.946+0000 7fde5a37e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde5414d1a0 0x7fde5414d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:05.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.946+0000 7fde5a37e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fde540d01c0 con 0x7fde3c004800 2024-03-17T23:49:05.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.946+0000 7fde5a37e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fde540cfee0 con 0x7fde5414d1a0 2024-03-17T23:49:05.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.946+0000 7fde5a37e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fde540d06e0 con 0x7fde5414cd20 2024-03-17T23:49:05.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.946+0000 7fde53fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde3c004800 0x7fde54150130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:05.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.946+0000 7fde53fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde3c004800 0x7fde54150130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43676/0 (socket says 172.21.15.150:43676) 2024-03-17T23:49:05.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.946+0000 7fde53fff640 1 -- 172.21.15.150:0/1373798133 learned_addr learned my addr 172.21.15.150:0/1373798133 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:05.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.946+0000 7fde537fe640 1 --2- 172.21.15.150:0/1373798133 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde5414cd20 0x7fde5406f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:05.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.946+0000 7fde5891b640 1 --2- 172.21.15.150:0/1373798133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde5414d1a0 0x7fde5414d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:05.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.946+0000 7fde53fff640 1 -- 172.21.15.150:0/1373798133 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde5414cd20 msgr2=0x7fde5406f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:05.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.946+0000 7fde53fff640 1 --2- 172.21.15.150:0/1373798133 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde5414cd20 0x7fde5406f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:05.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.946+0000 7fde53fff640 1 -- 172.21.15.150:0/1373798133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde5414d1a0 msgr2=0x7fde5414d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:05.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.946+0000 7fde53fff640 1 --2- 172.21.15.150:0/1373798133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde5414d1a0 0x7fde5414d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:05.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.946+0000 7fde53fff640 1 -- 172.21.15.150:0/1373798133 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde5406e8b0 con 0x7fde3c004800 2024-03-17T23:49:05.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.946+0000 7fde5891b640 1 --2- 172.21.15.150:0/1373798133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde5414d1a0 0x7fde5414d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:49:05.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.946+0000 7fde53fff640 1 --2- 172.21.15.150:0/1373798133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde3c004800 0x7fde54150130 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fde44011910 tx=0x7fde44011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:05.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.946+0000 7fde50ff9640 1 -- 172.21.15.150:0/1373798133 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde4400faf0 con 0x7fde3c004800 2024-03-17T23:49:05.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.946+0000 7fde50ff9640 1 -- 172.21.15.150:0/1373798133 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde4400fc90 con 0x7fde3c004800 2024-03-17T23:49:05.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.946+0000 7fde5a37e640 1 -- 172.21.15.150:0/1373798133 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde5406eb90 con 0x7fde3c004800 2024-03-17T23:49:05.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.946+0000 7fde50ff9640 1 -- 172.21.15.150:0/1373798133 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde44014440 con 0x7fde3c004800 2024-03-17T23:49:05.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.946+0000 7fde5a37e640 1 -- 172.21.15.150:0/1373798133 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde5406f150 con 0x7fde3c004800 2024-03-17T23:49:05.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.948+0000 7fde5a37e640 1 -- 172.21.15.150:0/1373798133 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde540681a0 con 0x7fde3c004800 2024-03-17T23:49:05.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.948+0000 7fde50ff9640 1 -- 172.21.15.150:0/1373798133 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fde44012040 con 0x7fde3c004800 2024-03-17T23:49:05.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.948+0000 7fde50ff9640 1 --2- 172.21.15.150:0/1373798133 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fde4006bbd0 0x7fde4006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:05.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.948+0000 7fde50ff9640 1 -- 172.21.15.150:0/1373798133 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5335+0+0 (secure 0 0 0) 0x7fde4408af30 con 0x7fde3c004800 2024-03-17T23:49:05.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.948+0000 7fde537fe640 1 --2- 172.21.15.150:0/1373798133 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fde4006bbd0 0x7fde4006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:05.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.949+0000 7fde537fe640 1 --2- 172.21.15.150:0/1373798133 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fde4006bbd0 0x7fde4006e090 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fde48033000 tx=0x7fde48009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:05.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:05.953+0000 7fde50ff9640 1 -- 172.21.15.150:0/1373798133 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fde44019030 con 0x7fde3c004800 2024-03-17T23:49:06.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.160+0000 7fde5a37e640 1 -- 172.21.15.150:0/1373798133 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde540008d0 con 0x7fde3c004800 2024-03-17T23:49:06.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.161+0000 7fde50ff9640 1 -- 172.21.15.150:0/1373798133 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+3881 (secure 0 0 0) 0x7fde44059c20 con 0x7fde3c004800 2024-03-17T23:49:06.161 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:06.162 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":18,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5177,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6834/3126446677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":3126446677},{"type":"v1","addr":"172.21.15.157:6834","nonce":3126446677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5472,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1992779366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1992779366},{"type":"v1","addr":"172.21.15.150:6836","nonce":1992779366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5487,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/3615467865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6833","nonce":3615467865},{"type":"v1","addr":"172.21.15.157:6835","nonce":3615467865}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":6105,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3515841248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":3515841248},{"type":"v1","addr":"172.21.15.150:6837","nonce":3515841248}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[]} 2024-03-17T23:49:06.162 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 18 2024-03-17T23:49:06.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.163+0000 7fde5a37e640 1 -- 172.21.15.150:0/1373798133 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fde4006bbd0 msgr2=0x7fde4006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:06.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.163+0000 7fde5a37e640 1 --2- 172.21.15.150:0/1373798133 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fde4006bbd0 0x7fde4006e090 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fde48033000 tx=0x7fde48009030 comp rx=0 tx=0).stop 2024-03-17T23:49:06.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.163+0000 7fde5a37e640 1 -- 172.21.15.150:0/1373798133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde3c004800 msgr2=0x7fde54150130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:06.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.163+0000 7fde5a37e640 1 --2- 172.21.15.150:0/1373798133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde3c004800 0x7fde54150130 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fde44011910 tx=0x7fde44011dd0 comp rx=0 tx=0).stop 2024-03-17T23:49:06.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.163+0000 7fde5a37e640 1 -- 172.21.15.150:0/1373798133 shutdown_connections 2024-03-17T23:49:06.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.163+0000 7fde5a37e640 1 --2- 172.21.15.150:0/1373798133 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fde4006bbd0 0x7fde4006e090 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:06.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.163+0000 7fde5a37e640 1 --2- 172.21.15.150:0/1373798133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fde5414d1a0 0x7fde5414d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:06.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.163+0000 7fde5a37e640 1 --2- 172.21.15.150:0/1373798133 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fde5414cd20 0x7fde5406f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:06.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.163+0000 7fde5a37e640 1 --2- 172.21.15.150:0/1373798133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fde3c004800 0x7fde54150130 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:06.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.163+0000 7fde5a37e640 1 -- 172.21.15.150:0/1373798133 >> 172.21.15.150:0/1373798133 conn(0x7fde540b3da0 msgr2=0x7fde540b0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:06.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.163+0000 7fde5a37e640 1 -- 172.21.15.150:0/1373798133 shutdown_connections 2024-03-17T23:49:06.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.163+0000 7fde5a37e640 1 -- 172.21.15.150:0/1373798133 wait complete. 2024-03-17T23:49:06.175 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-17T23:49:06.175 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-17T23:49:06.175 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-17T23:49:06.175 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:49:06.176 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-17T23:49:06.176 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-17T23:49:06.176 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-17T23:49:06.176 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:49:06.176 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-17T23:49:06.176 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-17T23:49:06.176 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-17T23:49:06.176 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:49:06.177 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-17T23:49:06.177 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-17T23:49:06.177 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-17T23:49:06.177 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:49:06.185 INFO:tasks.ceph.mds.a:Stopped 2024-03-17T23:49:06.185 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-17T23:49:06.186 INFO:tasks.ceph.mds.c:Stopped 2024-03-17T23:49:06.186 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-17T23:49:06.188 INFO:tasks.ceph.mds.b:Stopped 2024-03-17T23:49:06.188 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-17T23:49:06.189 INFO:tasks.ceph.mds.d:Stopped 2024-03-17T23:49:06.189 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-17T23:49:06.228 INFO:tasks.ceph.mds.a:Started 2024-03-17T23:49:06.229 DEBUG:teuthology.parallel:result is None 2024-03-17T23:49:06.229 INFO:tasks.ceph.mds.c:Started 2024-03-17T23:49:06.229 DEBUG:teuthology.parallel:result is None 2024-03-17T23:49:06.231 INFO:tasks.ceph.mds.b:Started 2024-03-17T23:49:06.231 DEBUG:teuthology.parallel:result is None 2024-03-17T23:49:06.232 INFO:tasks.ceph.mds.d:Started 2024-03-17T23:49:06.232 DEBUG:teuthology.parallel:result is None 2024-03-17T23:49:06.232 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-17T23:49:06.306 INFO:tasks.ceph.mds.a.smithi150.stdout:starting mds.a at 2024-03-17T23:49:06.306 INFO:tasks.ceph.mds.c.smithi150.stdout:starting mds.c at 2024-03-17T23:49:06.316 INFO:tasks.ceph.mds.d.smithi157.stdout:starting mds.d at 2024-03-17T23:49:06.316 INFO:tasks.ceph.mds.b.smithi157.stdout:starting mds.b at 2024-03-17T23:49:06.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.341+0000 7f114fdf3640 1 Processor -- start 2024-03-17T23:49:06.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.341+0000 7f114fdf3640 1 -- start start 2024-03-17T23:49:06.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.342+0000 7f114fdf3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f114815df60 0x7f11480d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:06.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.342+0000 7f114fdf3640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f11480d2630 0x7f11480b6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:06.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.342+0000 7f114fdf3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11480d2c50 con 0x7f11480d0bd0 2024-03-17T23:49:06.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.342+0000 7f114fdf3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f11480d1030 con 0x7f114815df60 2024-03-17T23:49:06.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.342+0000 7f114fdf3640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f11480d2f30 con 0x7f11480d2630 2024-03-17T23:49:06.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.342+0000 7f114d38e640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f11480d0bd0 0x7f11480d1f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55580/0 (socket says 172.21.15.150:55580) 2024-03-17T23:49:06.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.342+0000 7f114d38e640 1 -- 172.21.15.150:0/2406641557 learned_addr learned my addr 172.21.15.150:0/2406641557 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:06.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.342+0000 7f114db8f640 1 --2- 172.21.15.150:0/2406641557 >> v2:172.21.15.150:3300/0 conn(0x7f114815df60 0x7f11480d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:06.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.342+0000 7f114e390640 1 --2- 172.21.15.150:0/2406641557 >> v2:172.21.15.157:3300/0 conn(0x7f11480d2630 0x7f11480b6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:06.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.342+0000 7f114cb8d640 1 -- 172.21.15.150:0/2406641557 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 631100215 0 0) 0x7f11480d2c50 con 0x7f11480d0bd0 2024-03-17T23:49:06.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.342+0000 7f114cb8d640 1 -- 172.21.15.150:0/2406641557 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1134003520 con 0x7f11480d0bd0 2024-03-17T23:49:06.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.342+0000 7f114db8f640 1 -- 172.21.15.150:0/2406641557 >> v2:172.21.15.157:3300/0 conn(0x7f11480d2630 msgr2=0x7f11480b6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:06.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.342+0000 7f114db8f640 1 --2- 172.21.15.150:0/2406641557 >> v2:172.21.15.157:3300/0 conn(0x7f11480d2630 0x7f11480b6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:06.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.342+0000 7f114db8f640 1 -- 172.21.15.150:0/2406641557 >> v1:172.21.15.150:6789/0 conn(0x7f11480d0bd0 legacy=0x7f11480d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:06.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.342+0000 7f114db8f640 1 -- 172.21.15.150:0/2406641557 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11480d3210 con 0x7f114815df60 2024-03-17T23:49:06.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.343+0000 7f114db8f640 1 --2- 172.21.15.150:0/2406641557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f114815df60 0x7f11480d0690 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f114806db40 tx=0x7f114402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6202b99cb9c66984 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:06.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.343+0000 7f114cb8d640 1 -- 172.21.15.150:0/2406641557 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1144034070 con 0x7f114815df60 2024-03-17T23:49:06.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.343+0000 7f114cb8d640 1 -- 172.21.15.150:0/2406641557 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1144030610 con 0x7f114815df60 2024-03-17T23:49:06.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.343+0000 7f114cb8d640 1 -- 172.21.15.150:0/2406641557 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11440309e0 con 0x7f114815df60 2024-03-17T23:49:06.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.344+0000 7f114fdf3640 1 -- 172.21.15.150:0/2406641557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f114815df60 msgr2=0x7f11480d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:06.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.344+0000 7f114fdf3640 1 --2- 172.21.15.150:0/2406641557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f114815df60 0x7f11480d0690 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f114806db40 tx=0x7f114402b960 comp rx=0 tx=0).stop 2024-03-17T23:49:06.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.344+0000 7f114fdf3640 1 -- 172.21.15.150:0/2406641557 shutdown_connections 2024-03-17T23:49:06.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.344+0000 7f114fdf3640 1 --2- 172.21.15.150:0/2406641557 >> v2:172.21.15.157:3300/0 conn(0x7f11480d2630 0x7f11480b6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:06.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.344+0000 7f114fdf3640 1 --2- 172.21.15.150:0/2406641557 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f114815df60 0x7f11480d0690 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:06.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.344+0000 7f114fdf3640 1 -- 172.21.15.150:0/2406641557 >> 172.21.15.150:0/2406641557 conn(0x7f1148072ef0 msgr2=0x7f114806cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:06.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.344+0000 7f114fdf3640 1 -- 172.21.15.150:0/2406641557 shutdown_connections 2024-03-17T23:49:06.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.344+0000 7f114fdf3640 1 -- 172.21.15.150:0/2406641557 wait complete. 2024-03-17T23:49:06.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.345+0000 7f114fdf3640 1 Processor -- start 2024-03-17T23:49:06.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.345+0000 7f114fdf3640 1 -- start start 2024-03-17T23:49:06.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.346+0000 7f114fdf3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11480d0bd0 0x7f11480c8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:06.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.346+0000 7f114fdf3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11480d2630 0x7f11481f7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:06.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.346+0000 7f114fdf3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f114815df60 0x7f11481531f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:06.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.346+0000 7f114fdf3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f114815e4c0 con 0x7f114815df60 2024-03-17T23:49:06.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.346+0000 7f114fdf3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f114815e340 con 0x7f11480d2630 2024-03-17T23:49:06.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.346+0000 7f114e390640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f114815df60 0x7f11481531f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:06.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.346+0000 7f114db8f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11480d0bd0 0x7f11480c8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:06.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.346+0000 7f114e390640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f114815df60 0x7f11481531f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43728/0 (socket says 172.21.15.150:43728) 2024-03-17T23:49:06.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.346+0000 7f114e390640 1 -- 172.21.15.150:0/3331411765 learned_addr learned my addr 172.21.15.150:0/3331411765 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:06.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.346+0000 7f114fdf3640 1 -- 172.21.15.150:0/3331411765 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f11480b74e0 con 0x7f11480d0bd0 2024-03-17T23:49:06.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.346+0000 7f114d38e640 1 --2- 172.21.15.150:0/3331411765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11480d2630 0x7f11481f7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:06.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.346+0000 7f114e390640 1 -- 172.21.15.150:0/3331411765 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11480d0bd0 msgr2=0x7f11480c8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:06.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.346+0000 7f114e390640 1 --2- 172.21.15.150:0/3331411765 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11480d0bd0 0x7f11480c8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:06.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.346+0000 7f114e390640 1 -- 172.21.15.150:0/3331411765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11480d2630 msgr2=0x7f11481f7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:06.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.346+0000 7f114e390640 1 --2- 172.21.15.150:0/3331411765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11480d2630 0x7f11481f7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:06.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.346+0000 7f114e390640 1 -- 172.21.15.150:0/3331411765 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11481f7b20 con 0x7f114815df60 2024-03-17T23:49:06.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.346+0000 7f114db8f640 1 --2- 172.21.15.150:0/3331411765 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11480d0bd0 0x7f11480c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:06.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.347+0000 7f114e390640 1 --2- 172.21.15.150:0/3331411765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f114815df60 0x7f11481531f0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f114000efc0 tx=0x7f114000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:06.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.347+0000 7f113e7fc640 1 -- 172.21.15.150:0/3331411765 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1140018020 con 0x7f114815df60 2024-03-17T23:49:06.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.347+0000 7f113e7fc640 1 -- 172.21.15.150:0/3331411765 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1140014480 con 0x7f114815df60 2024-03-17T23:49:06.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.347+0000 7f114fdf3640 1 -- 172.21.15.150:0/3331411765 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11481f7e00 con 0x7f114815df60 2024-03-17T23:49:06.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.347+0000 7f113e7fc640 1 -- 172.21.15.150:0/3331411765 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1140014870 con 0x7f114815df60 2024-03-17T23:49:06.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.347+0000 7f114fdf3640 1 -- 172.21.15.150:0/3331411765 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1148152cd0 con 0x7f114815df60 2024-03-17T23:49:06.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.348+0000 7f114fdf3640 1 -- 172.21.15.150:0/3331411765 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11480681a0 con 0x7f114815df60 2024-03-17T23:49:06.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.348+0000 7f113e7fc640 1 -- 172.21.15.150:0/3331411765 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1140005d10 con 0x7f114815df60 2024-03-17T23:49:06.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.349+0000 7f113e7fc640 1 --2- 172.21.15.150:0/3331411765 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f112c06bbd0 0x7f112c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:06.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.349+0000 7f113e7fc640 1 -- 172.21.15.150:0/3331411765 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5335+0+0 (secure 0 0 0) 0x7f114008b5a0 con 0x7f114815df60 2024-03-17T23:49:06.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.349+0000 7f114db8f640 1 --2- 172.21.15.150:0/3331411765 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f112c06bbd0 0x7f112c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:06.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.349+0000 7f114db8f640 1 --2- 172.21.15.150:0/3331411765 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f112c06bbd0 0x7f112c06e090 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1144036000 tx=0x7f114403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:06.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.353+0000 7f113e7fc640 1 -- 172.21.15.150:0/3331411765 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1140022030 con 0x7f114815df60 2024-03-17T23:49:06.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:06.539+0000 7f114fdf3640 1 -- 172.21.15.150:0/3331411765 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f11480c9fd0 con 0x7f114815df60 2024-03-17T23:49:07.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.623+0000 7f113e7fc640 1 -- 172.21.15.150:0/3331411765 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v54) ==== 97+0+0 (secure 0 0 0) 0x7f1140010040 con 0x7f114815df60 2024-03-17T23:49:07.624 INFO:teuthology.orchestra.run.smithi150.stderr: removed all blocklist entries 2024-03-17T23:49:07.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.626+0000 7f114fdf3640 1 -- 172.21.15.150:0/3331411765 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f112c06bbd0 msgr2=0x7f112c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:07.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.626+0000 7f114fdf3640 1 --2- 172.21.15.150:0/3331411765 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f112c06bbd0 0x7f112c06e090 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1144036000 tx=0x7f114403b040 comp rx=0 tx=0).stop 2024-03-17T23:49:07.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.626+0000 7f114fdf3640 1 -- 172.21.15.150:0/3331411765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f114815df60 msgr2=0x7f11481531f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:07.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.626+0000 7f114fdf3640 1 --2- 172.21.15.150:0/3331411765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f114815df60 0x7f11481531f0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f114000efc0 tx=0x7f114000c490 comp rx=0 tx=0).stop 2024-03-17T23:49:07.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.627+0000 7f114fdf3640 1 -- 172.21.15.150:0/3331411765 shutdown_connections 2024-03-17T23:49:07.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.627+0000 7f114fdf3640 1 --2- 172.21.15.150:0/3331411765 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f114815df60 0x7f11481531f0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:07.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.627+0000 7f114fdf3640 1 --2- 172.21.15.150:0/3331411765 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f112c06bbd0 0x7f112c06e090 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:07.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.627+0000 7f114fdf3640 1 --2- 172.21.15.150:0/3331411765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11480d2630 0x7f11481f7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:07.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.627+0000 7f114fdf3640 1 --2- 172.21.15.150:0/3331411765 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f11480d0bd0 0x7f11480c8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:07.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.627+0000 7f114fdf3640 1 -- 172.21.15.150:0/3331411765 >> 172.21.15.150:0/3331411765 conn(0x7f1148072ef0 msgr2=0x7f11481552d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:07.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.627+0000 7f114fdf3640 1 -- 172.21.15.150:0/3331411765 shutdown_connections 2024-03-17T23:49:07.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.627+0000 7f114fdf3640 1 -- 172.21.15.150:0/3331411765 wait complete. 2024-03-17T23:49:07.641 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-17T23:49:07.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.746+0000 7f55baf3b640 1 Processor -- start 2024-03-17T23:49:07.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.746+0000 7f55baf3b640 1 -- start start 2024-03-17T23:49:07.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.747+0000 7f55baf3b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55b415de30 0x7f55b41619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:07.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.747+0000 7f55baf3b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f55b40b9860 0x7f55b40cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:07.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.747+0000 7f55baf3b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f55b40cef50 0x7f55b406c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:07.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.747+0000 7f55baf3b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f55b40cf3c0 con 0x7f55b40b9860 2024-03-17T23:49:07.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.747+0000 7f55baf3b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f55b4162940 con 0x7f55b40cef50 2024-03-17T23:49:07.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.747+0000 7f55baf3b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f55b40b9cc0 con 0x7f55b415de30 2024-03-17T23:49:07.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.747+0000 7f55b3fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f55b40b9860 0x7f55b40cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:07.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.747+0000 7f55b3fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f55b40b9860 0x7f55b40cea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43736/0 (socket says 172.21.15.150:43736) 2024-03-17T23:49:07.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.747+0000 7f55b3fff640 1 -- 172.21.15.150:0/4057581609 learned_addr learned my addr 172.21.15.150:0/4057581609 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:07.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.748+0000 7f55b8cd7640 1 --2- 172.21.15.150:0/4057581609 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55b415de30 0x7f55b41619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:07.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.748+0000 7f55b94d8640 1 --2- 172.21.15.150:0/4057581609 >> v2:172.21.15.157:3300/0 conn(0x7f55b40cef50 0x7f55b406c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:07.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.748+0000 7f55b3fff640 1 -- 172.21.15.150:0/4057581609 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55b415de30 msgr2=0x7f55b41619a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:07.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.748+0000 7f55b3fff640 1 --2- 172.21.15.150:0/4057581609 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55b415de30 0x7f55b41619a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:07.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.748+0000 7f55b3fff640 1 -- 172.21.15.150:0/4057581609 >> v2:172.21.15.157:3300/0 conn(0x7f55b40cef50 msgr2=0x7f55b406c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:07.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.748+0000 7f55b3fff640 1 --2- 172.21.15.150:0/4057581609 >> v2:172.21.15.157:3300/0 conn(0x7f55b40cef50 0x7f55b406c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:07.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.748+0000 7f55b3fff640 1 -- 172.21.15.150:0/4057581609 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55b41620a0 con 0x7f55b40b9860 2024-03-17T23:49:07.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.748+0000 7f55b3fff640 1 --2- 172.21.15.150:0/4057581609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b40b9860 0x7f55b40cea10 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f55b406dca0 tx=0x7f55a802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=687ed4011b277eb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:07.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.749+0000 7f55b37fe640 1 -- 172.21.15.150:0/4057581609 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a802e070 con 0x7f55b40b9860 2024-03-17T23:49:07.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.749+0000 7f55b37fe640 1 -- 172.21.15.150:0/4057581609 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55a8037070 con 0x7f55b40b9860 2024-03-17T23:49:07.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.749+0000 7f55b37fe640 1 -- 172.21.15.150:0/4057581609 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a8031a00 con 0x7f55b40b9860 2024-03-17T23:49:07.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.749+0000 7f55baf3b640 1 -- 172.21.15.150:0/4057581609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b40b9860 msgr2=0x7f55b40cea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:07.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.749+0000 7f55baf3b640 1 --2- 172.21.15.150:0/4057581609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b40b9860 0x7f55b40cea10 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f55b406dca0 tx=0x7f55a802b880 comp rx=0 tx=0).stop 2024-03-17T23:49:07.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.749+0000 7f55baf3b640 1 -- 172.21.15.150:0/4057581609 shutdown_connections 2024-03-17T23:49:07.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.749+0000 7f55baf3b640 1 --2- 172.21.15.150:0/4057581609 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b40b9860 0x7f55b40cea10 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:07.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.749+0000 7f55baf3b640 1 --2- 172.21.15.150:0/4057581609 >> v2:172.21.15.157:3300/0 conn(0x7f55b40cef50 0x7f55b406c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:07.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.749+0000 7f55baf3b640 1 --2- 172.21.15.150:0/4057581609 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55b415de30 0x7f55b41619a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:07.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.749+0000 7f55baf3b640 1 -- 172.21.15.150:0/4057581609 >> 172.21.15.150:0/4057581609 conn(0x7f55b40730b0 msgr2=0x7f55b40d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:07.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.749+0000 7f55baf3b640 1 -- 172.21.15.150:0/4057581609 shutdown_connections 2024-03-17T23:49:07.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.749+0000 7f55baf3b640 1 -- 172.21.15.150:0/4057581609 wait complete. 2024-03-17T23:49:07.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.750+0000 7f55baf3b640 1 Processor -- start 2024-03-17T23:49:07.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.750+0000 7f55baf3b640 1 -- start start 2024-03-17T23:49:07.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.751+0000 7f55baf3b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b40b9860 0x7f55b415ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:07.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.751+0000 7f55baf3b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55b40cef50 0x7f55b4158420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:07.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.751+0000 7f55baf3b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55b415de30 0x7f55b41e8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:07.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.751+0000 7f55baf3b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f55b40d2170 con 0x7f55b40b9860 2024-03-17T23:49:07.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.751+0000 7f55baf3b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f55b4162660 con 0x7f55b415de30 2024-03-17T23:49:07.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.751+0000 7f55baf3b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f55b415e210 con 0x7f55b40cef50 2024-03-17T23:49:07.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.751+0000 7f55b8cd7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b40b9860 0x7f55b415ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:07.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.751+0000 7f55b8cd7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b40b9860 0x7f55b415ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43750/0 (socket says 172.21.15.150:43750) 2024-03-17T23:49:07.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.751+0000 7f55b8cd7640 1 -- 172.21.15.150:0/4249150838 learned_addr learned my addr 172.21.15.150:0/4249150838 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:07.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.752+0000 7f55b3fff640 1 --2- 172.21.15.150:0/4249150838 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55b40cef50 0x7f55b4158420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:07.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.752+0000 7f55b94d8640 1 --2- 172.21.15.150:0/4249150838 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55b415de30 0x7f55b41e8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:07.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.752+0000 7f55b8cd7640 1 -- 172.21.15.150:0/4249150838 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55b40cef50 msgr2=0x7f55b4158420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:07.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.752+0000 7f55b8cd7640 1 --2- 172.21.15.150:0/4249150838 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55b40cef50 0x7f55b4158420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:07.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.752+0000 7f55b8cd7640 1 -- 172.21.15.150:0/4249150838 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55b415de30 msgr2=0x7f55b41e8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:07.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.752+0000 7f55b8cd7640 1 --2- 172.21.15.150:0/4249150838 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55b415de30 0x7f55b41e8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:07.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.752+0000 7f55b8cd7640 1 -- 172.21.15.150:0/4249150838 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55b40ccc20 con 0x7f55b40b9860 2024-03-17T23:49:07.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.752+0000 7f55b8cd7640 1 --2- 172.21.15.150:0/4249150838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b40b9860 0x7f55b415ea70 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f55a400d5f0 tx=0x7f55a400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:07.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.752+0000 7f55b17fa640 1 -- 172.21.15.150:0/4249150838 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a40125f0 con 0x7f55b40b9860 2024-03-17T23:49:07.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.752+0000 7f55b17fa640 1 -- 172.21.15.150:0/4249150838 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55a4019070 con 0x7f55b40b9860 2024-03-17T23:49:07.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.752+0000 7f55b17fa640 1 -- 172.21.15.150:0/4249150838 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a40129d0 con 0x7f55b40b9860 2024-03-17T23:49:07.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.752+0000 7f55baf3b640 1 -- 172.21.15.150:0/4249150838 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55b40ccf00 con 0x7f55b40b9860 2024-03-17T23:49:07.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.752+0000 7f55baf3b640 1 -- 172.21.15.150:0/4249150838 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55b41e8430 con 0x7f55b40b9860 2024-03-17T23:49:07.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.753+0000 7f55baf3b640 1 -- 172.21.15.150:0/4249150838 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55b40681a0 con 0x7f55b40b9860 2024-03-17T23:49:07.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.754+0000 7f55b17fa640 1 -- 172.21.15.150:0/4249150838 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f55a4012b70 con 0x7f55b40b9860 2024-03-17T23:49:07.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.754+0000 7f55b17fa640 1 --2- 172.21.15.150:0/4249150838 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f55a006bc00 0x7f55a006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:07.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.754+0000 7f55b3fff640 1 --2- 172.21.15.150:0/4249150838 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f55a006bc00 0x7f55a006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:07.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.754+0000 7f55b17fa640 1 -- 172.21.15.150:0/4249150838 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5249+0+0 (secure 0 0 0) 0x7f55a408abe0 con 0x7f55b40b9860 2024-03-17T23:49:07.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.754+0000 7f55b3fff640 1 --2- 172.21.15.150:0/4249150838 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f55a006bc00 0x7f55a006e0c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f55a8036000 tx=0x7f55a803d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:07.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.759+0000 7f55b17fa640 1 -- 172.21.15.150:0/4249150838 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f55a4059920 con 0x7f55b40b9860 2024-03-17T23:49:07.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.998+0000 7f55baf3b640 1 -- 172.21.15.150:0/4249150838 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f55b40d1910 con 0x7f55b40b9860 2024-03-17T23:49:07.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:07.998+0000 7f55b17fa640 1 -- 172.21.15.150:0/4249150838 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) ==== 80+0+5298 (secure 0 0 0) 0x7f55a4059220 con 0x7f55b40b9860 2024-03-17T23:49:07.999 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:07.999 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-17T23:49:07.999 INFO:teuthology.orchestra.run.smithi150.stdout: "auth_dump": [ 2024-03-17T23:49:07.999 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:49:07.999 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "mds.a", 2024-03-17T23:49:07.999 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQC3gPdl0bOfHRAAqIf9ExwsoDeTZAbP6JhU/w==", 2024-03-17T23:49:07.999 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:49:07.999 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-03-17T23:49:07.999 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow *", 2024-03-17T23:49:08.000 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow *", 2024-03-17T23:49:08.000 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:49:08.000 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:49:08.000 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:08.000 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:49:08.000 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "mds.b", 2024-03-17T23:49:08.000 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQC3gPdlSlEVKhAAAScr6WsFt5/c2N1ns90pOg==", 2024-03-17T23:49:08.000 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:49:08.000 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-03-17T23:49:08.000 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow *", 2024-03-17T23:49:08.000 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow *", 2024-03-17T23:49:08.000 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:49:08.000 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:49:08.000 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:08.001 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:49:08.001 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "mds.c", 2024-03-17T23:49:08.001 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQC3gPdlJtcBJRAAsxXTt1otrV0PIJnP9p6CBQ==", 2024-03-17T23:49:08.001 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:49:08.001 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-03-17T23:49:08.001 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow *", 2024-03-17T23:49:08.001 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow *", 2024-03-17T23:49:08.001 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:49:08.001 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:49:08.001 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:08.001 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:49:08.001 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "mds.d", 2024-03-17T23:49:08.001 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQC3gPdlt7+DMRAAzv9aOR4QsJBCNHpUgSNQPg==", 2024-03-17T23:49:08.002 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:49:08.002 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-03-17T23:49:08.002 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow *", 2024-03-17T23:49:08.002 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow *", 2024-03-17T23:49:08.002 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:49:08.002 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:49:08.002 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:08.002 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:49:08.002 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.0", 2024-03-17T23:49:08.002 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQC7gPdlymEuGRAA0wyRX+r9Pocq1nrDOjfTfQ==", 2024-03-17T23:49:08.002 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:49:08.002 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-03-17T23:49:08.002 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-03-17T23:49:08.002 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:49:08.003 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:49:08.003 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:08.003 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:49:08.003 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.1", 2024-03-17T23:49:08.003 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQC+gPdl87WwFxAAO0iNw6UDAJ1I3WxmTusxFQ==", 2024-03-17T23:49:08.003 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:49:08.003 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-03-17T23:49:08.003 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-03-17T23:49:08.003 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:49:08.003 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:49:08.003 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:08.003 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:49:08.003 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.2", 2024-03-17T23:49:08.003 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQDBgPdlYVQIFxAANc97g+jl1RHahJS9BS2Kgw==", 2024-03-17T23:49:08.004 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:49:08.004 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-03-17T23:49:08.004 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-03-17T23:49:08.004 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:49:08.004 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:49:08.004 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:08.004 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:49:08.004 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.3", 2024-03-17T23:49:08.004 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQDEgPdlmaOqFRAAoYhWrnWdTC4LK0LB1SfSwg==", 2024-03-17T23:49:08.004 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:49:08.004 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-03-17T23:49:08.004 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-03-17T23:49:08.004 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:49:08.004 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:49:08.005 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:08.005 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:49:08.005 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.4", 2024-03-17T23:49:08.005 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQDIgPdlxX5rIhAAXHKRWgfegyAAfb/57c/xvQ==", 2024-03-17T23:49:08.005 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:49:08.005 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-03-17T23:49:08.005 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-03-17T23:49:08.005 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:49:08.005 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:49:08.005 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:08.005 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:49:08.005 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.5", 2024-03-17T23:49:08.005 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQDLgPdlf6xmIBAAK1d5IdCTjOHaR6yf41n/pQ==", 2024-03-17T23:49:08.005 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:49:08.006 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-03-17T23:49:08.006 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-03-17T23:49:08.006 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:49:08.006 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:49:08.006 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:08.006 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:49:08.006 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.6", 2024-03-17T23:49:08.006 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQDOgPdljVrnHhAAtIBm9CDQeN44WpOpUF3g3Q==", 2024-03-17T23:49:08.006 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:49:08.006 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-03-17T23:49:08.006 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-03-17T23:49:08.006 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:49:08.006 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:49:08.006 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:08.006 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:49:08.007 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.7", 2024-03-17T23:49:08.007 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQDRgPdlZDqfHRAAkkbz3Blr04H2uQsXjrQgRA==", 2024-03-17T23:49:08.007 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:49:08.007 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-03-17T23:49:08.007 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-03-17T23:49:08.007 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:49:08.007 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:49:08.007 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:08.007 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:49:08.007 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.0", 2024-03-17T23:49:08.007 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQC3gPdlvlPJNRAAkUfetPJK4hZQG5rNiSDozw==", 2024-03-17T23:49:08.007 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:49:08.007 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-03-17T23:49:08.007 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow r", 2024-03-17T23:49:08.008 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow rw pool=cephfs_data" 2024-03-17T23:49:08.008 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:49:08.008 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:08.008 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:49:08.008 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.1", 2024-03-17T23:49:08.008 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQC4gPdlNqN0AxAA4FezHbC/cqNvSbsaThwfNw==", 2024-03-17T23:49:08.008 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:49:08.008 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-03-17T23:49:08.008 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow r", 2024-03-17T23:49:08.008 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow rw", 2024-03-17T23:49:08.008 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow rwx" 2024-03-17T23:49:08.008 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:49:08.008 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:08.008 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:49:08.008 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.2", 2024-03-17T23:49:08.009 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQC4gPdlvBwIABAA9kdWclUPy/mGtbT4dxfoxA==", 2024-03-17T23:49:08.009 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:49:08.009 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-03-17T23:49:08.009 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow r", 2024-03-17T23:49:08.009 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow rw", 2024-03-17T23:49:08.009 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow rwx" 2024-03-17T23:49:08.009 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:49:08.009 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:08.009 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:49:08.009 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.3", 2024-03-17T23:49:08.009 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQC4gPdlbNtBCRAADYXS1k+3rScOubs0QA7doQ==", 2024-03-17T23:49:08.009 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:49:08.009 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-03-17T23:49:08.009 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow r", 2024-03-17T23:49:08.010 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow rw", 2024-03-17T23:49:08.010 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow rwx" 2024-03-17T23:49:08.010 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:49:08.010 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:08.010 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:49:08.010 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.admin", 2024-03-17T23:49:08.010 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQC3gPdldgyOABAA2MdA5GGq5XNBML+72T9WSw==", 2024-03-17T23:49:08.010 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:49:08.010 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow *", 2024-03-17T23:49:08.010 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow *", 2024-03-17T23:49:08.010 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow *", 2024-03-17T23:49:08.010 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:49:08.010 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:49:08.010 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:08.011 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:49:08.011 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "mgr.x", 2024-03-17T23:49:08.011 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQC3gPdlPDo1GhAAO8CxqquWsZg4j7EipcmOcQ==", 2024-03-17T23:49:08.011 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:49:08.011 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow *", 2024-03-17T23:49:08.011 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile mgr", 2024-03-17T23:49:08.011 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:49:08.011 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:49:08.011 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:08.011 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:49:08.011 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "mgr.y", 2024-03-17T23:49:08.011 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQC3gPdlQ9jOFRAAo0eZ4CMoH7heIX+sa4IuCw==", 2024-03-17T23:49:08.011 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-03-17T23:49:08.011 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow *", 2024-03-17T23:49:08.011 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile mgr", 2024-03-17T23:49:08.012 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-03-17T23:49:08.012 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:49:08.012 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:49:08.012 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-03-17T23:49:08.012 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-17T23:49:08.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.000+0000 7f55baf3b640 1 -- 172.21.15.150:0/4249150838 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f55a006bc00 msgr2=0x7f55a006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:08.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.000+0000 7f55baf3b640 1 --2- 172.21.15.150:0/4249150838 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f55a006bc00 0x7f55a006e0c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f55a8036000 tx=0x7f55a803d040 comp rx=0 tx=0).stop 2024-03-17T23:49:08.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.000+0000 7f55baf3b640 1 -- 172.21.15.150:0/4249150838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b40b9860 msgr2=0x7f55b415ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:08.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.000+0000 7f55baf3b640 1 --2- 172.21.15.150:0/4249150838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b40b9860 0x7f55b415ea70 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f55a400d5f0 tx=0x7f55a400dab0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.000+0000 7f55baf3b640 1 -- 172.21.15.150:0/4249150838 shutdown_connections 2024-03-17T23:49:08.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.000+0000 7f55baf3b640 1 --2- 172.21.15.150:0/4249150838 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f55a006bc00 0x7f55a006e0c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.000+0000 7f55baf3b640 1 --2- 172.21.15.150:0/4249150838 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f55b415de30 0x7f55b41e8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.000+0000 7f55baf3b640 1 --2- 172.21.15.150:0/4249150838 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f55b40cef50 0x7f55b4158420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.000+0000 7f55baf3b640 1 --2- 172.21.15.150:0/4249150838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55b40b9860 0x7f55b415ea70 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.000+0000 7f55baf3b640 1 -- 172.21.15.150:0/4249150838 >> 172.21.15.150:0/4249150838 conn(0x7f55b40730b0 msgr2=0x7f55b40b8030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:08.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.000+0000 7f55baf3b640 1 -- 172.21.15.150:0/4249150838 shutdown_connections 2024-03-17T23:49:08.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.000+0000 7f55baf3b640 1 -- 172.21.15.150:0/4249150838 wait complete. 2024-03-17T23:49:08.015 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:49:08.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.115+0000 7f470b1e6640 1 Processor -- start 2024-03-17T23:49:08.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.115+0000 7f470b1e6640 1 -- start start 2024-03-17T23:49:08.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.116+0000 7f470b1e6640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f470415e1c0 0x7f4704067550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:08.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.116+0000 7f470b1e6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4704099c80 con 0x7f470406ec90 2024-03-17T23:49:08.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.116+0000 7f470b1e6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f470406b330 con 0x7f470415f720 2024-03-17T23:49:08.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.116+0000 7f470b1e6640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f4704099f60 con 0x7f470415e1c0 2024-03-17T23:49:08.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.116+0000 7f4709783640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f470406ec90 0x7f470415e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55596/0 (socket says 172.21.15.150:55596) 2024-03-17T23:49:08.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.116+0000 7f4709783640 1 -- 172.21.15.150:0/2307974341 learned_addr learned my addr 172.21.15.150:0/2307974341 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:08.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.116+0000 7f47037fe640 1 -- 172.21.15.150:0/2307974341 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3937742114 0 0) 0x7f4704099c80 con 0x7f470406ec90 2024-03-17T23:49:08.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.117+0000 7f4708f82640 1 --2- 172.21.15.150:0/2307974341 >> v2:172.21.15.157:3300/0 conn(0x7f470415e1c0 0x7f4704067550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:08.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.117+0000 7f47037fe640 1 -- 172.21.15.150:0/2307974341 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46ec003520 con 0x7f470406ec90 2024-03-17T23:49:08.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.117+0000 7f47037fe640 1 -- 172.21.15.150:0/2307974341 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4266729642 0 0) 0x7f46ec003520 con 0x7f470406ec90 2024-03-17T23:49:08.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.117+0000 7f47037fe640 1 -- 172.21.15.150:0/2307974341 >> v2:172.21.15.157:3300/0 conn(0x7f470415e1c0 msgr2=0x7f4704067550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:08.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.117+0000 7f47037fe640 1 --2- 172.21.15.150:0/2307974341 >> v2:172.21.15.157:3300/0 conn(0x7f470415e1c0 0x7f4704067550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.117+0000 7f47037fe640 1 -- 172.21.15.150:0/2307974341 >> v1:172.21.15.157:6789/0 conn(0x7f470415f720 legacy=0x7f470406e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:08.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.117+0000 7f47037fe640 1 -- 172.21.15.150:0/2307974341 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f470406b610 con 0x7f470406ec90 2024-03-17T23:49:08.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.117+0000 7f47037fe640 1 -- 172.21.15.150:0/2307974341 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f46fc0019f0 con 0x7f470406ec90 2024-03-17T23:49:08.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.117+0000 7f47037fe640 1 -- 172.21.15.150:0/2307974341 >> v1:172.21.15.150:6789/0 conn(0x7f470406ec90 legacy=0x7f470415e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:08.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.117+0000 7f47037fe640 1 --2- 172.21.15.150:0/2307974341 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46ec0039b0 0x7f46ec003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:08.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.117+0000 7f47037fe640 1 --2- 172.21.15.150:0/2307974341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46ec004800 0x7f46ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:08.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.117+0000 7f47037fe640 1 --2- 172.21.15.150:0/2307974341 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f46ec0051c0 0x7f46ec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:08.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.117+0000 7f47037fe640 1 -- 172.21.15.150:0/2307974341 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f46ec00c030 con 0x7f46ec004800 2024-03-17T23:49:08.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.117+0000 7f47037fe640 1 -- 172.21.15.150:0/2307974341 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f46ec00c1e0 con 0x7f46ec0051c0 2024-03-17T23:49:08.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.117+0000 7f47037fe640 1 -- 172.21.15.150:0/2307974341 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f46ec00c3e0 con 0x7f46ec0039b0 2024-03-17T23:49:08.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.117+0000 7f4703fff640 1 --2- 172.21.15.150:0/2307974341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46ec004800 0x7f46ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:08.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.117+0000 7f4708f82640 1 --2- 172.21.15.150:0/2307974341 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46ec0039b0 0x7f46ec003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:08.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.117+0000 7f4709783640 1 --2- 172.21.15.150:0/2307974341 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f46ec0051c0 0x7f46ec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:08.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.117+0000 7f4703fff640 1 -- 172.21.15.150:0/2307974341 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46ec0039b0 msgr2=0x7f46ec003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:08.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.117+0000 7f4703fff640 1 --2- 172.21.15.150:0/2307974341 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46ec0039b0 0x7f46ec003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.117+0000 7f4703fff640 1 -- 172.21.15.150:0/2307974341 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f46ec0051c0 msgr2=0x7f46ec00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:08.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.117+0000 7f4703fff640 1 --2- 172.21.15.150:0/2307974341 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f46ec0051c0 0x7f46ec00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.117+0000 7f4709783640 1 -- 172.21.15.150:0/2307974341 reap_dead start 2024-03-17T23:49:08.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.117+0000 7f4703fff640 1 -- 172.21.15.150:0/2307974341 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46ec003520 con 0x7f46ec004800 2024-03-17T23:49:08.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.118+0000 7f4703fff640 1 --2- 172.21.15.150:0/2307974341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46ec004800 0x7f46ec004c80 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f46f8001720 tx=0x7f46ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b74bba783c52598a server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:08.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.118+0000 7f47037fe640 1 -- 172.21.15.150:0/2307974341 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f8001370 con 0x7f46ec004800 2024-03-17T23:49:08.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.118+0000 7f47037fe640 1 -- 172.21.15.150:0/2307974341 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46f80052d0 con 0x7f46ec004800 2024-03-17T23:49:08.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.118+0000 7f47037fe640 1 -- 172.21.15.150:0/2307974341 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f8003830 con 0x7f46ec004800 2024-03-17T23:49:08.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.119+0000 7f470b1e6640 1 -- 172.21.15.150:0/2307974341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46ec004800 msgr2=0x7f46ec004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:08.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.119+0000 7f470b1e6640 1 --2- 172.21.15.150:0/2307974341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46ec004800 0x7f46ec004c80 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f46f8001720 tx=0x7f46ec005830 comp rx=0 tx=0).stop 2024-03-17T23:49:08.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.119+0000 7f470b1e6640 1 -- 172.21.15.150:0/2307974341 shutdown_connections 2024-03-17T23:49:08.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.119+0000 7f470b1e6640 1 --2- 172.21.15.150:0/2307974341 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f46ec004800 0x7f46ec004c80 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.119+0000 7f470b1e6640 1 -- 172.21.15.150:0/2307974341 >> 172.21.15.150:0/2307974341 conn(0x7f47040b3da0 msgr2=0x7f47040b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:08.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.119+0000 7f470b1e6640 1 -- 172.21.15.150:0/2307974341 shutdown_connections 2024-03-17T23:49:08.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.119+0000 7f470b1e6640 1 -- 172.21.15.150:0/2307974341 wait complete. 2024-03-17T23:49:08.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.120+0000 7f470b1e6640 1 Processor -- start 2024-03-17T23:49:08.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.120+0000 7f470b1e6640 1 -- start start 2024-03-17T23:49:08.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.121+0000 7f470b1e6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46ec004800 0x7f470413f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:08.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.121+0000 7f470b1e6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4704156890 0x7f47040ad3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:08.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.121+0000 7f470b1e6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47041537d0 0x7f4704153c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:08.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.121+0000 7f470b1e6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4704072ba0 con 0x7f47041537d0 2024-03-17T23:49:08.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.121+0000 7f470b1e6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4704072a20 con 0x7f4704156890 2024-03-17T23:49:08.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.121+0000 7f470b1e6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4704072d20 con 0x7f46ec004800 2024-03-17T23:49:08.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.121+0000 7f4709783640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47041537d0 0x7f4704153c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:08.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.121+0000 7f4709783640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47041537d0 0x7f4704153c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43760/0 (socket says 172.21.15.150:43760) 2024-03-17T23:49:08.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.121+0000 7f4709783640 1 -- 172.21.15.150:0/3230528175 learned_addr learned my addr 172.21.15.150:0/3230528175 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:08.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.121+0000 7f4708f82640 1 --2- 172.21.15.150:0/3230528175 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46ec004800 0x7f470413f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:08.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.121+0000 7f4703fff640 1 --2- 172.21.15.150:0/3230528175 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4704156890 0x7f47040ad3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:08.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.121+0000 7f4709783640 1 -- 172.21.15.150:0/3230528175 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46ec004800 msgr2=0x7f470413f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:08.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.121+0000 7f4709783640 1 --2- 172.21.15.150:0/3230528175 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46ec004800 0x7f470413f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.121+0000 7f4709783640 1 -- 172.21.15.150:0/3230528175 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4704156890 msgr2=0x7f47040ad3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:08.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.121+0000 7f4709783640 1 --2- 172.21.15.150:0/3230528175 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4704156890 0x7f47040ad3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.121+0000 7f4709783640 1 -- 172.21.15.150:0/3230528175 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f470413fa80 con 0x7f47041537d0 2024-03-17T23:49:08.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.121+0000 7f4709783640 1 --2- 172.21.15.150:0/3230528175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47041537d0 0x7f4704153c80 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f46fc000f80 tx=0x7f46fc0079d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:08.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.121+0000 7f47017fa640 1 -- 172.21.15.150:0/3230528175 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46fc00ab00 con 0x7f47041537d0 2024-03-17T23:49:08.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.122+0000 7f47017fa640 1 -- 172.21.15.150:0/3230528175 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46fc019070 con 0x7f47041537d0 2024-03-17T23:49:08.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.122+0000 7f470b1e6640 1 -- 172.21.15.150:0/3230528175 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47040adca0 con 0x7f47041537d0 2024-03-17T23:49:08.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.122+0000 7f47017fa640 1 -- 172.21.15.150:0/3230528175 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46fc0133f0 con 0x7f47041537d0 2024-03-17T23:49:08.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.122+0000 7f470b1e6640 1 -- 172.21.15.150:0/3230528175 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47040ae260 con 0x7f47041537d0 2024-03-17T23:49:08.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.123+0000 7f470b1e6640 1 -- 172.21.15.150:0/3230528175 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f470406f550 con 0x7f47041537d0 2024-03-17T23:49:08.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.123+0000 7f47017fa640 1 -- 172.21.15.150:0/3230528175 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f46fc013590 con 0x7f47041537d0 2024-03-17T23:49:08.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.123+0000 7f47017fa640 1 --2- 172.21.15.150:0/3230528175 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f46f006bc00 0x7f46f006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:08.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.124+0000 7f47017fa640 1 -- 172.21.15.150:0/3230528175 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5249+0+0 (secure 0 0 0) 0x7f46fc08ac20 con 0x7f47041537d0 2024-03-17T23:49:08.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.124+0000 7f4708f82640 1 --2- 172.21.15.150:0/3230528175 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f46f006bc00 0x7f46f006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:08.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.124+0000 7f4708f82640 1 --2- 172.21.15.150:0/3230528175 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f46f006bc00 0x7f46f006e0c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f46f4009bc0 tx=0x7f46f4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:08.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.128+0000 7f47017fa640 1 -- 172.21.15.150:0/3230528175 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f470406f550 con 0x7f47041537d0 2024-03-17T23:49:08.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.300+0000 7f470b1e6640 1 -- 172.21.15.150:0/3230528175 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f47040681a0 con 0x7f47041537d0 2024-03-17T23:49:08.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.301+0000 7f47017fa640 1 -- 172.21.15.150:0/3230528175 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) ==== 74+0+13200 (secure 0 0 0) 0x7f46fc0598d0 con 0x7f47041537d0 2024-03-17T23:49:08.302 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:08.302 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":54,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:49:07.623233+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:49:08.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.303+0000 7f470b1e6640 1 -- 172.21.15.150:0/3230528175 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f46f006bc00 msgr2=0x7f46f006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:08.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.303+0000 7f470b1e6640 1 --2- 172.21.15.150:0/3230528175 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f46f006bc00 0x7f46f006e0c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f46f4009bc0 tx=0x7f46f4009190 comp rx=0 tx=0).stop 2024-03-17T23:49:08.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.303+0000 7f470b1e6640 1 -- 172.21.15.150:0/3230528175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47041537d0 msgr2=0x7f4704153c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:08.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.303+0000 7f470b1e6640 1 --2- 172.21.15.150:0/3230528175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47041537d0 0x7f4704153c80 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f46fc000f80 tx=0x7f46fc0079d0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.303+0000 7f470b1e6640 1 -- 172.21.15.150:0/3230528175 shutdown_connections 2024-03-17T23:49:08.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.303+0000 7f470b1e6640 1 --2- 172.21.15.150:0/3230528175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47041537d0 0x7f4704153c80 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.303+0000 7f470b1e6640 1 --2- 172.21.15.150:0/3230528175 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f46f006bc00 0x7f46f006e0c0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.303+0000 7f470b1e6640 1 --2- 172.21.15.150:0/3230528175 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4704156890 0x7f47040ad3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.303+0000 7f470b1e6640 1 --2- 172.21.15.150:0/3230528175 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f46ec004800 0x7f470413f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.303+0000 7f470b1e6640 1 -- 172.21.15.150:0/3230528175 >> 172.21.15.150:0/3230528175 conn(0x7f47040b3da0 msgr2=0x7f47040cfe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:08.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.304+0000 7f470b1e6640 1 -- 172.21.15.150:0/3230528175 shutdown_connections 2024-03-17T23:49:08.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.304+0000 7f470b1e6640 1 -- 172.21.15.150:0/3230528175 wait complete. 2024-03-17T23:49:08.315 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-17T23:49:08.316 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:49:08.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.419+0000 7f73730f8640 1 Processor -- start 2024-03-17T23:49:08.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.419+0000 7f73730f8640 1 -- start start 2024-03-17T23:49:08.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.419+0000 7f73730f8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f736c144bb0 0x7f736c15ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:08.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.419+0000 7f73730f8640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f736c0ce6a0 0x7f736c165a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:08.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.419+0000 7f73730f8640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f736c145170 con 0x7f736c15d390 2024-03-17T23:49:08.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.419+0000 7f73730f8640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f736c145450 con 0x7f736c144bb0 2024-03-17T23:49:08.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.419+0000 7f73730f8640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f736c145730 con 0x7f736c0ce6a0 2024-03-17T23:49:08.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.420+0000 7f7370e94640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f736c144bb0 0x7f736c15ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:08.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.420+0000 7f7370e94640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f736c144bb0 0x7f736c15ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43762/0 (socket says 172.21.15.150:43762) 2024-03-17T23:49:08.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.420+0000 7f7370e94640 1 -- 172.21.15.150:0/1294669464 learned_addr learned my addr 172.21.15.150:0/1294669464 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:08.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.420+0000 7f7371695640 1 --2- 172.21.15.150:0/1294669464 >> v2:172.21.15.157:3300/0 conn(0x7f736c0ce6a0 0x7f736c165a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:08.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.420+0000 7f7370e94640 1 -- 172.21.15.150:0/1294669464 >> v2:172.21.15.157:3300/0 conn(0x7f736c0ce6a0 msgr2=0x7f736c165a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:08.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.420+0000 7f7370e94640 1 --2- 172.21.15.150:0/1294669464 >> v2:172.21.15.157:3300/0 conn(0x7f736c0ce6a0 0x7f736c165a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.420+0000 7f7370e94640 1 -- 172.21.15.150:0/1294669464 >> v1:172.21.15.157:6789/0 conn(0x7f736c15d390 legacy=0x7f736c0cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:08.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.420+0000 7f7370e94640 1 -- 172.21.15.150:0/1294669464 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f736c06bd60 con 0x7f736c144bb0 2024-03-17T23:49:08.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.421+0000 7f7370e94640 1 --2- 172.21.15.150:0/1294669464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f736c144bb0 0x7f736c15ce50 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f736c073130 tx=0x7f736402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff8fbbbd6ca2158e server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:08.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.421+0000 7f736b7fe640 1 -- 172.21.15.150:0/1294669464 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736402c9f0 con 0x7f736c144bb0 2024-03-17T23:49:08.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.421+0000 7f736b7fe640 1 -- 172.21.15.150:0/1294669464 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f736402cb90 con 0x7f736c144bb0 2024-03-17T23:49:08.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.421+0000 7f736b7fe640 1 -- 172.21.15.150:0/1294669464 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73640343f0 con 0x7f736c144bb0 2024-03-17T23:49:08.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.421+0000 7f73730f8640 1 -- 172.21.15.150:0/1294669464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f736c144bb0 msgr2=0x7f736c15ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:08.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.421+0000 7f73730f8640 1 --2- 172.21.15.150:0/1294669464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f736c144bb0 0x7f736c15ce50 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f736c073130 tx=0x7f736402be50 comp rx=0 tx=0).stop 2024-03-17T23:49:08.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.421+0000 7f73730f8640 1 -- 172.21.15.150:0/1294669464 shutdown_connections 2024-03-17T23:49:08.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.421+0000 7f73730f8640 1 --2- 172.21.15.150:0/1294669464 >> v2:172.21.15.157:3300/0 conn(0x7f736c0ce6a0 0x7f736c165a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.421+0000 7f73730f8640 1 --2- 172.21.15.150:0/1294669464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f736c144bb0 0x7f736c15ce50 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.421+0000 7f73730f8640 1 -- 172.21.15.150:0/1294669464 >> 172.21.15.150:0/1294669464 conn(0x7f736c06b190 msgr2=0x7f736c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:08.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.421+0000 7f73730f8640 1 -- 172.21.15.150:0/1294669464 shutdown_connections 2024-03-17T23:49:08.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.421+0000 7f73730f8640 1 -- 172.21.15.150:0/1294669464 wait complete. 2024-03-17T23:49:08.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.422+0000 7f73730f8640 1 Processor -- start 2024-03-17T23:49:08.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.422+0000 7f73730f8640 1 -- start start 2024-03-17T23:49:08.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.423+0000 7f73730f8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f736c0ce6a0 0x7f736c143730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:08.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.423+0000 7f73730f8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f736c144bb0 0x7f736c142650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:08.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.423+0000 7f73730f8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f736c15d390 0x7f736c159370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:08.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.423+0000 7f73730f8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f736c165f70 con 0x7f736c144bb0 2024-03-17T23:49:08.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.423+0000 7f73730f8640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f736c06f650 con 0x7f736c0ce6a0 2024-03-17T23:49:08.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.423+0000 7f73730f8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f736c1660f0 con 0x7f736c15d390 2024-03-17T23:49:08.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.423+0000 7f736bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f736c144bb0 0x7f736c142650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:08.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.423+0000 7f736bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f736c144bb0 0x7f736c142650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43764/0 (socket says 172.21.15.150:43764) 2024-03-17T23:49:08.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.423+0000 7f736bfff640 1 -- 172.21.15.150:0/1356435286 learned_addr learned my addr 172.21.15.150:0/1356435286 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:08.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.424+0000 7f7370e94640 1 --2- 172.21.15.150:0/1356435286 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f736c0ce6a0 0x7f736c143730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:08.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.424+0000 7f7371695640 1 --2- 172.21.15.150:0/1356435286 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f736c15d390 0x7f736c159370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:08.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.424+0000 7f736bfff640 1 -- 172.21.15.150:0/1356435286 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f736c15d390 msgr2=0x7f736c159370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:08.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.424+0000 7f736bfff640 1 --2- 172.21.15.150:0/1356435286 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f736c15d390 0x7f736c159370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.424+0000 7f736bfff640 1 -- 172.21.15.150:0/1356435286 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f736c0ce6a0 msgr2=0x7f736c143730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:08.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.424+0000 7f736bfff640 1 --2- 172.21.15.150:0/1356435286 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f736c0ce6a0 0x7f736c143730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.424+0000 7f736bfff640 1 -- 172.21.15.150:0/1356435286 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f736c142e30 con 0x7f736c144bb0 2024-03-17T23:49:08.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.424+0000 7f7370e94640 1 --2- 172.21.15.150:0/1356435286 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f736c0ce6a0 0x7f736c143730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:49:08.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.424+0000 7f736bfff640 1 --2- 172.21.15.150:0/1356435286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f736c144bb0 0x7f736c142650 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f735400c460 tx=0x7f735400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:08.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.424+0000 7f73697fa640 1 -- 172.21.15.150:0/1356435286 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7354016020 con 0x7f736c144bb0 2024-03-17T23:49:08.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.424+0000 7f73697fa640 1 -- 172.21.15.150:0/1356435286 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7354012870 con 0x7f736c144bb0 2024-03-17T23:49:08.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.424+0000 7f73730f8640 1 -- 172.21.15.150:0/1356435286 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f736c15a660 con 0x7f736c144bb0 2024-03-17T23:49:08.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.424+0000 7f73697fa640 1 -- 172.21.15.150:0/1356435286 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7354012c60 con 0x7f736c144bb0 2024-03-17T23:49:08.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.424+0000 7f73730f8640 1 -- 172.21.15.150:0/1356435286 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f736c142b90 con 0x7f736c144bb0 2024-03-17T23:49:08.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.425+0000 7f73730f8640 1 -- 172.21.15.150:0/1356435286 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f736c0681a0 con 0x7f736c144bb0 2024-03-17T23:49:08.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.425+0000 7f73697fa640 1 -- 172.21.15.150:0/1356435286 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7354019020 con 0x7f736c144bb0 2024-03-17T23:49:08.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.426+0000 7f73697fa640 1 --2- 172.21.15.150:0/1356435286 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f734c06bbd0 0x7f734c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:08.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.426+0000 7f73697fa640 1 -- 172.21.15.150:0/1356435286 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5249+0+0 (secure 0 0 0) 0x7f735408a570 con 0x7f736c144bb0 2024-03-17T23:49:08.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.426+0000 7f7370e94640 1 --2- 172.21.15.150:0/1356435286 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f734c06bbd0 0x7f734c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:08.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.426+0000 7f7370e94640 1 --2- 172.21.15.150:0/1356435286 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f734c06bbd0 0x7f734c06e090 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7364035000 tx=0x7f7364030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:08.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.430+0000 7f73697fa640 1 -- 172.21.15.150:0/1356435286 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7354020030 con 0x7f736c144bb0 2024-03-17T23:49:08.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.602+0000 7f73730f8640 1 -- 172.21.15.150:0/1356435286 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f736c0b4e40 con 0x7f736c144bb0 2024-03-17T23:49:08.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.603+0000 7f73697fa640 1 -- 172.21.15.150:0/1356435286 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) ==== 93+0+10 (secure 0 0 0) 0x7f73540592b0 con 0x7f736c144bb0 2024-03-17T23:49:08.603 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:49:08.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.605+0000 7f73730f8640 1 -- 172.21.15.150:0/1356435286 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f734c06bbd0 msgr2=0x7f734c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:08.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.605+0000 7f73730f8640 1 --2- 172.21.15.150:0/1356435286 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f734c06bbd0 0x7f734c06e090 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7364035000 tx=0x7f7364030000 comp rx=0 tx=0).stop 2024-03-17T23:49:08.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.605+0000 7f73730f8640 1 -- 172.21.15.150:0/1356435286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f736c144bb0 msgr2=0x7f736c142650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:08.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.605+0000 7f73730f8640 1 --2- 172.21.15.150:0/1356435286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f736c144bb0 0x7f736c142650 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f735400c460 tx=0x7f735400c920 comp rx=0 tx=0).stop 2024-03-17T23:49:08.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.605+0000 7f73730f8640 1 -- 172.21.15.150:0/1356435286 shutdown_connections 2024-03-17T23:49:08.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.605+0000 7f73730f8640 1 --2- 172.21.15.150:0/1356435286 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f736c15d390 0x7f736c159370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.605+0000 7f73730f8640 1 --2- 172.21.15.150:0/1356435286 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f736c144bb0 0x7f736c142650 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.605+0000 7f73730f8640 1 --2- 172.21.15.150:0/1356435286 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f734c06bbd0 0x7f734c06e090 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.605+0000 7f73730f8640 1 --2- 172.21.15.150:0/1356435286 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f736c0ce6a0 0x7f736c143730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.605+0000 7f73730f8640 1 -- 172.21.15.150:0/1356435286 >> 172.21.15.150:0/1356435286 conn(0x7f736c06b190 msgr2=0x7f736c143cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:08.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.605+0000 7f73730f8640 1 -- 172.21.15.150:0/1356435286 shutdown_connections 2024-03-17T23:49:08.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.605+0000 7f73730f8640 1 -- 172.21.15.150:0/1356435286 wait complete. 2024-03-17T23:49:08.616 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-17T23:49:08.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.717+0000 7f6c9271d640 1 Processor -- start 2024-03-17T23:49:08.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.717+0000 7f6c9271d640 1 -- start start 2024-03-17T23:49:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.717+0000 7f6c9271d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6c8c1619a0 0x7f6c8c0b9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.717+0000 7f6c9271d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c8c161dd0 con 0x7f6c8c15de30 2024-03-17T23:49:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.717+0000 7f6c9271d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c8c1620a0 con 0x7f6c8c0cf120 2024-03-17T23:49:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.717+0000 7f6c9271d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f6c8c162940 con 0x7f6c8c1619a0 2024-03-17T23:49:08.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.718+0000 7f6c8bfff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f6c8c15de30 0x7f6c8c0cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55604/0 (socket says 172.21.15.150:55604) 2024-03-17T23:49:08.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.718+0000 7f6c8bfff640 1 -- 172.21.15.150:0/2072128437 learned_addr learned my addr 172.21.15.150:0/2072128437 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:08.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.718+0000 7f6c90cba640 1 --2- 172.21.15.150:0/2072128437 >> v2:172.21.15.150:3300/0 conn(0x7f6c8c1619a0 0x7f6c8c0b9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:08.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.718+0000 7f6c90cba640 1 -- 172.21.15.150:0/2072128437 >> v1:172.21.15.157:6789/0 conn(0x7f6c8c0cf120 legacy=0x7f6c8c0d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:08.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.718+0000 7f6c90cba640 1 -- 172.21.15.150:0/2072128437 >> v1:172.21.15.150:6789/0 conn(0x7f6c8c15de30 legacy=0x7f6c8c0cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:08.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.718+0000 7f6c90cba640 1 -- 172.21.15.150:0/2072128437 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c8c162380 con 0x7f6c8c1619a0 2024-03-17T23:49:08.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.719+0000 7f6c90cba640 1 --2- 172.21.15.150:0/2072128437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c8c1619a0 0x7f6c8c0b9860 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f6c84009d90 tx=0x7f6c8402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b86ba64a6555726 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:08.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.719+0000 7f6c8affd640 1 -- 172.21.15.150:0/2072128437 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c84039070 con 0x7f6c8c1619a0 2024-03-17T23:49:08.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.719+0000 7f6c8affd640 1 -- 172.21.15.150:0/2072128437 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c8402cc60 con 0x7f6c8c1619a0 2024-03-17T23:49:08.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.719+0000 7f6c8affd640 1 -- 172.21.15.150:0/2072128437 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c84034660 con 0x7f6c8c1619a0 2024-03-17T23:49:08.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.719+0000 7f6c9271d640 1 -- 172.21.15.150:0/2072128437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c8c1619a0 msgr2=0x7f6c8c0b9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:08.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.719+0000 7f6c9271d640 1 --2- 172.21.15.150:0/2072128437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c8c1619a0 0x7f6c8c0b9860 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f6c84009d90 tx=0x7f6c8402beb0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.719+0000 7f6c9271d640 1 -- 172.21.15.150:0/2072128437 shutdown_connections 2024-03-17T23:49:08.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.719+0000 7f6c9271d640 1 --2- 172.21.15.150:0/2072128437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c8c1619a0 0x7f6c8c0b9860 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.719+0000 7f6c9271d640 1 -- 172.21.15.150:0/2072128437 >> 172.21.15.150:0/2072128437 conn(0x7f6c8c0730b0 msgr2=0x7f6c8c0d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:08.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.719+0000 7f6c9271d640 1 -- 172.21.15.150:0/2072128437 shutdown_connections 2024-03-17T23:49:08.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.719+0000 7f6c9271d640 1 -- 172.21.15.150:0/2072128437 wait complete. 2024-03-17T23:49:08.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.720+0000 7f6c9271d640 1 Processor -- start 2024-03-17T23:49:08.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.721+0000 7f6c9271d640 1 -- start start 2024-03-17T23:49:08.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.721+0000 7f6c9271d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c8c0cf120 0x7f6c8c1500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:08.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.721+0000 7f6c9271d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c8c15de30 0x7f6c8c0b5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:08.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.721+0000 7f6c9271d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c8c1619a0 0x7f6c8c0b5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:08.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.721+0000 7f6c9271d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c15e210 con 0x7f6c8c15de30 2024-03-17T23:49:08.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.721+0000 7f6c9271d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c0d2170 con 0x7f6c8c0cf120 2024-03-17T23:49:08.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.721+0000 7f6c9271d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6c8c0d2da0 con 0x7f6c8c1619a0 2024-03-17T23:49:08.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.722+0000 7f6c8b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c8c15de30 0x7f6c8c0b5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:08.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.722+0000 7f6c8b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c8c15de30 0x7f6c8c0b5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43776/0 (socket says 172.21.15.150:43776) 2024-03-17T23:49:08.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.722+0000 7f6c8b7fe640 1 -- 172.21.15.150:0/4154942308 learned_addr learned my addr 172.21.15.150:0/4154942308 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:08.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.722+0000 7f6c90cba640 1 --2- 172.21.15.150:0/4154942308 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c8c1619a0 0x7f6c8c0b5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:08.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.722+0000 7f6c8bfff640 1 --2- 172.21.15.150:0/4154942308 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c8c0cf120 0x7f6c8c1500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:08.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.722+0000 7f6c8b7fe640 1 -- 172.21.15.150:0/4154942308 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c8c1619a0 msgr2=0x7f6c8c0b5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:08.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.722+0000 7f6c8b7fe640 1 --2- 172.21.15.150:0/4154942308 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c8c1619a0 0x7f6c8c0b5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.722+0000 7f6c8b7fe640 1 -- 172.21.15.150:0/4154942308 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c8c0cf120 msgr2=0x7f6c8c1500a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:08.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.722+0000 7f6c8b7fe640 1 --2- 172.21.15.150:0/4154942308 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c8c0cf120 0x7f6c8c1500a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.722+0000 7f6c8b7fe640 1 -- 172.21.15.150:0/4154942308 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c8c1614e0 con 0x7f6c8c15de30 2024-03-17T23:49:08.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.722+0000 7f6c8b7fe640 1 --2- 172.21.15.150:0/4154942308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c8c15de30 0x7f6c8c0b5a20 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f6c7400c430 tx=0x7f6c7400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:08.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.722+0000 7f6c88ff9640 1 -- 172.21.15.150:0/4154942308 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c740052d0 con 0x7f6c8c15de30 2024-03-17T23:49:08.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.722+0000 7f6c88ff9640 1 -- 172.21.15.150:0/4154942308 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c74005490 con 0x7f6c8c15de30 2024-03-17T23:49:08.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.722+0000 7f6c9271d640 1 -- 172.21.15.150:0/4154942308 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c8c15e4c0 con 0x7f6c8c15de30 2024-03-17T23:49:08.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.722+0000 7f6c88ff9640 1 -- 172.21.15.150:0/4154942308 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c74007260 con 0x7f6c8c15de30 2024-03-17T23:49:08.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.722+0000 7f6c9271d640 1 -- 172.21.15.150:0/4154942308 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c8c15ea80 con 0x7f6c8c15de30 2024-03-17T23:49:08.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.723+0000 7f6c9271d640 1 -- 172.21.15.150:0/4154942308 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c8c0681a0 con 0x7f6c8c15de30 2024-03-17T23:49:08.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.724+0000 7f6c88ff9640 1 -- 172.21.15.150:0/4154942308 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6c74007400 con 0x7f6c8c15de30 2024-03-17T23:49:08.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.724+0000 7f6c88ff9640 1 --2- 172.21.15.150:0/4154942308 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6c6c06bbd0 0x7f6c6c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:08.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.724+0000 7f6c88ff9640 1 -- 172.21.15.150:0/4154942308 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5249+0+0 (secure 0 0 0) 0x7f6c74089d70 con 0x7f6c8c15de30 2024-03-17T23:49:08.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.724+0000 7f6c8bfff640 1 --2- 172.21.15.150:0/4154942308 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6c6c06bbd0 0x7f6c6c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:08.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.725+0000 7f6c8bfff640 1 --2- 172.21.15.150:0/4154942308 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6c6c06bbd0 0x7f6c6c06e090 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c005880 tx=0x7f6c7c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:08.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.728+0000 7f6c88ff9640 1 -- 172.21.15.150:0/4154942308 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6c74058ab0 con 0x7f6c8c15de30 2024-03-17T23:49:08.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.901+0000 7f6c9271d640 1 -- 172.21.15.150:0/4154942308 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f6c8c0d1910 con 0x7f6c8c15de30 2024-03-17T23:49:08.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.902+0000 7f6c88ff9640 1 -- 172.21.15.150:0/4154942308 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v54) ==== 84+0+65 (secure 0 0 0) 0x7f6c740583b0 con 0x7f6c8c15de30 2024-03-17T23:49:08.903 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:08.903 INFO:teuthology.orchestra.run.smithi150.stdout:[ 2024-03-17T23:49:08.903 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-03-17T23:49:08.903 INFO:teuthology.orchestra.run.smithi150.stdout: "poolnum": 1, 2024-03-17T23:49:08.903 INFO:teuthology.orchestra.run.smithi150.stdout: "poolname": ".mgr" 2024-03-17T23:49:08.903 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:49:08.903 INFO:teuthology.orchestra.run.smithi150.stdout:] 2024-03-17T23:49:08.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.903+0000 7f6c9271d640 1 -- 172.21.15.150:0/4154942308 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6c6c06bbd0 msgr2=0x7f6c6c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:08.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.903+0000 7f6c9271d640 1 --2- 172.21.15.150:0/4154942308 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6c6c06bbd0 0x7f6c6c06e090 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c005880 tx=0x7f6c7c009040 comp rx=0 tx=0).stop 2024-03-17T23:49:08.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.903+0000 7f6c9271d640 1 -- 172.21.15.150:0/4154942308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c8c15de30 msgr2=0x7f6c8c0b5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:08.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.903+0000 7f6c9271d640 1 --2- 172.21.15.150:0/4154942308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c8c15de30 0x7f6c8c0b5a20 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f6c7400c430 tx=0x7f6c7400c8f0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.903+0000 7f6c9271d640 1 -- 172.21.15.150:0/4154942308 shutdown_connections 2024-03-17T23:49:08.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.903+0000 7f6c9271d640 1 --2- 172.21.15.150:0/4154942308 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c8c1619a0 0x7f6c8c0b5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.903+0000 7f6c9271d640 1 --2- 172.21.15.150:0/4154942308 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c8c15de30 0x7f6c8c0b5a20 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.903+0000 7f6c9271d640 1 --2- 172.21.15.150:0/4154942308 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6c6c06bbd0 0x7f6c6c06e090 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.903+0000 7f6c9271d640 1 --2- 172.21.15.150:0/4154942308 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c8c0cf120 0x7f6c8c1500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:08.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.904+0000 7f6c9271d640 1 -- 172.21.15.150:0/4154942308 >> 172.21.15.150:0/4154942308 conn(0x7f6c8c0730b0 msgr2=0x7f6c8c0b8410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:08.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.904+0000 7f6c9271d640 1 -- 172.21.15.150:0/4154942308 shutdown_connections 2024-03-17T23:49:08.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:08.904+0000 7f6c9271d640 1 -- 172.21.15.150:0/4154942308 wait complete. 2024-03-17T23:49:08.915 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-17T23:49:08.915 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-17T23:49:09.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.020+0000 7f7d6cf7e640 1 Processor -- start 2024-03-17T23:49:09.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.020+0000 7f7d6cf7e640 1 -- start start 2024-03-17T23:49:09.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d6cf7e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7d680cd4d0 0x7f7d68072070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:09.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d6cf7e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d6815aff0 con 0x7f7d680cf590 2024-03-17T23:49:09.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d6cf7e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d6815b260 con 0x7f7d680725b0 2024-03-17T23:49:09.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d6cf7e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7d68072a10 con 0x7f7d680cd4d0 2024-03-17T23:49:09.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d66d9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f7d680cf590 0x7f7d680b3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55606/0 (socket says 172.21.15.150:55606) 2024-03-17T23:49:09.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d66d9d640 1 -- 172.21.15.150:0/685237649 learned_addr learned my addr 172.21.15.150:0/685237649 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:09.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d6659c640 1 --2- 172.21.15.150:0/685237649 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7d680cd4d0 0x7f7d68072070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:09.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d6559a640 1 -- 172.21.15.150:0/685237649 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1899319136 0 0) 0x7f7d6815aff0 con 0x7f7d680cf590 2024-03-17T23:49:09.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d6559a640 1 -- 172.21.15.150:0/685237649 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d50003520 con 0x7f7d680cf590 2024-03-17T23:49:09.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d6559a640 1 -- 172.21.15.150:0/685237649 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1946771446 0 0) 0x7f7d6815b260 con 0x7f7d680725b0 2024-03-17T23:49:09.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d6559a640 1 -- 172.21.15.150:0/685237649 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d6815aff0 con 0x7f7d680725b0 2024-03-17T23:49:09.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d6559a640 1 -- 172.21.15.150:0/685237649 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1163230941 0 0) 0x7f7d50003520 con 0x7f7d680cf590 2024-03-17T23:49:09.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d6559a640 1 -- 172.21.15.150:0/685237649 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7d680cd4d0 msgr2=0x7f7d68072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:09.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d6559a640 1 --2- 172.21.15.150:0/685237649 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7d680cd4d0 0x7f7d68072070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:09.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d6559a640 1 -- 172.21.15.150:0/685237649 >> v1:172.21.15.157:6789/0 conn(0x7f7d680725b0 legacy=0x7f7d680cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:09.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d6559a640 1 -- 172.21.15.150:0/685237649 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d6815b540 con 0x7f7d680cf590 2024-03-17T23:49:09.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d6559a640 1 -- 172.21.15.150:0/685237649 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f7d600019f0 con 0x7f7d680cf590 2024-03-17T23:49:09.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d6659c640 1 --2- 172.21.15.150:0/685237649 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7d680cd4d0 0x7f7d68072070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:09.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d6559a640 1 -- 172.21.15.150:0/685237649 >> v1:172.21.15.150:6789/0 conn(0x7f7d680cf590 legacy=0x7f7d680b3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:09.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d6559a640 1 --2- 172.21.15.150:0/685237649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d50003b70 0x7f7d50003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:09.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d6559a640 1 --2- 172.21.15.150:0/685237649 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7d50004800 0x7f7d50004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:09.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d6559a640 1 --2- 172.21.15.150:0/685237649 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d500051c0 0x7f7d5000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:09.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d6559a640 1 -- 172.21.15.150:0/685237649 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5000bfb0 con 0x7f7d50003b70 2024-03-17T23:49:09.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d6559a640 1 -- 172.21.15.150:0/685237649 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5000c130 con 0x7f7d500051c0 2024-03-17T23:49:09.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d6559a640 1 -- 172.21.15.150:0/685237649 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7d5000c330 con 0x7f7d50004800 2024-03-17T23:49:09.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d6659c640 1 --2- 172.21.15.150:0/685237649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d50003b70 0x7f7d50003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:09.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.021+0000 7f7d65d9b640 1 --2- 172.21.15.150:0/685237649 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7d50004800 0x7f7d50004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:09.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.022+0000 7f7d66d9d640 1 --2- 172.21.15.150:0/685237649 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d500051c0 0x7f7d5000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:09.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.022+0000 7f7d6659c640 1 -- 172.21.15.150:0/685237649 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7d50004800 msgr2=0x7f7d50004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:09.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.022+0000 7f7d6659c640 1 --2- 172.21.15.150:0/685237649 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7d50004800 0x7f7d50004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:09.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.022+0000 7f7d6659c640 1 -- 172.21.15.150:0/685237649 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d500051c0 msgr2=0x7f7d5000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:09.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.022+0000 7f7d6659c640 1 --2- 172.21.15.150:0/685237649 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d500051c0 0x7f7d5000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:09.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.022+0000 7f7d6659c640 1 -- 172.21.15.150:0/685237649 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d50003520 con 0x7f7d50003b70 2024-03-17T23:49:09.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.022+0000 7f7d6659c640 1 --2- 172.21.15.150:0/685237649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d50003b70 0x7f7d50003f70 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f7d6806c1e0 tx=0x7f7d50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c87799c72648b3e server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:09.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.023+0000 7f7d6559a640 1 -- 172.21.15.150:0/685237649 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d580097f0 con 0x7f7d50003b70 2024-03-17T23:49:09.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.023+0000 7f7d6559a640 1 -- 172.21.15.150:0/685237649 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d58004030 con 0x7f7d50003b70 2024-03-17T23:49:09.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.023+0000 7f7d6559a640 1 -- 172.21.15.150:0/685237649 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d580043e0 con 0x7f7d50003b70 2024-03-17T23:49:09.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.023+0000 7f7d6cf7e640 1 -- 172.21.15.150:0/685237649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d50003b70 msgr2=0x7f7d50003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:09.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.023+0000 7f7d6cf7e640 1 --2- 172.21.15.150:0/685237649 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d50003b70 0x7f7d50003f70 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f7d6806c1e0 tx=0x7f7d50005830 comp rx=0 tx=0).stop 2024-03-17T23:49:09.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.023+0000 7f7d66d9d640 1 -- 172.21.15.150:0/685237649 reap_dead start 2024-03-17T23:49:09.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.023+0000 7f7d6cf7e640 1 -- 172.21.15.150:0/685237649 shutdown_connections 2024-03-17T23:49:09.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.023+0000 7f7d6cf7e640 1 -- 172.21.15.150:0/685237649 >> 172.21.15.150:0/685237649 conn(0x7f7d680b1be0 msgr2=0x7f7d680b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:09.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.023+0000 7f7d6cf7e640 1 -- 172.21.15.150:0/685237649 shutdown_connections 2024-03-17T23:49:09.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.023+0000 7f7d6cf7e640 1 -- 172.21.15.150:0/685237649 wait complete. 2024-03-17T23:49:09.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.025+0000 7f7d6cf7e640 1 Processor -- start 2024-03-17T23:49:09.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.025+0000 7f7d6cf7e640 1 -- start start 2024-03-17T23:49:09.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.025+0000 7f7d6cf7e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7d50003b70 0x7f7d6814a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:09.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.025+0000 7f7d6cf7e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d68159930 0x7f7d68159db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:09.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.025+0000 7f7d6cf7e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d680ad2f0 0x7f7d680ad770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:09.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.025+0000 7f7d6cf7e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6815bde0 con 0x7f7d68159930 2024-03-17T23:49:09.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.025+0000 7f7d6cf7e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6815bb00 con 0x7f7d680ad2f0 2024-03-17T23:49:09.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.025+0000 7f7d6cf7e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7d680cd8b0 con 0x7f7d50003b70 2024-03-17T23:49:09.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.025+0000 7f7d65d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d68159930 0x7f7d68159db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:09.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.025+0000 7f7d65d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d68159930 0x7f7d68159db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43788/0 (socket says 172.21.15.150:43788) 2024-03-17T23:49:09.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.025+0000 7f7d65d9b640 1 -- 172.21.15.150:0/68582602 learned_addr learned my addr 172.21.15.150:0/68582602 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:09.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.026+0000 7f7d66d9d640 1 --2- 172.21.15.150:0/68582602 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d680ad2f0 0x7f7d680ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:09.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.026+0000 7f7d6659c640 1 --2- 172.21.15.150:0/68582602 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7d50003b70 0x7f7d6814a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:09.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.026+0000 7f7d65d9b640 1 -- 172.21.15.150:0/68582602 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7d50003b70 msgr2=0x7f7d6814a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:09.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.026+0000 7f7d65d9b640 1 --2- 172.21.15.150:0/68582602 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7d50003b70 0x7f7d6814a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:09.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.026+0000 7f7d65d9b640 1 -- 172.21.15.150:0/68582602 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d680ad2f0 msgr2=0x7f7d680ad770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:09.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.026+0000 7f7d65d9b640 1 --2- 172.21.15.150:0/68582602 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d680ad2f0 0x7f7d680ad770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:09.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.026+0000 7f7d65d9b640 1 -- 172.21.15.150:0/68582602 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d6814a8e0 con 0x7f7d68159930 2024-03-17T23:49:09.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.026+0000 7f7d65d9b640 1 --2- 172.21.15.150:0/68582602 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d68159930 0x7f7d68159db0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c00f7d0 tx=0x7f7d5c00fc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:09.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.026+0000 7f7d56ffd640 1 -- 172.21.15.150:0/68582602 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c00fe80 con 0x7f7d68159930 2024-03-17T23:49:09.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.026+0000 7f7d56ffd640 1 -- 172.21.15.150:0/68582602 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d5c018070 con 0x7f7d68159930 2024-03-17T23:49:09.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.026+0000 7f7d6cf7e640 1 -- 172.21.15.150:0/68582602 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d680ae080 con 0x7f7d68159930 2024-03-17T23:49:09.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.026+0000 7f7d56ffd640 1 -- 172.21.15.150:0/68582602 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c00d730 con 0x7f7d68159930 2024-03-17T23:49:09.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.026+0000 7f7d6cf7e640 1 -- 172.21.15.150:0/68582602 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d680ae640 con 0x7f7d68159930 2024-03-17T23:49:09.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.027+0000 7f7d6cf7e640 1 -- 172.21.15.150:0/68582602 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d680a8660 con 0x7f7d68159930 2024-03-17T23:49:09.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.028+0000 7f7d56ffd640 1 -- 172.21.15.150:0/68582602 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7d5c00d8d0 con 0x7f7d68159930 2024-03-17T23:49:09.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.028+0000 7f7d56ffd640 1 --2- 172.21.15.150:0/68582602 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7d4806bbd0 0x7f7d4806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:09.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.028+0000 7f7d56ffd640 1 -- 172.21.15.150:0/68582602 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5249+0+0 (secure 0 0 0) 0x7f7d5c08a850 con 0x7f7d68159930 2024-03-17T23:49:09.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.028+0000 7f7d6659c640 1 --2- 172.21.15.150:0/68582602 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7d4806bbd0 0x7f7d4806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:09.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.028+0000 7f7d6659c640 1 --2- 172.21.15.150:0/68582602 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7d4806bbd0 0x7f7d4806e090 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7d58035000 tx=0x7f7d58030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:09.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.032+0000 7f7d56ffd640 1 -- 172.21.15.150:0/68582602 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7d680a8660 con 0x7f7d68159930 2024-03-17T23:49:09.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.205+0000 7f7d6cf7e640 1 -- 172.21.15.150:0/68582602 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f7d680681a0 con 0x7f7d68159930 2024-03-17T23:49:09.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.629+0000 7f7d56ffd640 1 -- 172.21.15.150:0/68582602 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v55) ==== 138+0+0 (secure 0 0 0) 0x7f7d6814a8e0 con 0x7f7d68159930 2024-03-17T23:49:09.630 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_metadata' created 2024-03-17T23:49:09.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.631+0000 7f7d6cf7e640 1 -- 172.21.15.150:0/68582602 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7d4806bbd0 msgr2=0x7f7d4806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:09.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.631+0000 7f7d6cf7e640 1 --2- 172.21.15.150:0/68582602 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7d4806bbd0 0x7f7d4806e090 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7d58035000 tx=0x7f7d58030000 comp rx=0 tx=0).stop 2024-03-17T23:49:09.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.631+0000 7f7d6cf7e640 1 -- 172.21.15.150:0/68582602 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d68159930 msgr2=0x7f7d68159db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:09.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.631+0000 7f7d6cf7e640 1 --2- 172.21.15.150:0/68582602 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d68159930 0x7f7d68159db0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c00f7d0 tx=0x7f7d5c00fc90 comp rx=0 tx=0).stop 2024-03-17T23:49:09.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.631+0000 7f7d6cf7e640 1 -- 172.21.15.150:0/68582602 shutdown_connections 2024-03-17T23:49:09.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.631+0000 7f7d6cf7e640 1 --2- 172.21.15.150:0/68582602 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7d4806bbd0 0x7f7d4806e090 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:09.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.631+0000 7f7d6cf7e640 1 --2- 172.21.15.150:0/68582602 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d680ad2f0 0x7f7d680ad770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:09.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.631+0000 7f7d6cf7e640 1 --2- 172.21.15.150:0/68582602 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7d68159930 0x7f7d68159db0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:09.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.631+0000 7f7d6cf7e640 1 --2- 172.21.15.150:0/68582602 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7d50003b70 0x7f7d6814a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:09.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.631+0000 7f7d6cf7e640 1 -- 172.21.15.150:0/68582602 >> 172.21.15.150:0/68582602 conn(0x7f7d680b1be0 msgr2=0x7f7d6815ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:09.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.632+0000 7f7d6cf7e640 1 -- 172.21.15.150:0/68582602 shutdown_connections 2024-03-17T23:49:09.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.632+0000 7f7d6cf7e640 1 -- 172.21.15.150:0/68582602 wait complete. 2024-03-17T23:49:09.647 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-17T23:49:09.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.753+0000 7f2709ff4640 1 Processor -- start 2024-03-17T23:49:09.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.753+0000 7f2709ff4640 1 -- start start 2024-03-17T23:49:09.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.754+0000 7f2709ff4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f270415f3d0 0x7f27040cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:09.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.754+0000 7f2709ff4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f27040cc770 0x7f27041603c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:09.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.754+0000 7f2709ff4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27040ccc50 con 0x7f27040d0170 2024-03-17T23:49:09.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.754+0000 7f2709ff4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f270415f7b0 con 0x7f27040cc770 2024-03-17T23:49:09.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.754+0000 7f2709ff4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2704160900 con 0x7f270415f3d0 2024-03-17T23:49:09.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.754+0000 7f26faffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f27040cc770 0x7f27041603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:09.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.754+0000 7f26faffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f27040cc770 0x7f27041603c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43804/0 (socket says 172.21.15.150:43804) 2024-03-17T23:49:09.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.754+0000 7f26faffd640 1 -- 172.21.15.150:0/421022627 learned_addr learned my addr 172.21.15.150:0/421022627 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:09.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.755+0000 7f27037fe640 1 --2- 172.21.15.150:0/421022627 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f270415f3d0 0x7f27040cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:09.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.755+0000 7f26faffd640 1 -- 172.21.15.150:0/421022627 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f270415f3d0 msgr2=0x7f27040cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:09.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.755+0000 7f26faffd640 1 --2- 172.21.15.150:0/421022627 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f270415f3d0 0x7f27040cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:09.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.755+0000 7f26faffd640 1 -- 172.21.15.150:0/421022627 >> v1:172.21.15.150:6789/0 conn(0x7f27040d0170 legacy=0x7f27040c5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:09.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.755+0000 7f2702ffd640 1 -- 172.21.15.150:0/421022627 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3611424616 0 0) 0x7f27040ccc50 con 0x7f27040d0170 2024-03-17T23:49:09.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.755+0000 7f26faffd640 1 -- 172.21.15.150:0/421022627 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27040ccf30 con 0x7f27040cc770 2024-03-17T23:49:09.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.755+0000 7f27037fe640 1 --2- 172.21.15.150:0/421022627 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f270415f3d0 0x7f27040cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:49:09.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.755+0000 7f26faffd640 1 --2- 172.21.15.150:0/421022627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27040cc770 0x7f27041603c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f26ec009870 tx=0x7f26ec02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2bcf7d3fd747786 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:09.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.756+0000 7f2702ffd640 1 -- 172.21.15.150:0/421022627 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec034070 con 0x7f27040cc770 2024-03-17T23:49:09.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.756+0000 7f2702ffd640 1 -- 172.21.15.150:0/421022627 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26ec030840 con 0x7f27040cc770 2024-03-17T23:49:09.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.756+0000 7f2702ffd640 1 -- 172.21.15.150:0/421022627 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec030c10 con 0x7f27040cc770 2024-03-17T23:49:09.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.756+0000 7f2709ff4640 1 -- 172.21.15.150:0/421022627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27040cc770 msgr2=0x7f27041603c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:09.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.756+0000 7f2709ff4640 1 --2- 172.21.15.150:0/421022627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27040cc770 0x7f27041603c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f26ec009870 tx=0x7f26ec02b980 comp rx=0 tx=0).stop 2024-03-17T23:49:09.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.756+0000 7f2709ff4640 1 -- 172.21.15.150:0/421022627 shutdown_connections 2024-03-17T23:49:09.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.756+0000 7f2709ff4640 1 --2- 172.21.15.150:0/421022627 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27040cc770 0x7f27041603c0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:09.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.756+0000 7f2709ff4640 1 --2- 172.21.15.150:0/421022627 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f270415f3d0 0x7f27040cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:09.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.756+0000 7f2709ff4640 1 -- 172.21.15.150:0/421022627 >> 172.21.15.150:0/421022627 conn(0x7f27040b36c0 msgr2=0x7f27040b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:09.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.756+0000 7f2709ff4640 1 -- 172.21.15.150:0/421022627 shutdown_connections 2024-03-17T23:49:09.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.756+0000 7f2709ff4640 1 -- 172.21.15.150:0/421022627 wait complete. 2024-03-17T23:49:09.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.757+0000 7f2709ff4640 1 Processor -- start 2024-03-17T23:49:09.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.758+0000 7f2709ff4640 1 -- start start 2024-03-17T23:49:09.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.758+0000 7f2709ff4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27040cc770 0x7f270406dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:09.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.758+0000 7f2709ff4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27040d0170 0x7f270415b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:09.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.758+0000 7f2709ff4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f270415f3d0 0x7f27040ce650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:09.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.758+0000 7f2709ff4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f27040cd7d0 con 0x7f27040cc770 2024-03-17T23:49:09.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.758+0000 7f2709ff4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f27040cd4f0 con 0x7f27040d0170 2024-03-17T23:49:09.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.758+0000 7f2709ff4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f27040cfa30 con 0x7f270415f3d0 2024-03-17T23:49:09.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.758+0000 7f27037fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27040cc770 0x7f270406dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:09.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.758+0000 7f27037fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27040cc770 0x7f270406dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43812/0 (socket says 172.21.15.150:43812) 2024-03-17T23:49:09.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.758+0000 7f27037fe640 1 -- 172.21.15.150:0/4086371703 learned_addr learned my addr 172.21.15.150:0/4086371703 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:09.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.759+0000 7f27037fe640 1 -- 172.21.15.150:0/4086371703 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f270415f3d0 msgr2=0x7f27040ce650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:09.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.759+0000 7f2703fff640 1 --2- 172.21.15.150:0/4086371703 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f270415f3d0 0x7f27040ce650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:09.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.759+0000 7f27037fe640 1 --2- 172.21.15.150:0/4086371703 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f270415f3d0 0x7f27040ce650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:09.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.759+0000 7f26faffd640 1 --2- 172.21.15.150:0/4086371703 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27040d0170 0x7f270415b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:09.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.759+0000 7f27037fe640 1 -- 172.21.15.150:0/4086371703 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27040d0170 msgr2=0x7f270415b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:09.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.759+0000 7f27037fe640 1 --2- 172.21.15.150:0/4086371703 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27040d0170 0x7f270415b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:09.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.759+0000 7f27037fe640 1 -- 172.21.15.150:0/4086371703 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27040b5630 con 0x7f27040cc770 2024-03-17T23:49:09.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.759+0000 7f27037fe640 1 --2- 172.21.15.150:0/4086371703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27040cc770 0x7f270406dec0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f26f400c7f0 tx=0x7f26f400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:09.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.759+0000 7f2700ff9640 1 -- 172.21.15.150:0/4086371703 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f4015070 con 0x7f27040cc770 2024-03-17T23:49:09.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.759+0000 7f2700ff9640 1 -- 172.21.15.150:0/4086371703 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26f4007820 con 0x7f27040cc770 2024-03-17T23:49:09.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.759+0000 7f2709ff4640 1 -- 172.21.15.150:0/4086371703 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2704070950 con 0x7f27040cc770 2024-03-17T23:49:09.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.759+0000 7f2700ff9640 1 -- 172.21.15.150:0/4086371703 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f4007bd0 con 0x7f27040cc770 2024-03-17T23:49:09.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.759+0000 7f2709ff4640 1 -- 172.21.15.150:0/4086371703 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2704070f10 con 0x7f27040cc770 2024-03-17T23:49:09.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.760+0000 7f2709ff4640 1 -- 172.21.15.150:0/4086371703 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27040688b0 con 0x7f27040cc770 2024-03-17T23:49:09.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.760+0000 7f2700ff9640 1 -- 172.21.15.150:0/4086371703 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f26f4018070 con 0x7f27040cc770 2024-03-17T23:49:09.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.761+0000 7f2700ff9640 1 --2- 172.21.15.150:0/4086371703 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f26e406bbd0 0x7f26e406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:09.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.761+0000 7f2700ff9640 1 -- 172.21.15.150:0/4086371703 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5619+0+0 (secure 0 0 0) 0x7f26f408aad0 con 0x7f27040cc770 2024-03-17T23:49:09.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.761+0000 7f26faffd640 1 --2- 172.21.15.150:0/4086371703 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f26e406bbd0 0x7f26e406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:09.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.761+0000 7f26faffd640 1 --2- 172.21.15.150:0/4086371703 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f26e406bbd0 0x7f26e406e090 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f26ec036000 tx=0x7f26ec03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:09.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.765+0000 7f2700ff9640 1 -- 172.21.15.150:0/4086371703 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f26f40596a0 con 0x7f27040cc770 2024-03-17T23:49:09.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:09.937+0000 7f2709ff4640 1 -- 172.21.15.150:0/4086371703 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f27040b7b50 con 0x7f27040cc770 2024-03-17T23:49:10.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.635+0000 7f2700ff9640 1 -- 172.21.15.150:0/4086371703 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v56) ==== 170+0+0 (secure 0 0 0) 0x7f26f4058fa0 con 0x7f27040cc770 2024-03-17T23:49:10.636 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_data' created 2024-03-17T23:49:10.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.639+0000 7f2709ff4640 1 -- 172.21.15.150:0/4086371703 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f26e406bbd0 msgr2=0x7f26e406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:10.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.639+0000 7f2709ff4640 1 --2- 172.21.15.150:0/4086371703 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f26e406bbd0 0x7f26e406e090 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f26ec036000 tx=0x7f26ec03b040 comp rx=0 tx=0).stop 2024-03-17T23:49:10.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.639+0000 7f2709ff4640 1 -- 172.21.15.150:0/4086371703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27040cc770 msgr2=0x7f270406dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:10.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.639+0000 7f2709ff4640 1 --2- 172.21.15.150:0/4086371703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27040cc770 0x7f270406dec0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f26f400c7f0 tx=0x7f26f400ccb0 comp rx=0 tx=0).stop 2024-03-17T23:49:10.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.639+0000 7f2709ff4640 1 -- 172.21.15.150:0/4086371703 shutdown_connections 2024-03-17T23:49:10.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.639+0000 7f2709ff4640 1 --2- 172.21.15.150:0/4086371703 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f270415f3d0 0x7f27040ce650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:10.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.639+0000 7f2709ff4640 1 --2- 172.21.15.150:0/4086371703 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f26e406bbd0 0x7f26e406e090 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:10.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.639+0000 7f2709ff4640 1 --2- 172.21.15.150:0/4086371703 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f27040d0170 0x7f270415b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:10.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.639+0000 7f2709ff4640 1 --2- 172.21.15.150:0/4086371703 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27040cc770 0x7f270406dec0 secure :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f26f400c7f0 tx=0x7f26f400ccb0 comp rx=0 tx=0).stop 2024-03-17T23:49:10.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.639+0000 7f2709ff4640 1 -- 172.21.15.150:0/4086371703 >> 172.21.15.150:0/4086371703 conn(0x7f27040b36c0 msgr2=0x7f2704073b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:10.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.644+0000 7f2709ff4640 1 -- 172.21.15.150:0/4086371703 shutdown_connections 2024-03-17T23:49:10.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.649+0000 7f2709ff4640 1 -- 172.21.15.150:0/4086371703 wait complete. 2024-03-17T23:49:10.668 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-17T23:49:10.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.773+0000 7efc67ea1640 1 Processor -- start 2024-03-17T23:49:10.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.773+0000 7efc67ea1640 1 -- start start 2024-03-17T23:49:10.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.774+0000 7efc67ea1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efc6015bfd0 0x7efc600ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:10.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.774+0000 7efc67ea1640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7efc600cd250 0x7efc6015ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:10.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.774+0000 7efc67ea1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc600cf3b0 con 0x7efc60155320 2024-03-17T23:49:10.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.774+0000 7efc67ea1640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7efc6015c550 con 0x7efc6015bfd0 2024-03-17T23:49:10.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.774+0000 7efc67ea1640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7efc600cf690 con 0x7efc600cd250 2024-03-17T23:49:10.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.774+0000 7efc6543c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efc6015bfd0 0x7efc600ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:10.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.774+0000 7efc65c3d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7efc60155320 0x7efc6006dd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55632/0 (socket says 172.21.15.150:55632) 2024-03-17T23:49:10.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.774+0000 7efc6543c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efc6015bfd0 0x7efc600ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43818/0 (socket says 172.21.15.150:43818) 2024-03-17T23:49:10.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.774+0000 7efc65c3d640 1 -- 172.21.15.150:0/696015408 learned_addr learned my addr 172.21.15.150:0/696015408 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:10.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.774+0000 7efc6543c640 1 -- 172.21.15.150:0/696015408 >> v2:172.21.15.157:3300/0 conn(0x7efc600cd250 msgr2=0x7efc6015ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:10.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.774+0000 7efc6543c640 1 --2- 172.21.15.150:0/696015408 >> v2:172.21.15.157:3300/0 conn(0x7efc600cd250 0x7efc6015ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:10.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.774+0000 7efc6543c640 1 -- 172.21.15.150:0/696015408 >> v1:172.21.15.150:6789/0 conn(0x7efc60155320 legacy=0x7efc6006dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:10.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.774+0000 7efc6543c640 1 -- 172.21.15.150:0/696015408 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc6015c7a0 con 0x7efc6015bfd0 2024-03-17T23:49:10.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.774+0000 7efc64c3b640 1 -- 172.21.15.150:0/696015408 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 39262995 0 0) 0x7efc600cf3b0 con 0x7efc60155320 2024-03-17T23:49:10.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.775+0000 7efc6543c640 1 --2- 172.21.15.150:0/696015408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc6015bfd0 0x7efc600ccd10 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7efc6006c0f0 tx=0x7efc5002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f92ca946e18c1540 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:10.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.775+0000 7efc64c3b640 1 -- 172.21.15.150:0/696015408 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc50035020 con 0x7efc6015bfd0 2024-03-17T23:49:10.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.775+0000 7efc64c3b640 1 -- 172.21.15.150:0/696015408 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc50031830 con 0x7efc6015bfd0 2024-03-17T23:49:10.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.775+0000 7efc64c3b640 1 -- 172.21.15.150:0/696015408 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc50031c00 con 0x7efc6015bfd0 2024-03-17T23:49:10.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.776+0000 7efc67ea1640 1 -- 172.21.15.150:0/696015408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc6015bfd0 msgr2=0x7efc600ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:10.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.776+0000 7efc67ea1640 1 --2- 172.21.15.150:0/696015408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc6015bfd0 0x7efc600ccd10 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7efc6006c0f0 tx=0x7efc5002b8d0 comp rx=0 tx=0).stop 2024-03-17T23:49:10.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.776+0000 7efc67ea1640 1 -- 172.21.15.150:0/696015408 shutdown_connections 2024-03-17T23:49:10.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.776+0000 7efc67ea1640 1 --2- 172.21.15.150:0/696015408 >> v2:172.21.15.157:3300/0 conn(0x7efc600cd250 0x7efc6015ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:10.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.776+0000 7efc67ea1640 1 --2- 172.21.15.150:0/696015408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc6015bfd0 0x7efc600ccd10 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:10.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.776+0000 7efc67ea1640 1 -- 172.21.15.150:0/696015408 >> 172.21.15.150:0/696015408 conn(0x7efc6006b190 msgr2=0x7efc6006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:10.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.776+0000 7efc67ea1640 1 -- 172.21.15.150:0/696015408 shutdown_connections 2024-03-17T23:49:10.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.776+0000 7efc67ea1640 1 -- 172.21.15.150:0/696015408 wait complete. 2024-03-17T23:49:10.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.777+0000 7efc67ea1640 1 Processor -- start 2024-03-17T23:49:10.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.777+0000 7efc67ea1640 1 -- start start 2024-03-17T23:49:10.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.778+0000 7efc67ea1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efc600cd250 0x7efc6013b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:10.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.778+0000 7efc67ea1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc60155320 0x7efc6014cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:10.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.778+0000 7efc67ea1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efc6015bfd0 0x7efc60153560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:10.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.778+0000 7efc67ea1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7efc60155880 con 0x7efc60155320 2024-03-17T23:49:10.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.778+0000 7efc67ea1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7efc60155700 con 0x7efc600cd250 2024-03-17T23:49:10.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.778+0000 7efc67ea1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7efc6015d200 con 0x7efc6015bfd0 2024-03-17T23:49:10.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.778+0000 7efc6543c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc60155320 0x7efc6014cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:10.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.778+0000 7efc6543c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc60155320 0x7efc6014cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43832/0 (socket says 172.21.15.150:43832) 2024-03-17T23:49:10.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.778+0000 7efc6543c640 1 -- 172.21.15.150:0/1700116257 learned_addr learned my addr 172.21.15.150:0/1700116257 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:10.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.778+0000 7efc65c3d640 1 --2- 172.21.15.150:0/1700116257 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efc600cd250 0x7efc6013b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:10.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.778+0000 7efc6643e640 1 --2- 172.21.15.150:0/1700116257 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efc6015bfd0 0x7efc60153560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:10.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.778+0000 7efc6543c640 1 -- 172.21.15.150:0/1700116257 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efc6015bfd0 msgr2=0x7efc60153560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:10.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.778+0000 7efc6543c640 1 --2- 172.21.15.150:0/1700116257 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efc6015bfd0 0x7efc60153560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:10.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.778+0000 7efc6543c640 1 -- 172.21.15.150:0/1700116257 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efc600cd250 msgr2=0x7efc6013b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:10.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.778+0000 7efc6543c640 1 --2- 172.21.15.150:0/1700116257 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efc600cd250 0x7efc6013b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:10.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.778+0000 7efc6543c640 1 -- 172.21.15.150:0/1700116257 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc6014dbc0 con 0x7efc60155320 2024-03-17T23:49:10.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.778+0000 7efc6643e640 1 --2- 172.21.15.150:0/1700116257 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efc6015bfd0 0x7efc60153560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:49:10.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.778+0000 7efc65c3d640 1 --2- 172.21.15.150:0/1700116257 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efc600cd250 0x7efc6013b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:49:10.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.778+0000 7efc6543c640 1 --2- 172.21.15.150:0/1700116257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc60155320 0x7efc6014cc20 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7efc50000c00 tx=0x7efc500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:10.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.778+0000 7efc567fc640 1 -- 172.21.15.150:0/1700116257 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc50041070 con 0x7efc60155320 2024-03-17T23:49:10.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.778+0000 7efc567fc640 1 -- 172.21.15.150:0/1700116257 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc50004230 con 0x7efc60155320 2024-03-17T23:49:10.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.778+0000 7efc67ea1640 1 -- 172.21.15.150:0/1700116257 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc6014d600 con 0x7efc60155320 2024-03-17T23:49:10.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.778+0000 7efc567fc640 1 -- 172.21.15.150:0/1700116257 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc50035040 con 0x7efc60155320 2024-03-17T23:49:10.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.779+0000 7efc67ea1640 1 -- 172.21.15.150:0/1700116257 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc60166dc0 con 0x7efc60155320 2024-03-17T23:49:10.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.779+0000 7efc67ea1640 1 -- 172.21.15.150:0/1700116257 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc600681a0 con 0x7efc60155320 2024-03-17T23:49:10.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.780+0000 7efc567fc640 1 -- 172.21.15.150:0/1700116257 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7efc50038020 con 0x7efc60155320 2024-03-17T23:49:10.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.780+0000 7efc567fc640 1 --2- 172.21.15.150:0/1700116257 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7efc4406bc00 0x7efc4406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:10.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.780+0000 7efc567fc640 1 -- 172.21.15.150:0/1700116257 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6001+0+0 (secure 0 0 0) 0x7efc500ac640 con 0x7efc60155320 2024-03-17T23:49:10.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.780+0000 7efc65c3d640 1 --2- 172.21.15.150:0/1700116257 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7efc4406bc00 0x7efc4406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:10.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.781+0000 7efc65c3d640 1 --2- 172.21.15.150:0/1700116257 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7efc4406bc00 0x7efc4406e0c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7efc5c000f60 tx=0x7efc5c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:10.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.785+0000 7efc567fc640 1 -- 172.21.15.150:0/1700116257 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7efc5007b010 con 0x7efc60155320 2024-03-17T23:49:10.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:10.994+0000 7efc67ea1640 1 -- 172.21.15.150:0/1700116257 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7efc60072920 con 0x7efc60155320 2024-03-17T23:49:11.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.645+0000 7efc567fc640 1 -- 172.21.15.150:0/1700116257 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v20) ==== 172+0+0 (secure 0 0 0) 0x7efc5007a910 con 0x7efc60155320 2024-03-17T23:49:11.649 INFO:teuthology.orchestra.run.smithi150.stderr:new fs with metadata pool 8 and data pool 9 2024-03-17T23:49:11.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.654+0000 7efc67ea1640 1 -- 172.21.15.150:0/1700116257 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7efc4406bc00 msgr2=0x7efc4406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:11.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.654+0000 7efc67ea1640 1 --2- 172.21.15.150:0/1700116257 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7efc4406bc00 0x7efc4406e0c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7efc5c000f60 tx=0x7efc5c009040 comp rx=0 tx=0).stop 2024-03-17T23:49:11.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.654+0000 7efc67ea1640 1 -- 172.21.15.150:0/1700116257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc60155320 msgr2=0x7efc6014cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:11.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.654+0000 7efc67ea1640 1 --2- 172.21.15.150:0/1700116257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc60155320 0x7efc6014cc20 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7efc50000c00 tx=0x7efc500046c0 comp rx=0 tx=0).stop 2024-03-17T23:49:11.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.655+0000 7efc67ea1640 1 -- 172.21.15.150:0/1700116257 shutdown_connections 2024-03-17T23:49:11.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.655+0000 7efc67ea1640 1 --2- 172.21.15.150:0/1700116257 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7efc6015bfd0 0x7efc60153560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:11.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.655+0000 7efc67ea1640 1 --2- 172.21.15.150:0/1700116257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc60155320 0x7efc6014cc20 secure :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7efc50000c00 tx=0x7efc500046c0 comp rx=0 tx=0).stop 2024-03-17T23:49:11.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.655+0000 7efc67ea1640 1 --2- 172.21.15.150:0/1700116257 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7efc4406bc00 0x7efc4406e0c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:11.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.655+0000 7efc67ea1640 1 --2- 172.21.15.150:0/1700116257 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efc600cd250 0x7efc6013b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:11.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.655+0000 7efc67ea1640 1 -- 172.21.15.150:0/1700116257 >> 172.21.15.150:0/1700116257 conn(0x7efc6006b190 msgr2=0x7efc6014e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:11.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.665+0000 7efc67ea1640 1 -- 172.21.15.150:0/1700116257 shutdown_connections 2024-03-17T23:49:11.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.666+0000 7efc67ea1640 1 -- 172.21.15.150:0/1700116257 wait complete. 2024-03-17T23:49:11.680 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:49:11.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.783+0000 7f4c7f2c0640 1 Processor -- start 2024-03-17T23:49:11.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.783+0000 7f4c7f2c0640 1 -- start start 2024-03-17T23:49:11.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.784+0000 7f4c7f2c0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c7815bfd0 0x7f4c780ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:11.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.784+0000 7f4c7f2c0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c780cf3b0 con 0x7f4c780cd250 2024-03-17T23:49:11.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.784+0000 7f4c7f2c0640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c780cf690 con 0x7f4c78155320 2024-03-17T23:49:11.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.784+0000 7f4c7f2c0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4c7815c550 con 0x7f4c7815bfd0 2024-03-17T23:49:11.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.784+0000 7f4c7d85d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f4c780cd250 0x7f4c7815ccc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55636/0 (socket says 172.21.15.150:55636) 2024-03-17T23:49:11.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.784+0000 7f4c7d85d640 1 -- 172.21.15.150:0/322425309 learned_addr learned my addr 172.21.15.150:0/322425309 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:11.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.784+0000 7f4c7c85b640 1 --2- 172.21.15.150:0/322425309 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c7815bfd0 0x7f4c780ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:11.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.784+0000 7f4c6ffff640 1 -- 172.21.15.150:0/322425309 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 652574077 0 0) 0x7f4c780cf3b0 con 0x7f4c780cd250 2024-03-17T23:49:11.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.784+0000 7f4c6ffff640 1 -- 172.21.15.150:0/322425309 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c64003520 con 0x7f4c780cd250 2024-03-17T23:49:11.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.785+0000 7f4c6ffff640 1 -- 172.21.15.150:0/322425309 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1102459186 0 0) 0x7f4c780cf690 con 0x7f4c78155320 2024-03-17T23:49:11.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.785+0000 7f4c6ffff640 1 -- 172.21.15.150:0/322425309 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c780cf3b0 con 0x7f4c78155320 2024-03-17T23:49:11.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.785+0000 7f4c6ffff640 1 -- 172.21.15.150:0/322425309 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 271504357 0 0) 0x7f4c64003520 con 0x7f4c780cd250 2024-03-17T23:49:11.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.785+0000 7f4c7c85b640 1 -- 172.21.15.150:0/322425309 >> v1:172.21.15.157:6789/0 conn(0x7f4c78155320 legacy=0x7f4c7806dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:11.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.785+0000 7f4c7c85b640 1 -- 172.21.15.150:0/322425309 >> v1:172.21.15.150:6789/0 conn(0x7f4c780cd250 legacy=0x7f4c7815ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:11.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.785+0000 7f4c7c85b640 1 -- 172.21.15.150:0/322425309 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c7815c7a0 con 0x7f4c7815bfd0 2024-03-17T23:49:11.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.785+0000 7f4c7c85b640 1 --2- 172.21.15.150:0/322425309 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c7815bfd0 0x7f4c780ccd10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4c680097d0 tx=0x7f4c6802b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a21e4da0de7e4560 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:11.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.786+0000 7f4c6ffff640 1 -- 172.21.15.150:0/322425309 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c68037070 con 0x7f4c7815bfd0 2024-03-17T23:49:11.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.786+0000 7f4c6ffff640 1 -- 172.21.15.150:0/322425309 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c68030510 con 0x7f4c7815bfd0 2024-03-17T23:49:11.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.786+0000 7f4c6ffff640 1 -- 172.21.15.150:0/322425309 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c700019f0 con 0x7f4c7815bfd0 2024-03-17T23:49:11.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.786+0000 7f4c7f2c0640 1 -- 172.21.15.150:0/322425309 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c7815bfd0 msgr2=0x7f4c780ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:11.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.786+0000 7f4c7f2c0640 1 --2- 172.21.15.150:0/322425309 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c7815bfd0 0x7f4c780ccd10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4c680097d0 tx=0x7f4c6802b800 comp rx=0 tx=0).stop 2024-03-17T23:49:11.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.786+0000 7f4c7f2c0640 1 -- 172.21.15.150:0/322425309 shutdown_connections 2024-03-17T23:49:11.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.786+0000 7f4c7f2c0640 1 --2- 172.21.15.150:0/322425309 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c7815bfd0 0x7f4c780ccd10 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:11.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.786+0000 7f4c7f2c0640 1 -- 172.21.15.150:0/322425309 >> 172.21.15.150:0/322425309 conn(0x7f4c7806b190 msgr2=0x7f4c7806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:11.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.786+0000 7f4c7f2c0640 1 -- 172.21.15.150:0/322425309 shutdown_connections 2024-03-17T23:49:11.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.786+0000 7f4c7f2c0640 1 -- 172.21.15.150:0/322425309 wait complete. 2024-03-17T23:49:11.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.787+0000 7f4c7f2c0640 1 Processor -- start 2024-03-17T23:49:11.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.787+0000 7f4c7f2c0640 1 -- start start 2024-03-17T23:49:11.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.788+0000 7f4c7f2c0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c780cd250 0x7f4c7814a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:11.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.788+0000 7f4c7f2c0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c78155320 0x7f4c7815d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:11.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.788+0000 7f4c7f2c0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c7815bfd0 0x7f4c7815dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:11.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.788+0000 7f4c7f2c0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4c78155880 con 0x7f4c780cd250 2024-03-17T23:49:11.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.788+0000 7f4c7f2c0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4c78155700 con 0x7f4c7815bfd0 2024-03-17T23:49:11.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.788+0000 7f4c7f2c0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4c7815d3d0 con 0x7f4c78155320 2024-03-17T23:49:11.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.788+0000 7f4c7d05c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c780cd250 0x7f4c7814a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:11.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.788+0000 7f4c7d05c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c780cd250 0x7f4c7814a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43836/0 (socket says 172.21.15.150:43836) 2024-03-17T23:49:11.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.788+0000 7f4c7d05c640 1 -- 172.21.15.150:0/1434323567 learned_addr learned my addr 172.21.15.150:0/1434323567 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:11.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.788+0000 7f4c7c85b640 1 --2- 172.21.15.150:0/1434323567 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c78155320 0x7f4c7815d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:11.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.788+0000 7f4c7d85d640 1 --2- 172.21.15.150:0/1434323567 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c7815bfd0 0x7f4c7815dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:11.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.788+0000 7f4c7d05c640 1 -- 172.21.15.150:0/1434323567 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c78155320 msgr2=0x7f4c7815d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:11.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.788+0000 7f4c7d05c640 1 --2- 172.21.15.150:0/1434323567 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c78155320 0x7f4c7815d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:11.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.788+0000 7f4c7d05c640 1 -- 172.21.15.150:0/1434323567 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c7815bfd0 msgr2=0x7f4c7815dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:11.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.788+0000 7f4c7d05c640 1 --2- 172.21.15.150:0/1434323567 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c7815bfd0 0x7f4c7815dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:11.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.788+0000 7f4c7d05c640 1 -- 172.21.15.150:0/1434323567 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c7814aa30 con 0x7f4c780cd250 2024-03-17T23:49:11.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.789+0000 7f4c7d05c640 1 --2- 172.21.15.150:0/1434323567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c780cd250 0x7f4c7814a4f0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f4c7400c490 tx=0x7f4c7400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:11.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.789+0000 7f4c6dffb640 1 -- 172.21.15.150:0/1434323567 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c740052d0 con 0x7f4c780cd250 2024-03-17T23:49:11.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.789+0000 7f4c6dffb640 1 -- 172.21.15.150:0/1434323567 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c74005470 con 0x7f4c780cd250 2024-03-17T23:49:11.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.789+0000 7f4c6dffb640 1 -- 172.21.15.150:0/1434323567 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c74012870 con 0x7f4c780cd250 2024-03-17T23:49:11.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.789+0000 7f4c7f2c0640 1 -- 172.21.15.150:0/1434323567 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c7814bd00 con 0x7f4c780cd250 2024-03-17T23:49:11.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.789+0000 7f4c7f2c0640 1 -- 172.21.15.150:0/1434323567 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c7814c2c0 con 0x7f4c780cd250 2024-03-17T23:49:11.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.790+0000 7f4c7f2c0640 1 -- 172.21.15.150:0/1434323567 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c780681a0 con 0x7f4c780cd250 2024-03-17T23:49:11.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.790+0000 7f4c6dffb640 1 -- 172.21.15.150:0/1434323567 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4c74012a10 con 0x7f4c780cd250 2024-03-17T23:49:11.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.791+0000 7f4c6dffb640 1 --2- 172.21.15.150:0/1434323567 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4c5c06bbd0 0x7f4c5c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:11.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.791+0000 7f4c6dffb640 1 -- 172.21.15.150:0/1434323567 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6101+0+0 (secure 0 0 0) 0x7f4c7408a7e0 con 0x7f4c780cd250 2024-03-17T23:49:11.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.791+0000 7f4c7c85b640 1 --2- 172.21.15.150:0/1434323567 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4c5c06bbd0 0x7f4c5c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:11.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.791+0000 7f4c7c85b640 1 --2- 172.21.15.150:0/1434323567 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4c5c06bbd0 0x7f4c5c06e090 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4c68034000 tx=0x7f4c68008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:11.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.795+0000 7f4c6dffb640 1 -- 172.21.15.150:0/1434323567 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4c74058a60 con 0x7f4c780cd250 2024-03-17T23:49:11.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.969+0000 7f4c7f2c0640 1 -- 172.21.15.150:0/1434323567 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4c7806e760 con 0x7f4c780cd250 2024-03-17T23:49:11.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.970+0000 7f4c6dffb640 1 -- 172.21.15.150:0/1434323567 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+16987 (secure 0 0 0) 0x7f4c7405cb00 con 0x7f4c780cd250 2024-03-17T23:49:11.971 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:11.971 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":57,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:49:10.995750+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:49:09.223630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-17T23:49:09.956422+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:49:11.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.972+0000 7f4c7f2c0640 1 -- 172.21.15.150:0/1434323567 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4c5c06bbd0 msgr2=0x7f4c5c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:11.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.972+0000 7f4c7f2c0640 1 --2- 172.21.15.150:0/1434323567 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4c5c06bbd0 0x7f4c5c06e090 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4c68034000 tx=0x7f4c68008030 comp rx=0 tx=0).stop 2024-03-17T23:49:11.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.972+0000 7f4c7f2c0640 1 -- 172.21.15.150:0/1434323567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c780cd250 msgr2=0x7f4c7814a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:11.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.972+0000 7f4c7f2c0640 1 --2- 172.21.15.150:0/1434323567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c780cd250 0x7f4c7814a4f0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f4c7400c490 tx=0x7f4c7400c950 comp rx=0 tx=0).stop 2024-03-17T23:49:11.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.972+0000 7f4c7f2c0640 1 -- 172.21.15.150:0/1434323567 shutdown_connections 2024-03-17T23:49:11.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.972+0000 7f4c7f2c0640 1 --2- 172.21.15.150:0/1434323567 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4c5c06bbd0 0x7f4c5c06e090 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:11.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.972+0000 7f4c7f2c0640 1 --2- 172.21.15.150:0/1434323567 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c7815bfd0 0x7f4c7815dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:11.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.972+0000 7f4c7f2c0640 1 --2- 172.21.15.150:0/1434323567 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4c78155320 0x7f4c7815d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:11.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.972+0000 7f4c7f2c0640 1 --2- 172.21.15.150:0/1434323567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4c780cd250 0x7f4c7814a4f0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:11.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.972+0000 7f4c7f2c0640 1 -- 172.21.15.150:0/1434323567 >> 172.21.15.150:0/1434323567 conn(0x7f4c7806b190 msgr2=0x7f4c78073b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:11.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.973+0000 7f4c7f2c0640 1 -- 172.21.15.150:0/1434323567 shutdown_connections 2024-03-17T23:49:11.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:11.973+0000 7f4c7f2c0640 1 -- 172.21.15.150:0/1434323567 wait complete. 2024-03-17T23:49:11.984 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:49:12.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.084+0000 7f51ce463640 1 Processor -- start 2024-03-17T23:49:12.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.084+0000 7f51ce463640 1 -- start start 2024-03-17T23:49:12.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.085+0000 7f51ce463640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51c81514f0 0x7f51c815bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:12.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.085+0000 7f51ce463640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51c80ce790 con 0x7f51c806ecd0 2024-03-17T23:49:12.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.085+0000 7f51ce463640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51c815b050 con 0x7f51c815ab90 2024-03-17T23:49:12.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.085+0000 7f51ce463640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f51c815b330 con 0x7f51c81514f0 2024-03-17T23:49:12.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.085+0000 7f51cca00640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f51c806ecd0 0x7f51c80cec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55650/0 (socket says 172.21.15.150:55650) 2024-03-17T23:49:12.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.085+0000 7f51cca00640 1 -- 172.21.15.150:0/3068323357 learned_addr learned my addr 172.21.15.150:0/3068323357 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:12.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.085+0000 7f51c7fff640 1 --2- 172.21.15.150:0/3068323357 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51c81514f0 0x7f51c815bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:12.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.085+0000 7f51c6ffd640 1 -- 172.21.15.150:0/3068323357 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 301691851 0 0) 0x7f51c0001490 con 0x7f51c806ecd0 2024-03-17T23:49:12.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.085+0000 7f51c6ffd640 1 -- 172.21.15.150:0/3068323357 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51b0003520 con 0x7f51c806ecd0 2024-03-17T23:49:12.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.086+0000 7f51c6ffd640 1 -- 172.21.15.150:0/3068323357 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1351625254 0 0) 0x7f51b0003520 con 0x7f51c806ecd0 2024-03-17T23:49:12.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.086+0000 7f51c6ffd640 1 -- 172.21.15.150:0/3068323357 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51c81514f0 msgr2=0x7f51c815bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:12.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.086+0000 7f51c6ffd640 1 --2- 172.21.15.150:0/3068323357 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51c81514f0 0x7f51c815bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:12.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.086+0000 7f51c6ffd640 1 -- 172.21.15.150:0/3068323357 >> v1:172.21.15.157:6789/0 conn(0x7f51c815ab90 legacy=0x7f51c806e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:12.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.086+0000 7f51c6ffd640 1 -- 172.21.15.150:0/3068323357 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51c815b610 con 0x7f51c806ecd0 2024-03-17T23:49:12.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.086+0000 7f51c6ffd640 1 -- 172.21.15.150:0/3068323357 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f51c0001bc0 con 0x7f51c806ecd0 2024-03-17T23:49:12.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.086+0000 7f51c6ffd640 1 -- 172.21.15.150:0/3068323357 >> v1:172.21.15.150:6789/0 conn(0x7f51c806ecd0 legacy=0x7f51c80cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:12.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.086+0000 7f51c6ffd640 1 --2- 172.21.15.150:0/3068323357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51b00039e0 0x7f51b0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:12.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.086+0000 7f51c6ffd640 1 --2- 172.21.15.150:0/3068323357 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51c81514f0 0x7f51b0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:12.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.086+0000 7f51c6ffd640 1 --2- 172.21.15.150:0/3068323357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51b0004d40 0x7f51b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:12.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.086+0000 7f51c6ffd640 1 -- 172.21.15.150:0/3068323357 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f51c815b330 con 0x7f51b0004d40 2024-03-17T23:49:12.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.086+0000 7f51c6ffd640 1 -- 172.21.15.150:0/3068323357 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f51b000b9c0 con 0x7f51b00039e0 2024-03-17T23:49:12.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.086+0000 7f51c6ffd640 1 -- 172.21.15.150:0/3068323357 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f51b000bbc0 con 0x7f51c81514f0 2024-03-17T23:49:12.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.086+0000 7f51cca00640 1 --2- 172.21.15.150:0/3068323357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51b0004d40 0x7f51b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:12.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.086+0000 7f51c7fff640 1 --2- 172.21.15.150:0/3068323357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51b00039e0 0x7f51b0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:12.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.086+0000 7f51c77fe640 1 --2- 172.21.15.150:0/3068323357 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51c81514f0 0x7f51b0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:12.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.086+0000 7f51cca00640 1 -- 172.21.15.150:0/3068323357 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51c81514f0 msgr2=0x7f51b0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:12.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.086+0000 7f51cca00640 1 --2- 172.21.15.150:0/3068323357 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51c81514f0 0x7f51b0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:12.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.086+0000 7f51cca00640 1 -- 172.21.15.150:0/3068323357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51b00039e0 msgr2=0x7f51b0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:12.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.086+0000 7f51cca00640 1 --2- 172.21.15.150:0/3068323357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51b00039e0 0x7f51b0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:12.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.086+0000 7f51cca00640 1 -- 172.21.15.150:0/3068323357 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51b0003520 con 0x7f51b0004d40 2024-03-17T23:49:12.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.087+0000 7f51cca00640 1 --2- 172.21.15.150:0/3068323357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51b0004d40 0x7f51b00051c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f51c000aa50 tx=0x7f51b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa5411a341563e05 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:12.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.087+0000 7f51c6ffd640 1 -- 172.21.15.150:0/3068323357 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c0038070 con 0x7f51b0004d40 2024-03-17T23:49:12.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.087+0000 7f51c6ffd640 1 -- 172.21.15.150:0/3068323357 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51c0033410 con 0x7f51b0004d40 2024-03-17T23:49:12.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.087+0000 7f51c6ffd640 1 -- 172.21.15.150:0/3068323357 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c000a670 con 0x7f51b0004d40 2024-03-17T23:49:12.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.088+0000 7f51ce463640 1 -- 172.21.15.150:0/3068323357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51b0004d40 msgr2=0x7f51b00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:12.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.088+0000 7f51ce463640 1 --2- 172.21.15.150:0/3068323357 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51b0004d40 0x7f51b00051c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f51c000aa50 tx=0x7f51b0005830 comp rx=0 tx=0).stop 2024-03-17T23:49:12.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.088+0000 7f51cca00640 1 -- 172.21.15.150:0/3068323357 reap_dead start 2024-03-17T23:49:12.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.088+0000 7f51ce463640 1 -- 172.21.15.150:0/3068323357 shutdown_connections 2024-03-17T23:49:12.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.088+0000 7f51ce463640 1 -- 172.21.15.150:0/3068323357 >> 172.21.15.150:0/3068323357 conn(0x7f51c806b190 msgr2=0x7f51c806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:12.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.088+0000 7f51ce463640 1 -- 172.21.15.150:0/3068323357 shutdown_connections 2024-03-17T23:49:12.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.088+0000 7f51ce463640 1 -- 172.21.15.150:0/3068323357 wait complete. 2024-03-17T23:49:12.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.089+0000 7f51ce463640 1 Processor -- start 2024-03-17T23:49:12.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.089+0000 7f51ce463640 1 -- start start 2024-03-17T23:49:12.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.090+0000 7f51ce463640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51c8136e00 0x7f51c8137bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:12.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.090+0000 7f51ce463640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51c813bb60 0x7f51c8143070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:12.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.090+0000 7f51ce463640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51c81435b0 0x7f51c8151e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:12.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.090+0000 7f51ce463640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f51c81518d0 con 0x7f51c8136e00 2024-03-17T23:49:12.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.090+0000 7f51ce463640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f51c80af510 con 0x7f51c813bb60 2024-03-17T23:49:12.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.090+0000 7f51ce463640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f51c8151a50 con 0x7f51c81435b0 2024-03-17T23:49:12.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.090+0000 7f51c7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51c8136e00 0x7f51c8137bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:12.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.090+0000 7f51c7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51c8136e00 0x7f51c8137bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43842/0 (socket says 172.21.15.150:43842) 2024-03-17T23:49:12.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.090+0000 7f51c7fff640 1 -- 172.21.15.150:0/399553188 learned_addr learned my addr 172.21.15.150:0/399553188 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:12.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.090+0000 7f51c77fe640 1 --2- 172.21.15.150:0/399553188 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51c813bb60 0x7f51c8143070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:12.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.090+0000 7f51cca00640 1 --2- 172.21.15.150:0/399553188 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51c81435b0 0x7f51c8151e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:12.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.090+0000 7f51c7fff640 1 -- 172.21.15.150:0/399553188 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51c81435b0 msgr2=0x7f51c8151e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:12.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.090+0000 7f51c7fff640 1 --2- 172.21.15.150:0/399553188 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51c81435b0 0x7f51c8151e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:12.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.090+0000 7f51c7fff640 1 -- 172.21.15.150:0/399553188 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51c813bb60 msgr2=0x7f51c8143070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:12.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.090+0000 7f51c7fff640 1 --2- 172.21.15.150:0/399553188 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51c813bb60 0x7f51c8143070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:12.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.090+0000 7f51c7fff640 1 -- 172.21.15.150:0/399553188 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51c8143c40 con 0x7f51c8136e00 2024-03-17T23:49:12.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.090+0000 7f51c77fe640 1 --2- 172.21.15.150:0/399553188 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51c813bb60 0x7f51c8143070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:49:12.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.090+0000 7f51c7fff640 1 --2- 172.21.15.150:0/399553188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51c8136e00 0x7f51c8137bf0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f51b80119a0 tx=0x7f51b8011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:12.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.090+0000 7f51c4ff9640 1 -- 172.21.15.150:0/399553188 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b800faf0 con 0x7f51c8136e00 2024-03-17T23:49:12.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.090+0000 7f51c4ff9640 1 -- 172.21.15.150:0/399553188 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51b800fc90 con 0x7f51c8136e00 2024-03-17T23:49:12.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.090+0000 7f51c4ff9640 1 -- 172.21.15.150:0/399553188 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b8014440 con 0x7f51c8136e00 2024-03-17T23:49:12.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.091+0000 7f51ce463640 1 -- 172.21.15.150:0/399553188 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51c8143f20 con 0x7f51c8136e00 2024-03-17T23:49:12.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.091+0000 7f51ce463640 1 -- 172.21.15.150:0/399553188 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51c81376d0 con 0x7f51c8136e00 2024-03-17T23:49:12.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.091+0000 7f51ce463640 1 -- 172.21.15.150:0/399553188 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51c81371e0 con 0x7f51c8136e00 2024-03-17T23:49:12.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.092+0000 7f51c4ff9640 1 -- 172.21.15.150:0/399553188 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f51b8012040 con 0x7f51c8136e00 2024-03-17T23:49:12.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.092+0000 7f51c4ff9640 1 --2- 172.21.15.150:0/399553188 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f51b406bbd0 0x7f51b406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:12.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.092+0000 7f51c4ff9640 1 -- 172.21.15.150:0/399553188 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6101+0+0 (secure 0 0 0) 0x7f51b808b3a0 con 0x7f51c8136e00 2024-03-17T23:49:12.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.092+0000 7f51c77fe640 1 --2- 172.21.15.150:0/399553188 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f51b406bbd0 0x7f51b406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:12.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.093+0000 7f51c77fe640 1 --2- 172.21.15.150:0/399553188 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f51b406bbd0 0x7f51b406e090 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f51c813c270 tx=0x7f51bc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:12.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.097+0000 7f51c4ff9640 1 -- 172.21.15.150:0/399553188 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f51b8019030 con 0x7f51c8136e00 2024-03-17T23:49:12.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.270+0000 7f51ce463640 1 -- 172.21.15.150:0/399553188 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f51c80681a0 con 0x7f51c8136e00 2024-03-17T23:49:12.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.271+0000 7f51c4ff9640 1 -- 172.21.15.150:0/399553188 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+16987 (secure 0 0 0) 0x7f51b8059c70 con 0x7f51c8136e00 2024-03-17T23:49:12.272 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:12.272 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":57,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:49:10.995750+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:49:09.223630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-17T23:49:09.956422+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:49:12.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.273+0000 7f51ce463640 1 -- 172.21.15.150:0/399553188 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f51b406bbd0 msgr2=0x7f51b406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:12.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.273+0000 7f51ce463640 1 --2- 172.21.15.150:0/399553188 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f51b406bbd0 0x7f51b406e090 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f51c813c270 tx=0x7f51bc009040 comp rx=0 tx=0).stop 2024-03-17T23:49:12.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.273+0000 7f51ce463640 1 -- 172.21.15.150:0/399553188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51c8136e00 msgr2=0x7f51c8137bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:12.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.273+0000 7f51ce463640 1 --2- 172.21.15.150:0/399553188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51c8136e00 0x7f51c8137bf0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f51b80119a0 tx=0x7f51b8011e60 comp rx=0 tx=0).stop 2024-03-17T23:49:12.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.273+0000 7f51ce463640 1 -- 172.21.15.150:0/399553188 shutdown_connections 2024-03-17T23:49:12.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.273+0000 7f51ce463640 1 --2- 172.21.15.150:0/399553188 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f51c81435b0 0x7f51c8151e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:12.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.273+0000 7f51ce463640 1 --2- 172.21.15.150:0/399553188 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f51b406bbd0 0x7f51b406e090 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:12.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.273+0000 7f51ce463640 1 --2- 172.21.15.150:0/399553188 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f51c813bb60 0x7f51c8143070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:12.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.273+0000 7f51ce463640 1 --2- 172.21.15.150:0/399553188 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51c8136e00 0x7f51c8137bf0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:12.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.273+0000 7f51ce463640 1 -- 172.21.15.150:0/399553188 >> 172.21.15.150:0/399553188 conn(0x7f51c806b190 msgr2=0x7f51c815d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:12.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.273+0000 7f51ce463640 1 -- 172.21.15.150:0/399553188 shutdown_connections 2024-03-17T23:49:12.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.273+0000 7f51ce463640 1 -- 172.21.15.150:0/399553188 wait complete. 2024-03-17T23:49:12.286 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-17T23:49:12.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.389+0000 7fd268148640 1 Processor -- start 2024-03-17T23:49:12.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.390+0000 7fd268148640 1 -- start start 2024-03-17T23:49:12.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.390+0000 7fd268148640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd2600d0ce0 0x7fd2600b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:12.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.390+0000 7fd268148640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2600b2bf0 0x7fd2600d1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:12.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.390+0000 7fd268148640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd260161d30 con 0x7fd260161520 2024-03-17T23:49:12.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.390+0000 7fd268148640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fd2600b9bf0 con 0x7fd2600d0ce0 2024-03-17T23:49:12.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.390+0000 7fd268148640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd260162940 con 0x7fd2600b2bf0 2024-03-17T23:49:12.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.390+0000 7fd265ee4640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd260161520 0x7fd2600d05d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55652/0 (socket says 172.21.15.150:55652) 2024-03-17T23:49:12.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.390+0000 7fd2656e3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd2600d0ce0 0x7fd2600b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:12.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.390+0000 7fd265ee4640 1 -- 172.21.15.150:0/1607174328 learned_addr learned my addr 172.21.15.150:0/1607174328 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:12.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.391+0000 7fd2666e5640 1 --2- 172.21.15.150:0/1607174328 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2600b2bf0 0x7fd2600d1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:12.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.391+0000 7fd2656e3640 1 -- 172.21.15.150:0/1607174328 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2600b2bf0 msgr2=0x7fd2600d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:12.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.391+0000 7fd2656e3640 1 --2- 172.21.15.150:0/1607174328 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2600b2bf0 0x7fd2600d1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:12.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.391+0000 7fd2656e3640 1 -- 172.21.15.150:0/1607174328 >> v1:172.21.15.150:6789/0 conn(0x7fd260161520 legacy=0x7fd2600d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:12.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.391+0000 7fd2656e3640 1 -- 172.21.15.150:0/1607174328 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd260162010 con 0x7fd2600d0ce0 2024-03-17T23:49:12.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.391+0000 7fd264ee2640 1 -- 172.21.15.150:0/1607174328 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1945512719 0 0) 0x7fd260161d30 con 0x7fd260161520 2024-03-17T23:49:12.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.391+0000 7fd2656e3640 1 --2- 172.21.15.150:0/1607174328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2600d0ce0 0x7fd2600b96b0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd2540097a0 tx=0x7fd25402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15748554d3e9613b server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:12.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.392+0000 7fd264ee2640 1 -- 172.21.15.150:0/1607174328 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd254038070 con 0x7fd2600d0ce0 2024-03-17T23:49:12.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.392+0000 7fd264ee2640 1 -- 172.21.15.150:0/1607174328 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd254031700 con 0x7fd2600d0ce0 2024-03-17T23:49:12.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.392+0000 7fd264ee2640 1 -- 172.21.15.150:0/1607174328 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd254031ad0 con 0x7fd2600d0ce0 2024-03-17T23:49:12.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.392+0000 7fd268148640 1 -- 172.21.15.150:0/1607174328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2600d0ce0 msgr2=0x7fd2600b96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:12.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.392+0000 7fd268148640 1 --2- 172.21.15.150:0/1607174328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2600d0ce0 0x7fd2600b96b0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd2540097a0 tx=0x7fd25402b7a0 comp rx=0 tx=0).stop 2024-03-17T23:49:12.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.392+0000 7fd268148640 1 -- 172.21.15.150:0/1607174328 shutdown_connections 2024-03-17T23:49:12.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.392+0000 7fd268148640 1 --2- 172.21.15.150:0/1607174328 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd2600b2bf0 0x7fd2600d1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:12.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.392+0000 7fd268148640 1 --2- 172.21.15.150:0/1607174328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2600d0ce0 0x7fd2600b96b0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:12.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.392+0000 7fd268148640 1 -- 172.21.15.150:0/1607174328 >> 172.21.15.150:0/1607174328 conn(0x7fd26006ca00 msgr2=0x7fd26006ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:12.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.392+0000 7fd268148640 1 -- 172.21.15.150:0/1607174328 shutdown_connections 2024-03-17T23:49:12.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.392+0000 7fd268148640 1 -- 172.21.15.150:0/1607174328 wait complete. 2024-03-17T23:49:12.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.393+0000 7fd268148640 1 Processor -- start 2024-03-17T23:49:12.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.393+0000 7fd268148640 1 -- start start 2024-03-17T23:49:12.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.394+0000 7fd268148640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2600b2bf0 0x7fd2600b50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:12.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.394+0000 7fd268148640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2600d0ce0 0x7fd2600c9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:12.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.394+0000 7fd268148640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd260161520 0x7fd2600b3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:12.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.394+0000 7fd268148640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd260161900 con 0x7fd2600d0ce0 2024-03-17T23:49:12.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.394+0000 7fd268148640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd2601625d0 con 0x7fd2600b2bf0 2024-03-17T23:49:12.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.394+0000 7fd268148640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd2600d24e0 con 0x7fd260161520 2024-03-17T23:49:12.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.394+0000 7fd2656e3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2600d0ce0 0x7fd2600c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:12.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.394+0000 7fd2656e3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2600d0ce0 0x7fd2600c9910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43864/0 (socket says 172.21.15.150:43864) 2024-03-17T23:49:12.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.394+0000 7fd2656e3640 1 -- 172.21.15.150:0/1380378352 learned_addr learned my addr 172.21.15.150:0/1380378352 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:12.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.394+0000 7fd265ee4640 1 --2- 172.21.15.150:0/1380378352 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2600b2bf0 0x7fd2600b50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:12.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.395+0000 7fd2666e5640 1 --2- 172.21.15.150:0/1380378352 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd260161520 0x7fd2600b3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:12.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.395+0000 7fd2656e3640 1 -- 172.21.15.150:0/1380378352 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd260161520 msgr2=0x7fd2600b3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:12.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.395+0000 7fd2656e3640 1 --2- 172.21.15.150:0/1380378352 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd260161520 0x7fd2600b3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:12.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.395+0000 7fd2656e3640 1 -- 172.21.15.150:0/1380378352 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2600b2bf0 msgr2=0x7fd2600b50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:12.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.395+0000 7fd2656e3640 1 --2- 172.21.15.150:0/1380378352 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2600b2bf0 0x7fd2600b50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:12.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.395+0000 7fd2656e3640 1 -- 172.21.15.150:0/1380378352 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2600b5600 con 0x7fd2600d0ce0 2024-03-17T23:49:12.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.395+0000 7fd265ee4640 1 --2- 172.21.15.150:0/1380378352 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2600b2bf0 0x7fd2600b50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:49:12.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.395+0000 7fd2656e3640 1 --2- 172.21.15.150:0/1380378352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2600d0ce0 0x7fd2600c9910 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fd2540098c0 tx=0x7fd2540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:12.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.395+0000 7fd25e7fc640 1 -- 172.21.15.150:0/1380378352 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd254042070 con 0x7fd2600d0ce0 2024-03-17T23:49:12.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.395+0000 7fd25e7fc640 1 -- 172.21.15.150:0/1380378352 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd254033a30 con 0x7fd2600d0ce0 2024-03-17T23:49:12.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.395+0000 7fd25e7fc640 1 -- 172.21.15.150:0/1380378352 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd254038040 con 0x7fd2600d0ce0 2024-03-17T23:49:12.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.395+0000 7fd268148640 1 -- 172.21.15.150:0/1380378352 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2600b4300 con 0x7fd2600d0ce0 2024-03-17T23:49:12.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.395+0000 7fd268148640 1 -- 172.21.15.150:0/1380378352 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2600b48c0 con 0x7fd2600d0ce0 2024-03-17T23:49:12.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.396+0000 7fd268148640 1 -- 172.21.15.150:0/1380378352 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2600681a0 con 0x7fd2600d0ce0 2024-03-17T23:49:12.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.396+0000 7fd25e7fc640 1 -- 172.21.15.150:0/1380378352 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd254045070 con 0x7fd2600d0ce0 2024-03-17T23:49:12.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.397+0000 7fd25e7fc640 1 --2- 172.21.15.150:0/1380378352 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd24c06bbd0 0x7fd24c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:12.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.397+0000 7fd25e7fc640 1 -- 172.21.15.150:0/1380378352 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6101+0+0 (secure 0 0 0) 0x7fd2540ac890 con 0x7fd2600d0ce0 2024-03-17T23:49:12.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.397+0000 7fd265ee4640 1 --2- 172.21.15.150:0/1380378352 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd24c06bbd0 0x7fd24c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:12.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.397+0000 7fd265ee4640 1 --2- 172.21.15.150:0/1380378352 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd24c06bbd0 0x7fd24c06e090 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd250007fd0 tx=0x7fd250009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:12.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.401+0000 7fd25e7fc640 1 -- 172.21.15.150:0/1380378352 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd254030070 con 0x7fd2600d0ce0 2024-03-17T23:49:12.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.612+0000 7fd268148640 1 -- 172.21.15.150:0/1380378352 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fd2600d0a60 con 0x7fd2600d0ce0 2024-03-17T23:49:12.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.649+0000 7fd25e7fc640 1 -- 172.21.15.150:0/1380378352 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v21) ==== 118+0+0 (secure 0 0 0) 0x7fd25407b3a0 con 0x7fd2600d0ce0 2024-03-17T23:49:12.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.652+0000 7fd268148640 1 -- 172.21.15.150:0/1380378352 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd24c06bbd0 msgr2=0x7fd24c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:12.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.652+0000 7fd268148640 1 --2- 172.21.15.150:0/1380378352 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd24c06bbd0 0x7fd24c06e090 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd250007fd0 tx=0x7fd250009040 comp rx=0 tx=0).stop 2024-03-17T23:49:12.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.652+0000 7fd268148640 1 -- 172.21.15.150:0/1380378352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2600d0ce0 msgr2=0x7fd2600c9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:12.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.652+0000 7fd268148640 1 --2- 172.21.15.150:0/1380378352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2600d0ce0 0x7fd2600c9910 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fd2540098c0 tx=0x7fd2540046c0 comp rx=0 tx=0).stop 2024-03-17T23:49:12.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.652+0000 7fd268148640 1 -- 172.21.15.150:0/1380378352 shutdown_connections 2024-03-17T23:49:12.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.652+0000 7fd268148640 1 --2- 172.21.15.150:0/1380378352 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd260161520 0x7fd2600b3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:12.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.652+0000 7fd268148640 1 --2- 172.21.15.150:0/1380378352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd2600d0ce0 0x7fd2600c9910 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:12.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.652+0000 7fd268148640 1 --2- 172.21.15.150:0/1380378352 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd24c06bbd0 0x7fd24c06e090 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:12.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.652+0000 7fd268148640 1 --2- 172.21.15.150:0/1380378352 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2600b2bf0 0x7fd2600b50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:12.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.652+0000 7fd268148640 1 -- 172.21.15.150:0/1380378352 >> 172.21.15.150:0/1380378352 conn(0x7fd26006ca00 msgr2=0x7fd2600d2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:12.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.652+0000 7fd268148640 1 -- 172.21.15.150:0/1380378352 shutdown_connections 2024-03-17T23:49:12.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.652+0000 7fd268148640 1 -- 172.21.15.150:0/1380378352 wait complete. 2024-03-17T23:49:12.665 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-17T23:49:12.665 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-17T23:49:12.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.773+0000 7f131b552640 1 Processor -- start 2024-03-17T23:49:12.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.773+0000 7f131b552640 1 -- start start 2024-03-17T23:49:12.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.774+0000 7f131b552640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f131415ab90 0x7f131415bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:12.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.774+0000 7f131b552640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13140ce790 con 0x7f131406ecd0 2024-03-17T23:49:12.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.774+0000 7f131b552640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f131415b050 con 0x7f13141514f0 2024-03-17T23:49:12.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.774+0000 7f131b552640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f131415b330 con 0x7f131415ab90 2024-03-17T23:49:12.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.774+0000 7f1319aef640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f131406ecd0 0x7f13140cec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55664/0 (socket says 172.21.15.150:55664) 2024-03-17T23:49:12.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.774+0000 7f1319aef640 1 -- 172.21.15.150:0/2800786498 learned_addr learned my addr 172.21.15.150:0/2800786498 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:12.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.774+0000 7f1318aed640 1 --2- 172.21.15.150:0/2800786498 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f131415ab90 0x7f131415bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:12.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.774+0000 7f130bfff640 1 -- 172.21.15.150:0/2800786498 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2673805398 0 0) 0x7f130c001490 con 0x7f131406ecd0 2024-03-17T23:49:12.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.774+0000 7f130bfff640 1 -- 172.21.15.150:0/2800786498 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1300003520 con 0x7f131406ecd0 2024-03-17T23:49:12.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.774+0000 7f130bfff640 1 -- 172.21.15.150:0/2800786498 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2544621331 0 0) 0x7f131415b050 con 0x7f13141514f0 2024-03-17T23:49:12.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.774+0000 7f130bfff640 1 -- 172.21.15.150:0/2800786498 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f130c001490 con 0x7f13141514f0 2024-03-17T23:49:12.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.774+0000 7f1318aed640 1 -- 172.21.15.150:0/2800786498 >> v1:172.21.15.157:6789/0 conn(0x7f13141514f0 legacy=0x7f131406e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:12.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.774+0000 7f1318aed640 1 -- 172.21.15.150:0/2800786498 >> v1:172.21.15.150:6789/0 conn(0x7f131406ecd0 legacy=0x7f13140cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:12.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.774+0000 7f1318aed640 1 -- 172.21.15.150:0/2800786498 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f131415b610 con 0x7f131415ab90 2024-03-17T23:49:12.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.775+0000 7f1318aed640 1 --2- 172.21.15.150:0/2800786498 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f131415ab90 0x7f131415bd10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f13040097d0 tx=0x7f130402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7094b815a531f270 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:12.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.775+0000 7f130bfff640 1 -- 172.21.15.150:0/2800786498 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1304035020 con 0x7f131415ab90 2024-03-17T23:49:12.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.775+0000 7f130bfff640 1 -- 172.21.15.150:0/2800786498 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1304031890 con 0x7f131415ab90 2024-03-17T23:49:12.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.775+0000 7f130bfff640 1 -- 172.21.15.150:0/2800786498 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1304031c40 con 0x7f131415ab90 2024-03-17T23:49:12.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.776+0000 7f131b552640 1 -- 172.21.15.150:0/2800786498 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f131415ab90 msgr2=0x7f131415bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:12.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.776+0000 7f131b552640 1 --2- 172.21.15.150:0/2800786498 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f131415ab90 0x7f131415bd10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f13040097d0 tx=0x7f130402b930 comp rx=0 tx=0).stop 2024-03-17T23:49:12.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.776+0000 7f131b552640 1 -- 172.21.15.150:0/2800786498 shutdown_connections 2024-03-17T23:49:12.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.776+0000 7f131b552640 1 --2- 172.21.15.150:0/2800786498 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f131415ab90 0x7f131415bd10 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:12.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.776+0000 7f131b552640 1 -- 172.21.15.150:0/2800786498 >> 172.21.15.150:0/2800786498 conn(0x7f131406b190 msgr2=0x7f131406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:12.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.776+0000 7f131b552640 1 -- 172.21.15.150:0/2800786498 shutdown_connections 2024-03-17T23:49:12.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.776+0000 7f131b552640 1 -- 172.21.15.150:0/2800786498 wait complete. 2024-03-17T23:49:12.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.777+0000 7f131b552640 1 Processor -- start 2024-03-17T23:49:12.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.777+0000 7f131b552640 1 -- start start 2024-03-17T23:49:12.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.777+0000 7f131b552640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f131406ecd0 0x7f131413fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:12.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.778+0000 7f131b552640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13141514f0 0x7f131414ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:12.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.778+0000 7f131b552640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f131415ab90 0x7f13141f5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:12.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.778+0000 7f131b552640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f13141518d0 con 0x7f131406ecd0 2024-03-17T23:49:12.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.778+0000 7f131b552640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f13140af510 con 0x7f13141514f0 2024-03-17T23:49:12.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.778+0000 7f131b552640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1314151a50 con 0x7f131415ab90 2024-03-17T23:49:12.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.778+0000 7f13192ee640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f131406ecd0 0x7f131413fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:12.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.778+0000 7f13192ee640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f131406ecd0 0x7f131413fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43868/0 (socket says 172.21.15.150:43868) 2024-03-17T23:49:12.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.778+0000 7f13192ee640 1 -- 172.21.15.150:0/559042403 learned_addr learned my addr 172.21.15.150:0/559042403 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:12.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.778+0000 7f1319aef640 1 --2- 172.21.15.150:0/559042403 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f131415ab90 0x7f13141f5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:12.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.778+0000 7f1318aed640 1 --2- 172.21.15.150:0/559042403 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13141514f0 0x7f131414ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:12.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.778+0000 7f13192ee640 1 -- 172.21.15.150:0/559042403 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f131415ab90 msgr2=0x7f13141f5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:12.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.778+0000 7f13192ee640 1 --2- 172.21.15.150:0/559042403 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f131415ab90 0x7f13141f5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:12.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.778+0000 7f13192ee640 1 -- 172.21.15.150:0/559042403 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13141514f0 msgr2=0x7f131414ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:12.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.778+0000 7f13192ee640 1 --2- 172.21.15.150:0/559042403 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13141514f0 0x7f131414ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:12.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.778+0000 7f13192ee640 1 -- 172.21.15.150:0/559042403 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1314147b80 con 0x7f131406ecd0 2024-03-17T23:49:12.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.778+0000 7f13192ee640 1 --2- 172.21.15.150:0/559042403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f131406ecd0 0x7f131413fa00 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f131000c530 tx=0x7f131000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:12.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.778+0000 7f1309ffb640 1 -- 172.21.15.150:0/559042403 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13100052d0 con 0x7f131406ecd0 2024-03-17T23:49:12.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.778+0000 7f1309ffb640 1 -- 172.21.15.150:0/559042403 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1310005490 con 0x7f131406ecd0 2024-03-17T23:49:12.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.778+0000 7f131b552640 1 -- 172.21.15.150:0/559042403 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13141472e0 con 0x7f131406ecd0 2024-03-17T23:49:12.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.778+0000 7f1309ffb640 1 -- 172.21.15.150:0/559042403 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1310012870 con 0x7f131406ecd0 2024-03-17T23:49:12.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.779+0000 7f131b552640 1 -- 172.21.15.150:0/559042403 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13141478a0 con 0x7f131406ecd0 2024-03-17T23:49:12.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.780+0000 7f131b552640 1 -- 172.21.15.150:0/559042403 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13140681a0 con 0x7f131406ecd0 2024-03-17T23:49:12.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.780+0000 7f1309ffb640 1 -- 172.21.15.150:0/559042403 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1310012a10 con 0x7f131406ecd0 2024-03-17T23:49:12.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.780+0000 7f1309ffb640 1 --2- 172.21.15.150:0/559042403 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f12f806bc00 0x7f12f806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:12.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.780+0000 7f1309ffb640 1 -- 172.21.15.150:0/559042403 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f131008a6b0 con 0x7f131406ecd0 2024-03-17T23:49:12.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.780+0000 7f1318aed640 1 --2- 172.21.15.150:0/559042403 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f12f806bc00 0x7f12f806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:12.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.781+0000 7f1318aed640 1 --2- 172.21.15.150:0/559042403 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f12f806bc00 0x7f12f806e0c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1304036000 tx=0x7f130403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:12.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.784+0000 7f1309ffb640 1 -- 172.21.15.150:0/559042403 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f13100589d0 con 0x7f131406ecd0 2024-03-17T23:49:12.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:12.993+0000 7f131b552640 1 -- 172.21.15.150:0/559042403 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f131406def0 con 0x7f131406ecd0 2024-03-17T23:49:13.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.653+0000 7f1309ffb640 1 -- 172.21.15.150:0/559042403 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v22) ==== 122+0+0 (secure 0 0 0) 0x7f131005ca70 con 0x7f131406ecd0 2024-03-17T23:49:13.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.656+0000 7f131b552640 1 -- 172.21.15.150:0/559042403 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f12f806bc00 msgr2=0x7f12f806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:13.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.656+0000 7f131b552640 1 --2- 172.21.15.150:0/559042403 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f12f806bc00 0x7f12f806e0c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1304036000 tx=0x7f130403c040 comp rx=0 tx=0).stop 2024-03-17T23:49:13.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.656+0000 7f131b552640 1 -- 172.21.15.150:0/559042403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f131406ecd0 msgr2=0x7f131413fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:13.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.656+0000 7f131b552640 1 --2- 172.21.15.150:0/559042403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f131406ecd0 0x7f131413fa00 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f131000c530 tx=0x7f131000c9f0 comp rx=0 tx=0).stop 2024-03-17T23:49:13.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.656+0000 7f131b552640 1 -- 172.21.15.150:0/559042403 shutdown_connections 2024-03-17T23:49:13.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.656+0000 7f131b552640 1 --2- 172.21.15.150:0/559042403 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f131415ab90 0x7f13141f5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:13.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.656+0000 7f131b552640 1 --2- 172.21.15.150:0/559042403 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f12f806bc00 0x7f12f806e0c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:13.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.656+0000 7f131b552640 1 --2- 172.21.15.150:0/559042403 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13141514f0 0x7f131414ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:13.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.656+0000 7f131b552640 1 --2- 172.21.15.150:0/559042403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f131406ecd0 0x7f131413fa00 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:13.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.656+0000 7f131b552640 1 -- 172.21.15.150:0/559042403 >> 172.21.15.150:0/559042403 conn(0x7f131406b190 msgr2=0x7f1314071cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:13.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.656+0000 7f131b552640 1 -- 172.21.15.150:0/559042403 shutdown_connections 2024-03-17T23:49:13.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.656+0000 7f131b552640 1 -- 172.21.15.150:0/559042403 wait complete. 2024-03-17T23:49:13.670 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-17T23:49:13.671 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:49:13.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.777+0000 7f7ab456f640 1 Processor -- start 2024-03-17T23:49:13.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.777+0000 7f7ab456f640 1 -- start start 2024-03-17T23:49:13.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.778+0000 7f7ab456f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7aac0b27b0 0x7f7aac0af580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:13.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.778+0000 7f7ab456f640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7aac072240 0x7f7aac0726d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:13.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.778+0000 7f7ab456f640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7aac0b5270 con 0x7f7aac06f620 2024-03-17T23:49:13.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.778+0000 7f7ab456f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f7aac15e610 con 0x7f7aac0b27b0 2024-03-17T23:49:13.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.778+0000 7f7ab456f640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f7aac15e8f0 con 0x7f7aac072240 2024-03-17T23:49:13.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.778+0000 7f7ab1b0a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7aac0b27b0 0x7f7aac0af580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:13.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.778+0000 7f7ab1b0a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7aac0b27b0 0x7f7aac0af580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43884/0 (socket says 172.21.15.150:43884) 2024-03-17T23:49:13.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.778+0000 7f7ab1b0a640 1 -- 172.21.15.150:0/2846104080 learned_addr learned my addr 172.21.15.150:0/2846104080 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:13.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.778+0000 7f7ab2b0c640 1 --2- 172.21.15.150:0/2846104080 >> v2:172.21.15.157:3300/0 conn(0x7f7aac072240 0x7f7aac0726d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:13.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.778+0000 7f7ab1b0a640 1 -- 172.21.15.150:0/2846104080 >> v2:172.21.15.157:3300/0 conn(0x7f7aac072240 msgr2=0x7f7aac0726d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:13.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.778+0000 7f7ab1b0a640 1 --2- 172.21.15.150:0/2846104080 >> v2:172.21.15.157:3300/0 conn(0x7f7aac072240 0x7f7aac0726d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:13.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.778+0000 7f7ab1b0a640 1 -- 172.21.15.150:0/2846104080 >> v1:172.21.15.157:6789/0 conn(0x7f7aac06f620 legacy=0x7f7aac06fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:13.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.778+0000 7f7ab1b0a640 1 -- 172.21.15.150:0/2846104080 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aac15ebd0 con 0x7f7aac0b27b0 2024-03-17T23:49:13.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.779+0000 7f7ab1b0a640 1 --2- 172.21.15.150:0/2846104080 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7aac0b27b0 0x7f7aac0af580 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f7aac09df10 tx=0x7f7a9c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a9f3351e9488b22 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:13.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.779+0000 7f7ab1309640 1 -- 172.21.15.150:0/2846104080 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c034070 con 0x7f7aac0b27b0 2024-03-17T23:49:13.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.779+0000 7f7ab1309640 1 -- 172.21.15.150:0/2846104080 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a9c030860 con 0x7f7aac0b27b0 2024-03-17T23:49:13.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.779+0000 7f7ab1309640 1 -- 172.21.15.150:0/2846104080 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c030c30 con 0x7f7aac0b27b0 2024-03-17T23:49:13.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.780+0000 7f7ab456f640 1 -- 172.21.15.150:0/2846104080 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7aac0b27b0 msgr2=0x7f7aac0af580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:13.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.780+0000 7f7ab456f640 1 --2- 172.21.15.150:0/2846104080 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7aac0b27b0 0x7f7aac0af580 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f7aac09df10 tx=0x7f7a9c02b9a0 comp rx=0 tx=0).stop 2024-03-17T23:49:13.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.780+0000 7f7ab456f640 1 -- 172.21.15.150:0/2846104080 shutdown_connections 2024-03-17T23:49:13.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.780+0000 7f7ab456f640 1 --2- 172.21.15.150:0/2846104080 >> v2:172.21.15.157:3300/0 conn(0x7f7aac072240 0x7f7aac0726d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:13.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.780+0000 7f7ab456f640 1 --2- 172.21.15.150:0/2846104080 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7aac0b27b0 0x7f7aac0af580 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:13.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.780+0000 7f7ab456f640 1 -- 172.21.15.150:0/2846104080 >> 172.21.15.150:0/2846104080 conn(0x7f7aac0b1c90 msgr2=0x7f7aac0b20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:13.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.780+0000 7f7ab456f640 1 -- 172.21.15.150:0/2846104080 shutdown_connections 2024-03-17T23:49:13.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.780+0000 7f7ab456f640 1 -- 172.21.15.150:0/2846104080 wait complete. 2024-03-17T23:49:13.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.781+0000 7f7ab456f640 1 Processor -- start 2024-03-17T23:49:13.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.781+0000 7f7ab456f640 1 -- start start 2024-03-17T23:49:13.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.781+0000 7f7ab456f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7aac06f620 0x7f7aac160940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:13.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.781+0000 7f7ab456f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7aac072240 0x7f7aac0ae5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:13.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.781+0000 7f7ab456f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7aac0b27b0 0x7f7aac0aeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:13.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.782+0000 7f7ab456f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7aac15e300 con 0x7f7aac072240 2024-03-17T23:49:13.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.782+0000 7f7ab456f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7aac15f190 con 0x7f7aac06f620 2024-03-17T23:49:13.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.782+0000 7f7ab456f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7aac0cc9b0 con 0x7f7aac0b27b0 2024-03-17T23:49:13.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.782+0000 7f7ab1b0a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7aac072240 0x7f7aac0ae5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:13.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.782+0000 7f7ab1b0a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7aac072240 0x7f7aac0ae5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43900/0 (socket says 172.21.15.150:43900) 2024-03-17T23:49:13.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.782+0000 7f7ab1b0a640 1 -- 172.21.15.150:0/1635160820 learned_addr learned my addr 172.21.15.150:0/1635160820 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:13.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.782+0000 7f7ab230b640 1 --2- 172.21.15.150:0/1635160820 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7aac06f620 0x7f7aac160940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:13.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.782+0000 7f7ab2b0c640 1 --2- 172.21.15.150:0/1635160820 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7aac0b27b0 0x7f7aac0aeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:13.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.782+0000 7f7ab1b0a640 1 -- 172.21.15.150:0/1635160820 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7aac0b27b0 msgr2=0x7f7aac0aeb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:13.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.782+0000 7f7ab1b0a640 1 --2- 172.21.15.150:0/1635160820 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7aac0b27b0 0x7f7aac0aeb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:13.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.782+0000 7f7ab1b0a640 1 -- 172.21.15.150:0/1635160820 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7aac06f620 msgr2=0x7f7aac160940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:13.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.782+0000 7f7ab1b0a640 1 --2- 172.21.15.150:0/1635160820 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7aac06f620 0x7f7aac160940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:13.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.782+0000 7f7ab1b0a640 1 -- 172.21.15.150:0/1635160820 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aac154ba0 con 0x7f7aac072240 2024-03-17T23:49:13.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.782+0000 7f7ab1b0a640 1 --2- 172.21.15.150:0/1635160820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7aac072240 0x7f7aac0ae5e0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c0090b0 tx=0x7f7a9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:13.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.782+0000 7f7aa2ffd640 1 -- 172.21.15.150:0/1635160820 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c034040 con 0x7f7aac072240 2024-03-17T23:49:13.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.782+0000 7f7aa2ffd640 1 -- 172.21.15.150:0/1635160820 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a9c03ada0 con 0x7f7aac072240 2024-03-17T23:49:13.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.782+0000 7f7ab456f640 1 -- 172.21.15.150:0/1635160820 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7aac15b520 con 0x7f7aac072240 2024-03-17T23:49:13.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.782+0000 7f7aa2ffd640 1 -- 172.21.15.150:0/1635160820 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c044610 con 0x7f7aac072240 2024-03-17T23:49:13.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.782+0000 7f7ab456f640 1 -- 172.21.15.150:0/1635160820 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7aac15bae0 con 0x7f7aac072240 2024-03-17T23:49:13.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.783+0000 7f7ab456f640 1 -- 172.21.15.150:0/1635160820 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7aac0681a0 con 0x7f7aac072240 2024-03-17T23:49:13.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.784+0000 7f7aa2ffd640 1 -- 172.21.15.150:0/1635160820 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7a9c037070 con 0x7f7aac072240 2024-03-17T23:49:13.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.784+0000 7f7aa2ffd640 1 --2- 172.21.15.150:0/1635160820 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7a9006bbd0 0x7f7a9006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:13.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.784+0000 7f7aa2ffd640 1 -- 172.21.15.150:0/1635160820 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f7a9c0acc90 con 0x7f7aac072240 2024-03-17T23:49:13.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.784+0000 7f7ab230b640 1 --2- 172.21.15.150:0/1635160820 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7a9006bbd0 0x7f7a9006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:13.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.785+0000 7f7ab230b640 1 --2- 172.21.15.150:0/1635160820 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7a9006bbd0 0x7f7a9006e090 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8001780 tx=0x7f7aa8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:13.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.788+0000 7f7aa2ffd640 1 -- 172.21.15.150:0/1635160820 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7a9c03f070 con 0x7f7aac072240 2024-03-17T23:49:13.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.998+0000 7f7ab456f640 1 -- 172.21.15.150:0/1635160820 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7aac072a10 con 0x7f7aac072240 2024-03-17T23:49:13.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:13.998+0000 7f7aa2ffd640 1 -- 172.21.15.150:0/1635160820 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+5187 (secure 0 0 0) 0x7f7a9c07afa0 con 0x7f7aac072240 2024-03-17T23:49:13.999 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:13.999 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":22,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/2082413564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2082413564},{"type":"v1","addr":"172.21.15.150:6836","nonce":2082413564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":6043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/2350077845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":2350077845},{"type":"v1","addr":"172.21.15.150:6837","nonce":2350077845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":6052,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6834/137606728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":137606728},{"type":"v1","addr":"172.21.15.157:6834","nonce":137606728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-17T23:49:10.996124+0000","modified":"2024-03-17T23:49:12.994518+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6835/479821580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6833","nonce":479821580},{"type":"v1","addr":"172.21.15.157:6835","nonce":479821580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6204,"qdb_cluster":[6204]},"id":3}]} 2024-03-17T23:49:13.999 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 22 2024-03-17T23:49:14.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.000+0000 7f7ab456f640 1 -- 172.21.15.150:0/1635160820 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7a9006bbd0 msgr2=0x7f7a9006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:14.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.000+0000 7f7ab456f640 1 --2- 172.21.15.150:0/1635160820 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7a9006bbd0 0x7f7a9006e090 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8001780 tx=0x7f7aa8009040 comp rx=0 tx=0).stop 2024-03-17T23:49:14.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.000+0000 7f7ab456f640 1 -- 172.21.15.150:0/1635160820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7aac072240 msgr2=0x7f7aac0ae5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:14.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.000+0000 7f7ab456f640 1 --2- 172.21.15.150:0/1635160820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7aac072240 0x7f7aac0ae5e0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c0090b0 tx=0x7f7a9c0046c0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.001+0000 7f7ab456f640 1 -- 172.21.15.150:0/1635160820 shutdown_connections 2024-03-17T23:49:14.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.001+0000 7f7ab456f640 1 --2- 172.21.15.150:0/1635160820 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7aac0b27b0 0x7f7aac0aeb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.001+0000 7f7ab456f640 1 --2- 172.21.15.150:0/1635160820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7aac072240 0x7f7aac0ae5e0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.001+0000 7f7ab456f640 1 --2- 172.21.15.150:0/1635160820 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7a9006bbd0 0x7f7a9006e090 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.001+0000 7f7ab456f640 1 --2- 172.21.15.150:0/1635160820 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7aac06f620 0x7f7aac160940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.001+0000 7f7ab456f640 1 -- 172.21.15.150:0/1635160820 >> 172.21.15.150:0/1635160820 conn(0x7f7aac0b1c90 msgr2=0x7f7aac0d00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:14.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.001+0000 7f7ab456f640 1 -- 172.21.15.150:0/1635160820 shutdown_connections 2024-03-17T23:49:14.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.001+0000 7f7ab456f640 1 -- 172.21.15.150:0/1635160820 wait complete. 2024-03-17T23:49:14.012 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:49:14.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.111+0000 7f3f576c0640 1 Processor -- start 2024-03-17T23:49:14.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.112+0000 7f3f576c0640 1 -- start start 2024-03-17T23:49:14.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.112+0000 7f3f576c0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f500cf120 0x7f3f501619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:14.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.112+0000 7f3f576c0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f500b9cb0 con 0x7f3f500b9860 2024-03-17T23:49:14.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.112+0000 7f3f576c0640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f501620a0 con 0x7f3f5015de30 2024-03-17T23:49:14.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.112+0000 7f3f576c0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3f50162940 con 0x7f3f500cf120 2024-03-17T23:49:14.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.112+0000 7f3f55c5d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f3f500b9860 0x7f3f500d2690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55678/0 (socket says 172.21.15.150:55678) 2024-03-17T23:49:14.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.113+0000 7f3f55c5d640 1 -- 172.21.15.150:0/3050039809 learned_addr learned my addr 172.21.15.150:0/3050039809 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:14.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.113+0000 7f3f54c5b640 1 --2- 172.21.15.150:0/3050039809 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f500cf120 0x7f3f501619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:14.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.113+0000 7f3f47fff640 1 -- 172.21.15.150:0/3050039809 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 321312530 0 0) 0x7f3f500b9cb0 con 0x7f3f500b9860 2024-03-17T23:49:14.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.113+0000 7f3f47fff640 1 -- 172.21.15.150:0/3050039809 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f3c003520 con 0x7f3f500b9860 2024-03-17T23:49:14.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.113+0000 7f3f47fff640 1 -- 172.21.15.150:0/3050039809 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 191248954 0 0) 0x7f3f501620a0 con 0x7f3f5015de30 2024-03-17T23:49:14.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.113+0000 7f3f47fff640 1 -- 172.21.15.150:0/3050039809 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f500b9cb0 con 0x7f3f5015de30 2024-03-17T23:49:14.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.113+0000 7f3f54c5b640 1 -- 172.21.15.150:0/3050039809 >> v1:172.21.15.157:6789/0 conn(0x7f3f5015de30 legacy=0x7f3f500cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:14.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.113+0000 7f3f54c5b640 1 -- 172.21.15.150:0/3050039809 >> v1:172.21.15.150:6789/0 conn(0x7f3f500b9860 legacy=0x7f3f500d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:14.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.113+0000 7f3f54c5b640 1 -- 172.21.15.150:0/3050039809 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f50162380 con 0x7f3f500cf120 2024-03-17T23:49:14.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.114+0000 7f3f54c5b640 1 --2- 172.21.15.150:0/3050039809 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f500cf120 0x7f3f501619a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3f400097d0 tx=0x7f3f4002b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e877b51e744cb6e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:14.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.114+0000 7f3f47fff640 1 -- 172.21.15.150:0/3050039809 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f40035020 con 0x7f3f500cf120 2024-03-17T23:49:14.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.114+0000 7f3f47fff640 1 -- 172.21.15.150:0/3050039809 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f40031800 con 0x7f3f500cf120 2024-03-17T23:49:14.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.114+0000 7f3f47fff640 1 -- 172.21.15.150:0/3050039809 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f40031bd0 con 0x7f3f500cf120 2024-03-17T23:49:14.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.114+0000 7f3f576c0640 1 -- 172.21.15.150:0/3050039809 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f500cf120 msgr2=0x7f3f501619a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:14.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.114+0000 7f3f576c0640 1 --2- 172.21.15.150:0/3050039809 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f500cf120 0x7f3f501619a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3f400097d0 tx=0x7f3f4002b8a0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.114+0000 7f3f576c0640 1 -- 172.21.15.150:0/3050039809 shutdown_connections 2024-03-17T23:49:14.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.114+0000 7f3f576c0640 1 --2- 172.21.15.150:0/3050039809 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f500cf120 0x7f3f501619a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.114+0000 7f3f576c0640 1 -- 172.21.15.150:0/3050039809 >> 172.21.15.150:0/3050039809 conn(0x7f3f500730b0 msgr2=0x7f3f500d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:14.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.115+0000 7f3f576c0640 1 -- 172.21.15.150:0/3050039809 shutdown_connections 2024-03-17T23:49:14.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.115+0000 7f3f576c0640 1 -- 172.21.15.150:0/3050039809 wait complete. 2024-03-17T23:49:14.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.116+0000 7f3f576c0640 1 Processor -- start 2024-03-17T23:49:14.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.116+0000 7f3f576c0640 1 -- start start 2024-03-17T23:49:14.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.116+0000 7f3f576c0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f500b9860 0x7f3f500b56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:14.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.116+0000 7f3f576c0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f500cf120 0x7f3f50159280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:14.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.116+0000 7f3f576c0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f5015de30 0x7f3f500cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:14.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.116+0000 7f3f576c0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5015e210 con 0x7f3f500b9860 2024-03-17T23:49:14.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.116+0000 7f3f576c0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3f500d2170 con 0x7f3f5015de30 2024-03-17T23:49:14.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.116+0000 7f3f576c0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3f500d2da0 con 0x7f3f500cf120 2024-03-17T23:49:14.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.116+0000 7f3f5545c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f500b9860 0x7f3f500b56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:14.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.116+0000 7f3f5545c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f500b9860 0x7f3f500b56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43902/0 (socket says 172.21.15.150:43902) 2024-03-17T23:49:14.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.116+0000 7f3f5545c640 1 -- 172.21.15.150:0/3250848862 learned_addr learned my addr 172.21.15.150:0/3250848862 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:14.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.117+0000 7f3f55c5d640 1 --2- 172.21.15.150:0/3250848862 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f5015de30 0x7f3f500cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:14.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.117+0000 7f3f54c5b640 1 --2- 172.21.15.150:0/3250848862 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f500cf120 0x7f3f50159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:14.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.117+0000 7f3f5545c640 1 -- 172.21.15.150:0/3250848862 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f500cf120 msgr2=0x7f3f50159280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:14.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.117+0000 7f3f5545c640 1 --2- 172.21.15.150:0/3250848862 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f500cf120 0x7f3f50159280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.117+0000 7f3f5545c640 1 -- 172.21.15.150:0/3250848862 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f5015de30 msgr2=0x7f3f500cb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:14.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.117+0000 7f3f5545c640 1 --2- 172.21.15.150:0/3250848862 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f5015de30 0x7f3f500cb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.117+0000 7f3f5545c640 1 -- 172.21.15.150:0/3250848862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f500cd0d0 con 0x7f3f500b9860 2024-03-17T23:49:14.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.117+0000 7f3f5545c640 1 --2- 172.21.15.150:0/3250848862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f500b9860 0x7f3f500b56f0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f3f4800c7a0 tx=0x7f3f4800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:14.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.117+0000 7f3f45ffb640 1 -- 172.21.15.150:0/3250848862 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f48003b70 con 0x7f3f500b9860 2024-03-17T23:49:14.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.117+0000 7f3f45ffb640 1 -- 172.21.15.150:0/3250848862 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f48003d30 con 0x7f3f500b9860 2024-03-17T23:49:14.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.117+0000 7f3f576c0640 1 -- 172.21.15.150:0/3250848862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f501597c0 con 0x7f3f500b9860 2024-03-17T23:49:14.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.117+0000 7f3f45ffb640 1 -- 172.21.15.150:0/3250848862 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4800aa30 con 0x7f3f500b9860 2024-03-17T23:49:14.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.117+0000 7f3f576c0640 1 -- 172.21.15.150:0/3250848862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f500cc310 con 0x7f3f500b9860 2024-03-17T23:49:14.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.118+0000 7f3f576c0640 1 -- 172.21.15.150:0/3250848862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f500681a0 con 0x7f3f500b9860 2024-03-17T23:49:14.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.118+0000 7f3f45ffb640 1 -- 172.21.15.150:0/3250848862 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3f4800abd0 con 0x7f3f500b9860 2024-03-17T23:49:14.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.119+0000 7f3f45ffb640 1 --2- 172.21.15.150:0/3250848862 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3f3406bbd0 0x7f3f3406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:14.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.119+0000 7f3f45ffb640 1 -- 172.21.15.150:0/3250848862 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f3f4808a970 con 0x7f3f500b9860 2024-03-17T23:49:14.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.119+0000 7f3f54c5b640 1 --2- 172.21.15.150:0/3250848862 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3f3406bbd0 0x7f3f3406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:14.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.119+0000 7f3f54c5b640 1 --2- 172.21.15.150:0/3250848862 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3f3406bbd0 0x7f3f3406e090 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3f40036000 tx=0x7f3f40008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:14.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.123+0000 7f3f45ffb640 1 -- 172.21.15.150:0/3250848862 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3f48058d10 con 0x7f3f500b9860 2024-03-17T23:49:14.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.297+0000 7f3f576c0640 1 -- 172.21.15.150:0/3250848862 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3f5015d880 con 0x7f3f500b9860 2024-03-17T23:49:14.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.298+0000 7f3f45ffb640 1 -- 172.21.15.150:0/3250848862 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) ==== 74+0+16974 (secure 0 0 0) 0x7f3f4805cdb0 con 0x7f3f500b9860 2024-03-17T23:49:14.299 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:14.299 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":58,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:49:12.646307+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:49:09.223630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-17T23:49:09.956422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:49:14.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.301+0000 7f3f576c0640 1 -- 172.21.15.150:0/3250848862 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3f3406bbd0 msgr2=0x7f3f3406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:14.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.301+0000 7f3f576c0640 1 --2- 172.21.15.150:0/3250848862 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3f3406bbd0 0x7f3f3406e090 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3f40036000 tx=0x7f3f40008030 comp rx=0 tx=0).stop 2024-03-17T23:49:14.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.301+0000 7f3f576c0640 1 -- 172.21.15.150:0/3250848862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f500b9860 msgr2=0x7f3f500b56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:14.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.301+0000 7f3f576c0640 1 --2- 172.21.15.150:0/3250848862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f500b9860 0x7f3f500b56f0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f3f4800c7a0 tx=0x7f3f4800cc60 comp rx=0 tx=0).stop 2024-03-17T23:49:14.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.301+0000 7f3f576c0640 1 -- 172.21.15.150:0/3250848862 shutdown_connections 2024-03-17T23:49:14.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.301+0000 7f3f576c0640 1 --2- 172.21.15.150:0/3250848862 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3f3406bbd0 0x7f3f3406e090 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.301+0000 7f3f576c0640 1 --2- 172.21.15.150:0/3250848862 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f5015de30 0x7f3f500cb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.301+0000 7f3f576c0640 1 --2- 172.21.15.150:0/3250848862 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f500cf120 0x7f3f50159280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.301+0000 7f3f576c0640 1 --2- 172.21.15.150:0/3250848862 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f500b9860 0x7f3f500b56f0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.301+0000 7f3f576c0640 1 -- 172.21.15.150:0/3250848862 >> 172.21.15.150:0/3250848862 conn(0x7f3f500730b0 msgr2=0x7f3f5015bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:14.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.301+0000 7f3f576c0640 1 -- 172.21.15.150:0/3250848862 shutdown_connections 2024-03-17T23:49:14.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.301+0000 7f3f576c0640 1 -- 172.21.15.150:0/3250848862 wait complete. 2024-03-17T23:49:14.312 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-17T23:49:14.313 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-17T23:49:14.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.412+0000 7f4ecd00c640 1 Processor -- start 2024-03-17T23:49:14.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.412+0000 7f4ecd00c640 1 -- start start 2024-03-17T23:49:14.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.413+0000 7f4ecd00c640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4ec815e060 0x7f4ec80d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:14.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.413+0000 7f4ecd00c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ec80b57d0 0x7f4ec80b5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:14.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.413+0000 7f4ecd00c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ec80d2890 con 0x7f4ec806c6e0 2024-03-17T23:49:14.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.413+0000 7f4ecd00c640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f4ec80d2b70 con 0x7f4ec815e060 2024-03-17T23:49:14.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.413+0000 7f4ecd00c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4ec80d2e50 con 0x7f4ec80b57d0 2024-03-17T23:49:14.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.413+0000 7f4ec659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ec80b57d0 0x7f4ec80b5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:14.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.413+0000 7f4ec6d9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f4ec815e060 0x7f4ec80d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:14.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.413+0000 7f4ec659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ec80b57d0 0x7f4ec80b5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:35010/0 (socket says 172.21.15.150:35010) 2024-03-17T23:49:14.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.413+0000 7f4ec659c640 1 -- 172.21.15.150:0/1448186842 learned_addr learned my addr 172.21.15.150:0/1448186842 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:14.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.413+0000 7f4ec759e640 1 --1- 172.21.15.150:0/1448186842 >> v1:172.21.15.157:6789/0 conn(0x7f4ec806c6e0 0x7f4ec806caf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:57302/0 (socket says 172.21.15.150:57302) 2024-03-17T23:49:14.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.414+0000 7f4ec659c640 1 -- 172.21.15.150:0/1448186842 >> v2:172.21.15.157:3300/0 conn(0x7f4ec815e060 msgr2=0x7f4ec80d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:14.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.414+0000 7f4ec659c640 1 --2- 172.21.15.150:0/1448186842 >> v2:172.21.15.157:3300/0 conn(0x7f4ec815e060 0x7f4ec80d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.414+0000 7f4ec659c640 1 -- 172.21.15.150:0/1448186842 >> v1:172.21.15.157:6789/0 conn(0x7f4ec806c6e0 legacy=0x7f4ec806caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:14.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.414+0000 7f4ec659c640 1 -- 172.21.15.150:0/1448186842 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec80d3130 con 0x7f4ec80b57d0 2024-03-17T23:49:14.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.414+0000 7f4ec659c640 1 --2- 172.21.15.150:0/1448186842 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ec80b57d0 0x7f4ec80b5c60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc009870 tx=0x7f4ebc02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa658652d5191ff9 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:14.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.414+0000 7f4ec5d9b640 1 -- 172.21.15.150:0/1448186842 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc035020 con 0x7f4ec80b57d0 2024-03-17T23:49:14.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.415+0000 7f4ec5d9b640 1 -- 172.21.15.150:0/1448186842 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ebc031640 con 0x7f4ec80b57d0 2024-03-17T23:49:14.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.415+0000 7f4ec5d9b640 1 -- 172.21.15.150:0/1448186842 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc02d040 con 0x7f4ec80b57d0 2024-03-17T23:49:14.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.415+0000 7f4ecd00c640 1 -- 172.21.15.150:0/1448186842 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ec80b57d0 msgr2=0x7f4ec80b5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:14.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.415+0000 7f4ecd00c640 1 --2- 172.21.15.150:0/1448186842 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ec80b57d0 0x7f4ec80b5c60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc009870 tx=0x7f4ebc02b8f0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.415+0000 7f4ecd00c640 1 -- 172.21.15.150:0/1448186842 shutdown_connections 2024-03-17T23:49:14.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.415+0000 7f4ecd00c640 1 --2- 172.21.15.150:0/1448186842 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ec80b57d0 0x7f4ec80b5c60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.415+0000 7f4ecd00c640 1 --2- 172.21.15.150:0/1448186842 >> v2:172.21.15.157:3300/0 conn(0x7f4ec815e060 0x7f4ec80d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.415+0000 7f4ecd00c640 1 -- 172.21.15.150:0/1448186842 >> 172.21.15.150:0/1448186842 conn(0x7f4ec8073200 msgr2=0x7f4ec80d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:14.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.415+0000 7f4ecd00c640 1 -- 172.21.15.150:0/1448186842 shutdown_connections 2024-03-17T23:49:14.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.415+0000 7f4ecd00c640 1 -- 172.21.15.150:0/1448186842 wait complete. 2024-03-17T23:49:14.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.417+0000 7f4ecd00c640 1 Processor -- start 2024-03-17T23:49:14.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.417+0000 7f4ecd00c640 1 -- start start 2024-03-17T23:49:14.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.417+0000 7f4ecd00c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ec806c6e0 0x7f4ec80b49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:14.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.417+0000 7f4ecd00c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ec80b57d0 0x7f4ec81532e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:14.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.417+0000 7f4ecd00c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ec815e060 0x7f4ec8153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:14.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.417+0000 7f4ecd00c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec815e440 con 0x7f4ec815e060 2024-03-17T23:49:14.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.417+0000 7f4ecd00c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec80b7f00 con 0x7f4ec80b57d0 2024-03-17T23:49:14.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.417+0000 7f4ecd00c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4ec80d2460 con 0x7f4ec806c6e0 2024-03-17T23:49:14.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.417+0000 7f4ec759e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ec815e060 0x7f4ec8153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:14.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.417+0000 7f4ec6d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ec806c6e0 0x7f4ec80b49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:14.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.418+0000 7f4ec659c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ec80b57d0 0x7f4ec81532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:14.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.418+0000 7f4ec6d9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ec806c6e0 0x7f4ec80b49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:35018/0 (socket says 172.21.15.150:35018) 2024-03-17T23:49:14.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.418+0000 7f4ec6d9d640 1 -- 172.21.15.150:0/1748880078 learned_addr learned my addr 172.21.15.150:0/1748880078 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:14.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.418+0000 7f4ec759e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ec815e060 0x7f4ec8153e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43918/0 (socket says 172.21.15.150:43918) 2024-03-17T23:49:14.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.418+0000 7f4ec6d9d640 1 -- 172.21.15.150:0/1748880078 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ec80b57d0 msgr2=0x7f4ec81532e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:14.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.418+0000 7f4ec6d9d640 1 --2- 172.21.15.150:0/1748880078 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ec80b57d0 0x7f4ec81532e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.418+0000 7f4ec6d9d640 1 -- 172.21.15.150:0/1748880078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ec815e060 msgr2=0x7f4ec8153e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:14.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.418+0000 7f4ec6d9d640 1 --2- 172.21.15.150:0/1748880078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ec815e060 0x7f4ec8153e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.418+0000 7f4ec6d9d640 1 -- 172.21.15.150:0/1748880078 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec8156510 con 0x7f4ec806c6e0 2024-03-17T23:49:14.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.418+0000 7f4ec759e640 1 --2- 172.21.15.150:0/1748880078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ec815e060 0x7f4ec8153e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:14.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.418+0000 7f4ec6d9d640 1 --2- 172.21.15.150:0/1748880078 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ec806c6e0 0x7f4ec80b49d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4eb800efc0 tx=0x7f4eb800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:14.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.418+0000 7f4eaf7fe640 1 -- 172.21.15.150:0/1748880078 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb8018020 con 0x7f4ec806c6e0 2024-03-17T23:49:14.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.418+0000 7f4eaf7fe640 1 -- 172.21.15.150:0/1748880078 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4eb8014480 con 0x7f4ec806c6e0 2024-03-17T23:49:14.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.418+0000 7f4ecd00c640 1 -- 172.21.15.150:0/1748880078 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ec8156790 con 0x7f4ec806c6e0 2024-03-17T23:49:14.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.418+0000 7f4eaf7fe640 1 -- 172.21.15.150:0/1748880078 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb8014850 con 0x7f4ec806c6e0 2024-03-17T23:49:14.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.418+0000 7f4ecd00c640 1 -- 172.21.15.150:0/1748880078 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ec8156d50 con 0x7f4ec806c6e0 2024-03-17T23:49:14.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.419+0000 7f4ecd00c640 1 -- 172.21.15.150:0/1748880078 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ec80688b0 con 0x7f4ec806c6e0 2024-03-17T23:49:14.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.420+0000 7f4eaf7fe640 1 -- 172.21.15.150:0/1748880078 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4eb8005d10 con 0x7f4ec806c6e0 2024-03-17T23:49:14.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.420+0000 7f4eaf7fe640 1 --2- 172.21.15.150:0/1748880078 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4eb406bbd0 0x7f4eb406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:14.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.420+0000 7f4eaf7fe640 1 -- 172.21.15.150:0/1748880078 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f4eb808b660 con 0x7f4ec806c6e0 2024-03-17T23:49:14.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.420+0000 7f4ec659c640 1 --2- 172.21.15.150:0/1748880078 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4eb406bbd0 0x7f4eb406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:14.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.421+0000 7f4ec659c640 1 --2- 172.21.15.150:0/1748880078 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4eb406bbd0 0x7f4eb406e090 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc036000 tx=0x7f4ebc03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:14.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.429+0000 7f4eaf7fe640 1 -- 172.21.15.150:0/1748880078 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4eb8022070 con 0x7f4ec806c6e0 2024-03-17T23:49:14.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.599+0000 7f4ecd00c640 1 -- 172.21.15.150:0/1748880078 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4ec80d3ff0 con 0x7f4eb406bbd0 2024-03-17T23:49:14.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.604+0000 7f4eaf7fe640 1 -- 172.21.15.150:0/1748880078 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+245797 (secure 0 0 0) 0x7f4ebc084040 con 0x7f4eb406bbd0 2024-03-17T23:49:14.608 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:14.609 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-03-17T23:49:14.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.607+0000 7f4ecd00c640 1 -- 172.21.15.150:0/1748880078 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4eb406bbd0 msgr2=0x7f4eb406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:14.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.607+0000 7f4ecd00c640 1 --2- 172.21.15.150:0/1748880078 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4eb406bbd0 0x7f4eb406e090 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc036000 tx=0x7f4ebc03c040 comp rx=0 tx=0).stop 2024-03-17T23:49:14.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.607+0000 7f4ecd00c640 1 -- 172.21.15.150:0/1748880078 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ec806c6e0 msgr2=0x7f4ec80b49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:14.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.607+0000 7f4ecd00c640 1 --2- 172.21.15.150:0/1748880078 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ec806c6e0 0x7f4ec80b49d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4eb800efc0 tx=0x7f4eb800c490 comp rx=0 tx=0).stop 2024-03-17T23:49:14.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.607+0000 7f4ecd00c640 1 -- 172.21.15.150:0/1748880078 shutdown_connections 2024-03-17T23:49:14.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.607+0000 7f4ecd00c640 1 --2- 172.21.15.150:0/1748880078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ec815e060 0x7f4ec8153e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.607+0000 7f4ecd00c640 1 --2- 172.21.15.150:0/1748880078 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4eb406bbd0 0x7f4eb406e090 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.607+0000 7f4ecd00c640 1 --2- 172.21.15.150:0/1748880078 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4ec80b57d0 0x7f4ec81532e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.607+0000 7f4ecd00c640 1 --2- 172.21.15.150:0/1748880078 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4ec806c6e0 0x7f4ec80b49d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.607+0000 7f4ecd00c640 1 -- 172.21.15.150:0/1748880078 >> 172.21.15.150:0/1748880078 conn(0x7f4ec8073200 msgr2=0x7f4ec80b3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:14.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.607+0000 7f4ecd00c640 1 -- 172.21.15.150:0/1748880078 shutdown_connections 2024-03-17T23:49:14.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.607+0000 7f4ecd00c640 1 -- 172.21.15.150:0/1748880078 wait complete. 2024-03-17T23:49:14.619 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-03-17T23:49:13.624322+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592140,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":188,"num_write_kb":2421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":112,"acting":112,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":6,"kb":754974720,"kb_used":239188,"kb_used_data":6372,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735532,"statfs":{"total":773094113280,"available":772849184768,"internally_reserved":0,"allocated":6524928,"data_stored":3587795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1772,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000782"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.670271+0000","last_change":"2024-03-17T23:49:11.667507+0000","last_active":"2024-03-17T23:49:11.670271+0000","last_peered":"2024-03-17T23:49:11.670271+0000","last_clean":"2024-03-17T23:49:11.670271+0000","last_became_active":"2024-03-17T23:49:11.666238+0000","last_became_peered":"2024-03-17T23:49:11.666238+0000","last_unstale":"2024-03-17T23:49:11.670271+0000","last_undegraded":"2024-03-17T23:49:11.670271+0000","last_fullsized":"2024-03-17T23:49:11.670271+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:15.111002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.663256+0000","last_change":"2024-03-17T23:49:11.660542+0000","last_active":"2024-03-17T23:49:11.663256+0000","last_peered":"2024-03-17T23:49:11.663256+0000","last_clean":"2024-03-17T23:49:11.663256+0000","last_became_active":"2024-03-17T23:49:11.660272+0000","last_became_peered":"2024-03-17T23:49:11.660272+0000","last_unstale":"2024-03-17T23:49:11.663256+0000","last_undegraded":"2024-03-17T23:49:11.663256+0000","last_fullsized":"2024-03-17T23:49:11.663256+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:52:42.007552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.658166+0000","last_change":"2024-03-17T23:49:11.655317+0000","last_active":"2024-03-17T23:49:11.658166+0000","last_peered":"2024-03-17T23:49:11.658166+0000","last_clean":"2024-03-17T23:49:11.658166+0000","last_became_active":"2024-03-17T23:49:11.655056+0000","last_became_peered":"2024-03-17T23:49:11.655056+0000","last_unstale":"2024-03-17T23:49:11.658166+0000","last_undegraded":"2024-03-17T23:49:11.658166+0000","last_fullsized":"2024-03-17T23:49:11.658166+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:00:05.868962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659258+0000","last_change":"2024-03-17T23:49:11.657708+0000","last_active":"2024-03-17T23:49:11.659258+0000","last_peered":"2024-03-17T23:49:11.659258+0000","last_clean":"2024-03-17T23:49:11.659258+0000","last_became_active":"2024-03-17T23:49:11.656418+0000","last_became_peered":"2024-03-17T23:49:11.656418+0000","last_unstale":"2024-03-17T23:49:11.659258+0000","last_undegraded":"2024-03-17T23:49:11.659258+0000","last_fullsized":"2024-03-17T23:49:11.659258+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T23:59:25.825110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.652513+0000","last_change":"2024-03-17T23:49:11.645251+0000","last_active":"2024-03-17T23:49:11.652513+0000","last_peered":"2024-03-17T23:49:11.652513+0000","last_clean":"2024-03-17T23:49:11.652513+0000","last_became_active":"2024-03-17T23:49:11.644999+0000","last_became_peered":"2024-03-17T23:49:11.644999+0000","last_unstale":"2024-03-17T23:49:11.652513+0000","last_undegraded":"2024-03-17T23:49:11.652513+0000","last_fullsized":"2024-03-17T23:49:11.652513+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:10:05.668007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.652609+0000","last_change":"2024-03-17T23:49:11.648149+0000","last_active":"2024-03-17T23:49:11.652609+0000","last_peered":"2024-03-17T23:49:11.652609+0000","last_clean":"2024-03-17T23:49:11.652609+0000","last_became_active":"2024-03-17T23:49:11.647887+0000","last_became_peered":"2024-03-17T23:49:11.647887+0000","last_unstale":"2024-03-17T23:49:11.652609+0000","last_undegraded":"2024-03-17T23:49:11.652609+0000","last_fullsized":"2024-03-17T23:49:11.652609+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:16:38.296758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.660678+0000","last_change":"2024-03-17T23:49:11.655458+0000","last_active":"2024-03-17T23:49:11.660678+0000","last_peered":"2024-03-17T23:49:11.660678+0000","last_clean":"2024-03-17T23:49:11.660678+0000","last_became_active":"2024-03-17T23:49:11.655181+0000","last_became_peered":"2024-03-17T23:49:11.655181+0000","last_unstale":"2024-03-17T23:49:11.660678+0000","last_undegraded":"2024-03-17T23:49:11.660678+0000","last_fullsized":"2024-03-17T23:49:11.660678+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:55:32.443067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.664194+0000","last_change":"2024-03-17T23:49:11.656693+0000","last_active":"2024-03-17T23:49:11.664194+0000","last_peered":"2024-03-17T23:49:11.664194+0000","last_clean":"2024-03-17T23:49:11.664194+0000","last_became_active":"2024-03-17T23:49:11.656400+0000","last_became_peered":"2024-03-17T23:49:11.656400+0000","last_unstale":"2024-03-17T23:49:11.664194+0000","last_undegraded":"2024-03-17T23:49:11.664194+0000","last_fullsized":"2024-03-17T23:49:11.664194+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:54:01.212249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.655981+0000","last_change":"2024-03-17T23:49:11.652508+0000","last_active":"2024-03-17T23:49:11.655981+0000","last_peered":"2024-03-17T23:49:11.655981+0000","last_clean":"2024-03-17T23:49:11.655981+0000","last_became_active":"2024-03-17T23:49:11.652153+0000","last_became_peered":"2024-03-17T23:49:11.652153+0000","last_unstale":"2024-03-17T23:49:11.655981+0000","last_undegraded":"2024-03-17T23:49:11.655981+0000","last_fullsized":"2024-03-17T23:49:11.655981+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:35.755434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.650975+0000","last_change":"2024-03-17T23:49:11.647575+0000","last_active":"2024-03-17T23:49:11.650975+0000","last_peered":"2024-03-17T23:49:11.650975+0000","last_clean":"2024-03-17T23:49:11.650975+0000","last_became_active":"2024-03-17T23:49:11.647313+0000","last_became_peered":"2024-03-17T23:49:11.647313+0000","last_unstale":"2024-03-17T23:49:11.650975+0000","last_undegraded":"2024-03-17T23:49:11.650975+0000","last_fullsized":"2024-03-17T23:49:11.650975+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:37.609834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.670209+0000","last_change":"2024-03-17T23:49:11.668183+0000","last_active":"2024-03-17T23:49:11.670209+0000","last_peered":"2024-03-17T23:49:11.670209+0000","last_clean":"2024-03-17T23:49:11.670209+0000","last_became_active":"2024-03-17T23:49:11.667577+0000","last_became_peered":"2024-03-17T23:49:11.667577+0000","last_unstale":"2024-03-17T23:49:11.670209+0000","last_undegraded":"2024-03-17T23:49:11.670209+0000","last_fullsized":"2024-03-17T23:49:11.670209+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T23:59:25.825110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.657425+0000","last_change":"2024-03-17T23:49:11.656210+0000","last_active":"2024-03-17T23:49:11.657425+0000","last_peered":"2024-03-17T23:49:11.657425+0000","last_clean":"2024-03-17T23:49:11.657425+0000","last_became_active":"2024-03-17T23:49:11.655629+0000","last_became_peered":"2024-03-17T23:49:11.655629+0000","last_unstale":"2024-03-17T23:49:11.657425+0000","last_undegraded":"2024-03-17T23:49:11.657425+0000","last_fullsized":"2024-03-17T23:49:11.657425+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:09:15.871249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.660884+0000","last_change":"2024-03-17T23:49:11.658319+0000","last_active":"2024-03-17T23:49:11.660884+0000","last_peered":"2024-03-17T23:49:11.660884+0000","last_clean":"2024-03-17T23:49:11.660884+0000","last_became_active":"2024-03-17T23:49:11.656866+0000","last_became_peered":"2024-03-17T23:49:11.656866+0000","last_unstale":"2024-03-17T23:49:11.660884+0000","last_undegraded":"2024-03-17T23:49:11.660884+0000","last_fullsized":"2024-03-17T23:49:11.660884+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:26:49.763854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.664486+0000","last_change":"2024-03-17T23:49:11.658960+0000","last_active":"2024-03-17T23:49:11.664486+0000","last_peered":"2024-03-17T23:49:11.664486+0000","last_clean":"2024-03-17T23:49:11.664486+0000","last_became_active":"2024-03-17T23:49:11.658695+0000","last_became_peered":"2024-03-17T23:49:11.658695+0000","last_unstale":"2024-03-17T23:49:11.664486+0000","last_undegraded":"2024-03-17T23:49:11.664486+0000","last_fullsized":"2024-03-17T23:49:11.664486+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:07:08.139888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.655891+0000","last_change":"2024-03-17T23:49:11.654139+0000","last_active":"2024-03-17T23:49:11.655891+0000","last_peered":"2024-03-17T23:49:11.655891+0000","last_clean":"2024-03-17T23:49:11.655891+0000","last_became_active":"2024-03-17T23:49:11.653838+0000","last_became_peered":"2024-03-17T23:49:11.653838+0000","last_unstale":"2024-03-17T23:49:11.655891+0000","last_undegraded":"2024-03-17T23:49:11.655891+0000","last_fullsized":"2024-03-17T23:49:11.655891+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:00:05.868962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.670150+0000","last_change":"2024-03-17T23:49:11.667572+0000","last_active":"2024-03-17T23:49:11.670150+0000","last_peered":"2024-03-17T23:49:11.670150+0000","last_clean":"2024-03-17T23:49:11.670150+0000","last_became_active":"2024-03-17T23:49:11.666249+0000","last_became_peered":"2024-03-17T23:49:11.666249+0000","last_unstale":"2024-03-17T23:49:11.670150+0000","last_undegraded":"2024-03-17T23:49:11.670150+0000","last_fullsized":"2024-03-17T23:49:11.670150+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:00:05.868962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.661421+0000","last_change":"2024-03-17T23:49:11.656653+0000","last_active":"2024-03-17T23:49:11.661421+0000","last_peered":"2024-03-17T23:49:11.661421+0000","last_clean":"2024-03-17T23:49:11.661421+0000","last_became_active":"2024-03-17T23:49:11.656335+0000","last_became_peered":"2024-03-17T23:49:11.656335+0000","last_unstale":"2024-03-17T23:49:11.661421+0000","last_undegraded":"2024-03-17T23:49:11.661421+0000","last_fullsized":"2024-03-17T23:49:11.661421+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:21:49.823973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1c","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.669900+0000","last_change":"2024-03-17T23:49:10.654468+0000","last_active":"2024-03-17T23:49:11.669900+0000","last_peered":"2024-03-17T23:49:11.669900+0000","last_clean":"2024-03-17T23:49:11.669900+0000","last_became_active":"2024-03-17T23:49:10.652810+0000","last_became_peered":"2024-03-17T23:49:10.652810+0000","last_unstale":"2024-03-17T23:49:11.669900+0000","last_undegraded":"2024-03-17T23:49:11.669900+0000","last_fullsized":"2024-03-17T23:49:11.669900+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:36.605162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1b","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.664427+0000","last_change":"2024-03-17T23:49:10.653909+0000","last_active":"2024-03-17T23:49:11.664427+0000","last_peered":"2024-03-17T23:49:11.664427+0000","last_clean":"2024-03-17T23:49:11.664427+0000","last_became_active":"2024-03-17T23:49:10.653648+0000","last_became_peered":"2024-03-17T23:49:10.653648+0000","last_unstale":"2024-03-17T23:49:11.664427+0000","last_undegraded":"2024-03-17T23:49:11.664427+0000","last_fullsized":"2024-03-17T23:49:11.664427+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:04.395229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.664451+0000","last_change":"2024-03-17T23:49:11.655809+0000","last_active":"2024-03-17T23:49:11.664451+0000","last_peered":"2024-03-17T23:49:11.664451+0000","last_clean":"2024-03-17T23:49:11.664451+0000","last_became_active":"2024-03-17T23:49:11.655510+0000","last_became_peered":"2024-03-17T23:49:11.655510+0000","last_unstale":"2024-03-17T23:49:11.664451+0000","last_undegraded":"2024-03-17T23:49:11.664451+0000","last_fullsized":"2024-03-17T23:49:11.664451+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:59:37.841369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.648060+0000","last_change":"2024-03-17T23:49:10.656254+0000","last_active":"2024-03-17T23:49:11.648060+0000","last_peered":"2024-03-17T23:49:11.648060+0000","last_clean":"2024-03-17T23:49:11.648060+0000","last_became_active":"2024-03-17T23:49:10.655915+0000","last_became_peered":"2024-03-17T23:49:10.655915+0000","last_unstale":"2024-03-17T23:49:11.648060+0000","last_undegraded":"2024-03-17T23:49:11.648060+0000","last_fullsized":"2024-03-17T23:49:11.648060+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:26:48.759181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.643323+0000","last_change":"2024-03-17T23:49:10.649005+0000","last_active":"2024-03-17T23:49:11.643323+0000","last_peered":"2024-03-17T23:49:11.643323+0000","last_clean":"2024-03-17T23:49:11.643323+0000","last_became_active":"2024-03-17T23:49:10.648717+0000","last_became_peered":"2024-03-17T23:49:10.648717+0000","last_unstale":"2024-03-17T23:49:11.643323+0000","last_undegraded":"2024-03-17T23:49:11.643323+0000","last_fullsized":"2024-03-17T23:49:11.643323+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:26:48.759181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659768+0000","last_change":"2024-03-17T23:49:11.658818+0000","last_active":"2024-03-17T23:49:11.659768+0000","last_peered":"2024-03-17T23:49:11.659768+0000","last_clean":"2024-03-17T23:49:11.659768+0000","last_became_active":"2024-03-17T23:49:11.658546+0000","last_became_peered":"2024-03-17T23:49:11.658546+0000","last_unstale":"2024-03-17T23:49:11.659768+0000","last_undegraded":"2024-03-17T23:49:11.659768+0000","last_fullsized":"2024-03-17T23:49:11.659768+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:15.111002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.648333+0000","last_change":"2024-03-17T23:49:10.658240+0000","last_active":"2024-03-17T23:49:11.648333+0000","last_peered":"2024-03-17T23:49:11.648333+0000","last_clean":"2024-03-17T23:49:11.648333+0000","last_became_active":"2024-03-17T23:49:10.657882+0000","last_became_peered":"2024-03-17T23:49:10.657882+0000","last_unstale":"2024-03-17T23:49:11.648333+0000","last_undegraded":"2024-03-17T23:49:11.648333+0000","last_fullsized":"2024-03-17T23:49:11.648333+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:16:37.292086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.665442+0000","last_change":"2024-03-17T23:49:11.660908+0000","last_active":"2024-03-17T23:49:11.665442+0000","last_peered":"2024-03-17T23:49:11.665442+0000","last_clean":"2024-03-17T23:49:11.665442+0000","last_became_active":"2024-03-17T23:49:11.660644+0000","last_became_peered":"2024-03-17T23:49:11.660644+0000","last_unstale":"2024-03-17T23:49:11.665442+0000","last_undegraded":"2024-03-17T23:49:11.665442+0000","last_fullsized":"2024-03-17T23:49:11.665442+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:24:35.182836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659276+0000","last_change":"2024-03-17T23:49:11.657785+0000","last_active":"2024-03-17T23:49:11.659276+0000","last_peered":"2024-03-17T23:49:11.659276+0000","last_clean":"2024-03-17T23:49:11.659276+0000","last_became_active":"2024-03-17T23:49:11.657040+0000","last_became_peered":"2024-03-17T23:49:11.657040+0000","last_unstale":"2024-03-17T23:49:11.659276+0000","last_undegraded":"2024-03-17T23:49:11.659276+0000","last_fullsized":"2024-03-17T23:49:11.659276+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:05.399901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"57'3","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.672409+0000","last_change":"2024-03-17T23:49:10.656304+0000","last_active":"2024-03-17T23:49:11.672409+0000","last_peered":"2024-03-17T23:49:11.672409+0000","last_clean":"2024-03-17T23:49:11.672409+0000","last_became_active":"2024-03-17T23:49:10.655795+0000","last_became_peered":"2024-03-17T23:49:10.655795+0000","last_unstale":"2024-03-17T23:49:11.672409+0000","last_undegraded":"2024-03-17T23:49:11.672409+0000","last_fullsized":"2024-03-17T23:49:11.672409+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:57:35.440980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.663919+0000","last_change":"2024-03-17T23:49:11.660244+0000","last_active":"2024-03-17T23:49:11.663919+0000","last_peered":"2024-03-17T23:49:11.663919+0000","last_clean":"2024-03-17T23:49:11.663919+0000","last_became_active":"2024-03-17T23:49:11.659982+0000","last_became_peered":"2024-03-17T23:49:11.659982+0000","last_unstale":"2024-03-17T23:49:11.663919+0000","last_undegraded":"2024-03-17T23:49:11.663919+0000","last_fullsized":"2024-03-17T23:49:11.663919+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:05:20.724503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"57'6","reported_seq":18,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.675052+0000","last_change":"2024-03-17T23:49:10.650050+0000","last_active":"2024-03-17T23:49:11.675052+0000","last_peered":"2024-03-17T23:49:11.675052+0000","last_clean":"2024-03-17T23:49:11.675052+0000","last_became_active":"2024-03-17T23:49:10.649505+0000","last_became_peered":"2024-03-17T23:49:10.649505+0000","last_unstale":"2024-03-17T23:49:11.675052+0000","last_undegraded":"2024-03-17T23:49:11.675052+0000","last_fullsized":"2024-03-17T23:49:11.675052+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:24:34.178164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.662957+0000","last_change":"2024-03-17T23:49:11.659236+0000","last_active":"2024-03-17T23:49:11.662957+0000","last_peered":"2024-03-17T23:49:11.662957+0000","last_clean":"2024-03-17T23:49:11.662957+0000","last_became_active":"2024-03-17T23:49:11.658962+0000","last_became_peered":"2024-03-17T23:49:11.658962+0000","last_unstale":"2024-03-17T23:49:11.662957+0000","last_undegraded":"2024-03-17T23:49:11.662957+0000","last_fullsized":"2024-03-17T23:49:11.662957+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:48:27.835084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.670115+0000","last_change":"2024-03-17T23:49:11.667510+0000","last_active":"2024-03-17T23:49:11.670115+0000","last_peered":"2024-03-17T23:49:11.670115+0000","last_clean":"2024-03-17T23:49:11.670115+0000","last_became_active":"2024-03-17T23:49:11.665443+0000","last_became_peered":"2024-03-17T23:49:11.665443+0000","last_unstale":"2024-03-17T23:49:11.670115+0000","last_undegraded":"2024-03-17T23:49:11.670115+0000","last_fullsized":"2024-03-17T23:49:11.670115+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:35.755434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.642062+0000","last_change":"2024-03-17T23:49:10.650246+0000","last_active":"2024-03-17T23:49:11.642062+0000","last_peered":"2024-03-17T23:49:11.642062+0000","last_clean":"2024-03-17T23:49:11.642062+0000","last_became_active":"2024-03-17T23:49:10.649837+0000","last_became_peered":"2024-03-17T23:49:10.649837+0000","last_unstale":"2024-03-17T23:49:11.642062+0000","last_undegraded":"2024-03-17T23:49:11.642062+0000","last_fullsized":"2024-03-17T23:49:11.642062+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:08:24.371365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659904+0000","last_change":"2024-03-17T23:49:11.655903+0000","last_active":"2024-03-17T23:49:11.659904+0000","last_peered":"2024-03-17T23:49:11.659904+0000","last_clean":"2024-03-17T23:49:11.659904+0000","last_became_active":"2024-03-17T23:49:11.655521+0000","last_became_peered":"2024-03-17T23:49:11.655521+0000","last_unstale":"2024-03-17T23:49:11.659904+0000","last_undegraded":"2024-03-17T23:49:11.659904+0000","last_fullsized":"2024-03-17T23:49:11.659904+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:55:32.443067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.648464+0000","last_change":"2024-03-17T23:49:10.651590+0000","last_active":"2024-03-17T23:49:11.648464+0000","last_peered":"2024-03-17T23:49:11.648464+0000","last_clean":"2024-03-17T23:49:11.648464+0000","last_became_active":"2024-03-17T23:49:10.651300+0000","last_became_peered":"2024-03-17T23:49:10.651300+0000","last_unstale":"2024-03-17T23:49:11.648464+0000","last_undegraded":"2024-03-17T23:49:11.648464+0000","last_fullsized":"2024-03-17T23:49:11.648464+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:05:19.719830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659701+0000","last_change":"2024-03-17T23:49:11.657665+0000","last_active":"2024-03-17T23:49:11.659701+0000","last_peered":"2024-03-17T23:49:11.659701+0000","last_clean":"2024-03-17T23:49:11.659701+0000","last_became_active":"2024-03-17T23:49:11.657348+0000","last_became_peered":"2024-03-17T23:49:11.657348+0000","last_unstale":"2024-03-17T23:49:11.659701+0000","last_undegraded":"2024-03-17T23:49:11.659701+0000","last_fullsized":"2024-03-17T23:49:11.659701+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:05.399901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"57'8","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.672231+0000","last_change":"2024-03-17T23:49:10.654069+0000","last_active":"2024-03-17T23:49:11.672231+0000","last_peered":"2024-03-17T23:49:11.672231+0000","last_clean":"2024-03-17T23:49:11.672231+0000","last_became_active":"2024-03-17T23:49:10.653800+0000","last_became_peered":"2024-03-17T23:49:10.653800+0000","last_unstale":"2024-03-17T23:49:11.672231+0000","last_undegraded":"2024-03-17T23:49:11.672231+0000","last_fullsized":"2024-03-17T23:49:11.672231+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:10:04.663334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.670182+0000","last_change":"2024-03-17T23:49:11.667630+0000","last_active":"2024-03-17T23:49:11.670182+0000","last_peered":"2024-03-17T23:49:11.670182+0000","last_clean":"2024-03-17T23:49:11.670182+0000","last_became_active":"2024-03-17T23:49:11.665756+0000","last_became_peered":"2024-03-17T23:49:11.665756+0000","last_unstale":"2024-03-17T23:49:11.670182+0000","last_undegraded":"2024-03-17T23:49:11.670182+0000","last_fullsized":"2024-03-17T23:49:11.670182+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:55:32.443067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659682+0000","last_change":"2024-03-17T23:49:11.657629+0000","last_active":"2024-03-17T23:49:11.659682+0000","last_peered":"2024-03-17T23:49:11.659682+0000","last_clean":"2024-03-17T23:49:11.659682+0000","last_became_active":"2024-03-17T23:49:11.657356+0000","last_became_peered":"2024-03-17T23:49:11.657356+0000","last_unstale":"2024-03-17T23:49:11.659682+0000","last_undegraded":"2024-03-17T23:49:11.659682+0000","last_fullsized":"2024-03-17T23:49:11.659682+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:15.111002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.d","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.669750+0000","last_change":"2024-03-17T23:49:10.652457+0000","last_active":"2024-03-17T23:49:11.669750+0000","last_peered":"2024-03-17T23:49:11.669750+0000","last_clean":"2024-03-17T23:49:11.669750+0000","last_became_active":"2024-03-17T23:49:10.652105+0000","last_became_peered":"2024-03-17T23:49:10.652105+0000","last_unstale":"2024-03-17T23:49:11.669750+0000","last_undegraded":"2024-03-17T23:49:11.669750+0000","last_fullsized":"2024-03-17T23:49:11.669750+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:24:34.178164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.665510+0000","last_change":"2024-03-17T23:49:11.661804+0000","last_active":"2024-03-17T23:49:11.665510+0000","last_peered":"2024-03-17T23:49:11.665510+0000","last_clean":"2024-03-17T23:49:11.665510+0000","last_became_active":"2024-03-17T23:49:11.661518+0000","last_became_peered":"2024-03-17T23:49:11.661518+0000","last_unstale":"2024-03-17T23:49:11.665510+0000","last_undegraded":"2024-03-17T23:49:11.665510+0000","last_fullsized":"2024-03-17T23:49:11.665510+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:37.609834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.662996+0000","last_change":"2024-03-17T23:49:11.659323+0000","last_active":"2024-03-17T23:49:11.662996+0000","last_peered":"2024-03-17T23:49:11.662996+0000","last_clean":"2024-03-17T23:49:11.662996+0000","last_became_active":"2024-03-17T23:49:11.659047+0000","last_became_peered":"2024-03-17T23:49:11.659047+0000","last_unstale":"2024-03-17T23:49:11.662996+0000","last_undegraded":"2024-03-17T23:49:11.662996+0000","last_fullsized":"2024-03-17T23:49:11.662996+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:11:49.300093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.639827+0000","last_change":"2024-03-17T23:49:10.655445+0000","last_active":"2024-03-17T23:49:11.639827+0000","last_peered":"2024-03-17T23:49:11.639827+0000","last_clean":"2024-03-17T23:49:11.639827+0000","last_became_active":"2024-03-17T23:49:10.654632+0000","last_became_peered":"2024-03-17T23:49:10.654632+0000","last_unstale":"2024-03-17T23:49:11.639827+0000","last_undegraded":"2024-03-17T23:49:11.639827+0000","last_fullsized":"2024-03-17T23:49:11.639827+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:54:00.207576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.647483+0000","last_change":"2024-03-17T23:49:10.654321+0000","last_active":"2024-03-17T23:49:11.647483+0000","last_peered":"2024-03-17T23:49:11.647483+0000","last_clean":"2024-03-17T23:49:11.647483+0000","last_became_active":"2024-03-17T23:49:10.653691+0000","last_became_peered":"2024-03-17T23:49:10.653691+0000","last_unstale":"2024-03-17T23:49:11.647483+0000","last_undegraded":"2024-03-17T23:49:11.647483+0000","last_fullsized":"2024-03-17T23:49:11.647483+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:04.395229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.663081+0000","last_change":"2024-03-17T23:49:11.660090+0000","last_active":"2024-03-17T23:49:11.663081+0000","last_peered":"2024-03-17T23:49:11.663081+0000","last_clean":"2024-03-17T23:49:11.663081+0000","last_became_active":"2024-03-17T23:49:11.659121+0000","last_became_peered":"2024-03-17T23:49:11.659121+0000","last_unstale":"2024-03-17T23:49:11.663081+0000","last_undegraded":"2024-03-17T23:49:11.663081+0000","last_fullsized":"2024-03-17T23:49:11.663081+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:23:45.083399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.675037+0000","last_change":"2024-03-17T23:49:10.655569+0000","last_active":"2024-03-17T23:49:11.675037+0000","last_peered":"2024-03-17T23:49:11.675037+0000","last_clean":"2024-03-17T23:49:11.675037+0000","last_became_active":"2024-03-17T23:49:10.655284+0000","last_became_peered":"2024-03-17T23:49:10.655284+0000","last_unstale":"2024-03-17T23:49:11.675037+0000","last_undegraded":"2024-03-17T23:49:11.675037+0000","last_fullsized":"2024-03-17T23:49:11.675037+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:07:07.135215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659651+0000","last_change":"2024-03-17T23:49:11.655737+0000","last_active":"2024-03-17T23:49:11.659651+0000","last_peered":"2024-03-17T23:49:11.659651+0000","last_clean":"2024-03-17T23:49:11.659651+0000","last_became_active":"2024-03-17T23:49:11.655464+0000","last_became_peered":"2024-03-17T23:49:11.655464+0000","last_unstale":"2024-03-17T23:49:11.659651+0000","last_undegraded":"2024-03-17T23:49:11.659651+0000","last_fullsized":"2024-03-17T23:49:11.659651+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T23:59:25.825110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.652228+0000","last_change":"2024-03-17T23:49:11.649056+0000","last_active":"2024-03-17T23:49:11.652228+0000","last_peered":"2024-03-17T23:49:11.652228+0000","last_clean":"2024-03-17T23:49:11.652228+0000","last_became_active":"2024-03-17T23:49:11.648791+0000","last_became_peered":"2024-03-17T23:49:11.648791+0000","last_unstale":"2024-03-17T23:49:11.652228+0000","last_undegraded":"2024-03-17T23:49:11.652228+0000","last_fullsized":"2024-03-17T23:49:11.652228+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:09:15.871249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.642103+0000","last_change":"2024-03-17T23:49:10.648679+0000","last_active":"2024-03-17T23:49:11.642103+0000","last_peered":"2024-03-17T23:49:11.642103+0000","last_clean":"2024-03-17T23:49:11.642103+0000","last_became_active":"2024-03-17T23:49:10.648407+0000","last_became_peered":"2024-03-17T23:49:10.648407+0000","last_unstale":"2024-03-17T23:49:11.642103+0000","last_undegraded":"2024-03-17T23:49:11.642103+0000","last_fullsized":"2024-03-17T23:49:11.642103+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:38:18.229949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.649926+0000","last_change":"2024-03-17T23:49:10.655937+0000","last_active":"2024-03-17T23:49:11.649926+0000","last_peered":"2024-03-17T23:49:11.649926+0000","last_clean":"2024-03-17T23:49:11.649926+0000","last_became_active":"2024-03-17T23:49:10.654317+0000","last_became_peered":"2024-03-17T23:49:10.654317+0000","last_unstale":"2024-03-17T23:49:11.649926+0000","last_undegraded":"2024-03-17T23:49:11.649926+0000","last_fullsized":"2024-03-17T23:49:11.649926+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:34.750762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.639931+0000","last_change":"2024-03-17T23:49:10.655613+0000","last_active":"2024-03-17T23:49:11.639931+0000","last_peered":"2024-03-17T23:49:11.639931+0000","last_clean":"2024-03-17T23:49:11.639931+0000","last_became_active":"2024-03-17T23:49:10.655134+0000","last_became_peered":"2024-03-17T23:49:10.655134+0000","last_unstale":"2024-03-17T23:49:11.639931+0000","last_undegraded":"2024-03-17T23:49:11.639931+0000","last_fullsized":"2024-03-17T23:49:11.639931+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:48:38.320666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.660923+0000","last_change":"2024-03-17T23:49:11.656942+0000","last_active":"2024-03-17T23:49:11.660923+0000","last_peered":"2024-03-17T23:49:11.660923+0000","last_clean":"2024-03-17T23:49:11.660923+0000","last_became_active":"2024-03-17T23:49:11.656612+0000","last_became_peered":"2024-03-17T23:49:11.656612+0000","last_unstale":"2024-03-17T23:49:11.660923+0000","last_undegraded":"2024-03-17T23:49:11.660923+0000","last_fullsized":"2024-03-17T23:49:11.660923+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:21:49.823973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.643561+0000","last_change":"2024-03-17T23:49:10.647432+0000","last_active":"2024-03-17T23:49:11.643561+0000","last_peered":"2024-03-17T23:49:11.643561+0000","last_clean":"2024-03-17T23:49:11.643561+0000","last_became_active":"2024-03-17T23:49:10.647160+0000","last_became_peered":"2024-03-17T23:49:10.647160+0000","last_unstale":"2024-03-17T23:49:11.643561+0000","last_undegraded":"2024-03-17T23:49:11.643561+0000","last_fullsized":"2024-03-17T23:49:11.643561+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:05:19.719830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.651916+0000","last_change":"2024-03-17T23:49:11.645350+0000","last_active":"2024-03-17T23:49:11.651916+0000","last_peered":"2024-03-17T23:49:11.651916+0000","last_clean":"2024-03-17T23:49:11.651916+0000","last_became_active":"2024-03-17T23:49:11.645101+0000","last_became_peered":"2024-03-17T23:49:11.645101+0000","last_unstale":"2024-03-17T23:49:11.651916+0000","last_undegraded":"2024-03-17T23:49:11.651916+0000","last_fullsized":"2024-03-17T23:49:11.651916+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:57:36.445652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.639772+0000","last_change":"2024-03-17T23:49:10.655236+0000","last_active":"2024-03-17T23:49:11.639772+0000","last_peered":"2024-03-17T23:49:11.639772+0000","last_clean":"2024-03-17T23:49:11.639772+0000","last_became_active":"2024-03-17T23:49:10.654952+0000","last_became_peered":"2024-03-17T23:49:10.654952+0000","last_unstale":"2024-03-17T23:49:11.639772+0000","last_undegraded":"2024-03-17T23:49:11.639772+0000","last_fullsized":"2024-03-17T23:49:11.639772+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:59:36.836697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.641904+0000","last_change":"2024-03-17T23:49:10.653100+0000","last_active":"2024-03-17T23:49:11.641904+0000","last_peered":"2024-03-17T23:49:11.641904+0000","last_clean":"2024-03-17T23:49:11.641904+0000","last_became_active":"2024-03-17T23:49:10.652809+0000","last_became_peered":"2024-03-17T23:49:10.652809+0000","last_unstale":"2024-03-17T23:49:11.641904+0000","last_undegraded":"2024-03-17T23:49:11.641904+0000","last_fullsized":"2024-03-17T23:49:11.641904+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:51:40.737241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":238,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.645564+0000","last_change":"2024-03-17T23:46:44.768668+0000","last_active":"2024-03-17T23:49:11.645564+0000","last_peered":"2024-03-17T23:49:11.645564+0000","last_clean":"2024-03-17T23:49:11.645564+0000","last_became_active":"2024-03-17T23:46:44.767826+0000","last_became_peered":"2024-03-17T23:46:44.767826+0000","last_unstale":"2024-03-17T23:49:11.645564+0000","last_undegraded":"2024-03-17T23:49:11.645564+0000","last_fullsized":"2024-03-17T23:49:11.645564+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_clean_scrub_stamp":"2024-03-17T23:46:43.744878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:54:22.286155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":70,"acting":70,"num_store_stats":5},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":1772,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23,"ondisk_log_size":23,"up":40,"acting":40,"num_store_stats":5},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":14,"seq":60129542177,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27644,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":630784,"data_stored":283907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542177,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27640,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":630784,"data_stored":283907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28232,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343608,"statfs":{"total":96636764160,"available":96607854592,"internally_reserved":0,"allocated":1236992,"data_stored":879794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574881,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32432,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339408,"statfs":{"total":96636764160,"available":96603553792,"internally_reserved":0,"allocated":1282048,"data_stored":899989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31844,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339996,"statfs":{"total":96636764160,"available":96604155904,"internally_reserved":0,"allocated":679936,"data_stored":309461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574881,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":688128,"data_stored":310113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574881,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":692224,"data_stored":311073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":684032,"data_stored":309551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-17T23:49:14.624 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-17T23:49:14.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.760+0000 7ffb5fadd640 1 Processor -- start 2024-03-17T23:49:14.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.760+0000 7ffb5fadd640 1 -- start start 2024-03-17T23:49:14.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.761+0000 7ffb5fadd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ffb58155320 0x7ffb5815bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:14.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.761+0000 7ffb5fadd640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ffb5806dd30 0x7ffb580ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:14.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.761+0000 7ffb5fadd640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb580cf3b0 con 0x7ffb580cd250 2024-03-17T23:49:14.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.761+0000 7ffb5fadd640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7ffb580cd6c0 con 0x7ffb58155320 2024-03-17T23:49:14.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.761+0000 7ffb5fadd640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ffb5815c550 con 0x7ffb5806dd30 2024-03-17T23:49:14.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.761+0000 7ffb5d879640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ffb58155320 0x7ffb5815bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:14.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.761+0000 7ffb5d879640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ffb58155320 0x7ffb5815bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43922/0 (socket says 172.21.15.150:43922) 2024-03-17T23:49:14.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.761+0000 7ffb5d879640 1 -- 172.21.15.150:0/1760193870 learned_addr learned my addr 172.21.15.150:0/1760193870 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:14.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.761+0000 7ffb5d078640 1 --2- 172.21.15.150:0/1760193870 >> v2:172.21.15.157:3300/0 conn(0x7ffb5806dd30 0x7ffb580ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:14.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.761+0000 7ffb5d879640 1 -- 172.21.15.150:0/1760193870 >> v2:172.21.15.157:3300/0 conn(0x7ffb5806dd30 msgr2=0x7ffb580ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:14.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.761+0000 7ffb5d879640 1 --2- 172.21.15.150:0/1760193870 >> v2:172.21.15.157:3300/0 conn(0x7ffb5806dd30 0x7ffb580ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.761+0000 7ffb5d879640 1 -- 172.21.15.150:0/1760193870 >> v1:172.21.15.157:6789/0 conn(0x7ffb580cd250 legacy=0x7ffb5815e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:14.763 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.761+0000 7ffb5d879640 1 -- 172.21.15.150:0/1760193870 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb580cf690 con 0x7ffb58155320 2024-03-17T23:49:14.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.762+0000 7ffb5d879640 1 --2- 172.21.15.150:0/1760193870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb58155320 0x7ffb5815bfd0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ffb58073500 tx=0x7ffb4c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=488ae33f0a703e38 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:14.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.762+0000 7ffb5c877640 1 -- 172.21.15.150:0/1760193870 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c034070 con 0x7ffb58155320 2024-03-17T23:49:14.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.762+0000 7ffb5c877640 1 -- 172.21.15.150:0/1760193870 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb4c030860 con 0x7ffb58155320 2024-03-17T23:49:14.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.762+0000 7ffb5c877640 1 -- 172.21.15.150:0/1760193870 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c030c30 con 0x7ffb58155320 2024-03-17T23:49:14.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.762+0000 7ffb5fadd640 1 -- 172.21.15.150:0/1760193870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb58155320 msgr2=0x7ffb5815bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:14.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.762+0000 7ffb5fadd640 1 --2- 172.21.15.150:0/1760193870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb58155320 0x7ffb5815bfd0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ffb58073500 tx=0x7ffb4c02b9a0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.762+0000 7ffb5fadd640 1 -- 172.21.15.150:0/1760193870 shutdown_connections 2024-03-17T23:49:14.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.762+0000 7ffb5fadd640 1 --2- 172.21.15.150:0/1760193870 >> v2:172.21.15.157:3300/0 conn(0x7ffb5806dd30 0x7ffb580ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.762+0000 7ffb5fadd640 1 --2- 172.21.15.150:0/1760193870 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb58155320 0x7ffb5815bfd0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.762+0000 7ffb5fadd640 1 -- 172.21.15.150:0/1760193870 >> 172.21.15.150:0/1760193870 conn(0x7ffb5806b190 msgr2=0x7ffb5806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:14.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.763+0000 7ffb5fadd640 1 -- 172.21.15.150:0/1760193870 shutdown_connections 2024-03-17T23:49:14.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.763+0000 7ffb5fadd640 1 -- 172.21.15.150:0/1760193870 wait complete. 2024-03-17T23:49:14.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.764+0000 7ffb5fadd640 1 Processor -- start 2024-03-17T23:49:14.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.764+0000 7ffb5fadd640 1 -- start start 2024-03-17T23:49:14.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.764+0000 7ffb5fadd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb5806dd30 0x7ffb58146340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:14.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.764+0000 7ffb5fadd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb580cd250 0x7ffb580c5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:14.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.764+0000 7ffb5fadd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb58155320 0x7ffb580c63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:14.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.764+0000 7ffb5fadd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ffb5806e190 con 0x7ffb58155320 2024-03-17T23:49:14.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.764+0000 7ffb5fadd640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ffb580cf060 con 0x7ffb5806dd30 2024-03-17T23:49:14.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.764+0000 7ffb5fadd640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ffb58155700 con 0x7ffb580cd250 2024-03-17T23:49:14.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.765+0000 7ffb5e07a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb58155320 0x7ffb580c63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:14.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.765+0000 7ffb5e07a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb58155320 0x7ffb580c63c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43924/0 (socket says 172.21.15.150:43924) 2024-03-17T23:49:14.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.765+0000 7ffb5e07a640 1 -- 172.21.15.150:0/3370036030 learned_addr learned my addr 172.21.15.150:0/3370036030 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:14.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.765+0000 7ffb5d879640 1 --2- 172.21.15.150:0/3370036030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb5806dd30 0x7ffb58146340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:14.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.765+0000 7ffb5d078640 1 --2- 172.21.15.150:0/3370036030 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb580cd250 0x7ffb580c5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:14.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.765+0000 7ffb5e07a640 1 -- 172.21.15.150:0/3370036030 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb580cd250 msgr2=0x7ffb580c5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:14.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.765+0000 7ffb5e07a640 1 --2- 172.21.15.150:0/3370036030 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb580cd250 0x7ffb580c5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.765+0000 7ffb5e07a640 1 -- 172.21.15.150:0/3370036030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb5806dd30 msgr2=0x7ffb58146340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:14.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.765+0000 7ffb5e07a640 1 --2- 172.21.15.150:0/3370036030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb5806dd30 0x7ffb58146340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.765+0000 7ffb5e07a640 1 -- 172.21.15.150:0/3370036030 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb580c6da0 con 0x7ffb58155320 2024-03-17T23:49:14.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.765+0000 7ffb5e07a640 1 --2- 172.21.15.150:0/3370036030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb58155320 0x7ffb580c63c0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ffb5000c7a0 tx=0x7ffb5000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:14.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.765+0000 7ffb4a7fc640 1 -- 172.21.15.150:0/3370036030 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb50003b70 con 0x7ffb58155320 2024-03-17T23:49:14.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.765+0000 7ffb4a7fc640 1 -- 172.21.15.150:0/3370036030 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb50003d30 con 0x7ffb58155320 2024-03-17T23:49:14.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.765+0000 7ffb4a7fc640 1 -- 172.21.15.150:0/3370036030 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb5000aa90 con 0x7ffb58155320 2024-03-17T23:49:14.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.765+0000 7ffb5fadd640 1 -- 172.21.15.150:0/3370036030 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb580c4b40 con 0x7ffb58155320 2024-03-17T23:49:14.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.765+0000 7ffb5fadd640 1 -- 172.21.15.150:0/3370036030 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb580c5b60 con 0x7ffb58155320 2024-03-17T23:49:14.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.766+0000 7ffb5fadd640 1 -- 172.21.15.150:0/3370036030 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb580681a0 con 0x7ffb58155320 2024-03-17T23:49:14.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.767+0000 7ffb4a7fc640 1 -- 172.21.15.150:0/3370036030 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ffb5000ac30 con 0x7ffb58155320 2024-03-17T23:49:14.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.767+0000 7ffb4a7fc640 1 --2- 172.21.15.150:0/3370036030 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ffb3c06bc00 0x7ffb3c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:14.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.767+0000 7ffb4a7fc640 1 -- 172.21.15.150:0/3370036030 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7ffb5008a970 con 0x7ffb58155320 2024-03-17T23:49:14.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.767+0000 7ffb5d879640 1 --2- 172.21.15.150:0/3370036030 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ffb3c06bc00 0x7ffb3c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:14.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.767+0000 7ffb5d879640 1 --2- 172.21.15.150:0/3370036030 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ffb3c06bc00 0x7ffb3c06e0c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c036000 tx=0x7ffb4c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:14.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.771+0000 7ffb4a7fc640 1 -- 172.21.15.150:0/3370036030 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ffb50058d10 con 0x7ffb58155320 2024-03-17T23:49:14.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.941+0000 7ffb5fadd640 1 -- 172.21.15.150:0/3370036030 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ffb58154690 con 0x7ffb3c06bc00 2024-03-17T23:49:14.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.946+0000 7ffb4a7fc640 1 -- 172.21.15.150:0/3370036030 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+245797 (secure 0 0 0) 0x7ffb58154690 con 0x7ffb3c06bc00 2024-03-17T23:49:14.950 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:14.951 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-03-17T23:49:14.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.949+0000 7ffb5fadd640 1 -- 172.21.15.150:0/3370036030 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ffb3c06bc00 msgr2=0x7ffb3c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:14.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.949+0000 7ffb5fadd640 1 --2- 172.21.15.150:0/3370036030 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ffb3c06bc00 0x7ffb3c06e0c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c036000 tx=0x7ffb4c03b040 comp rx=0 tx=0).stop 2024-03-17T23:49:14.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.949+0000 7ffb5fadd640 1 -- 172.21.15.150:0/3370036030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb58155320 msgr2=0x7ffb580c63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:14.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.949+0000 7ffb5fadd640 1 --2- 172.21.15.150:0/3370036030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb58155320 0x7ffb580c63c0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ffb5000c7a0 tx=0x7ffb5000cc60 comp rx=0 tx=0).stop 2024-03-17T23:49:14.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.949+0000 7ffb5fadd640 1 -- 172.21.15.150:0/3370036030 shutdown_connections 2024-03-17T23:49:14.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.949+0000 7ffb5fadd640 1 --2- 172.21.15.150:0/3370036030 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb58155320 0x7ffb580c63c0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.950+0000 7ffb5fadd640 1 --2- 172.21.15.150:0/3370036030 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ffb580cd250 0x7ffb580c5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.950+0000 7ffb5fadd640 1 --2- 172.21.15.150:0/3370036030 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ffb3c06bc00 0x7ffb3c06e0c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.950+0000 7ffb5fadd640 1 --2- 172.21.15.150:0/3370036030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb5806dd30 0x7ffb58146340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:14.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.950+0000 7ffb5fadd640 1 -- 172.21.15.150:0/3370036030 >> 172.21.15.150:0/3370036030 conn(0x7ffb5806b190 msgr2=0x7ffb580b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:14.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.950+0000 7ffb5fadd640 1 -- 172.21.15.150:0/3370036030 shutdown_connections 2024-03-17T23:49:14.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:14.950+0000 7ffb5fadd640 1 -- 172.21.15.150:0/3370036030 wait complete. 2024-03-17T23:49:14.962 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-03-17T23:49:13.624322+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592140,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":188,"num_write_kb":2421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":112,"acting":112,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":6,"kb":754974720,"kb_used":239188,"kb_used_data":6372,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735532,"statfs":{"total":773094113280,"available":772849184768,"internally_reserved":0,"allocated":6524928,"data_stored":3587795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1772,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000782"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.670271+0000","last_change":"2024-03-17T23:49:11.667507+0000","last_active":"2024-03-17T23:49:11.670271+0000","last_peered":"2024-03-17T23:49:11.670271+0000","last_clean":"2024-03-17T23:49:11.670271+0000","last_became_active":"2024-03-17T23:49:11.666238+0000","last_became_peered":"2024-03-17T23:49:11.666238+0000","last_unstale":"2024-03-17T23:49:11.670271+0000","last_undegraded":"2024-03-17T23:49:11.670271+0000","last_fullsized":"2024-03-17T23:49:11.670271+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:15.111002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.663256+0000","last_change":"2024-03-17T23:49:11.660542+0000","last_active":"2024-03-17T23:49:11.663256+0000","last_peered":"2024-03-17T23:49:11.663256+0000","last_clean":"2024-03-17T23:49:11.663256+0000","last_became_active":"2024-03-17T23:49:11.660272+0000","last_became_peered":"2024-03-17T23:49:11.660272+0000","last_unstale":"2024-03-17T23:49:11.663256+0000","last_undegraded":"2024-03-17T23:49:11.663256+0000","last_fullsized":"2024-03-17T23:49:11.663256+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:52:42.007552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.658166+0000","last_change":"2024-03-17T23:49:11.655317+0000","last_active":"2024-03-17T23:49:11.658166+0000","last_peered":"2024-03-17T23:49:11.658166+0000","last_clean":"2024-03-17T23:49:11.658166+0000","last_became_active":"2024-03-17T23:49:11.655056+0000","last_became_peered":"2024-03-17T23:49:11.655056+0000","last_unstale":"2024-03-17T23:49:11.658166+0000","last_undegraded":"2024-03-17T23:49:11.658166+0000","last_fullsized":"2024-03-17T23:49:11.658166+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:00:05.868962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659258+0000","last_change":"2024-03-17T23:49:11.657708+0000","last_active":"2024-03-17T23:49:11.659258+0000","last_peered":"2024-03-17T23:49:11.659258+0000","last_clean":"2024-03-17T23:49:11.659258+0000","last_became_active":"2024-03-17T23:49:11.656418+0000","last_became_peered":"2024-03-17T23:49:11.656418+0000","last_unstale":"2024-03-17T23:49:11.659258+0000","last_undegraded":"2024-03-17T23:49:11.659258+0000","last_fullsized":"2024-03-17T23:49:11.659258+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T23:59:25.825110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.652513+0000","last_change":"2024-03-17T23:49:11.645251+0000","last_active":"2024-03-17T23:49:11.652513+0000","last_peered":"2024-03-17T23:49:11.652513+0000","last_clean":"2024-03-17T23:49:11.652513+0000","last_became_active":"2024-03-17T23:49:11.644999+0000","last_became_peered":"2024-03-17T23:49:11.644999+0000","last_unstale":"2024-03-17T23:49:11.652513+0000","last_undegraded":"2024-03-17T23:49:11.652513+0000","last_fullsized":"2024-03-17T23:49:11.652513+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:10:05.668007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.652609+0000","last_change":"2024-03-17T23:49:11.648149+0000","last_active":"2024-03-17T23:49:11.652609+0000","last_peered":"2024-03-17T23:49:11.652609+0000","last_clean":"2024-03-17T23:49:11.652609+0000","last_became_active":"2024-03-17T23:49:11.647887+0000","last_became_peered":"2024-03-17T23:49:11.647887+0000","last_unstale":"2024-03-17T23:49:11.652609+0000","last_undegraded":"2024-03-17T23:49:11.652609+0000","last_fullsized":"2024-03-17T23:49:11.652609+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:16:38.296758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.660678+0000","last_change":"2024-03-17T23:49:11.655458+0000","last_active":"2024-03-17T23:49:11.660678+0000","last_peered":"2024-03-17T23:49:11.660678+0000","last_clean":"2024-03-17T23:49:11.660678+0000","last_became_active":"2024-03-17T23:49:11.655181+0000","last_became_peered":"2024-03-17T23:49:11.655181+0000","last_unstale":"2024-03-17T23:49:11.660678+0000","last_undegraded":"2024-03-17T23:49:11.660678+0000","last_fullsized":"2024-03-17T23:49:11.660678+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:55:32.443067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.664194+0000","last_change":"2024-03-17T23:49:11.656693+0000","last_active":"2024-03-17T23:49:11.664194+0000","last_peered":"2024-03-17T23:49:11.664194+0000","last_clean":"2024-03-17T23:49:11.664194+0000","last_became_active":"2024-03-17T23:49:11.656400+0000","last_became_peered":"2024-03-17T23:49:11.656400+0000","last_unstale":"2024-03-17T23:49:11.664194+0000","last_undegraded":"2024-03-17T23:49:11.664194+0000","last_fullsized":"2024-03-17T23:49:11.664194+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:54:01.212249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.655981+0000","last_change":"2024-03-17T23:49:11.652508+0000","last_active":"2024-03-17T23:49:11.655981+0000","last_peered":"2024-03-17T23:49:11.655981+0000","last_clean":"2024-03-17T23:49:11.655981+0000","last_became_active":"2024-03-17T23:49:11.652153+0000","last_became_peered":"2024-03-17T23:49:11.652153+0000","last_unstale":"2024-03-17T23:49:11.655981+0000","last_undegraded":"2024-03-17T23:49:11.655981+0000","last_fullsized":"2024-03-17T23:49:11.655981+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:35.755434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.650975+0000","last_change":"2024-03-17T23:49:11.647575+0000","last_active":"2024-03-17T23:49:11.650975+0000","last_peered":"2024-03-17T23:49:11.650975+0000","last_clean":"2024-03-17T23:49:11.650975+0000","last_became_active":"2024-03-17T23:49:11.647313+0000","last_became_peered":"2024-03-17T23:49:11.647313+0000","last_unstale":"2024-03-17T23:49:11.650975+0000","last_undegraded":"2024-03-17T23:49:11.650975+0000","last_fullsized":"2024-03-17T23:49:11.650975+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:37.609834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.670209+0000","last_change":"2024-03-17T23:49:11.668183+0000","last_active":"2024-03-17T23:49:11.670209+0000","last_peered":"2024-03-17T23:49:11.670209+0000","last_clean":"2024-03-17T23:49:11.670209+0000","last_became_active":"2024-03-17T23:49:11.667577+0000","last_became_peered":"2024-03-17T23:49:11.667577+0000","last_unstale":"2024-03-17T23:49:11.670209+0000","last_undegraded":"2024-03-17T23:49:11.670209+0000","last_fullsized":"2024-03-17T23:49:11.670209+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T23:59:25.825110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.657425+0000","last_change":"2024-03-17T23:49:11.656210+0000","last_active":"2024-03-17T23:49:11.657425+0000","last_peered":"2024-03-17T23:49:11.657425+0000","last_clean":"2024-03-17T23:49:11.657425+0000","last_became_active":"2024-03-17T23:49:11.655629+0000","last_became_peered":"2024-03-17T23:49:11.655629+0000","last_unstale":"2024-03-17T23:49:11.657425+0000","last_undegraded":"2024-03-17T23:49:11.657425+0000","last_fullsized":"2024-03-17T23:49:11.657425+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:09:15.871249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.660884+0000","last_change":"2024-03-17T23:49:11.658319+0000","last_active":"2024-03-17T23:49:11.660884+0000","last_peered":"2024-03-17T23:49:11.660884+0000","last_clean":"2024-03-17T23:49:11.660884+0000","last_became_active":"2024-03-17T23:49:11.656866+0000","last_became_peered":"2024-03-17T23:49:11.656866+0000","last_unstale":"2024-03-17T23:49:11.660884+0000","last_undegraded":"2024-03-17T23:49:11.660884+0000","last_fullsized":"2024-03-17T23:49:11.660884+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:26:49.763854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.664486+0000","last_change":"2024-03-17T23:49:11.658960+0000","last_active":"2024-03-17T23:49:11.664486+0000","last_peered":"2024-03-17T23:49:11.664486+0000","last_clean":"2024-03-17T23:49:11.664486+0000","last_became_active":"2024-03-17T23:49:11.658695+0000","last_became_peered":"2024-03-17T23:49:11.658695+0000","last_unstale":"2024-03-17T23:49:11.664486+0000","last_undegraded":"2024-03-17T23:49:11.664486+0000","last_fullsized":"2024-03-17T23:49:11.664486+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:07:08.139888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.655891+0000","last_change":"2024-03-17T23:49:11.654139+0000","last_active":"2024-03-17T23:49:11.655891+0000","last_peered":"2024-03-17T23:49:11.655891+0000","last_clean":"2024-03-17T23:49:11.655891+0000","last_became_active":"2024-03-17T23:49:11.653838+0000","last_became_peered":"2024-03-17T23:49:11.653838+0000","last_unstale":"2024-03-17T23:49:11.655891+0000","last_undegraded":"2024-03-17T23:49:11.655891+0000","last_fullsized":"2024-03-17T23:49:11.655891+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:00:05.868962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.670150+0000","last_change":"2024-03-17T23:49:11.667572+0000","last_active":"2024-03-17T23:49:11.670150+0000","last_peered":"2024-03-17T23:49:11.670150+0000","last_clean":"2024-03-17T23:49:11.670150+0000","last_became_active":"2024-03-17T23:49:11.666249+0000","last_became_peered":"2024-03-17T23:49:11.666249+0000","last_unstale":"2024-03-17T23:49:11.670150+0000","last_undegraded":"2024-03-17T23:49:11.670150+0000","last_fullsized":"2024-03-17T23:49:11.670150+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:00:05.868962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.661421+0000","last_change":"2024-03-17T23:49:11.656653+0000","last_active":"2024-03-17T23:49:11.661421+0000","last_peered":"2024-03-17T23:49:11.661421+0000","last_clean":"2024-03-17T23:49:11.661421+0000","last_became_active":"2024-03-17T23:49:11.656335+0000","last_became_peered":"2024-03-17T23:49:11.656335+0000","last_unstale":"2024-03-17T23:49:11.661421+0000","last_undegraded":"2024-03-17T23:49:11.661421+0000","last_fullsized":"2024-03-17T23:49:11.661421+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:21:49.823973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1c","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.669900+0000","last_change":"2024-03-17T23:49:10.654468+0000","last_active":"2024-03-17T23:49:11.669900+0000","last_peered":"2024-03-17T23:49:11.669900+0000","last_clean":"2024-03-17T23:49:11.669900+0000","last_became_active":"2024-03-17T23:49:10.652810+0000","last_became_peered":"2024-03-17T23:49:10.652810+0000","last_unstale":"2024-03-17T23:49:11.669900+0000","last_undegraded":"2024-03-17T23:49:11.669900+0000","last_fullsized":"2024-03-17T23:49:11.669900+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:36.605162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1b","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.664427+0000","last_change":"2024-03-17T23:49:10.653909+0000","last_active":"2024-03-17T23:49:11.664427+0000","last_peered":"2024-03-17T23:49:11.664427+0000","last_clean":"2024-03-17T23:49:11.664427+0000","last_became_active":"2024-03-17T23:49:10.653648+0000","last_became_peered":"2024-03-17T23:49:10.653648+0000","last_unstale":"2024-03-17T23:49:11.664427+0000","last_undegraded":"2024-03-17T23:49:11.664427+0000","last_fullsized":"2024-03-17T23:49:11.664427+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:04.395229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.664451+0000","last_change":"2024-03-17T23:49:11.655809+0000","last_active":"2024-03-17T23:49:11.664451+0000","last_peered":"2024-03-17T23:49:11.664451+0000","last_clean":"2024-03-17T23:49:11.664451+0000","last_became_active":"2024-03-17T23:49:11.655510+0000","last_became_peered":"2024-03-17T23:49:11.655510+0000","last_unstale":"2024-03-17T23:49:11.664451+0000","last_undegraded":"2024-03-17T23:49:11.664451+0000","last_fullsized":"2024-03-17T23:49:11.664451+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:59:37.841369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.648060+0000","last_change":"2024-03-17T23:49:10.656254+0000","last_active":"2024-03-17T23:49:11.648060+0000","last_peered":"2024-03-17T23:49:11.648060+0000","last_clean":"2024-03-17T23:49:11.648060+0000","last_became_active":"2024-03-17T23:49:10.655915+0000","last_became_peered":"2024-03-17T23:49:10.655915+0000","last_unstale":"2024-03-17T23:49:11.648060+0000","last_undegraded":"2024-03-17T23:49:11.648060+0000","last_fullsized":"2024-03-17T23:49:11.648060+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:26:48.759181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.643323+0000","last_change":"2024-03-17T23:49:10.649005+0000","last_active":"2024-03-17T23:49:11.643323+0000","last_peered":"2024-03-17T23:49:11.643323+0000","last_clean":"2024-03-17T23:49:11.643323+0000","last_became_active":"2024-03-17T23:49:10.648717+0000","last_became_peered":"2024-03-17T23:49:10.648717+0000","last_unstale":"2024-03-17T23:49:11.643323+0000","last_undegraded":"2024-03-17T23:49:11.643323+0000","last_fullsized":"2024-03-17T23:49:11.643323+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:26:48.759181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659768+0000","last_change":"2024-03-17T23:49:11.658818+0000","last_active":"2024-03-17T23:49:11.659768+0000","last_peered":"2024-03-17T23:49:11.659768+0000","last_clean":"2024-03-17T23:49:11.659768+0000","last_became_active":"2024-03-17T23:49:11.658546+0000","last_became_peered":"2024-03-17T23:49:11.658546+0000","last_unstale":"2024-03-17T23:49:11.659768+0000","last_undegraded":"2024-03-17T23:49:11.659768+0000","last_fullsized":"2024-03-17T23:49:11.659768+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:15.111002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.648333+0000","last_change":"2024-03-17T23:49:10.658240+0000","last_active":"2024-03-17T23:49:11.648333+0000","last_peered":"2024-03-17T23:49:11.648333+0000","last_clean":"2024-03-17T23:49:11.648333+0000","last_became_active":"2024-03-17T23:49:10.657882+0000","last_became_peered":"2024-03-17T23:49:10.657882+0000","last_unstale":"2024-03-17T23:49:11.648333+0000","last_undegraded":"2024-03-17T23:49:11.648333+0000","last_fullsized":"2024-03-17T23:49:11.648333+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:16:37.292086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.665442+0000","last_change":"2024-03-17T23:49:11.660908+0000","last_active":"2024-03-17T23:49:11.665442+0000","last_peered":"2024-03-17T23:49:11.665442+0000","last_clean":"2024-03-17T23:49:11.665442+0000","last_became_active":"2024-03-17T23:49:11.660644+0000","last_became_peered":"2024-03-17T23:49:11.660644+0000","last_unstale":"2024-03-17T23:49:11.665442+0000","last_undegraded":"2024-03-17T23:49:11.665442+0000","last_fullsized":"2024-03-17T23:49:11.665442+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:24:35.182836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659276+0000","last_change":"2024-03-17T23:49:11.657785+0000","last_active":"2024-03-17T23:49:11.659276+0000","last_peered":"2024-03-17T23:49:11.659276+0000","last_clean":"2024-03-17T23:49:11.659276+0000","last_became_active":"2024-03-17T23:49:11.657040+0000","last_became_peered":"2024-03-17T23:49:11.657040+0000","last_unstale":"2024-03-17T23:49:11.659276+0000","last_undegraded":"2024-03-17T23:49:11.659276+0000","last_fullsized":"2024-03-17T23:49:11.659276+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:05.399901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"57'3","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.672409+0000","last_change":"2024-03-17T23:49:10.656304+0000","last_active":"2024-03-17T23:49:11.672409+0000","last_peered":"2024-03-17T23:49:11.672409+0000","last_clean":"2024-03-17T23:49:11.672409+0000","last_became_active":"2024-03-17T23:49:10.655795+0000","last_became_peered":"2024-03-17T23:49:10.655795+0000","last_unstale":"2024-03-17T23:49:11.672409+0000","last_undegraded":"2024-03-17T23:49:11.672409+0000","last_fullsized":"2024-03-17T23:49:11.672409+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:57:35.440980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.663919+0000","last_change":"2024-03-17T23:49:11.660244+0000","last_active":"2024-03-17T23:49:11.663919+0000","last_peered":"2024-03-17T23:49:11.663919+0000","last_clean":"2024-03-17T23:49:11.663919+0000","last_became_active":"2024-03-17T23:49:11.659982+0000","last_became_peered":"2024-03-17T23:49:11.659982+0000","last_unstale":"2024-03-17T23:49:11.663919+0000","last_undegraded":"2024-03-17T23:49:11.663919+0000","last_fullsized":"2024-03-17T23:49:11.663919+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:05:20.724503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"57'6","reported_seq":18,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.675052+0000","last_change":"2024-03-17T23:49:10.650050+0000","last_active":"2024-03-17T23:49:11.675052+0000","last_peered":"2024-03-17T23:49:11.675052+0000","last_clean":"2024-03-17T23:49:11.675052+0000","last_became_active":"2024-03-17T23:49:10.649505+0000","last_became_peered":"2024-03-17T23:49:10.649505+0000","last_unstale":"2024-03-17T23:49:11.675052+0000","last_undegraded":"2024-03-17T23:49:11.675052+0000","last_fullsized":"2024-03-17T23:49:11.675052+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:24:34.178164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.662957+0000","last_change":"2024-03-17T23:49:11.659236+0000","last_active":"2024-03-17T23:49:11.662957+0000","last_peered":"2024-03-17T23:49:11.662957+0000","last_clean":"2024-03-17T23:49:11.662957+0000","last_became_active":"2024-03-17T23:49:11.658962+0000","last_became_peered":"2024-03-17T23:49:11.658962+0000","last_unstale":"2024-03-17T23:49:11.662957+0000","last_undegraded":"2024-03-17T23:49:11.662957+0000","last_fullsized":"2024-03-17T23:49:11.662957+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:48:27.835084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.670115+0000","last_change":"2024-03-17T23:49:11.667510+0000","last_active":"2024-03-17T23:49:11.670115+0000","last_peered":"2024-03-17T23:49:11.670115+0000","last_clean":"2024-03-17T23:49:11.670115+0000","last_became_active":"2024-03-17T23:49:11.665443+0000","last_became_peered":"2024-03-17T23:49:11.665443+0000","last_unstale":"2024-03-17T23:49:11.670115+0000","last_undegraded":"2024-03-17T23:49:11.670115+0000","last_fullsized":"2024-03-17T23:49:11.670115+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:35.755434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.642062+0000","last_change":"2024-03-17T23:49:10.650246+0000","last_active":"2024-03-17T23:49:11.642062+0000","last_peered":"2024-03-17T23:49:11.642062+0000","last_clean":"2024-03-17T23:49:11.642062+0000","last_became_active":"2024-03-17T23:49:10.649837+0000","last_became_peered":"2024-03-17T23:49:10.649837+0000","last_unstale":"2024-03-17T23:49:11.642062+0000","last_undegraded":"2024-03-17T23:49:11.642062+0000","last_fullsized":"2024-03-17T23:49:11.642062+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:08:24.371365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659904+0000","last_change":"2024-03-17T23:49:11.655903+0000","last_active":"2024-03-17T23:49:11.659904+0000","last_peered":"2024-03-17T23:49:11.659904+0000","last_clean":"2024-03-17T23:49:11.659904+0000","last_became_active":"2024-03-17T23:49:11.655521+0000","last_became_peered":"2024-03-17T23:49:11.655521+0000","last_unstale":"2024-03-17T23:49:11.659904+0000","last_undegraded":"2024-03-17T23:49:11.659904+0000","last_fullsized":"2024-03-17T23:49:11.659904+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:55:32.443067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.648464+0000","last_change":"2024-03-17T23:49:10.651590+0000","last_active":"2024-03-17T23:49:11.648464+0000","last_peered":"2024-03-17T23:49:11.648464+0000","last_clean":"2024-03-17T23:49:11.648464+0000","last_became_active":"2024-03-17T23:49:10.651300+0000","last_became_peered":"2024-03-17T23:49:10.651300+0000","last_unstale":"2024-03-17T23:49:11.648464+0000","last_undegraded":"2024-03-17T23:49:11.648464+0000","last_fullsized":"2024-03-17T23:49:11.648464+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:05:19.719830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659701+0000","last_change":"2024-03-17T23:49:11.657665+0000","last_active":"2024-03-17T23:49:11.659701+0000","last_peered":"2024-03-17T23:49:11.659701+0000","last_clean":"2024-03-17T23:49:11.659701+0000","last_became_active":"2024-03-17T23:49:11.657348+0000","last_became_peered":"2024-03-17T23:49:11.657348+0000","last_unstale":"2024-03-17T23:49:11.659701+0000","last_undegraded":"2024-03-17T23:49:11.659701+0000","last_fullsized":"2024-03-17T23:49:11.659701+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:05.399901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"57'8","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.672231+0000","last_change":"2024-03-17T23:49:10.654069+0000","last_active":"2024-03-17T23:49:11.672231+0000","last_peered":"2024-03-17T23:49:11.672231+0000","last_clean":"2024-03-17T23:49:11.672231+0000","last_became_active":"2024-03-17T23:49:10.653800+0000","last_became_peered":"2024-03-17T23:49:10.653800+0000","last_unstale":"2024-03-17T23:49:11.672231+0000","last_undegraded":"2024-03-17T23:49:11.672231+0000","last_fullsized":"2024-03-17T23:49:11.672231+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:10:04.663334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.670182+0000","last_change":"2024-03-17T23:49:11.667630+0000","last_active":"2024-03-17T23:49:11.670182+0000","last_peered":"2024-03-17T23:49:11.670182+0000","last_clean":"2024-03-17T23:49:11.670182+0000","last_became_active":"2024-03-17T23:49:11.665756+0000","last_became_peered":"2024-03-17T23:49:11.665756+0000","last_unstale":"2024-03-17T23:49:11.670182+0000","last_undegraded":"2024-03-17T23:49:11.670182+0000","last_fullsized":"2024-03-17T23:49:11.670182+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:55:32.443067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659682+0000","last_change":"2024-03-17T23:49:11.657629+0000","last_active":"2024-03-17T23:49:11.659682+0000","last_peered":"2024-03-17T23:49:11.659682+0000","last_clean":"2024-03-17T23:49:11.659682+0000","last_became_active":"2024-03-17T23:49:11.657356+0000","last_became_peered":"2024-03-17T23:49:11.657356+0000","last_unstale":"2024-03-17T23:49:11.659682+0000","last_undegraded":"2024-03-17T23:49:11.659682+0000","last_fullsized":"2024-03-17T23:49:11.659682+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:15.111002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.d","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.669750+0000","last_change":"2024-03-17T23:49:10.652457+0000","last_active":"2024-03-17T23:49:11.669750+0000","last_peered":"2024-03-17T23:49:11.669750+0000","last_clean":"2024-03-17T23:49:11.669750+0000","last_became_active":"2024-03-17T23:49:10.652105+0000","last_became_peered":"2024-03-17T23:49:10.652105+0000","last_unstale":"2024-03-17T23:49:11.669750+0000","last_undegraded":"2024-03-17T23:49:11.669750+0000","last_fullsized":"2024-03-17T23:49:11.669750+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:24:34.178164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.665510+0000","last_change":"2024-03-17T23:49:11.661804+0000","last_active":"2024-03-17T23:49:11.665510+0000","last_peered":"2024-03-17T23:49:11.665510+0000","last_clean":"2024-03-17T23:49:11.665510+0000","last_became_active":"2024-03-17T23:49:11.661518+0000","last_became_peered":"2024-03-17T23:49:11.661518+0000","last_unstale":"2024-03-17T23:49:11.665510+0000","last_undegraded":"2024-03-17T23:49:11.665510+0000","last_fullsized":"2024-03-17T23:49:11.665510+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:37.609834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.662996+0000","last_change":"2024-03-17T23:49:11.659323+0000","last_active":"2024-03-17T23:49:11.662996+0000","last_peered":"2024-03-17T23:49:11.662996+0000","last_clean":"2024-03-17T23:49:11.662996+0000","last_became_active":"2024-03-17T23:49:11.659047+0000","last_became_peered":"2024-03-17T23:49:11.659047+0000","last_unstale":"2024-03-17T23:49:11.662996+0000","last_undegraded":"2024-03-17T23:49:11.662996+0000","last_fullsized":"2024-03-17T23:49:11.662996+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:11:49.300093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.639827+0000","last_change":"2024-03-17T23:49:10.655445+0000","last_active":"2024-03-17T23:49:11.639827+0000","last_peered":"2024-03-17T23:49:11.639827+0000","last_clean":"2024-03-17T23:49:11.639827+0000","last_became_active":"2024-03-17T23:49:10.654632+0000","last_became_peered":"2024-03-17T23:49:10.654632+0000","last_unstale":"2024-03-17T23:49:11.639827+0000","last_undegraded":"2024-03-17T23:49:11.639827+0000","last_fullsized":"2024-03-17T23:49:11.639827+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:54:00.207576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.647483+0000","last_change":"2024-03-17T23:49:10.654321+0000","last_active":"2024-03-17T23:49:11.647483+0000","last_peered":"2024-03-17T23:49:11.647483+0000","last_clean":"2024-03-17T23:49:11.647483+0000","last_became_active":"2024-03-17T23:49:10.653691+0000","last_became_peered":"2024-03-17T23:49:10.653691+0000","last_unstale":"2024-03-17T23:49:11.647483+0000","last_undegraded":"2024-03-17T23:49:11.647483+0000","last_fullsized":"2024-03-17T23:49:11.647483+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:04.395229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.663081+0000","last_change":"2024-03-17T23:49:11.660090+0000","last_active":"2024-03-17T23:49:11.663081+0000","last_peered":"2024-03-17T23:49:11.663081+0000","last_clean":"2024-03-17T23:49:11.663081+0000","last_became_active":"2024-03-17T23:49:11.659121+0000","last_became_peered":"2024-03-17T23:49:11.659121+0000","last_unstale":"2024-03-17T23:49:11.663081+0000","last_undegraded":"2024-03-17T23:49:11.663081+0000","last_fullsized":"2024-03-17T23:49:11.663081+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:23:45.083399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.675037+0000","last_change":"2024-03-17T23:49:10.655569+0000","last_active":"2024-03-17T23:49:11.675037+0000","last_peered":"2024-03-17T23:49:11.675037+0000","last_clean":"2024-03-17T23:49:11.675037+0000","last_became_active":"2024-03-17T23:49:10.655284+0000","last_became_peered":"2024-03-17T23:49:10.655284+0000","last_unstale":"2024-03-17T23:49:11.675037+0000","last_undegraded":"2024-03-17T23:49:11.675037+0000","last_fullsized":"2024-03-17T23:49:11.675037+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:07:07.135215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659651+0000","last_change":"2024-03-17T23:49:11.655737+0000","last_active":"2024-03-17T23:49:11.659651+0000","last_peered":"2024-03-17T23:49:11.659651+0000","last_clean":"2024-03-17T23:49:11.659651+0000","last_became_active":"2024-03-17T23:49:11.655464+0000","last_became_peered":"2024-03-17T23:49:11.655464+0000","last_unstale":"2024-03-17T23:49:11.659651+0000","last_undegraded":"2024-03-17T23:49:11.659651+0000","last_fullsized":"2024-03-17T23:49:11.659651+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T23:59:25.825110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.652228+0000","last_change":"2024-03-17T23:49:11.649056+0000","last_active":"2024-03-17T23:49:11.652228+0000","last_peered":"2024-03-17T23:49:11.652228+0000","last_clean":"2024-03-17T23:49:11.652228+0000","last_became_active":"2024-03-17T23:49:11.648791+0000","last_became_peered":"2024-03-17T23:49:11.648791+0000","last_unstale":"2024-03-17T23:49:11.652228+0000","last_undegraded":"2024-03-17T23:49:11.652228+0000","last_fullsized":"2024-03-17T23:49:11.652228+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:09:15.871249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.642103+0000","last_change":"2024-03-17T23:49:10.648679+0000","last_active":"2024-03-17T23:49:11.642103+0000","last_peered":"2024-03-17T23:49:11.642103+0000","last_clean":"2024-03-17T23:49:11.642103+0000","last_became_active":"2024-03-17T23:49:10.648407+0000","last_became_peered":"2024-03-17T23:49:10.648407+0000","last_unstale":"2024-03-17T23:49:11.642103+0000","last_undegraded":"2024-03-17T23:49:11.642103+0000","last_fullsized":"2024-03-17T23:49:11.642103+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:38:18.229949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.649926+0000","last_change":"2024-03-17T23:49:10.655937+0000","last_active":"2024-03-17T23:49:11.649926+0000","last_peered":"2024-03-17T23:49:11.649926+0000","last_clean":"2024-03-17T23:49:11.649926+0000","last_became_active":"2024-03-17T23:49:10.654317+0000","last_became_peered":"2024-03-17T23:49:10.654317+0000","last_unstale":"2024-03-17T23:49:11.649926+0000","last_undegraded":"2024-03-17T23:49:11.649926+0000","last_fullsized":"2024-03-17T23:49:11.649926+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:34.750762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.639931+0000","last_change":"2024-03-17T23:49:10.655613+0000","last_active":"2024-03-17T23:49:11.639931+0000","last_peered":"2024-03-17T23:49:11.639931+0000","last_clean":"2024-03-17T23:49:11.639931+0000","last_became_active":"2024-03-17T23:49:10.655134+0000","last_became_peered":"2024-03-17T23:49:10.655134+0000","last_unstale":"2024-03-17T23:49:11.639931+0000","last_undegraded":"2024-03-17T23:49:11.639931+0000","last_fullsized":"2024-03-17T23:49:11.639931+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:48:38.320666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.660923+0000","last_change":"2024-03-17T23:49:11.656942+0000","last_active":"2024-03-17T23:49:11.660923+0000","last_peered":"2024-03-17T23:49:11.660923+0000","last_clean":"2024-03-17T23:49:11.660923+0000","last_became_active":"2024-03-17T23:49:11.656612+0000","last_became_peered":"2024-03-17T23:49:11.656612+0000","last_unstale":"2024-03-17T23:49:11.660923+0000","last_undegraded":"2024-03-17T23:49:11.660923+0000","last_fullsized":"2024-03-17T23:49:11.660923+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:21:49.823973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.643561+0000","last_change":"2024-03-17T23:49:10.647432+0000","last_active":"2024-03-17T23:49:11.643561+0000","last_peered":"2024-03-17T23:49:11.643561+0000","last_clean":"2024-03-17T23:49:11.643561+0000","last_became_active":"2024-03-17T23:49:10.647160+0000","last_became_peered":"2024-03-17T23:49:10.647160+0000","last_unstale":"2024-03-17T23:49:11.643561+0000","last_undegraded":"2024-03-17T23:49:11.643561+0000","last_fullsized":"2024-03-17T23:49:11.643561+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:05:19.719830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.651916+0000","last_change":"2024-03-17T23:49:11.645350+0000","last_active":"2024-03-17T23:49:11.651916+0000","last_peered":"2024-03-17T23:49:11.651916+0000","last_clean":"2024-03-17T23:49:11.651916+0000","last_became_active":"2024-03-17T23:49:11.645101+0000","last_became_peered":"2024-03-17T23:49:11.645101+0000","last_unstale":"2024-03-17T23:49:11.651916+0000","last_undegraded":"2024-03-17T23:49:11.651916+0000","last_fullsized":"2024-03-17T23:49:11.651916+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:57:36.445652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.639772+0000","last_change":"2024-03-17T23:49:10.655236+0000","last_active":"2024-03-17T23:49:11.639772+0000","last_peered":"2024-03-17T23:49:11.639772+0000","last_clean":"2024-03-17T23:49:11.639772+0000","last_became_active":"2024-03-17T23:49:10.654952+0000","last_became_peered":"2024-03-17T23:49:10.654952+0000","last_unstale":"2024-03-17T23:49:11.639772+0000","last_undegraded":"2024-03-17T23:49:11.639772+0000","last_fullsized":"2024-03-17T23:49:11.639772+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:59:36.836697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.641904+0000","last_change":"2024-03-17T23:49:10.653100+0000","last_active":"2024-03-17T23:49:11.641904+0000","last_peered":"2024-03-17T23:49:11.641904+0000","last_clean":"2024-03-17T23:49:11.641904+0000","last_became_active":"2024-03-17T23:49:10.652809+0000","last_became_peered":"2024-03-17T23:49:10.652809+0000","last_unstale":"2024-03-17T23:49:11.641904+0000","last_undegraded":"2024-03-17T23:49:11.641904+0000","last_fullsized":"2024-03-17T23:49:11.641904+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:51:40.737241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":238,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.645564+0000","last_change":"2024-03-17T23:46:44.768668+0000","last_active":"2024-03-17T23:49:11.645564+0000","last_peered":"2024-03-17T23:49:11.645564+0000","last_clean":"2024-03-17T23:49:11.645564+0000","last_became_active":"2024-03-17T23:46:44.767826+0000","last_became_peered":"2024-03-17T23:46:44.767826+0000","last_unstale":"2024-03-17T23:49:11.645564+0000","last_undegraded":"2024-03-17T23:49:11.645564+0000","last_fullsized":"2024-03-17T23:49:11.645564+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_clean_scrub_stamp":"2024-03-17T23:46:43.744878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:54:22.286155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":70,"acting":70,"num_store_stats":5},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":1772,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23,"ondisk_log_size":23,"up":40,"acting":40,"num_store_stats":5},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":14,"seq":60129542177,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27644,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":630784,"data_stored":283907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542177,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27640,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":630784,"data_stored":283907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28232,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343608,"statfs":{"total":96636764160,"available":96607854592,"internally_reserved":0,"allocated":1236992,"data_stored":879794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574881,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32432,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339408,"statfs":{"total":96636764160,"available":96603553792,"internally_reserved":0,"allocated":1282048,"data_stored":899989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31844,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339996,"statfs":{"total":96636764160,"available":96604155904,"internally_reserved":0,"allocated":679936,"data_stored":309461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574881,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":688128,"data_stored":310113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574881,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":692224,"data_stored":311073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":684032,"data_stored":309551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-17T23:49:14.967 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-17T23:49:15.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.102+0000 7f0dc4dc4640 1 Processor -- start 2024-03-17T23:49:15.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.102+0000 7f0dc4dc4640 1 -- start start 2024-03-17T23:49:15.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.103+0000 7f0dc4dc4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0dc01514f0 0x7f0dc015bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:15.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.103+0000 7f0dc4dc4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f0dc006ecd0 0x7f0dc006f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:15.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.103+0000 7f0dc4dc4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dc00ce790 con 0x7f0dc015ab90 2024-03-17T23:49:15.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.103+0000 7f0dc4dc4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f0dc006fba0 con 0x7f0dc006ecd0 2024-03-17T23:49:15.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.103+0000 7f0dc4dc4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0dc015b050 con 0x7f0dc01514f0 2024-03-17T23:49:15.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.103+0000 7f0dbdd9b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f0dc015ab90 0x7f0dc006e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55682/0 (socket says 172.21.15.150:55682) 2024-03-17T23:49:15.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.103+0000 7f0dbdd9b640 1 -- 172.21.15.150:0/328199575 learned_addr learned my addr 172.21.15.150:0/328199575 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:15.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.103+0000 7f0dbed9d640 1 --2- 172.21.15.150:0/328199575 >> v2:172.21.15.157:3300/0 conn(0x7f0dc006ecd0 0x7f0dc006f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:15.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.103+0000 7f0dbe59c640 1 --2- 172.21.15.150:0/328199575 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0dc01514f0 0x7f0dc015bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:15.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.103+0000 7f0dbd59a640 1 -- 172.21.15.150:0/328199575 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3268978022 0 0) 0x7f0db4001110 con 0x7f0dc015ab90 2024-03-17T23:49:15.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.103+0000 7f0dbd59a640 1 -- 172.21.15.150:0/328199575 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0da8003520 con 0x7f0dc015ab90 2024-03-17T23:49:15.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.103+0000 7f0dbe59c640 1 -- 172.21.15.150:0/328199575 >> v2:172.21.15.157:3300/0 conn(0x7f0dc006ecd0 msgr2=0x7f0dc006f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:15.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.103+0000 7f0dbe59c640 1 --2- 172.21.15.150:0/328199575 >> v2:172.21.15.157:3300/0 conn(0x7f0dc006ecd0 0x7f0dc006f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.103+0000 7f0dbe59c640 1 -- 172.21.15.150:0/328199575 >> v1:172.21.15.150:6789/0 conn(0x7f0dc015ab90 legacy=0x7f0dc006e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:15.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.103+0000 7f0dbe59c640 1 -- 172.21.15.150:0/328199575 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc015b330 con 0x7f0dc01514f0 2024-03-17T23:49:15.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.104+0000 7f0dbe59c640 1 --2- 172.21.15.150:0/328199575 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0dc01514f0 0x7f0dc015bd10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0db00097a0 tx=0x7f0db002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46e204a825b89fde server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:15.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.104+0000 7f0dbd59a640 1 -- 172.21.15.150:0/328199575 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db0035020 con 0x7f0dc01514f0 2024-03-17T23:49:15.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.104+0000 7f0dbd59a640 1 -- 172.21.15.150:0/328199575 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0db0031810 con 0x7f0dc01514f0 2024-03-17T23:49:15.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.104+0000 7f0dbd59a640 1 -- 172.21.15.150:0/328199575 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db0031bc0 con 0x7f0dc01514f0 2024-03-17T23:49:15.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.105+0000 7f0dc4dc4640 1 -- 172.21.15.150:0/328199575 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0dc01514f0 msgr2=0x7f0dc015bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:15.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.105+0000 7f0dc4dc4640 1 --2- 172.21.15.150:0/328199575 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0dc01514f0 0x7f0dc015bd10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0db00097a0 tx=0x7f0db002b8b0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.105+0000 7f0dc4dc4640 1 -- 172.21.15.150:0/328199575 shutdown_connections 2024-03-17T23:49:15.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.105+0000 7f0dc4dc4640 1 --2- 172.21.15.150:0/328199575 >> v2:172.21.15.157:3300/0 conn(0x7f0dc006ecd0 0x7f0dc006f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.105+0000 7f0dc4dc4640 1 --2- 172.21.15.150:0/328199575 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0dc01514f0 0x7f0dc015bd10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.105+0000 7f0dc4dc4640 1 -- 172.21.15.150:0/328199575 >> 172.21.15.150:0/328199575 conn(0x7f0dc006b190 msgr2=0x7f0dc006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:15.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.105+0000 7f0dc4dc4640 1 -- 172.21.15.150:0/328199575 shutdown_connections 2024-03-17T23:49:15.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.105+0000 7f0dc4dc4640 1 -- 172.21.15.150:0/328199575 wait complete. 2024-03-17T23:49:15.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.106+0000 7f0dc4dc4640 1 Processor -- start 2024-03-17T23:49:15.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.106+0000 7f0dc4dc4640 1 -- start start 2024-03-17T23:49:15.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.107+0000 7f0dc4dc4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0dc006ecd0 0x7f0dc00aed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:15.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.107+0000 7f0dc4dc4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0dc01514f0 0x7f0dc0144870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:15.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.107+0000 7f0dc4dc4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0dc015ab90 0x7f0dc014a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:15.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.107+0000 7f0dc4dc4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc00af510 con 0x7f0dc01514f0 2024-03-17T23:49:15.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.107+0000 7f0dc4dc4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc015b8f0 con 0x7f0dc006ecd0 2024-03-17T23:49:15.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.107+0000 7f0dc4dc4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0dc01518d0 con 0x7f0dc015ab90 2024-03-17T23:49:15.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.107+0000 7f0dbdd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0dc01514f0 0x7f0dc0144870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:15.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.107+0000 7f0dbdd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0dc01514f0 0x7f0dc0144870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43928/0 (socket says 172.21.15.150:43928) 2024-03-17T23:49:15.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.107+0000 7f0dbdd9b640 1 -- 172.21.15.150:0/168611919 learned_addr learned my addr 172.21.15.150:0/168611919 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:15.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.107+0000 7f0dbe59c640 1 --2- 172.21.15.150:0/168611919 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0dc006ecd0 0x7f0dc00aed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:15.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.107+0000 7f0dbed9d640 1 --2- 172.21.15.150:0/168611919 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0dc015ab90 0x7f0dc014a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:15.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.107+0000 7f0dbdd9b640 1 -- 172.21.15.150:0/168611919 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0dc015ab90 msgr2=0x7f0dc014a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:15.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.107+0000 7f0dbdd9b640 1 --2- 172.21.15.150:0/168611919 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0dc015ab90 0x7f0dc014a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.107+0000 7f0dbdd9b640 1 -- 172.21.15.150:0/168611919 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0dc006ecd0 msgr2=0x7f0dc00aed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:15.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.107+0000 7f0dbdd9b640 1 --2- 172.21.15.150:0/168611919 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0dc006ecd0 0x7f0dc00aed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.107+0000 7f0dbdd9b640 1 -- 172.21.15.150:0/168611919 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc014ee20 con 0x7f0dc01514f0 2024-03-17T23:49:15.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.107+0000 7f0dbdd9b640 1 --2- 172.21.15.150:0/168611919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0dc01514f0 0x7f0dc0144870 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f0db400c3a0 tx=0x7f0db400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:15.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.107+0000 7f0daeffd640 1 -- 172.21.15.150:0/168611919 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db4017070 con 0x7f0dc01514f0 2024-03-17T23:49:15.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.108+0000 7f0dc4dc4640 1 -- 172.21.15.150:0/168611919 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dc014f0c0 con 0x7f0dc01514f0 2024-03-17T23:49:15.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.108+0000 7f0daeffd640 1 -- 172.21.15.150:0/168611919 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0db4005ce0 con 0x7f0dc01514f0 2024-03-17T23:49:15.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.108+0000 7f0daeffd640 1 -- 172.21.15.150:0/168611919 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db4007310 con 0x7f0dc01514f0 2024-03-17T23:49:15.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.108+0000 7f0dc4dc4640 1 -- 172.21.15.150:0/168611919 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dc014f680 con 0x7f0dc01514f0 2024-03-17T23:49:15.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.108+0000 7f0dc4dc4640 1 -- 172.21.15.150:0/168611919 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dc00681a0 con 0x7f0dc01514f0 2024-03-17T23:49:15.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.109+0000 7f0daeffd640 1 -- 172.21.15.150:0/168611919 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0db401a070 con 0x7f0dc01514f0 2024-03-17T23:49:15.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.109+0000 7f0daeffd640 1 --2- 172.21.15.150:0/168611919 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0da006bbd0 0x7f0da006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:15.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.109+0000 7f0daeffd640 1 -- 172.21.15.150:0/168611919 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f0db4089fd0 con 0x7f0dc01514f0 2024-03-17T23:49:15.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.109+0000 7f0dbe59c640 1 --2- 172.21.15.150:0/168611919 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0da006bbd0 0x7f0da006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:15.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.110+0000 7f0dbe59c640 1 --2- 172.21.15.150:0/168611919 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0da006bbd0 0x7f0da006e090 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0db0036000 tx=0x7f0db0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:15.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.114+0000 7f0daeffd640 1 -- 172.21.15.150:0/168611919 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0dc00ce790 con 0x7f0dc01514f0 2024-03-17T23:49:15.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.283+0000 7f0dc4dc4640 1 -- 172.21.15.150:0/168611919 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0dc006bf90 con 0x7f0da006bbd0 2024-03-17T23:49:15.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.289+0000 7f0daeffd640 1 -- 172.21.15.150:0/168611919 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+245797 (secure 0 0 0) 0x7f0dc006bf90 con 0x7f0da006bbd0 2024-03-17T23:49:15.292 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:15.293 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-03-17T23:49:15.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.292+0000 7f0dc4dc4640 1 -- 172.21.15.150:0/168611919 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0da006bbd0 msgr2=0x7f0da006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:15.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.292+0000 7f0dc4dc4640 1 --2- 172.21.15.150:0/168611919 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0da006bbd0 0x7f0da006e090 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0db0036000 tx=0x7f0db0008030 comp rx=0 tx=0).stop 2024-03-17T23:49:15.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.292+0000 7f0dc4dc4640 1 -- 172.21.15.150:0/168611919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0dc01514f0 msgr2=0x7f0dc0144870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:15.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.292+0000 7f0dc4dc4640 1 --2- 172.21.15.150:0/168611919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0dc01514f0 0x7f0dc0144870 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f0db400c3a0 tx=0x7f0db400c860 comp rx=0 tx=0).stop 2024-03-17T23:49:15.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.292+0000 7f0dc4dc4640 1 -- 172.21.15.150:0/168611919 shutdown_connections 2024-03-17T23:49:15.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.292+0000 7f0dc4dc4640 1 --2- 172.21.15.150:0/168611919 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0dc015ab90 0x7f0dc014a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.292+0000 7f0dc4dc4640 1 --2- 172.21.15.150:0/168611919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0dc01514f0 0x7f0dc0144870 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.292+0000 7f0dc4dc4640 1 --2- 172.21.15.150:0/168611919 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0da006bbd0 0x7f0da006e090 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.292+0000 7f0dc4dc4640 1 --2- 172.21.15.150:0/168611919 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0dc006ecd0 0x7f0dc00aed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.292+0000 7f0dc4dc4640 1 -- 172.21.15.150:0/168611919 >> 172.21.15.150:0/168611919 conn(0x7f0dc006b190 msgr2=0x7f0dc015ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:15.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.292+0000 7f0dc4dc4640 1 -- 172.21.15.150:0/168611919 shutdown_connections 2024-03-17T23:49:15.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.292+0000 7f0dc4dc4640 1 -- 172.21.15.150:0/168611919 wait complete. 2024-03-17T23:49:15.304 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-03-17T23:49:13.624322+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592140,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":188,"num_write_kb":2421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":112,"acting":112,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":6,"kb":754974720,"kb_used":239188,"kb_used_data":6372,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735532,"statfs":{"total":773094113280,"available":772849184768,"internally_reserved":0,"allocated":6524928,"data_stored":3587795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1772,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000782"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.670271+0000","last_change":"2024-03-17T23:49:11.667507+0000","last_active":"2024-03-17T23:49:11.670271+0000","last_peered":"2024-03-17T23:49:11.670271+0000","last_clean":"2024-03-17T23:49:11.670271+0000","last_became_active":"2024-03-17T23:49:11.666238+0000","last_became_peered":"2024-03-17T23:49:11.666238+0000","last_unstale":"2024-03-17T23:49:11.670271+0000","last_undegraded":"2024-03-17T23:49:11.670271+0000","last_fullsized":"2024-03-17T23:49:11.670271+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:15.111002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.663256+0000","last_change":"2024-03-17T23:49:11.660542+0000","last_active":"2024-03-17T23:49:11.663256+0000","last_peered":"2024-03-17T23:49:11.663256+0000","last_clean":"2024-03-17T23:49:11.663256+0000","last_became_active":"2024-03-17T23:49:11.660272+0000","last_became_peered":"2024-03-17T23:49:11.660272+0000","last_unstale":"2024-03-17T23:49:11.663256+0000","last_undegraded":"2024-03-17T23:49:11.663256+0000","last_fullsized":"2024-03-17T23:49:11.663256+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:52:42.007552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.658166+0000","last_change":"2024-03-17T23:49:11.655317+0000","last_active":"2024-03-17T23:49:11.658166+0000","last_peered":"2024-03-17T23:49:11.658166+0000","last_clean":"2024-03-17T23:49:11.658166+0000","last_became_active":"2024-03-17T23:49:11.655056+0000","last_became_peered":"2024-03-17T23:49:11.655056+0000","last_unstale":"2024-03-17T23:49:11.658166+0000","last_undegraded":"2024-03-17T23:49:11.658166+0000","last_fullsized":"2024-03-17T23:49:11.658166+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:00:05.868962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659258+0000","last_change":"2024-03-17T23:49:11.657708+0000","last_active":"2024-03-17T23:49:11.659258+0000","last_peered":"2024-03-17T23:49:11.659258+0000","last_clean":"2024-03-17T23:49:11.659258+0000","last_became_active":"2024-03-17T23:49:11.656418+0000","last_became_peered":"2024-03-17T23:49:11.656418+0000","last_unstale":"2024-03-17T23:49:11.659258+0000","last_undegraded":"2024-03-17T23:49:11.659258+0000","last_fullsized":"2024-03-17T23:49:11.659258+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T23:59:25.825110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.652513+0000","last_change":"2024-03-17T23:49:11.645251+0000","last_active":"2024-03-17T23:49:11.652513+0000","last_peered":"2024-03-17T23:49:11.652513+0000","last_clean":"2024-03-17T23:49:11.652513+0000","last_became_active":"2024-03-17T23:49:11.644999+0000","last_became_peered":"2024-03-17T23:49:11.644999+0000","last_unstale":"2024-03-17T23:49:11.652513+0000","last_undegraded":"2024-03-17T23:49:11.652513+0000","last_fullsized":"2024-03-17T23:49:11.652513+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:10:05.668007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.652609+0000","last_change":"2024-03-17T23:49:11.648149+0000","last_active":"2024-03-17T23:49:11.652609+0000","last_peered":"2024-03-17T23:49:11.652609+0000","last_clean":"2024-03-17T23:49:11.652609+0000","last_became_active":"2024-03-17T23:49:11.647887+0000","last_became_peered":"2024-03-17T23:49:11.647887+0000","last_unstale":"2024-03-17T23:49:11.652609+0000","last_undegraded":"2024-03-17T23:49:11.652609+0000","last_fullsized":"2024-03-17T23:49:11.652609+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:16:38.296758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.660678+0000","last_change":"2024-03-17T23:49:11.655458+0000","last_active":"2024-03-17T23:49:11.660678+0000","last_peered":"2024-03-17T23:49:11.660678+0000","last_clean":"2024-03-17T23:49:11.660678+0000","last_became_active":"2024-03-17T23:49:11.655181+0000","last_became_peered":"2024-03-17T23:49:11.655181+0000","last_unstale":"2024-03-17T23:49:11.660678+0000","last_undegraded":"2024-03-17T23:49:11.660678+0000","last_fullsized":"2024-03-17T23:49:11.660678+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:55:32.443067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.664194+0000","last_change":"2024-03-17T23:49:11.656693+0000","last_active":"2024-03-17T23:49:11.664194+0000","last_peered":"2024-03-17T23:49:11.664194+0000","last_clean":"2024-03-17T23:49:11.664194+0000","last_became_active":"2024-03-17T23:49:11.656400+0000","last_became_peered":"2024-03-17T23:49:11.656400+0000","last_unstale":"2024-03-17T23:49:11.664194+0000","last_undegraded":"2024-03-17T23:49:11.664194+0000","last_fullsized":"2024-03-17T23:49:11.664194+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:54:01.212249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.655981+0000","last_change":"2024-03-17T23:49:11.652508+0000","last_active":"2024-03-17T23:49:11.655981+0000","last_peered":"2024-03-17T23:49:11.655981+0000","last_clean":"2024-03-17T23:49:11.655981+0000","last_became_active":"2024-03-17T23:49:11.652153+0000","last_became_peered":"2024-03-17T23:49:11.652153+0000","last_unstale":"2024-03-17T23:49:11.655981+0000","last_undegraded":"2024-03-17T23:49:11.655981+0000","last_fullsized":"2024-03-17T23:49:11.655981+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:35.755434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.650975+0000","last_change":"2024-03-17T23:49:11.647575+0000","last_active":"2024-03-17T23:49:11.650975+0000","last_peered":"2024-03-17T23:49:11.650975+0000","last_clean":"2024-03-17T23:49:11.650975+0000","last_became_active":"2024-03-17T23:49:11.647313+0000","last_became_peered":"2024-03-17T23:49:11.647313+0000","last_unstale":"2024-03-17T23:49:11.650975+0000","last_undegraded":"2024-03-17T23:49:11.650975+0000","last_fullsized":"2024-03-17T23:49:11.650975+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:37.609834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.670209+0000","last_change":"2024-03-17T23:49:11.668183+0000","last_active":"2024-03-17T23:49:11.670209+0000","last_peered":"2024-03-17T23:49:11.670209+0000","last_clean":"2024-03-17T23:49:11.670209+0000","last_became_active":"2024-03-17T23:49:11.667577+0000","last_became_peered":"2024-03-17T23:49:11.667577+0000","last_unstale":"2024-03-17T23:49:11.670209+0000","last_undegraded":"2024-03-17T23:49:11.670209+0000","last_fullsized":"2024-03-17T23:49:11.670209+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T23:59:25.825110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.657425+0000","last_change":"2024-03-17T23:49:11.656210+0000","last_active":"2024-03-17T23:49:11.657425+0000","last_peered":"2024-03-17T23:49:11.657425+0000","last_clean":"2024-03-17T23:49:11.657425+0000","last_became_active":"2024-03-17T23:49:11.655629+0000","last_became_peered":"2024-03-17T23:49:11.655629+0000","last_unstale":"2024-03-17T23:49:11.657425+0000","last_undegraded":"2024-03-17T23:49:11.657425+0000","last_fullsized":"2024-03-17T23:49:11.657425+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:09:15.871249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.660884+0000","last_change":"2024-03-17T23:49:11.658319+0000","last_active":"2024-03-17T23:49:11.660884+0000","last_peered":"2024-03-17T23:49:11.660884+0000","last_clean":"2024-03-17T23:49:11.660884+0000","last_became_active":"2024-03-17T23:49:11.656866+0000","last_became_peered":"2024-03-17T23:49:11.656866+0000","last_unstale":"2024-03-17T23:49:11.660884+0000","last_undegraded":"2024-03-17T23:49:11.660884+0000","last_fullsized":"2024-03-17T23:49:11.660884+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:26:49.763854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.664486+0000","last_change":"2024-03-17T23:49:11.658960+0000","last_active":"2024-03-17T23:49:11.664486+0000","last_peered":"2024-03-17T23:49:11.664486+0000","last_clean":"2024-03-17T23:49:11.664486+0000","last_became_active":"2024-03-17T23:49:11.658695+0000","last_became_peered":"2024-03-17T23:49:11.658695+0000","last_unstale":"2024-03-17T23:49:11.664486+0000","last_undegraded":"2024-03-17T23:49:11.664486+0000","last_fullsized":"2024-03-17T23:49:11.664486+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:07:08.139888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.655891+0000","last_change":"2024-03-17T23:49:11.654139+0000","last_active":"2024-03-17T23:49:11.655891+0000","last_peered":"2024-03-17T23:49:11.655891+0000","last_clean":"2024-03-17T23:49:11.655891+0000","last_became_active":"2024-03-17T23:49:11.653838+0000","last_became_peered":"2024-03-17T23:49:11.653838+0000","last_unstale":"2024-03-17T23:49:11.655891+0000","last_undegraded":"2024-03-17T23:49:11.655891+0000","last_fullsized":"2024-03-17T23:49:11.655891+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:00:05.868962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.670150+0000","last_change":"2024-03-17T23:49:11.667572+0000","last_active":"2024-03-17T23:49:11.670150+0000","last_peered":"2024-03-17T23:49:11.670150+0000","last_clean":"2024-03-17T23:49:11.670150+0000","last_became_active":"2024-03-17T23:49:11.666249+0000","last_became_peered":"2024-03-17T23:49:11.666249+0000","last_unstale":"2024-03-17T23:49:11.670150+0000","last_undegraded":"2024-03-17T23:49:11.670150+0000","last_fullsized":"2024-03-17T23:49:11.670150+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:00:05.868962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.661421+0000","last_change":"2024-03-17T23:49:11.656653+0000","last_active":"2024-03-17T23:49:11.661421+0000","last_peered":"2024-03-17T23:49:11.661421+0000","last_clean":"2024-03-17T23:49:11.661421+0000","last_became_active":"2024-03-17T23:49:11.656335+0000","last_became_peered":"2024-03-17T23:49:11.656335+0000","last_unstale":"2024-03-17T23:49:11.661421+0000","last_undegraded":"2024-03-17T23:49:11.661421+0000","last_fullsized":"2024-03-17T23:49:11.661421+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:21:49.823973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1c","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.669900+0000","last_change":"2024-03-17T23:49:10.654468+0000","last_active":"2024-03-17T23:49:11.669900+0000","last_peered":"2024-03-17T23:49:11.669900+0000","last_clean":"2024-03-17T23:49:11.669900+0000","last_became_active":"2024-03-17T23:49:10.652810+0000","last_became_peered":"2024-03-17T23:49:10.652810+0000","last_unstale":"2024-03-17T23:49:11.669900+0000","last_undegraded":"2024-03-17T23:49:11.669900+0000","last_fullsized":"2024-03-17T23:49:11.669900+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:36.605162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1b","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.664427+0000","last_change":"2024-03-17T23:49:10.653909+0000","last_active":"2024-03-17T23:49:11.664427+0000","last_peered":"2024-03-17T23:49:11.664427+0000","last_clean":"2024-03-17T23:49:11.664427+0000","last_became_active":"2024-03-17T23:49:10.653648+0000","last_became_peered":"2024-03-17T23:49:10.653648+0000","last_unstale":"2024-03-17T23:49:11.664427+0000","last_undegraded":"2024-03-17T23:49:11.664427+0000","last_fullsized":"2024-03-17T23:49:11.664427+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:04.395229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.664451+0000","last_change":"2024-03-17T23:49:11.655809+0000","last_active":"2024-03-17T23:49:11.664451+0000","last_peered":"2024-03-17T23:49:11.664451+0000","last_clean":"2024-03-17T23:49:11.664451+0000","last_became_active":"2024-03-17T23:49:11.655510+0000","last_became_peered":"2024-03-17T23:49:11.655510+0000","last_unstale":"2024-03-17T23:49:11.664451+0000","last_undegraded":"2024-03-17T23:49:11.664451+0000","last_fullsized":"2024-03-17T23:49:11.664451+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:59:37.841369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.648060+0000","last_change":"2024-03-17T23:49:10.656254+0000","last_active":"2024-03-17T23:49:11.648060+0000","last_peered":"2024-03-17T23:49:11.648060+0000","last_clean":"2024-03-17T23:49:11.648060+0000","last_became_active":"2024-03-17T23:49:10.655915+0000","last_became_peered":"2024-03-17T23:49:10.655915+0000","last_unstale":"2024-03-17T23:49:11.648060+0000","last_undegraded":"2024-03-17T23:49:11.648060+0000","last_fullsized":"2024-03-17T23:49:11.648060+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:26:48.759181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.643323+0000","last_change":"2024-03-17T23:49:10.649005+0000","last_active":"2024-03-17T23:49:11.643323+0000","last_peered":"2024-03-17T23:49:11.643323+0000","last_clean":"2024-03-17T23:49:11.643323+0000","last_became_active":"2024-03-17T23:49:10.648717+0000","last_became_peered":"2024-03-17T23:49:10.648717+0000","last_unstale":"2024-03-17T23:49:11.643323+0000","last_undegraded":"2024-03-17T23:49:11.643323+0000","last_fullsized":"2024-03-17T23:49:11.643323+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:26:48.759181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659768+0000","last_change":"2024-03-17T23:49:11.658818+0000","last_active":"2024-03-17T23:49:11.659768+0000","last_peered":"2024-03-17T23:49:11.659768+0000","last_clean":"2024-03-17T23:49:11.659768+0000","last_became_active":"2024-03-17T23:49:11.658546+0000","last_became_peered":"2024-03-17T23:49:11.658546+0000","last_unstale":"2024-03-17T23:49:11.659768+0000","last_undegraded":"2024-03-17T23:49:11.659768+0000","last_fullsized":"2024-03-17T23:49:11.659768+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:15.111002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.648333+0000","last_change":"2024-03-17T23:49:10.658240+0000","last_active":"2024-03-17T23:49:11.648333+0000","last_peered":"2024-03-17T23:49:11.648333+0000","last_clean":"2024-03-17T23:49:11.648333+0000","last_became_active":"2024-03-17T23:49:10.657882+0000","last_became_peered":"2024-03-17T23:49:10.657882+0000","last_unstale":"2024-03-17T23:49:11.648333+0000","last_undegraded":"2024-03-17T23:49:11.648333+0000","last_fullsized":"2024-03-17T23:49:11.648333+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:16:37.292086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.665442+0000","last_change":"2024-03-17T23:49:11.660908+0000","last_active":"2024-03-17T23:49:11.665442+0000","last_peered":"2024-03-17T23:49:11.665442+0000","last_clean":"2024-03-17T23:49:11.665442+0000","last_became_active":"2024-03-17T23:49:11.660644+0000","last_became_peered":"2024-03-17T23:49:11.660644+0000","last_unstale":"2024-03-17T23:49:11.665442+0000","last_undegraded":"2024-03-17T23:49:11.665442+0000","last_fullsized":"2024-03-17T23:49:11.665442+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:24:35.182836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659276+0000","last_change":"2024-03-17T23:49:11.657785+0000","last_active":"2024-03-17T23:49:11.659276+0000","last_peered":"2024-03-17T23:49:11.659276+0000","last_clean":"2024-03-17T23:49:11.659276+0000","last_became_active":"2024-03-17T23:49:11.657040+0000","last_became_peered":"2024-03-17T23:49:11.657040+0000","last_unstale":"2024-03-17T23:49:11.659276+0000","last_undegraded":"2024-03-17T23:49:11.659276+0000","last_fullsized":"2024-03-17T23:49:11.659276+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:05.399901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"57'3","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.672409+0000","last_change":"2024-03-17T23:49:10.656304+0000","last_active":"2024-03-17T23:49:11.672409+0000","last_peered":"2024-03-17T23:49:11.672409+0000","last_clean":"2024-03-17T23:49:11.672409+0000","last_became_active":"2024-03-17T23:49:10.655795+0000","last_became_peered":"2024-03-17T23:49:10.655795+0000","last_unstale":"2024-03-17T23:49:11.672409+0000","last_undegraded":"2024-03-17T23:49:11.672409+0000","last_fullsized":"2024-03-17T23:49:11.672409+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:57:35.440980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.663919+0000","last_change":"2024-03-17T23:49:11.660244+0000","last_active":"2024-03-17T23:49:11.663919+0000","last_peered":"2024-03-17T23:49:11.663919+0000","last_clean":"2024-03-17T23:49:11.663919+0000","last_became_active":"2024-03-17T23:49:11.659982+0000","last_became_peered":"2024-03-17T23:49:11.659982+0000","last_unstale":"2024-03-17T23:49:11.663919+0000","last_undegraded":"2024-03-17T23:49:11.663919+0000","last_fullsized":"2024-03-17T23:49:11.663919+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:05:20.724503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"57'6","reported_seq":18,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.675052+0000","last_change":"2024-03-17T23:49:10.650050+0000","last_active":"2024-03-17T23:49:11.675052+0000","last_peered":"2024-03-17T23:49:11.675052+0000","last_clean":"2024-03-17T23:49:11.675052+0000","last_became_active":"2024-03-17T23:49:10.649505+0000","last_became_peered":"2024-03-17T23:49:10.649505+0000","last_unstale":"2024-03-17T23:49:11.675052+0000","last_undegraded":"2024-03-17T23:49:11.675052+0000","last_fullsized":"2024-03-17T23:49:11.675052+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:24:34.178164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.662957+0000","last_change":"2024-03-17T23:49:11.659236+0000","last_active":"2024-03-17T23:49:11.662957+0000","last_peered":"2024-03-17T23:49:11.662957+0000","last_clean":"2024-03-17T23:49:11.662957+0000","last_became_active":"2024-03-17T23:49:11.658962+0000","last_became_peered":"2024-03-17T23:49:11.658962+0000","last_unstale":"2024-03-17T23:49:11.662957+0000","last_undegraded":"2024-03-17T23:49:11.662957+0000","last_fullsized":"2024-03-17T23:49:11.662957+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:48:27.835084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.670115+0000","last_change":"2024-03-17T23:49:11.667510+0000","last_active":"2024-03-17T23:49:11.670115+0000","last_peered":"2024-03-17T23:49:11.670115+0000","last_clean":"2024-03-17T23:49:11.670115+0000","last_became_active":"2024-03-17T23:49:11.665443+0000","last_became_peered":"2024-03-17T23:49:11.665443+0000","last_unstale":"2024-03-17T23:49:11.670115+0000","last_undegraded":"2024-03-17T23:49:11.670115+0000","last_fullsized":"2024-03-17T23:49:11.670115+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:35.755434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.642062+0000","last_change":"2024-03-17T23:49:10.650246+0000","last_active":"2024-03-17T23:49:11.642062+0000","last_peered":"2024-03-17T23:49:11.642062+0000","last_clean":"2024-03-17T23:49:11.642062+0000","last_became_active":"2024-03-17T23:49:10.649837+0000","last_became_peered":"2024-03-17T23:49:10.649837+0000","last_unstale":"2024-03-17T23:49:11.642062+0000","last_undegraded":"2024-03-17T23:49:11.642062+0000","last_fullsized":"2024-03-17T23:49:11.642062+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:08:24.371365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659904+0000","last_change":"2024-03-17T23:49:11.655903+0000","last_active":"2024-03-17T23:49:11.659904+0000","last_peered":"2024-03-17T23:49:11.659904+0000","last_clean":"2024-03-17T23:49:11.659904+0000","last_became_active":"2024-03-17T23:49:11.655521+0000","last_became_peered":"2024-03-17T23:49:11.655521+0000","last_unstale":"2024-03-17T23:49:11.659904+0000","last_undegraded":"2024-03-17T23:49:11.659904+0000","last_fullsized":"2024-03-17T23:49:11.659904+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:55:32.443067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.648464+0000","last_change":"2024-03-17T23:49:10.651590+0000","last_active":"2024-03-17T23:49:11.648464+0000","last_peered":"2024-03-17T23:49:11.648464+0000","last_clean":"2024-03-17T23:49:11.648464+0000","last_became_active":"2024-03-17T23:49:10.651300+0000","last_became_peered":"2024-03-17T23:49:10.651300+0000","last_unstale":"2024-03-17T23:49:11.648464+0000","last_undegraded":"2024-03-17T23:49:11.648464+0000","last_fullsized":"2024-03-17T23:49:11.648464+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:05:19.719830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659701+0000","last_change":"2024-03-17T23:49:11.657665+0000","last_active":"2024-03-17T23:49:11.659701+0000","last_peered":"2024-03-17T23:49:11.659701+0000","last_clean":"2024-03-17T23:49:11.659701+0000","last_became_active":"2024-03-17T23:49:11.657348+0000","last_became_peered":"2024-03-17T23:49:11.657348+0000","last_unstale":"2024-03-17T23:49:11.659701+0000","last_undegraded":"2024-03-17T23:49:11.659701+0000","last_fullsized":"2024-03-17T23:49:11.659701+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:05.399901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"57'8","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.672231+0000","last_change":"2024-03-17T23:49:10.654069+0000","last_active":"2024-03-17T23:49:11.672231+0000","last_peered":"2024-03-17T23:49:11.672231+0000","last_clean":"2024-03-17T23:49:11.672231+0000","last_became_active":"2024-03-17T23:49:10.653800+0000","last_became_peered":"2024-03-17T23:49:10.653800+0000","last_unstale":"2024-03-17T23:49:11.672231+0000","last_undegraded":"2024-03-17T23:49:11.672231+0000","last_fullsized":"2024-03-17T23:49:11.672231+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:10:04.663334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.670182+0000","last_change":"2024-03-17T23:49:11.667630+0000","last_active":"2024-03-17T23:49:11.670182+0000","last_peered":"2024-03-17T23:49:11.670182+0000","last_clean":"2024-03-17T23:49:11.670182+0000","last_became_active":"2024-03-17T23:49:11.665756+0000","last_became_peered":"2024-03-17T23:49:11.665756+0000","last_unstale":"2024-03-17T23:49:11.670182+0000","last_undegraded":"2024-03-17T23:49:11.670182+0000","last_fullsized":"2024-03-17T23:49:11.670182+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:55:32.443067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659682+0000","last_change":"2024-03-17T23:49:11.657629+0000","last_active":"2024-03-17T23:49:11.659682+0000","last_peered":"2024-03-17T23:49:11.659682+0000","last_clean":"2024-03-17T23:49:11.659682+0000","last_became_active":"2024-03-17T23:49:11.657356+0000","last_became_peered":"2024-03-17T23:49:11.657356+0000","last_unstale":"2024-03-17T23:49:11.659682+0000","last_undegraded":"2024-03-17T23:49:11.659682+0000","last_fullsized":"2024-03-17T23:49:11.659682+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:15.111002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.d","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.669750+0000","last_change":"2024-03-17T23:49:10.652457+0000","last_active":"2024-03-17T23:49:11.669750+0000","last_peered":"2024-03-17T23:49:11.669750+0000","last_clean":"2024-03-17T23:49:11.669750+0000","last_became_active":"2024-03-17T23:49:10.652105+0000","last_became_peered":"2024-03-17T23:49:10.652105+0000","last_unstale":"2024-03-17T23:49:11.669750+0000","last_undegraded":"2024-03-17T23:49:11.669750+0000","last_fullsized":"2024-03-17T23:49:11.669750+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:24:34.178164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.665510+0000","last_change":"2024-03-17T23:49:11.661804+0000","last_active":"2024-03-17T23:49:11.665510+0000","last_peered":"2024-03-17T23:49:11.665510+0000","last_clean":"2024-03-17T23:49:11.665510+0000","last_became_active":"2024-03-17T23:49:11.661518+0000","last_became_peered":"2024-03-17T23:49:11.661518+0000","last_unstale":"2024-03-17T23:49:11.665510+0000","last_undegraded":"2024-03-17T23:49:11.665510+0000","last_fullsized":"2024-03-17T23:49:11.665510+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:37.609834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.662996+0000","last_change":"2024-03-17T23:49:11.659323+0000","last_active":"2024-03-17T23:49:11.662996+0000","last_peered":"2024-03-17T23:49:11.662996+0000","last_clean":"2024-03-17T23:49:11.662996+0000","last_became_active":"2024-03-17T23:49:11.659047+0000","last_became_peered":"2024-03-17T23:49:11.659047+0000","last_unstale":"2024-03-17T23:49:11.662996+0000","last_undegraded":"2024-03-17T23:49:11.662996+0000","last_fullsized":"2024-03-17T23:49:11.662996+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:11:49.300093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.639827+0000","last_change":"2024-03-17T23:49:10.655445+0000","last_active":"2024-03-17T23:49:11.639827+0000","last_peered":"2024-03-17T23:49:11.639827+0000","last_clean":"2024-03-17T23:49:11.639827+0000","last_became_active":"2024-03-17T23:49:10.654632+0000","last_became_peered":"2024-03-17T23:49:10.654632+0000","last_unstale":"2024-03-17T23:49:11.639827+0000","last_undegraded":"2024-03-17T23:49:11.639827+0000","last_fullsized":"2024-03-17T23:49:11.639827+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:54:00.207576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.647483+0000","last_change":"2024-03-17T23:49:10.654321+0000","last_active":"2024-03-17T23:49:11.647483+0000","last_peered":"2024-03-17T23:49:11.647483+0000","last_clean":"2024-03-17T23:49:11.647483+0000","last_became_active":"2024-03-17T23:49:10.653691+0000","last_became_peered":"2024-03-17T23:49:10.653691+0000","last_unstale":"2024-03-17T23:49:11.647483+0000","last_undegraded":"2024-03-17T23:49:11.647483+0000","last_fullsized":"2024-03-17T23:49:11.647483+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:04.395229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.663081+0000","last_change":"2024-03-17T23:49:11.660090+0000","last_active":"2024-03-17T23:49:11.663081+0000","last_peered":"2024-03-17T23:49:11.663081+0000","last_clean":"2024-03-17T23:49:11.663081+0000","last_became_active":"2024-03-17T23:49:11.659121+0000","last_became_peered":"2024-03-17T23:49:11.659121+0000","last_unstale":"2024-03-17T23:49:11.663081+0000","last_undegraded":"2024-03-17T23:49:11.663081+0000","last_fullsized":"2024-03-17T23:49:11.663081+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:23:45.083399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.675037+0000","last_change":"2024-03-17T23:49:10.655569+0000","last_active":"2024-03-17T23:49:11.675037+0000","last_peered":"2024-03-17T23:49:11.675037+0000","last_clean":"2024-03-17T23:49:11.675037+0000","last_became_active":"2024-03-17T23:49:10.655284+0000","last_became_peered":"2024-03-17T23:49:10.655284+0000","last_unstale":"2024-03-17T23:49:11.675037+0000","last_undegraded":"2024-03-17T23:49:11.675037+0000","last_fullsized":"2024-03-17T23:49:11.675037+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:07:07.135215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659651+0000","last_change":"2024-03-17T23:49:11.655737+0000","last_active":"2024-03-17T23:49:11.659651+0000","last_peered":"2024-03-17T23:49:11.659651+0000","last_clean":"2024-03-17T23:49:11.659651+0000","last_became_active":"2024-03-17T23:49:11.655464+0000","last_became_peered":"2024-03-17T23:49:11.655464+0000","last_unstale":"2024-03-17T23:49:11.659651+0000","last_undegraded":"2024-03-17T23:49:11.659651+0000","last_fullsized":"2024-03-17T23:49:11.659651+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T23:59:25.825110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.652228+0000","last_change":"2024-03-17T23:49:11.649056+0000","last_active":"2024-03-17T23:49:11.652228+0000","last_peered":"2024-03-17T23:49:11.652228+0000","last_clean":"2024-03-17T23:49:11.652228+0000","last_became_active":"2024-03-17T23:49:11.648791+0000","last_became_peered":"2024-03-17T23:49:11.648791+0000","last_unstale":"2024-03-17T23:49:11.652228+0000","last_undegraded":"2024-03-17T23:49:11.652228+0000","last_fullsized":"2024-03-17T23:49:11.652228+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:09:15.871249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.642103+0000","last_change":"2024-03-17T23:49:10.648679+0000","last_active":"2024-03-17T23:49:11.642103+0000","last_peered":"2024-03-17T23:49:11.642103+0000","last_clean":"2024-03-17T23:49:11.642103+0000","last_became_active":"2024-03-17T23:49:10.648407+0000","last_became_peered":"2024-03-17T23:49:10.648407+0000","last_unstale":"2024-03-17T23:49:11.642103+0000","last_undegraded":"2024-03-17T23:49:11.642103+0000","last_fullsized":"2024-03-17T23:49:11.642103+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:38:18.229949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.649926+0000","last_change":"2024-03-17T23:49:10.655937+0000","last_active":"2024-03-17T23:49:11.649926+0000","last_peered":"2024-03-17T23:49:11.649926+0000","last_clean":"2024-03-17T23:49:11.649926+0000","last_became_active":"2024-03-17T23:49:10.654317+0000","last_became_peered":"2024-03-17T23:49:10.654317+0000","last_unstale":"2024-03-17T23:49:11.649926+0000","last_undegraded":"2024-03-17T23:49:11.649926+0000","last_fullsized":"2024-03-17T23:49:11.649926+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:34.750762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.639931+0000","last_change":"2024-03-17T23:49:10.655613+0000","last_active":"2024-03-17T23:49:11.639931+0000","last_peered":"2024-03-17T23:49:11.639931+0000","last_clean":"2024-03-17T23:49:11.639931+0000","last_became_active":"2024-03-17T23:49:10.655134+0000","last_became_peered":"2024-03-17T23:49:10.655134+0000","last_unstale":"2024-03-17T23:49:11.639931+0000","last_undegraded":"2024-03-17T23:49:11.639931+0000","last_fullsized":"2024-03-17T23:49:11.639931+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:48:38.320666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.660923+0000","last_change":"2024-03-17T23:49:11.656942+0000","last_active":"2024-03-17T23:49:11.660923+0000","last_peered":"2024-03-17T23:49:11.660923+0000","last_clean":"2024-03-17T23:49:11.660923+0000","last_became_active":"2024-03-17T23:49:11.656612+0000","last_became_peered":"2024-03-17T23:49:11.656612+0000","last_unstale":"2024-03-17T23:49:11.660923+0000","last_undegraded":"2024-03-17T23:49:11.660923+0000","last_fullsized":"2024-03-17T23:49:11.660923+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:21:49.823973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.643561+0000","last_change":"2024-03-17T23:49:10.647432+0000","last_active":"2024-03-17T23:49:11.643561+0000","last_peered":"2024-03-17T23:49:11.643561+0000","last_clean":"2024-03-17T23:49:11.643561+0000","last_became_active":"2024-03-17T23:49:10.647160+0000","last_became_peered":"2024-03-17T23:49:10.647160+0000","last_unstale":"2024-03-17T23:49:11.643561+0000","last_undegraded":"2024-03-17T23:49:11.643561+0000","last_fullsized":"2024-03-17T23:49:11.643561+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:05:19.719830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.651916+0000","last_change":"2024-03-17T23:49:11.645350+0000","last_active":"2024-03-17T23:49:11.651916+0000","last_peered":"2024-03-17T23:49:11.651916+0000","last_clean":"2024-03-17T23:49:11.651916+0000","last_became_active":"2024-03-17T23:49:11.645101+0000","last_became_peered":"2024-03-17T23:49:11.645101+0000","last_unstale":"2024-03-17T23:49:11.651916+0000","last_undegraded":"2024-03-17T23:49:11.651916+0000","last_fullsized":"2024-03-17T23:49:11.651916+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:57:36.445652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.639772+0000","last_change":"2024-03-17T23:49:10.655236+0000","last_active":"2024-03-17T23:49:11.639772+0000","last_peered":"2024-03-17T23:49:11.639772+0000","last_clean":"2024-03-17T23:49:11.639772+0000","last_became_active":"2024-03-17T23:49:10.654952+0000","last_became_peered":"2024-03-17T23:49:10.654952+0000","last_unstale":"2024-03-17T23:49:11.639772+0000","last_undegraded":"2024-03-17T23:49:11.639772+0000","last_fullsized":"2024-03-17T23:49:11.639772+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:59:36.836697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:09.625866+0000","last_change":"2024-03-17T23:49:09.625866+0000","last_active":"2024-03-17T23:49:09.625866+0000","last_peered":"2024-03-17T23:49:09.625866+0000","last_clean":"2024-03-17T23:49:09.625866+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:09.625866+0000","last_undegraded":"2024-03-17T23:49:09.625866+0000","last_fullsized":"2024-03-17T23:49:09.625866+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-17T23:49:10.630539+0000","last_change":"2024-03-17T23:49:10.630539+0000","last_active":"2024-03-17T23:49:10.630539+0000","last_peered":"2024-03-17T23:49:10.630539+0000","last_clean":"2024-03-17T23:49:10.630539+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-17T23:49:10.630539+0000","last_undegraded":"2024-03-17T23:49:10.630539+0000","last_fullsized":"2024-03-17T23:49:10.630539+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.641904+0000","last_change":"2024-03-17T23:49:10.653100+0000","last_active":"2024-03-17T23:49:11.641904+0000","last_peered":"2024-03-17T23:49:11.641904+0000","last_clean":"2024-03-17T23:49:11.641904+0000","last_became_active":"2024-03-17T23:49:10.652809+0000","last_became_peered":"2024-03-17T23:49:10.652809+0000","last_unstale":"2024-03-17T23:49:11.641904+0000","last_undegraded":"2024-03-17T23:49:11.641904+0000","last_fullsized":"2024-03-17T23:49:11.641904+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:51:40.737241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":238,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.645564+0000","last_change":"2024-03-17T23:46:44.768668+0000","last_active":"2024-03-17T23:49:11.645564+0000","last_peered":"2024-03-17T23:49:11.645564+0000","last_clean":"2024-03-17T23:49:11.645564+0000","last_became_active":"2024-03-17T23:46:44.767826+0000","last_became_peered":"2024-03-17T23:46:44.767826+0000","last_unstale":"2024-03-17T23:49:11.645564+0000","last_undegraded":"2024-03-17T23:49:11.645564+0000","last_fullsized":"2024-03-17T23:49:11.645564+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_clean_scrub_stamp":"2024-03-17T23:46:43.744878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:54:22.286155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":70,"acting":70,"num_store_stats":5},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":1772,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23,"ondisk_log_size":23,"up":40,"acting":40,"num_store_stats":5},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":14,"seq":60129542177,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27644,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":630784,"data_stored":283907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542177,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27640,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":630784,"data_stored":283907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28232,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343608,"statfs":{"total":96636764160,"available":96607854592,"internally_reserved":0,"allocated":1236992,"data_stored":879794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574881,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32432,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339408,"statfs":{"total":96636764160,"available":96603553792,"internally_reserved":0,"allocated":1282048,"data_stored":899989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31844,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339996,"statfs":{"total":96636764160,"available":96604155904,"internally_reserved":0,"allocated":679936,"data_stored":309461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574881,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":688128,"data_stored":310113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574881,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":692224,"data_stored":311073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":684032,"data_stored":309551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-17T23:49:15.308 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3f is not active+clean 2024-03-17T23:49:15.308 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.308 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3e is not active+clean 2024-03-17T23:49:15.308 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.309 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3b is not active+clean 2024-03-17T23:49:15.309 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.309 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3a is not active+clean 2024-03-17T23:49:15.309 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.309 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.38 is not active+clean 2024-03-17T23:49:15.309 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.309 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.30 is not active+clean 2024-03-17T23:49:15.309 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.309 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2f is not active+clean 2024-03-17T23:49:15.309 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.309 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2d is not active+clean 2024-03-17T23:49:15.309 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.310 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2c is not active+clean 2024-03-17T23:49:15.310 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.310 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2a is not active+clean 2024-03-17T23:49:15.310 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.310 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.27 is not active+clean 2024-03-17T23:49:15.310 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.310 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.25 is not active+clean 2024-03-17T23:49:15.310 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.310 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.24 is not active+clean 2024-03-17T23:49:15.314 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.314 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.23 is not active+clean 2024-03-17T23:49:15.314 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.314 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.21 is not active+clean 2024-03-17T23:49:15.314 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.314 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.20 is not active+clean 2024-03-17T23:49:15.314 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.314 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1e is not active+clean 2024-03-17T23:49:15.314 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.314 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1f is not active+clean 2024-03-17T23:49:15.315 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:09.625866+0000', 'last_change': '2024-03-17T23:49:09.625866+0000', 'last_active': '2024-03-17T23:49:09.625866+0000', 'last_peered': '2024-03-17T23:49:09.625866+0000', 'last_clean': '2024-03-17T23:49:09.625866+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:09.625866+0000', 'last_undegraded': '2024-03-17T23:49:09.625866+0000', 'last_fullsized': '2024-03-17T23:49:09.625866+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.315 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1e is not active+clean 2024-03-17T23:49:15.315 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:09.625866+0000', 'last_change': '2024-03-17T23:49:09.625866+0000', 'last_active': '2024-03-17T23:49:09.625866+0000', 'last_peered': '2024-03-17T23:49:09.625866+0000', 'last_clean': '2024-03-17T23:49:09.625866+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:09.625866+0000', 'last_undegraded': '2024-03-17T23:49:09.625866+0000', 'last_fullsized': '2024-03-17T23:49:09.625866+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.315 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1c is not active+clean 2024-03-17T23:49:15.315 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.315 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1d is not active+clean 2024-03-17T23:49:15.315 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:09.625866+0000', 'last_change': '2024-03-17T23:49:09.625866+0000', 'last_active': '2024-03-17T23:49:09.625866+0000', 'last_peered': '2024-03-17T23:49:09.625866+0000', 'last_clean': '2024-03-17T23:49:09.625866+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:09.625866+0000', 'last_undegraded': '2024-03-17T23:49:09.625866+0000', 'last_fullsized': '2024-03-17T23:49:09.625866+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.315 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1d is not active+clean 2024-03-17T23:49:15.315 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.315 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1a is not active+clean 2024-03-17T23:49:15.315 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.315 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1b is not active+clean 2024-03-17T23:49:15.316 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.316 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1a is not active+clean 2024-03-17T23:49:15.316 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:09.625866+0000', 'last_change': '2024-03-17T23:49:09.625866+0000', 'last_active': '2024-03-17T23:49:09.625866+0000', 'last_peered': '2024-03-17T23:49:09.625866+0000', 'last_clean': '2024-03-17T23:49:09.625866+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:09.625866+0000', 'last_undegraded': '2024-03-17T23:49:09.625866+0000', 'last_fullsized': '2024-03-17T23:49:09.625866+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.316 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.19 is not active+clean 2024-03-17T23:49:15.316 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.316 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.16 is not active+clean 2024-03-17T23:49:15.316 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:09.625866+0000', 'last_change': '2024-03-17T23:49:09.625866+0000', 'last_active': '2024-03-17T23:49:09.625866+0000', 'last_peered': '2024-03-17T23:49:09.625866+0000', 'last_clean': '2024-03-17T23:49:09.625866+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:09.625866+0000', 'last_undegraded': '2024-03-17T23:49:09.625866+0000', 'last_fullsized': '2024-03-17T23:49:09.625866+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.316 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.13 is not active+clean 2024-03-17T23:49:15.316 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:09.625866+0000', 'last_change': '2024-03-17T23:49:09.625866+0000', 'last_active': '2024-03-17T23:49:09.625866+0000', 'last_peered': '2024-03-17T23:49:09.625866+0000', 'last_clean': '2024-03-17T23:49:09.625866+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:09.625866+0000', 'last_undegraded': '2024-03-17T23:49:09.625866+0000', 'last_fullsized': '2024-03-17T23:49:09.625866+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.316 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.f is not active+clean 2024-03-17T23:49:15.316 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:09.625866+0000', 'last_change': '2024-03-17T23:49:09.625866+0000', 'last_active': '2024-03-17T23:49:09.625866+0000', 'last_peered': '2024-03-17T23:49:09.625866+0000', 'last_clean': '2024-03-17T23:49:09.625866+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:09.625866+0000', 'last_undegraded': '2024-03-17T23:49:09.625866+0000', 'last_fullsized': '2024-03-17T23:49:09.625866+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.317 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.e is not active+clean 2024-03-17T23:49:15.317 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:09.625866+0000', 'last_change': '2024-03-17T23:49:09.625866+0000', 'last_active': '2024-03-17T23:49:09.625866+0000', 'last_peered': '2024-03-17T23:49:09.625866+0000', 'last_clean': '2024-03-17T23:49:09.625866+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:09.625866+0000', 'last_undegraded': '2024-03-17T23:49:09.625866+0000', 'last_fullsized': '2024-03-17T23:49:09.625866+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.317 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.c is not active+clean 2024-03-17T23:49:15.317 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.317 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.c is not active+clean 2024-03-17T23:49:15.317 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:09.625866+0000', 'last_change': '2024-03-17T23:49:09.625866+0000', 'last_active': '2024-03-17T23:49:09.625866+0000', 'last_peered': '2024-03-17T23:49:09.625866+0000', 'last_clean': '2024-03-17T23:49:09.625866+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:09.625866+0000', 'last_undegraded': '2024-03-17T23:49:09.625866+0000', 'last_fullsized': '2024-03-17T23:49:09.625866+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.317 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.b is not active+clean 2024-03-17T23:49:15.317 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:09.625866+0000', 'last_change': '2024-03-17T23:49:09.625866+0000', 'last_active': '2024-03-17T23:49:09.625866+0000', 'last_peered': '2024-03-17T23:49:09.625866+0000', 'last_clean': '2024-03-17T23:49:09.625866+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:09.625866+0000', 'last_undegraded': '2024-03-17T23:49:09.625866+0000', 'last_fullsized': '2024-03-17T23:49:09.625866+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.317 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.b is not active+clean 2024-03-17T23:49:15.317 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.318 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.9 is not active+clean 2024-03-17T23:49:15.318 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.318 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.6 is not active+clean 2024-03-17T23:49:15.318 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:09.625866+0000', 'last_change': '2024-03-17T23:49:09.625866+0000', 'last_active': '2024-03-17T23:49:09.625866+0000', 'last_peered': '2024-03-17T23:49:09.625866+0000', 'last_clean': '2024-03-17T23:49:09.625866+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:09.625866+0000', 'last_undegraded': '2024-03-17T23:49:09.625866+0000', 'last_fullsized': '2024-03-17T23:49:09.625866+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.318 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.5 is not active+clean 2024-03-17T23:49:15.318 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.318 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2 is not active+clean 2024-03-17T23:49:15.318 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.318 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1 is not active+clean 2024-03-17T23:49:15.318 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.319 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.0 is not active+clean 2024-03-17T23:49:15.319 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:09.625866+0000', 'last_change': '2024-03-17T23:49:09.625866+0000', 'last_active': '2024-03-17T23:49:09.625866+0000', 'last_peered': '2024-03-17T23:49:09.625866+0000', 'last_clean': '2024-03-17T23:49:09.625866+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:09.625866+0000', 'last_undegraded': '2024-03-17T23:49:09.625866+0000', 'last_fullsized': '2024-03-17T23:49:09.625866+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:09.625866+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.319 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.8 is not active+clean 2024-03-17T23:49:15.319 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-17T23:49:10.630539+0000', 'last_change': '2024-03-17T23:49:10.630539+0000', 'last_active': '2024-03-17T23:49:10.630539+0000', 'last_peered': '2024-03-17T23:49:10.630539+0000', 'last_clean': '2024-03-17T23:49:10.630539+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-17T23:49:10.630539+0000', 'last_undegraded': '2024-03-17T23:49:10.630539+0000', 'last_fullsized': '2024-03-17T23:49:10.630539+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'last_clean_scrub_stamp': '2024-03-17T23:49:10.630539+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-17T23:49:15.319 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-17T23:49:15.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.444+0000 7f07dd922640 1 Processor -- start 2024-03-17T23:49:15.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.444+0000 7f07dd922640 1 -- start start 2024-03-17T23:49:15.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.445+0000 7f07dd922640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f07d80cf120 0x7f07d81619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:15.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.445+0000 7f07dd922640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07d81620a0 con 0x7f07d815de30 2024-03-17T23:49:15.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.445+0000 7f07dd922640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07d8162380 con 0x7f07d80b9860 2024-03-17T23:49:15.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.445+0000 7f07dd922640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f07d8162940 con 0x7f07d80cf120 2024-03-17T23:49:15.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.445+0000 7f07d6ffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f07d815de30 0x7f07d80cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55698/0 (socket says 172.21.15.150:55698) 2024-03-17T23:49:15.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.445+0000 7f07d6ffd640 1 -- 172.21.15.150:0/3906519004 learned_addr learned my addr 172.21.15.150:0/3906519004 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:15.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.445+0000 7f07d67fc640 1 --2- 172.21.15.150:0/3906519004 >> v2:172.21.15.157:3300/0 conn(0x7f07d80cf120 0x7f07d81619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:15.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.445+0000 7f07d5ffb640 1 -- 172.21.15.150:0/3906519004 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2417429734 0 0) 0x7f07d81620a0 con 0x7f07d815de30 2024-03-17T23:49:15.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.445+0000 7f07d5ffb640 1 -- 172.21.15.150:0/3906519004 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07c0003520 con 0x7f07d815de30 2024-03-17T23:49:15.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.445+0000 7f07d67fc640 1 -- 172.21.15.150:0/3906519004 >> v1:172.21.15.157:6789/0 conn(0x7f07d80b9860 legacy=0x7f07d80d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:15.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.445+0000 7f07d67fc640 1 -- 172.21.15.150:0/3906519004 >> v1:172.21.15.150:6789/0 conn(0x7f07d815de30 legacy=0x7f07d80cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:15.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.446+0000 7f07d67fc640 1 -- 172.21.15.150:0/3906519004 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d8162660 con 0x7f07d80cf120 2024-03-17T23:49:15.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.446+0000 7f07d67fc640 1 --2- 172.21.15.150:0/3906519004 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07d80cf120 0x7f07d81619a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f07cc0097a0 tx=0x7f07cc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=791601cb43496bc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:15.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.446+0000 7f07d5ffb640 1 -- 172.21.15.150:0/3906519004 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07cc035020 con 0x7f07d80cf120 2024-03-17T23:49:15.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.446+0000 7f07d5ffb640 1 -- 172.21.15.150:0/3906519004 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07cc031830 con 0x7f07d80cf120 2024-03-17T23:49:15.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.446+0000 7f07d5ffb640 1 -- 172.21.15.150:0/3906519004 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07cc031be0 con 0x7f07d80cf120 2024-03-17T23:49:15.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.447+0000 7f07dd922640 1 -- 172.21.15.150:0/3906519004 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07d80cf120 msgr2=0x7f07d81619a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:15.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.447+0000 7f07dd922640 1 --2- 172.21.15.150:0/3906519004 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07d80cf120 0x7f07d81619a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f07cc0097a0 tx=0x7f07cc02b8d0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.447+0000 7f07dd922640 1 -- 172.21.15.150:0/3906519004 shutdown_connections 2024-03-17T23:49:15.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.447+0000 7f07dd922640 1 --2- 172.21.15.150:0/3906519004 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07d80cf120 0x7f07d81619a0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.447+0000 7f07dd922640 1 -- 172.21.15.150:0/3906519004 >> 172.21.15.150:0/3906519004 conn(0x7f07d80730b0 msgr2=0x7f07d80d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:15.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.447+0000 7f07dd922640 1 -- 172.21.15.150:0/3906519004 shutdown_connections 2024-03-17T23:49:15.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.447+0000 7f07dd922640 1 -- 172.21.15.150:0/3906519004 wait complete. 2024-03-17T23:49:15.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.448+0000 7f07dd922640 1 Processor -- start 2024-03-17T23:49:15.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.448+0000 7f07dd922640 1 -- start start 2024-03-17T23:49:15.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.449+0000 7f07dd922640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07d80b9860 0x7f07d815fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:15.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.449+0000 7f07dd922640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07d80cf120 0x7f07d80cb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:15.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.449+0000 7f07dd922640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f07d815de30 0x7f07d814b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:15.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.449+0000 7f07dd922640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f07d815e210 con 0x7f07d80b9860 2024-03-17T23:49:15.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.449+0000 7f07dd922640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f07d80b9c90 con 0x7f07d80cf120 2024-03-17T23:49:15.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.449+0000 7f07dd922640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f07d80d2da0 con 0x7f07d815de30 2024-03-17T23:49:15.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.449+0000 7f07d6ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07d80b9860 0x7f07d815fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:15.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.449+0000 7f07d6ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07d80b9860 0x7f07d815fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43944/0 (socket says 172.21.15.150:43944) 2024-03-17T23:49:15.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.449+0000 7f07d6ffd640 1 -- 172.21.15.150:0/739508473 learned_addr learned my addr 172.21.15.150:0/739508473 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:15.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.449+0000 7f07d67fc640 1 --2- 172.21.15.150:0/739508473 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07d80cf120 0x7f07d80cb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:15.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.449+0000 7f07d77fe640 1 --2- 172.21.15.150:0/739508473 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f07d815de30 0x7f07d814b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:15.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.449+0000 7f07d6ffd640 1 -- 172.21.15.150:0/739508473 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f07d815de30 msgr2=0x7f07d814b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:15.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.449+0000 7f07d6ffd640 1 --2- 172.21.15.150:0/739508473 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f07d815de30 0x7f07d814b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.449+0000 7f07d6ffd640 1 -- 172.21.15.150:0/739508473 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07d80cf120 msgr2=0x7f07d80cb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:15.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.449+0000 7f07d6ffd640 1 --2- 172.21.15.150:0/739508473 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07d80cf120 0x7f07d80cb180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.449+0000 7f07d6ffd640 1 -- 172.21.15.150:0/739508473 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d814b950 con 0x7f07d80b9860 2024-03-17T23:49:15.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.450+0000 7f07d6ffd640 1 --2- 172.21.15.150:0/739508473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07d80b9860 0x7f07d815fc30 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f07c800cbf0 tx=0x7f07c800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:15.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.450+0000 7f07bffff640 1 -- 172.21.15.150:0/739508473 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c8017070 con 0x7f07d80b9860 2024-03-17T23:49:15.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.450+0000 7f07bffff640 1 -- 172.21.15.150:0/739508473 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07c800ad40 con 0x7f07d80b9860 2024-03-17T23:49:15.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.450+0000 7f07bffff640 1 -- 172.21.15.150:0/739508473 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c8012620 con 0x7f07d80b9860 2024-03-17T23:49:15.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.450+0000 7f07dd922640 1 -- 172.21.15.150:0/739508473 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07d8160330 con 0x7f07d80b9860 2024-03-17T23:49:15.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.450+0000 7f07dd922640 1 -- 172.21.15.150:0/739508473 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07d81608f0 con 0x7f07d80b9860 2024-03-17T23:49:15.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.451+0000 7f07dd922640 1 -- 172.21.15.150:0/739508473 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07d80681a0 con 0x7f07d80b9860 2024-03-17T23:49:15.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.451+0000 7f07bffff640 1 -- 172.21.15.150:0/739508473 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f07c801a070 con 0x7f07d80b9860 2024-03-17T23:49:15.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.452+0000 7f07bffff640 1 --2- 172.21.15.150:0/739508473 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f07c406bbd0 0x7f07c406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:15.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.452+0000 7f07bffff640 1 -- 172.21.15.150:0/739508473 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f07c808a9c0 con 0x7f07d80b9860 2024-03-17T23:49:15.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.452+0000 7f07d67fc640 1 --2- 172.21.15.150:0/739508473 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f07c406bbd0 0x7f07c406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:15.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.452+0000 7f07d67fc640 1 --2- 172.21.15.150:0/739508473 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f07c406bbd0 0x7f07c406e090 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f07cc036000 tx=0x7f07cc03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:15.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.456+0000 7f07bffff640 1 -- 172.21.15.150:0/739508473 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f07c808ce20 con 0x7f07d80b9860 2024-03-17T23:49:15.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.697+0000 7f07dd922640 1 -- 172.21.15.150:0/739508473 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f07d80d1910 con 0x7f07d80b9860 2024-03-17T23:49:15.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.697+0000 7f07bffff640 1 -- 172.21.15.150:0/739508473 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1432 (secure 0 0 0) 0x7f07c8058cd0 con 0x7f07d80b9860 2024-03-17T23:49:15.698 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:15.698 INFO:teuthology.orchestra.run.smithi150.stdout:{"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":164,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":58,"num_osds":8,"num_up_osds":8,"osd_up_since":1710719202,"num_in_osds":8,"osd_in_since":1710719193,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":245112832,"bytes_avail":772849000448,"bytes_total":773094113280,"write_bytes_sec":2391,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":22,"id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"b","status":"up:active","gid":6204}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-03-17T23:48:59.622075+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-17T23:49:15.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.699+0000 7f07dd922640 1 -- 172.21.15.150:0/739508473 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f07c406bbd0 msgr2=0x7f07c406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:15.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.699+0000 7f07dd922640 1 --2- 172.21.15.150:0/739508473 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f07c406bbd0 0x7f07c406e090 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f07cc036000 tx=0x7f07cc03d040 comp rx=0 tx=0).stop 2024-03-17T23:49:15.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.699+0000 7f07dd922640 1 -- 172.21.15.150:0/739508473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07d80b9860 msgr2=0x7f07d815fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:15.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.699+0000 7f07dd922640 1 --2- 172.21.15.150:0/739508473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07d80b9860 0x7f07d815fc30 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f07c800cbf0 tx=0x7f07c800a430 comp rx=0 tx=0).stop 2024-03-17T23:49:15.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.699+0000 7f07dd922640 1 -- 172.21.15.150:0/739508473 shutdown_connections 2024-03-17T23:49:15.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.699+0000 7f07dd922640 1 --2- 172.21.15.150:0/739508473 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f07d815de30 0x7f07d814b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.699+0000 7f07dd922640 1 --2- 172.21.15.150:0/739508473 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f07c406bbd0 0x7f07c406e090 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.699+0000 7f07dd922640 1 --2- 172.21.15.150:0/739508473 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f07d80cf120 0x7f07d80cb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.699+0000 7f07dd922640 1 --2- 172.21.15.150:0/739508473 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07d80b9860 0x7f07d815fc30 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.699+0000 7f07dd922640 1 -- 172.21.15.150:0/739508473 >> 172.21.15.150:0/739508473 conn(0x7f07d80730b0 msgr2=0x7f07d80ce6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:15.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.699+0000 7f07dd922640 1 -- 172.21.15.150:0/739508473 shutdown_connections 2024-03-17T23:49:15.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.699+0000 7f07dd922640 1 -- 172.21.15.150:0/739508473 wait complete. 2024-03-17T23:49:15.711 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-17T23:49:15.711 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-17T23:49:15.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.812+0000 7f83e4b9d640 1 Processor -- start 2024-03-17T23:49:15.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.812+0000 7f83e4b9d640 1 -- start start 2024-03-17T23:49:15.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.813+0000 7f83e4b9d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f83e0147880 0x7f83e015ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:15.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.813+0000 7f83e4b9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f83e006b8a0 0x7f83e01619d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:15.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.813+0000 7f83e4b9d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83e00b55f0 con 0x7f83e00b2060 2024-03-17T23:49:15.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.813+0000 7f83e4b9d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f83e00b3990 con 0x7f83e006b8a0 2024-03-17T23:49:15.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.813+0000 7f83e4b9d640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f83e015f370 con 0x7f83e0147880 2024-03-17T23:49:15.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.813+0000 7f83dffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f83e006b8a0 0x7f83e01619d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:15.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.813+0000 7f83dffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f83e006b8a0 0x7f83e01619d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45510/0 (socket says 172.21.15.150:45510) 2024-03-17T23:49:15.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.813+0000 7f83dffff640 1 -- 172.21.15.150:0/3526702789 learned_addr learned my addr 172.21.15.150:0/3526702789 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:15.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.813+0000 7f83d6ffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f83e00b2060 0x7f83e006b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:51764/0 (socket says 172.21.15.150:51764) 2024-03-17T23:49:15.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.813+0000 7f83df7fe640 1 --2- 172.21.15.150:0/3526702789 >> v2:172.21.15.157:3300/0 conn(0x7f83e0147880 0x7f83e015ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:15.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.814+0000 7f83df7fe640 1 -- 172.21.15.150:0/3526702789 >> v2:172.21.15.150:3300/0 conn(0x7f83e006b8a0 msgr2=0x7f83e01619d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:15.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.814+0000 7f83df7fe640 1 --2- 172.21.15.150:0/3526702789 >> v2:172.21.15.150:3300/0 conn(0x7f83e006b8a0 0x7f83e01619d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.814+0000 7f83df7fe640 1 -- 172.21.15.150:0/3526702789 >> v1:172.21.15.150:6789/0 conn(0x7f83e00b2060 legacy=0x7f83e006b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:15.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.814+0000 7f83df7fe640 1 -- 172.21.15.150:0/3526702789 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83e00b3be0 con 0x7f83e0147880 2024-03-17T23:49:15.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.814+0000 7f83deffd640 1 -- 172.21.15.150:0/3526702789 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 580514617 0 0) 0x7f83e00b55f0 con 0x7f83e00b2060 2024-03-17T23:49:15.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.814+0000 7f83dffff640 1 --2- 172.21.15.150:0/3526702789 >> v2:172.21.15.150:3300/0 conn(0x7f83e006b8a0 0x7f83e01619d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:15.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.814+0000 7f83df7fe640 1 --2- 172.21.15.150:0/3526702789 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f83e0147880 0x7f83e015ee30 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f83e006ec40 tx=0x7f83d002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cf2c220f787284 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:15.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.815+0000 7f83deffd640 1 -- 172.21.15.150:0/3526702789 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d00315a0 con 0x7f83e0147880 2024-03-17T23:49:15.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.815+0000 7f83deffd640 1 -- 172.21.15.150:0/3526702789 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83d0037070 con 0x7f83e0147880 2024-03-17T23:49:15.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.815+0000 7f83deffd640 1 -- 172.21.15.150:0/3526702789 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d0031960 con 0x7f83e0147880 2024-03-17T23:49:15.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.815+0000 7f83e4b9d640 1 -- 172.21.15.150:0/3526702789 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f83e0147880 msgr2=0x7f83e015ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:15.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.815+0000 7f83e4b9d640 1 --2- 172.21.15.150:0/3526702789 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f83e0147880 0x7f83e015ee30 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f83e006ec40 tx=0x7f83d002b840 comp rx=0 tx=0).stop 2024-03-17T23:49:15.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.815+0000 7f83e4b9d640 1 -- 172.21.15.150:0/3526702789 shutdown_connections 2024-03-17T23:49:15.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.815+0000 7f83e4b9d640 1 --2- 172.21.15.150:0/3526702789 >> v2:172.21.15.150:3300/0 conn(0x7f83e006b8a0 0x7f83e01619d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.815+0000 7f83e4b9d640 1 --2- 172.21.15.150:0/3526702789 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f83e0147880 0x7f83e015ee30 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.815+0000 7f83e4b9d640 1 -- 172.21.15.150:0/3526702789 >> 172.21.15.150:0/3526702789 conn(0x7f83e00b60d0 msgr2=0x7f83e0099a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:15.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.815+0000 7f83e4b9d640 1 -- 172.21.15.150:0/3526702789 shutdown_connections 2024-03-17T23:49:15.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.815+0000 7f83e4b9d640 1 -- 172.21.15.150:0/3526702789 wait complete. 2024-03-17T23:49:15.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.816+0000 7f83e4b9d640 1 Processor -- start 2024-03-17T23:49:15.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.817+0000 7f83e4b9d640 1 -- start start 2024-03-17T23:49:15.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.817+0000 7f83e4b9d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f83e006b8a0 0x7f83e013d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:15.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.817+0000 7f83e4b9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83e00b2060 0x7f83e00af230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:15.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.817+0000 7f83e4b9d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f83e0147880 0x7f83e009c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:15.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.817+0000 7f83e4b9d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f83e015f5a0 con 0x7f83e00b2060 2024-03-17T23:49:15.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.817+0000 7f83e4b9d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f83e00b5890 con 0x7f83e006b8a0 2024-03-17T23:49:15.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.817+0000 7f83e4b9d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f83e015f720 con 0x7f83e0147880 2024-03-17T23:49:15.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.817+0000 7f83d6ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83e00b2060 0x7f83e00af230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:15.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.817+0000 7f83d6ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83e00b2060 0x7f83e00af230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45518/0 (socket says 172.21.15.150:45518) 2024-03-17T23:49:15.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.817+0000 7f83d6ffd640 1 -- 172.21.15.150:0/4083765115 learned_addr learned my addr 172.21.15.150:0/4083765115 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:15.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.817+0000 7f83df7fe640 1 --2- 172.21.15.150:0/4083765115 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f83e006b8a0 0x7f83e013d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:15.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.817+0000 7f83dffff640 1 --2- 172.21.15.150:0/4083765115 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f83e0147880 0x7f83e009c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:15.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.818+0000 7f83d6ffd640 1 -- 172.21.15.150:0/4083765115 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f83e0147880 msgr2=0x7f83e009c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:15.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.818+0000 7f83d6ffd640 1 --2- 172.21.15.150:0/4083765115 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f83e0147880 0x7f83e009c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.818+0000 7f83d6ffd640 1 -- 172.21.15.150:0/4083765115 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f83e006b8a0 msgr2=0x7f83e013d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:15.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.818+0000 7f83d6ffd640 1 --2- 172.21.15.150:0/4083765115 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f83e006b8a0 0x7f83e013d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:15.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.818+0000 7f83d6ffd640 1 -- 172.21.15.150:0/4083765115 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83e009e9c0 con 0x7f83e00b2060 2024-03-17T23:49:15.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.818+0000 7f83d6ffd640 1 --2- 172.21.15.150:0/4083765115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83e00b2060 0x7f83e00af230 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f83c800c410 tx=0x7f83c800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:15.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.818+0000 7f83dcff9640 1 -- 172.21.15.150:0/4083765115 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c80074e0 con 0x7f83e00b2060 2024-03-17T23:49:15.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.818+0000 7f83dcff9640 1 -- 172.21.15.150:0/4083765115 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83c80047e0 con 0x7f83e00b2060 2024-03-17T23:49:15.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.818+0000 7f83e4b9d640 1 -- 172.21.15.150:0/4083765115 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83e00adeb0 con 0x7f83e00b2060 2024-03-17T23:49:15.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.818+0000 7f83dcff9640 1 -- 172.21.15.150:0/4083765115 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c8004bd0 con 0x7f83e00b2060 2024-03-17T23:49:15.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.818+0000 7f83e4b9d640 1 -- 172.21.15.150:0/4083765115 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83e00ae470 con 0x7f83e00b2060 2024-03-17T23:49:15.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.819+0000 7f83e4b9d640 1 -- 172.21.15.150:0/4083765115 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83e00688b0 con 0x7f83e00b2060 2024-03-17T23:49:15.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.819+0000 7f83dcff9640 1 -- 172.21.15.150:0/4083765115 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f83c8011070 con 0x7f83e00b2060 2024-03-17T23:49:15.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.820+0000 7f83dcff9640 1 --2- 172.21.15.150:0/4083765115 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f83c006bbd0 0x7f83c006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:15.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.820+0000 7f83dcff9640 1 -- 172.21.15.150:0/4083765115 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f83c808a640 con 0x7f83e00b2060 2024-03-17T23:49:15.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.820+0000 7f83df7fe640 1 --2- 172.21.15.150:0/4083765115 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f83c006bbd0 0x7f83c006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:15.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.820+0000 7f83df7fe640 1 --2- 172.21.15.150:0/4083765115 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f83c006bbd0 0x7f83c006e090 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f83d0036000 tx=0x7f83d0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:15.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.824+0000 7f83dcff9640 1 -- 172.21.15.150:0/4083765115 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f83c80589e0 con 0x7f83e00b2060 2024-03-17T23:49:15.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.994+0000 7f83e4b9d640 1 -- 172.21.15.150:0/4083765115 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f83e015ec10 con 0x7f83c006bbd0 2024-03-17T23:49:15.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:15.999+0000 7f83dcff9640 1 -- 172.21.15.150:0/4083765115 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251918 (secure 0 0 0) 0x7f83e015ec10 con 0x7f83c006bbd0 2024-03-17T23:49:16.002 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:16.003 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-03-17T23:49:16.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:16.002+0000 7f83e4b9d640 1 -- 172.21.15.150:0/4083765115 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f83c006bbd0 msgr2=0x7f83c006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:16.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:16.002+0000 7f83e4b9d640 1 --2- 172.21.15.150:0/4083765115 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f83c006bbd0 0x7f83c006e090 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f83d0036000 tx=0x7f83d0008030 comp rx=0 tx=0).stop 2024-03-17T23:49:16.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:16.002+0000 7f83e4b9d640 1 -- 172.21.15.150:0/4083765115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83e00b2060 msgr2=0x7f83e00af230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:16.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:16.002+0000 7f83e4b9d640 1 --2- 172.21.15.150:0/4083765115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83e00b2060 0x7f83e00af230 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f83c800c410 tx=0x7f83c800c8d0 comp rx=0 tx=0).stop 2024-03-17T23:49:16.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:16.002+0000 7f83e4b9d640 1 -- 172.21.15.150:0/4083765115 shutdown_connections 2024-03-17T23:49:16.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:16.002+0000 7f83e4b9d640 1 --2- 172.21.15.150:0/4083765115 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f83e0147880 0x7f83e009c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:16.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:16.002+0000 7f83e4b9d640 1 --2- 172.21.15.150:0/4083765115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f83e00b2060 0x7f83e00af230 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:16.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:16.002+0000 7f83e4b9d640 1 --2- 172.21.15.150:0/4083765115 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f83c006bbd0 0x7f83c006e090 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:16.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:16.002+0000 7f83e4b9d640 1 --2- 172.21.15.150:0/4083765115 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f83e006b8a0 0x7f83e013d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:16.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:16.002+0000 7f83e4b9d640 1 -- 172.21.15.150:0/4083765115 >> 172.21.15.150:0/4083765115 conn(0x7f83e00b60d0 msgr2=0x7f83e00b2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:16.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:16.002+0000 7f83e4b9d640 1 -- 172.21.15.150:0/4083765115 shutdown_connections 2024-03-17T23:49:16.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:16.002+0000 7f83e4b9d640 1 -- 172.21.15.150:0/4083765115 wait complete. 2024-03-17T23:49:16.014 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2024-03-17T23:49:15.624702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":132,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239368,"kb_used_data":6560,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735352,"statfs":{"total":773094113280,"available":772849000448,"internally_reserved":0,"allocated":6717440,"data_stored":3674628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.993937"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.652693+0000","last_change":"2024-03-17T23:49:11.657789+0000","last_active":"2024-03-17T23:49:12.652693+0000","last_peered":"2024-03-17T23:49:12.652693+0000","last_clean":"2024-03-17T23:49:12.652693+0000","last_became_active":"2024-03-17T23:49:11.657538+0000","last_became_peered":"2024-03-17T23:49:11.657538+0000","last_unstale":"2024-03-17T23:49:12.652693+0000","last_undegraded":"2024-03-17T23:49:12.652693+0000","last_fullsized":"2024-03-17T23:49:12.652693+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:57:19.872599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531008+0000","last_change":"2024-03-17T23:49:11.653495+0000","last_active":"2024-03-17T23:49:13.531008+0000","last_peered":"2024-03-17T23:49:13.531008+0000","last_clean":"2024-03-17T23:49:13.531008+0000","last_became_active":"2024-03-17T23:49:11.653244+0000","last_became_peered":"2024-03-17T23:49:11.653244+0000","last_unstale":"2024-03-17T23:49:13.531008+0000","last_undegraded":"2024-03-17T23:49:13.531008+0000","last_fullsized":"2024-03-17T23:49:13.531008+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:11:54.641346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.670271+0000","last_change":"2024-03-17T23:49:11.667507+0000","last_active":"2024-03-17T23:49:11.670271+0000","last_peered":"2024-03-17T23:49:11.670271+0000","last_clean":"2024-03-17T23:49:11.670271+0000","last_became_active":"2024-03-17T23:49:11.666238+0000","last_became_peered":"2024-03-17T23:49:11.666238+0000","last_unstale":"2024-03-17T23:49:11.670271+0000","last_undegraded":"2024-03-17T23:49:11.670271+0000","last_fullsized":"2024-03-17T23:49:11.670271+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:15.111002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.663256+0000","last_change":"2024-03-17T23:49:11.660542+0000","last_active":"2024-03-17T23:49:11.663256+0000","last_peered":"2024-03-17T23:49:11.663256+0000","last_clean":"2024-03-17T23:49:11.663256+0000","last_became_active":"2024-03-17T23:49:11.660272+0000","last_became_peered":"2024-03-17T23:49:11.660272+0000","last_unstale":"2024-03-17T23:49:11.663256+0000","last_undegraded":"2024-03-17T23:49:11.663256+0000","last_fullsized":"2024-03-17T23:49:11.663256+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:52:42.007552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.652530+0000","last_change":"2024-03-17T23:49:11.648970+0000","last_active":"2024-03-17T23:49:12.652530+0000","last_peered":"2024-03-17T23:49:12.652530+0000","last_clean":"2024-03-17T23:49:12.652530+0000","last_became_active":"2024-03-17T23:49:11.648699+0000","last_became_peered":"2024-03-17T23:49:11.648699+0000","last_unstale":"2024-03-17T23:49:12.652530+0000","last_undegraded":"2024-03-17T23:49:12.652530+0000","last_fullsized":"2024-03-17T23:49:12.652530+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:24:41.316664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.656797+0000","last_change":"2024-03-17T23:49:11.665421+0000","last_active":"2024-03-17T23:49:12.656797+0000","last_peered":"2024-03-17T23:49:12.656797+0000","last_clean":"2024-03-17T23:49:12.656797+0000","last_became_active":"2024-03-17T23:49:11.664791+0000","last_became_peered":"2024-03-17T23:49:11.664791+0000","last_unstale":"2024-03-17T23:49:12.656797+0000","last_undegraded":"2024-03-17T23:49:12.656797+0000","last_fullsized":"2024-03-17T23:49:12.656797+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:38:28.202171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.658166+0000","last_change":"2024-03-17T23:49:11.655317+0000","last_active":"2024-03-17T23:49:11.658166+0000","last_peered":"2024-03-17T23:49:11.658166+0000","last_clean":"2024-03-17T23:49:11.658166+0000","last_became_active":"2024-03-17T23:49:11.655056+0000","last_became_peered":"2024-03-17T23:49:11.655056+0000","last_unstale":"2024-03-17T23:49:11.658166+0000","last_undegraded":"2024-03-17T23:49:11.658166+0000","last_fullsized":"2024-03-17T23:49:11.658166+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:00:05.868962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531333+0000","last_change":"2024-03-17T23:49:11.658083+0000","last_active":"2024-03-17T23:49:13.531333+0000","last_peered":"2024-03-17T23:49:13.531333+0000","last_clean":"2024-03-17T23:49:13.531333+0000","last_became_active":"2024-03-17T23:49:11.657418+0000","last_became_peered":"2024-03-17T23:49:11.657418+0000","last_unstale":"2024-03-17T23:49:13.531333+0000","last_undegraded":"2024-03-17T23:49:13.531333+0000","last_fullsized":"2024-03-17T23:49:13.531333+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:16:16.395414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659258+0000","last_change":"2024-03-17T23:49:11.657708+0000","last_active":"2024-03-17T23:49:11.659258+0000","last_peered":"2024-03-17T23:49:11.659258+0000","last_clean":"2024-03-17T23:49:11.659258+0000","last_became_active":"2024-03-17T23:49:11.656418+0000","last_became_peered":"2024-03-17T23:49:11.656418+0000","last_unstale":"2024-03-17T23:49:11.659258+0000","last_undegraded":"2024-03-17T23:49:11.659258+0000","last_fullsized":"2024-03-17T23:49:11.659258+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T23:59:25.825110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.652513+0000","last_change":"2024-03-17T23:49:11.645251+0000","last_active":"2024-03-17T23:49:11.652513+0000","last_peered":"2024-03-17T23:49:11.652513+0000","last_clean":"2024-03-17T23:49:11.652513+0000","last_became_active":"2024-03-17T23:49:11.644999+0000","last_became_peered":"2024-03-17T23:49:11.644999+0000","last_unstale":"2024-03-17T23:49:11.652513+0000","last_undegraded":"2024-03-17T23:49:11.652513+0000","last_fullsized":"2024-03-17T23:49:11.652513+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:10:05.668007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.652609+0000","last_change":"2024-03-17T23:49:11.648149+0000","last_active":"2024-03-17T23:49:11.652609+0000","last_peered":"2024-03-17T23:49:11.652609+0000","last_clean":"2024-03-17T23:49:11.652609+0000","last_became_active":"2024-03-17T23:49:11.647887+0000","last_became_peered":"2024-03-17T23:49:11.647887+0000","last_unstale":"2024-03-17T23:49:11.652609+0000","last_undegraded":"2024-03-17T23:49:11.652609+0000","last_fullsized":"2024-03-17T23:49:11.652609+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:16:38.296758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.660678+0000","last_change":"2024-03-17T23:49:11.655458+0000","last_active":"2024-03-17T23:49:11.660678+0000","last_peered":"2024-03-17T23:49:11.660678+0000","last_clean":"2024-03-17T23:49:11.660678+0000","last_became_active":"2024-03-17T23:49:11.655181+0000","last_became_peered":"2024-03-17T23:49:11.655181+0000","last_unstale":"2024-03-17T23:49:11.660678+0000","last_undegraded":"2024-03-17T23:49:11.660678+0000","last_fullsized":"2024-03-17T23:49:11.660678+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:55:32.443067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.664194+0000","last_change":"2024-03-17T23:49:11.656693+0000","last_active":"2024-03-17T23:49:11.664194+0000","last_peered":"2024-03-17T23:49:11.664194+0000","last_clean":"2024-03-17T23:49:11.664194+0000","last_became_active":"2024-03-17T23:49:11.656400+0000","last_became_peered":"2024-03-17T23:49:11.656400+0000","last_unstale":"2024-03-17T23:49:11.664194+0000","last_undegraded":"2024-03-17T23:49:11.664194+0000","last_fullsized":"2024-03-17T23:49:11.664194+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:54:01.212249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.655981+0000","last_change":"2024-03-17T23:49:11.652508+0000","last_active":"2024-03-17T23:49:11.655981+0000","last_peered":"2024-03-17T23:49:11.655981+0000","last_clean":"2024-03-17T23:49:11.655981+0000","last_became_active":"2024-03-17T23:49:11.652153+0000","last_became_peered":"2024-03-17T23:49:11.652153+0000","last_unstale":"2024-03-17T23:49:11.655981+0000","last_undegraded":"2024-03-17T23:49:11.655981+0000","last_fullsized":"2024-03-17T23:49:11.655981+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:35.755434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.650975+0000","last_change":"2024-03-17T23:49:11.647575+0000","last_active":"2024-03-17T23:49:11.650975+0000","last_peered":"2024-03-17T23:49:11.650975+0000","last_clean":"2024-03-17T23:49:11.650975+0000","last_became_active":"2024-03-17T23:49:11.647313+0000","last_became_peered":"2024-03-17T23:49:11.647313+0000","last_unstale":"2024-03-17T23:49:11.650975+0000","last_undegraded":"2024-03-17T23:49:11.650975+0000","last_fullsized":"2024-03-17T23:49:11.650975+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:37.609834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.656786+0000","last_change":"2024-03-17T23:49:11.665738+0000","last_active":"2024-03-17T23:49:12.656786+0000","last_peered":"2024-03-17T23:49:12.656786+0000","last_clean":"2024-03-17T23:49:12.656786+0000","last_became_active":"2024-03-17T23:49:11.665255+0000","last_became_peered":"2024-03-17T23:49:11.665255+0000","last_unstale":"2024-03-17T23:49:12.656786+0000","last_undegraded":"2024-03-17T23:49:12.656786+0000","last_fullsized":"2024-03-17T23:49:12.656786+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:42:40.988689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531384+0000","last_change":"2024-03-17T23:49:11.654570+0000","last_active":"2024-03-17T23:49:13.531384+0000","last_peered":"2024-03-17T23:49:13.531384+0000","last_clean":"2024-03-17T23:49:13.531384+0000","last_became_active":"2024-03-17T23:49:11.654322+0000","last_became_peered":"2024-03-17T23:49:11.654322+0000","last_unstale":"2024-03-17T23:49:13.531384+0000","last_undegraded":"2024-03-17T23:49:13.531384+0000","last_fullsized":"2024-03-17T23:49:13.531384+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:01:11.561015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.670209+0000","last_change":"2024-03-17T23:49:11.668183+0000","last_active":"2024-03-17T23:49:11.670209+0000","last_peered":"2024-03-17T23:49:11.670209+0000","last_clean":"2024-03-17T23:49:11.670209+0000","last_became_active":"2024-03-17T23:49:11.667577+0000","last_became_peered":"2024-03-17T23:49:11.667577+0000","last_unstale":"2024-03-17T23:49:11.670209+0000","last_undegraded":"2024-03-17T23:49:11.670209+0000","last_fullsized":"2024-03-17T23:49:11.670209+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T23:59:25.825110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.652561+0000","last_change":"2024-03-17T23:49:11.655445+0000","last_active":"2024-03-17T23:49:12.652561+0000","last_peered":"2024-03-17T23:49:12.652561+0000","last_clean":"2024-03-17T23:49:12.652561+0000","last_became_active":"2024-03-17T23:49:11.655177+0000","last_became_peered":"2024-03-17T23:49:11.655177+0000","last_unstale":"2024-03-17T23:49:12.652561+0000","last_undegraded":"2024-03-17T23:49:12.652561+0000","last_fullsized":"2024-03-17T23:49:12.652561+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:36:05.463772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531709+0000","last_change":"2024-03-17T23:49:11.659640+0000","last_active":"2024-03-17T23:49:13.531709+0000","last_peered":"2024-03-17T23:49:13.531709+0000","last_clean":"2024-03-17T23:49:13.531709+0000","last_became_active":"2024-03-17T23:49:11.659386+0000","last_became_peered":"2024-03-17T23:49:11.659386+0000","last_unstale":"2024-03-17T23:49:13.531709+0000","last_undegraded":"2024-03-17T23:49:13.531709+0000","last_fullsized":"2024-03-17T23:49:13.531709+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:04:57.089593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.657425+0000","last_change":"2024-03-17T23:49:11.656210+0000","last_active":"2024-03-17T23:49:11.657425+0000","last_peered":"2024-03-17T23:49:11.657425+0000","last_clean":"2024-03-17T23:49:11.657425+0000","last_became_active":"2024-03-17T23:49:11.655629+0000","last_became_peered":"2024-03-17T23:49:11.655629+0000","last_unstale":"2024-03-17T23:49:11.657425+0000","last_undegraded":"2024-03-17T23:49:11.657425+0000","last_fullsized":"2024-03-17T23:49:11.657425+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:09:15.871249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.530914+0000","last_change":"2024-03-17T23:49:11.657524+0000","last_active":"2024-03-17T23:49:13.530914+0000","last_peered":"2024-03-17T23:49:13.530914+0000","last_clean":"2024-03-17T23:49:13.530914+0000","last_became_active":"2024-03-17T23:49:11.657227+0000","last_became_peered":"2024-03-17T23:49:11.657227+0000","last_unstale":"2024-03-17T23:49:13.530914+0000","last_undegraded":"2024-03-17T23:49:13.530914+0000","last_fullsized":"2024-03-17T23:49:13.530914+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:05:09.392406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.660884+0000","last_change":"2024-03-17T23:49:11.658319+0000","last_active":"2024-03-17T23:49:11.660884+0000","last_peered":"2024-03-17T23:49:11.660884+0000","last_clean":"2024-03-17T23:49:11.660884+0000","last_became_active":"2024-03-17T23:49:11.656866+0000","last_became_peered":"2024-03-17T23:49:11.656866+0000","last_unstale":"2024-03-17T23:49:11.660884+0000","last_undegraded":"2024-03-17T23:49:11.660884+0000","last_fullsized":"2024-03-17T23:49:11.660884+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:26:49.763854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.664486+0000","last_change":"2024-03-17T23:49:11.658960+0000","last_active":"2024-03-17T23:49:11.664486+0000","last_peered":"2024-03-17T23:49:11.664486+0000","last_clean":"2024-03-17T23:49:11.664486+0000","last_became_active":"2024-03-17T23:49:11.658695+0000","last_became_peered":"2024-03-17T23:49:11.658695+0000","last_unstale":"2024-03-17T23:49:11.664486+0000","last_undegraded":"2024-03-17T23:49:11.664486+0000","last_fullsized":"2024-03-17T23:49:11.664486+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:07:08.139888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.652724+0000","last_change":"2024-03-17T23:49:11.661827+0000","last_active":"2024-03-17T23:49:12.652724+0000","last_peered":"2024-03-17T23:49:12.652724+0000","last_clean":"2024-03-17T23:49:12.652724+0000","last_became_active":"2024-03-17T23:49:11.661536+0000","last_became_peered":"2024-03-17T23:49:11.661536+0000","last_unstale":"2024-03-17T23:49:12.652724+0000","last_undegraded":"2024-03-17T23:49:12.652724+0000","last_fullsized":"2024-03-17T23:49:12.652724+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:42:06.022555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.655891+0000","last_change":"2024-03-17T23:49:11.654139+0000","last_active":"2024-03-17T23:49:11.655891+0000","last_peered":"2024-03-17T23:49:11.655891+0000","last_clean":"2024-03-17T23:49:11.655891+0000","last_became_active":"2024-03-17T23:49:11.653838+0000","last_became_peered":"2024-03-17T23:49:11.653838+0000","last_unstale":"2024-03-17T23:49:11.655891+0000","last_undegraded":"2024-03-17T23:49:11.655891+0000","last_fullsized":"2024-03-17T23:49:11.655891+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:00:05.868962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.657337+0000","last_change":"2024-03-17T23:49:11.655872+0000","last_active":"2024-03-17T23:49:12.657337+0000","last_peered":"2024-03-17T23:49:12.657337+0000","last_clean":"2024-03-17T23:49:12.657337+0000","last_became_active":"2024-03-17T23:49:11.654787+0000","last_became_peered":"2024-03-17T23:49:11.654787+0000","last_unstale":"2024-03-17T23:49:12.657337+0000","last_undegraded":"2024-03-17T23:49:12.657337+0000","last_fullsized":"2024-03-17T23:49:12.657337+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:41:43.506586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.652643+0000","last_change":"2024-03-17T23:49:11.657164+0000","last_active":"2024-03-17T23:49:12.652643+0000","last_peered":"2024-03-17T23:49:12.652643+0000","last_clean":"2024-03-17T23:49:12.652643+0000","last_became_active":"2024-03-17T23:49:11.656855+0000","last_became_peered":"2024-03-17T23:49:11.656855+0000","last_unstale":"2024-03-17T23:49:12.652643+0000","last_undegraded":"2024-03-17T23:49:12.652643+0000","last_fullsized":"2024-03-17T23:49:12.652643+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:07:00.700235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531614+0000","last_change":"2024-03-17T23:49:11.653476+0000","last_active":"2024-03-17T23:49:13.531614+0000","last_peered":"2024-03-17T23:49:13.531614+0000","last_clean":"2024-03-17T23:49:13.531614+0000","last_became_active":"2024-03-17T23:49:11.653189+0000","last_became_peered":"2024-03-17T23:49:11.653189+0000","last_unstale":"2024-03-17T23:49:13.531614+0000","last_undegraded":"2024-03-17T23:49:13.531614+0000","last_fullsized":"2024-03-17T23:49:13.531614+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T23:50:29.132627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.670150+0000","last_change":"2024-03-17T23:49:11.667572+0000","last_active":"2024-03-17T23:49:11.670150+0000","last_peered":"2024-03-17T23:49:11.670150+0000","last_clean":"2024-03-17T23:49:11.670150+0000","last_became_active":"2024-03-17T23:49:11.666249+0000","last_became_peered":"2024-03-17T23:49:11.666249+0000","last_unstale":"2024-03-17T23:49:11.670150+0000","last_undegraded":"2024-03-17T23:49:11.670150+0000","last_fullsized":"2024-03-17T23:49:11.670150+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:00:05.868962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.652416+0000","last_change":"2024-03-17T23:49:11.661039+0000","last_active":"2024-03-17T23:49:12.652416+0000","last_peered":"2024-03-17T23:49:12.652416+0000","last_clean":"2024-03-17T23:49:12.652416+0000","last_became_active":"2024-03-17T23:49:11.660765+0000","last_became_peered":"2024-03-17T23:49:11.660765+0000","last_unstale":"2024-03-17T23:49:12.652416+0000","last_undegraded":"2024-03-17T23:49:12.652416+0000","last_fullsized":"2024-03-17T23:49:12.652416+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:25:25.661450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.656858+0000","last_change":"2024-03-17T23:49:11.665506+0000","last_active":"2024-03-17T23:49:12.656858+0000","last_peered":"2024-03-17T23:49:12.656858+0000","last_clean":"2024-03-17T23:49:12.656858+0000","last_became_active":"2024-03-17T23:49:11.664919+0000","last_became_peered":"2024-03-17T23:49:11.664919+0000","last_unstale":"2024-03-17T23:49:12.656858+0000","last_undegraded":"2024-03-17T23:49:12.656858+0000","last_fullsized":"2024-03-17T23:49:12.656858+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:00:38.498038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.657299+0000","last_change":"2024-03-17T23:49:11.656201+0000","last_active":"2024-03-17T23:49:12.657299+0000","last_peered":"2024-03-17T23:49:12.657299+0000","last_clean":"2024-03-17T23:49:12.657299+0000","last_became_active":"2024-03-17T23:49:11.655858+0000","last_became_peered":"2024-03-17T23:49:11.655858+0000","last_unstale":"2024-03-17T23:49:12.657299+0000","last_undegraded":"2024-03-17T23:49:12.657299+0000","last_fullsized":"2024-03-17T23:49:12.657299+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:39:45.444066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"57'4","reported_seq":18,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.657280+0000","last_change":"2024-03-17T23:49:10.652792+0000","last_active":"2024-03-17T23:49:12.657280+0000","last_peered":"2024-03-17T23:49:12.657280+0000","last_clean":"2024-03-17T23:49:12.657280+0000","last_became_active":"2024-03-17T23:49:10.652498+0000","last_became_peered":"2024-03-17T23:49:10.652498+0000","last_unstale":"2024-03-17T23:49:12.657280+0000","last_undegraded":"2024-03-17T23:49:12.657280+0000","last_fullsized":"2024-03-17T23:49:12.657280+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:14.106330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.661421+0000","last_change":"2024-03-17T23:49:11.656653+0000","last_active":"2024-03-17T23:49:11.661421+0000","last_peered":"2024-03-17T23:49:11.661421+0000","last_clean":"2024-03-17T23:49:11.661421+0000","last_became_active":"2024-03-17T23:49:11.656335+0000","last_became_peered":"2024-03-17T23:49:11.656335+0000","last_unstale":"2024-03-17T23:49:11.661421+0000","last_undegraded":"2024-03-17T23:49:11.661421+0000","last_fullsized":"2024-03-17T23:49:11.661421+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:21:49.823973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531085+0000","last_change":"2024-03-17T23:49:10.652668+0000","last_active":"2024-03-17T23:49:13.531085+0000","last_peered":"2024-03-17T23:49:13.531085+0000","last_clean":"2024-03-17T23:49:13.531085+0000","last_became_active":"2024-03-17T23:49:10.652257+0000","last_became_peered":"2024-03-17T23:49:10.652257+0000","last_unstale":"2024-03-17T23:49:13.531085+0000","last_undegraded":"2024-03-17T23:49:13.531085+0000","last_fullsized":"2024-03-17T23:49:13.531085+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:39:44.439394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.657074+0000","last_change":"2024-03-17T23:49:11.661266+0000","last_active":"2024-03-17T23:49:12.657074+0000","last_peered":"2024-03-17T23:49:12.657074+0000","last_clean":"2024-03-17T23:49:12.657074+0000","last_became_active":"2024-03-17T23:49:11.660990+0000","last_became_peered":"2024-03-17T23:49:11.660990+0000","last_unstale":"2024-03-17T23:49:12.657074+0000","last_undegraded":"2024-03-17T23:49:12.657074+0000","last_fullsized":"2024-03-17T23:49:12.657074+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:37:56.896972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.657389+0000","last_change":"2024-03-17T23:49:10.653477+0000","last_active":"2024-03-17T23:49:12.657389+0000","last_peered":"2024-03-17T23:49:12.657389+0000","last_clean":"2024-03-17T23:49:12.657389+0000","last_became_active":"2024-03-17T23:49:10.653143+0000","last_became_peered":"2024-03-17T23:49:10.653143+0000","last_unstale":"2024-03-17T23:49:12.657389+0000","last_undegraded":"2024-03-17T23:49:12.657389+0000","last_fullsized":"2024-03-17T23:49:12.657389+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T23:59:24.820437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.657139+0000","last_change":"2024-03-17T23:49:11.659380+0000","last_active":"2024-03-17T23:49:12.657139+0000","last_peered":"2024-03-17T23:49:12.657139+0000","last_clean":"2024-03-17T23:49:12.657139+0000","last_became_active":"2024-03-17T23:49:11.658920+0000","last_became_peered":"2024-03-17T23:49:11.658920+0000","last_unstale":"2024-03-17T23:49:12.657139+0000","last_undegraded":"2024-03-17T23:49:12.657139+0000","last_fullsized":"2024-03-17T23:49:12.657139+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:09:04.313151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.669900+0000","last_change":"2024-03-17T23:49:10.654468+0000","last_active":"2024-03-17T23:49:11.669900+0000","last_peered":"2024-03-17T23:49:11.669900+0000","last_clean":"2024-03-17T23:49:11.669900+0000","last_became_active":"2024-03-17T23:49:10.652810+0000","last_became_peered":"2024-03-17T23:49:10.652810+0000","last_unstale":"2024-03-17T23:49:11.669900+0000","last_undegraded":"2024-03-17T23:49:11.669900+0000","last_fullsized":"2024-03-17T23:49:11.669900+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:36.605162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531022+0000","last_change":"2024-03-17T23:49:11.656823+0000","last_active":"2024-03-17T23:49:13.531022+0000","last_peered":"2024-03-17T23:49:13.531022+0000","last_clean":"2024-03-17T23:49:13.531022+0000","last_became_active":"2024-03-17T23:49:11.656546+0000","last_became_peered":"2024-03-17T23:49:11.656546+0000","last_unstale":"2024-03-17T23:49:13.531022+0000","last_undegraded":"2024-03-17T23:49:13.531022+0000","last_fullsized":"2024-03-17T23:49:13.531022+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:22:21.200213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.664427+0000","last_change":"2024-03-17T23:49:10.653909+0000","last_active":"2024-03-17T23:49:11.664427+0000","last_peered":"2024-03-17T23:49:11.664427+0000","last_clean":"2024-03-17T23:49:11.664427+0000","last_became_active":"2024-03-17T23:49:10.653648+0000","last_became_peered":"2024-03-17T23:49:10.653648+0000","last_unstale":"2024-03-17T23:49:11.664427+0000","last_undegraded":"2024-03-17T23:49:11.664427+0000","last_fullsized":"2024-03-17T23:49:11.664427+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:04.395229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531539+0000","last_change":"2024-03-17T23:49:11.659042+0000","last_active":"2024-03-17T23:49:13.531539+0000","last_peered":"2024-03-17T23:49:13.531539+0000","last_clean":"2024-03-17T23:49:13.531539+0000","last_became_active":"2024-03-17T23:49:11.658786+0000","last_became_peered":"2024-03-17T23:49:11.658786+0000","last_unstale":"2024-03-17T23:49:13.531539+0000","last_undegraded":"2024-03-17T23:49:13.531539+0000","last_fullsized":"2024-03-17T23:49:13.531539+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:09:46.753563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.656898+0000","last_change":"2024-03-17T23:49:10.645546+0000","last_active":"2024-03-17T23:49:12.656898+0000","last_peered":"2024-03-17T23:49:12.656898+0000","last_clean":"2024-03-17T23:49:12.656898+0000","last_became_active":"2024-03-17T23:49:10.645217+0000","last_became_peered":"2024-03-17T23:49:10.645217+0000","last_unstale":"2024-03-17T23:49:12.656898+0000","last_undegraded":"2024-03-17T23:49:12.656898+0000","last_fullsized":"2024-03-17T23:49:12.656898+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:48:26.830412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.664451+0000","last_change":"2024-03-17T23:49:11.655809+0000","last_active":"2024-03-17T23:49:11.664451+0000","last_peered":"2024-03-17T23:49:11.664451+0000","last_clean":"2024-03-17T23:49:11.664451+0000","last_became_active":"2024-03-17T23:49:11.655510+0000","last_became_peered":"2024-03-17T23:49:11.655510+0000","last_unstale":"2024-03-17T23:49:11.664451+0000","last_undegraded":"2024-03-17T23:49:11.664451+0000","last_fullsized":"2024-03-17T23:49:11.664451+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:59:37.841369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.648060+0000","last_change":"2024-03-17T23:49:10.656254+0000","last_active":"2024-03-17T23:49:11.648060+0000","last_peered":"2024-03-17T23:49:11.648060+0000","last_clean":"2024-03-17T23:49:11.648060+0000","last_became_active":"2024-03-17T23:49:10.655915+0000","last_became_peered":"2024-03-17T23:49:10.655915+0000","last_unstale":"2024-03-17T23:49:11.648060+0000","last_undegraded":"2024-03-17T23:49:11.648060+0000","last_fullsized":"2024-03-17T23:49:11.648060+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:26:48.759181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531727+0000","last_change":"2024-03-17T23:49:11.654251+0000","last_active":"2024-03-17T23:49:13.531727+0000","last_peered":"2024-03-17T23:49:13.531727+0000","last_clean":"2024-03-17T23:49:13.531727+0000","last_became_active":"2024-03-17T23:49:11.654004+0000","last_became_peered":"2024-03-17T23:49:11.654004+0000","last_unstale":"2024-03-17T23:49:13.531727+0000","last_undegraded":"2024-03-17T23:49:13.531727+0000","last_fullsized":"2024-03-17T23:49:13.531727+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:39:15.399081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.643323+0000","last_change":"2024-03-17T23:49:10.649005+0000","last_active":"2024-03-17T23:49:11.643323+0000","last_peered":"2024-03-17T23:49:11.643323+0000","last_clean":"2024-03-17T23:49:11.643323+0000","last_became_active":"2024-03-17T23:49:10.648717+0000","last_became_peered":"2024-03-17T23:49:10.648717+0000","last_unstale":"2024-03-17T23:49:11.643323+0000","last_undegraded":"2024-03-17T23:49:11.643323+0000","last_fullsized":"2024-03-17T23:49:11.643323+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:26:48.759181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659768+0000","last_change":"2024-03-17T23:49:11.658818+0000","last_active":"2024-03-17T23:49:11.659768+0000","last_peered":"2024-03-17T23:49:11.659768+0000","last_clean":"2024-03-17T23:49:11.659768+0000","last_became_active":"2024-03-17T23:49:11.658546+0000","last_became_peered":"2024-03-17T23:49:11.658546+0000","last_unstale":"2024-03-17T23:49:11.659768+0000","last_undegraded":"2024-03-17T23:49:11.659768+0000","last_fullsized":"2024-03-17T23:49:11.659768+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:15.111002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.648333+0000","last_change":"2024-03-17T23:49:10.658240+0000","last_active":"2024-03-17T23:49:11.648333+0000","last_peered":"2024-03-17T23:49:11.648333+0000","last_clean":"2024-03-17T23:49:11.648333+0000","last_became_active":"2024-03-17T23:49:10.657882+0000","last_became_peered":"2024-03-17T23:49:10.657882+0000","last_unstale":"2024-03-17T23:49:11.648333+0000","last_undegraded":"2024-03-17T23:49:11.648333+0000","last_fullsized":"2024-03-17T23:49:11.648333+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:16:37.292086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.665442+0000","last_change":"2024-03-17T23:49:11.660908+0000","last_active":"2024-03-17T23:49:11.665442+0000","last_peered":"2024-03-17T23:49:11.665442+0000","last_clean":"2024-03-17T23:49:11.665442+0000","last_became_active":"2024-03-17T23:49:11.660644+0000","last_became_peered":"2024-03-17T23:49:11.660644+0000","last_unstale":"2024-03-17T23:49:11.665442+0000","last_undegraded":"2024-03-17T23:49:11.665442+0000","last_fullsized":"2024-03-17T23:49:11.665442+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:24:35.182836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531096+0000","last_change":"2024-03-17T23:49:10.655338+0000","last_active":"2024-03-17T23:49:13.531096+0000","last_peered":"2024-03-17T23:49:13.531096+0000","last_clean":"2024-03-17T23:49:13.531096+0000","last_became_active":"2024-03-17T23:49:10.655070+0000","last_became_peered":"2024-03-17T23:49:10.655070+0000","last_unstale":"2024-03-17T23:49:13.531096+0000","last_undegraded":"2024-03-17T23:49:13.531096+0000","last_fullsized":"2024-03-17T23:49:13.531096+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:41:42.501913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659276+0000","last_change":"2024-03-17T23:49:11.657785+0000","last_active":"2024-03-17T23:49:11.659276+0000","last_peered":"2024-03-17T23:49:11.659276+0000","last_clean":"2024-03-17T23:49:11.659276+0000","last_became_active":"2024-03-17T23:49:11.657040+0000","last_became_peered":"2024-03-17T23:49:11.657040+0000","last_unstale":"2024-03-17T23:49:11.659276+0000","last_undegraded":"2024-03-17T23:49:11.659276+0000","last_fullsized":"2024-03-17T23:49:11.659276+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:05.399901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"57'3","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.672409+0000","last_change":"2024-03-17T23:49:10.656304+0000","last_active":"2024-03-17T23:49:11.672409+0000","last_peered":"2024-03-17T23:49:11.672409+0000","last_clean":"2024-03-17T23:49:11.672409+0000","last_became_active":"2024-03-17T23:49:10.655795+0000","last_became_peered":"2024-03-17T23:49:10.655795+0000","last_unstale":"2024-03-17T23:49:11.672409+0000","last_undegraded":"2024-03-17T23:49:11.672409+0000","last_fullsized":"2024-03-17T23:49:11.672409+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:57:35.440980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.663919+0000","last_change":"2024-03-17T23:49:11.660244+0000","last_active":"2024-03-17T23:49:11.663919+0000","last_peered":"2024-03-17T23:49:11.663919+0000","last_clean":"2024-03-17T23:49:11.663919+0000","last_became_active":"2024-03-17T23:49:11.659982+0000","last_became_peered":"2024-03-17T23:49:11.659982+0000","last_unstale":"2024-03-17T23:49:11.663919+0000","last_undegraded":"2024-03-17T23:49:11.663919+0000","last_fullsized":"2024-03-17T23:49:11.663919+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:05:20.724503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"57'6","reported_seq":18,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.675052+0000","last_change":"2024-03-17T23:49:10.650050+0000","last_active":"2024-03-17T23:49:11.675052+0000","last_peered":"2024-03-17T23:49:11.675052+0000","last_clean":"2024-03-17T23:49:11.675052+0000","last_became_active":"2024-03-17T23:49:10.649505+0000","last_became_peered":"2024-03-17T23:49:10.649505+0000","last_unstale":"2024-03-17T23:49:11.675052+0000","last_undegraded":"2024-03-17T23:49:11.675052+0000","last_fullsized":"2024-03-17T23:49:11.675052+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:24:34.178164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.662957+0000","last_change":"2024-03-17T23:49:11.659236+0000","last_active":"2024-03-17T23:49:11.662957+0000","last_peered":"2024-03-17T23:49:11.662957+0000","last_clean":"2024-03-17T23:49:11.662957+0000","last_became_active":"2024-03-17T23:49:11.658962+0000","last_became_peered":"2024-03-17T23:49:11.658962+0000","last_unstale":"2024-03-17T23:49:11.662957+0000","last_undegraded":"2024-03-17T23:49:11.662957+0000","last_fullsized":"2024-03-17T23:49:11.662957+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:48:27.835084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531494+0000","last_change":"2024-03-17T23:49:10.657312+0000","last_active":"2024-03-17T23:49:13.531494+0000","last_peered":"2024-03-17T23:49:13.531494+0000","last_clean":"2024-03-17T23:49:13.531494+0000","last_became_active":"2024-03-17T23:49:10.657002+0000","last_became_peered":"2024-03-17T23:49:10.657002+0000","last_unstale":"2024-03-17T23:49:13.531494+0000","last_undegraded":"2024-03-17T23:49:13.531494+0000","last_fullsized":"2024-03-17T23:49:13.531494+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:19:39.198683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.670115+0000","last_change":"2024-03-17T23:49:11.667510+0000","last_active":"2024-03-17T23:49:11.670115+0000","last_peered":"2024-03-17T23:49:11.670115+0000","last_clean":"2024-03-17T23:49:11.670115+0000","last_became_active":"2024-03-17T23:49:11.665443+0000","last_became_peered":"2024-03-17T23:49:11.665443+0000","last_unstale":"2024-03-17T23:49:11.670115+0000","last_undegraded":"2024-03-17T23:49:11.670115+0000","last_fullsized":"2024-03-17T23:49:11.670115+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:35.755434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.642062+0000","last_change":"2024-03-17T23:49:10.650246+0000","last_active":"2024-03-17T23:49:11.642062+0000","last_peered":"2024-03-17T23:49:11.642062+0000","last_clean":"2024-03-17T23:49:11.642062+0000","last_became_active":"2024-03-17T23:49:10.649837+0000","last_became_peered":"2024-03-17T23:49:10.649837+0000","last_unstale":"2024-03-17T23:49:11.642062+0000","last_undegraded":"2024-03-17T23:49:11.642062+0000","last_fullsized":"2024-03-17T23:49:11.642062+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:08:24.371365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659904+0000","last_change":"2024-03-17T23:49:11.655903+0000","last_active":"2024-03-17T23:49:11.659904+0000","last_peered":"2024-03-17T23:49:11.659904+0000","last_clean":"2024-03-17T23:49:11.659904+0000","last_became_active":"2024-03-17T23:49:11.655521+0000","last_became_peered":"2024-03-17T23:49:11.655521+0000","last_unstale":"2024-03-17T23:49:11.659904+0000","last_undegraded":"2024-03-17T23:49:11.659904+0000","last_fullsized":"2024-03-17T23:49:11.659904+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:55:32.443067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.648464+0000","last_change":"2024-03-17T23:49:10.651590+0000","last_active":"2024-03-17T23:49:11.648464+0000","last_peered":"2024-03-17T23:49:11.648464+0000","last_clean":"2024-03-17T23:49:11.648464+0000","last_became_active":"2024-03-17T23:49:10.651300+0000","last_became_peered":"2024-03-17T23:49:10.651300+0000","last_unstale":"2024-03-17T23:49:11.648464+0000","last_undegraded":"2024-03-17T23:49:11.648464+0000","last_fullsized":"2024-03-17T23:49:11.648464+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:05:19.719830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659701+0000","last_change":"2024-03-17T23:49:11.657665+0000","last_active":"2024-03-17T23:49:11.659701+0000","last_peered":"2024-03-17T23:49:11.659701+0000","last_clean":"2024-03-17T23:49:11.659701+0000","last_became_active":"2024-03-17T23:49:11.657348+0000","last_became_peered":"2024-03-17T23:49:11.657348+0000","last_unstale":"2024-03-17T23:49:11.659701+0000","last_undegraded":"2024-03-17T23:49:11.659701+0000","last_fullsized":"2024-03-17T23:49:11.659701+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:05.399901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"57'8","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.672231+0000","last_change":"2024-03-17T23:49:10.654069+0000","last_active":"2024-03-17T23:49:11.672231+0000","last_peered":"2024-03-17T23:49:11.672231+0000","last_clean":"2024-03-17T23:49:11.672231+0000","last_became_active":"2024-03-17T23:49:10.653800+0000","last_became_peered":"2024-03-17T23:49:10.653800+0000","last_unstale":"2024-03-17T23:49:11.672231+0000","last_undegraded":"2024-03-17T23:49:11.672231+0000","last_fullsized":"2024-03-17T23:49:11.672231+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:10:04.663334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.670182+0000","last_change":"2024-03-17T23:49:11.667630+0000","last_active":"2024-03-17T23:49:11.670182+0000","last_peered":"2024-03-17T23:49:11.670182+0000","last_clean":"2024-03-17T23:49:11.670182+0000","last_became_active":"2024-03-17T23:49:11.665756+0000","last_became_peered":"2024-03-17T23:49:11.665756+0000","last_unstale":"2024-03-17T23:49:11.670182+0000","last_undegraded":"2024-03-17T23:49:11.670182+0000","last_fullsized":"2024-03-17T23:49:11.670182+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:55:32.443067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531371+0000","last_change":"2024-03-17T23:49:10.655436+0000","last_active":"2024-03-17T23:49:13.531371+0000","last_peered":"2024-03-17T23:49:13.531371+0000","last_clean":"2024-03-17T23:49:13.531371+0000","last_became_active":"2024-03-17T23:49:10.655114+0000","last_became_peered":"2024-03-17T23:49:10.655114+0000","last_unstale":"2024-03-17T23:49:13.531371+0000","last_undegraded":"2024-03-17T23:49:13.531371+0000","last_fullsized":"2024-03-17T23:49:13.531371+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:05:23.558519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659682+0000","last_change":"2024-03-17T23:49:11.657629+0000","last_active":"2024-03-17T23:49:11.659682+0000","last_peered":"2024-03-17T23:49:11.659682+0000","last_clean":"2024-03-17T23:49:11.659682+0000","last_became_active":"2024-03-17T23:49:11.657356+0000","last_became_peered":"2024-03-17T23:49:11.657356+0000","last_unstale":"2024-03-17T23:49:11.659682+0000","last_undegraded":"2024-03-17T23:49:11.659682+0000","last_fullsized":"2024-03-17T23:49:11.659682+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:15.111002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.657242+0000","last_change":"2024-03-17T23:49:10.653706+0000","last_active":"2024-03-17T23:49:12.657242+0000","last_peered":"2024-03-17T23:49:12.657242+0000","last_clean":"2024-03-17T23:49:12.657242+0000","last_became_active":"2024-03-17T23:49:10.653133+0000","last_became_peered":"2024-03-17T23:49:10.653133+0000","last_unstale":"2024-03-17T23:49:12.657242+0000","last_undegraded":"2024-03-17T23:49:12.657242+0000","last_fullsized":"2024-03-17T23:49:12.657242+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:00:04.864289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531727+0000","last_change":"2024-03-17T23:49:11.649960+0000","last_active":"2024-03-17T23:49:13.531727+0000","last_peered":"2024-03-17T23:49:13.531727+0000","last_clean":"2024-03-17T23:49:13.531727+0000","last_became_active":"2024-03-17T23:49:11.649688+0000","last_became_peered":"2024-03-17T23:49:11.649688+0000","last_unstale":"2024-03-17T23:49:13.531727+0000","last_undegraded":"2024-03-17T23:49:13.531727+0000","last_fullsized":"2024-03-17T23:49:13.531727+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:21:14.621063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"57'1","reported_seq":13,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.669750+0000","last_change":"2024-03-17T23:49:10.652457+0000","last_active":"2024-03-17T23:49:11.669750+0000","last_peered":"2024-03-17T23:49:11.669750+0000","last_clean":"2024-03-17T23:49:11.669750+0000","last_became_active":"2024-03-17T23:49:10.652105+0000","last_became_peered":"2024-03-17T23:49:10.652105+0000","last_unstale":"2024-03-17T23:49:11.669750+0000","last_undegraded":"2024-03-17T23:49:11.669750+0000","last_fullsized":"2024-03-17T23:49:11.669750+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:24:34.178164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.665510+0000","last_change":"2024-03-17T23:49:11.661804+0000","last_active":"2024-03-17T23:49:11.665510+0000","last_peered":"2024-03-17T23:49:11.665510+0000","last_clean":"2024-03-17T23:49:11.665510+0000","last_became_active":"2024-03-17T23:49:11.661518+0000","last_became_peered":"2024-03-17T23:49:11.661518+0000","last_unstale":"2024-03-17T23:49:11.665510+0000","last_undegraded":"2024-03-17T23:49:11.665510+0000","last_fullsized":"2024-03-17T23:49:11.665510+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:37.609834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.657185+0000","last_change":"2024-03-17T23:49:10.652713+0000","last_active":"2024-03-17T23:49:12.657185+0000","last_peered":"2024-03-17T23:49:12.657185+0000","last_clean":"2024-03-17T23:49:12.657185+0000","last_became_active":"2024-03-17T23:49:10.652449+0000","last_became_peered":"2024-03-17T23:49:10.652449+0000","last_unstale":"2024-03-17T23:49:12.657185+0000","last_undegraded":"2024-03-17T23:49:12.657185+0000","last_fullsized":"2024-03-17T23:49:12.657185+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:34.750762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.662996+0000","last_change":"2024-03-17T23:49:11.659323+0000","last_active":"2024-03-17T23:49:11.662996+0000","last_peered":"2024-03-17T23:49:11.662996+0000","last_clean":"2024-03-17T23:49:11.662996+0000","last_became_active":"2024-03-17T23:49:11.659047+0000","last_became_peered":"2024-03-17T23:49:11.659047+0000","last_unstale":"2024-03-17T23:49:11.662996+0000","last_undegraded":"2024-03-17T23:49:11.662996+0000","last_fullsized":"2024-03-17T23:49:11.662996+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:11:49.300093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531663+0000","last_change":"2024-03-17T23:49:10.653875+0000","last_active":"2024-03-17T23:49:13.531663+0000","last_peered":"2024-03-17T23:49:13.531663+0000","last_clean":"2024-03-17T23:49:13.531663+0000","last_became_active":"2024-03-17T23:49:10.653617+0000","last_became_peered":"2024-03-17T23:49:10.653617+0000","last_unstale":"2024-03-17T23:49:13.531663+0000","last_undegraded":"2024-03-17T23:49:13.531663+0000","last_fullsized":"2024-03-17T23:49:13.531663+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:09:03.308479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.657098+0000","last_change":"2024-03-17T23:49:11.655254+0000","last_active":"2024-03-17T23:49:12.657098+0000","last_peered":"2024-03-17T23:49:12.657098+0000","last_clean":"2024-03-17T23:49:12.657098+0000","last_became_active":"2024-03-17T23:49:11.654992+0000","last_became_peered":"2024-03-17T23:49:11.654992+0000","last_unstale":"2024-03-17T23:49:12.657098+0000","last_undegraded":"2024-03-17T23:49:12.657098+0000","last_fullsized":"2024-03-17T23:49:12.657098+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:19:40.203356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.639827+0000","last_change":"2024-03-17T23:49:10.655445+0000","last_active":"2024-03-17T23:49:11.639827+0000","last_peered":"2024-03-17T23:49:11.639827+0000","last_clean":"2024-03-17T23:49:11.639827+0000","last_became_active":"2024-03-17T23:49:10.654632+0000","last_became_peered":"2024-03-17T23:49:10.654632+0000","last_unstale":"2024-03-17T23:49:11.639827+0000","last_undegraded":"2024-03-17T23:49:11.639827+0000","last_fullsized":"2024-03-17T23:49:11.639827+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:54:00.207576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.656848+0000","last_change":"2024-03-17T23:49:11.666849+0000","last_active":"2024-03-17T23:49:12.656848+0000","last_peered":"2024-03-17T23:49:12.656848+0000","last_clean":"2024-03-17T23:49:12.656848+0000","last_became_active":"2024-03-17T23:49:11.666568+0000","last_became_peered":"2024-03-17T23:49:11.666568+0000","last_unstale":"2024-03-17T23:49:12.656848+0000","last_undegraded":"2024-03-17T23:49:12.656848+0000","last_fullsized":"2024-03-17T23:49:12.656848+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:58:45.165907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.647483+0000","last_change":"2024-03-17T23:49:10.654321+0000","last_active":"2024-03-17T23:49:11.647483+0000","last_peered":"2024-03-17T23:49:11.647483+0000","last_clean":"2024-03-17T23:49:11.647483+0000","last_became_active":"2024-03-17T23:49:10.653691+0000","last_became_peered":"2024-03-17T23:49:10.653691+0000","last_unstale":"2024-03-17T23:49:11.647483+0000","last_undegraded":"2024-03-17T23:49:11.647483+0000","last_fullsized":"2024-03-17T23:49:11.647483+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:04.395229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.663081+0000","last_change":"2024-03-17T23:49:11.660090+0000","last_active":"2024-03-17T23:49:11.663081+0000","last_peered":"2024-03-17T23:49:11.663081+0000","last_clean":"2024-03-17T23:49:11.663081+0000","last_became_active":"2024-03-17T23:49:11.659121+0000","last_became_peered":"2024-03-17T23:49:11.659121+0000","last_unstale":"2024-03-17T23:49:11.663081+0000","last_undegraded":"2024-03-17T23:49:11.663081+0000","last_fullsized":"2024-03-17T23:49:11.663081+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:23:45.083399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.675037+0000","last_change":"2024-03-17T23:49:10.655569+0000","last_active":"2024-03-17T23:49:11.675037+0000","last_peered":"2024-03-17T23:49:11.675037+0000","last_clean":"2024-03-17T23:49:11.675037+0000","last_became_active":"2024-03-17T23:49:10.655284+0000","last_became_peered":"2024-03-17T23:49:10.655284+0000","last_unstale":"2024-03-17T23:49:11.675037+0000","last_undegraded":"2024-03-17T23:49:11.675037+0000","last_fullsized":"2024-03-17T23:49:11.675037+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:07:07.135215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659651+0000","last_change":"2024-03-17T23:49:11.655737+0000","last_active":"2024-03-17T23:49:11.659651+0000","last_peered":"2024-03-17T23:49:11.659651+0000","last_clean":"2024-03-17T23:49:11.659651+0000","last_became_active":"2024-03-17T23:49:11.655464+0000","last_became_peered":"2024-03-17T23:49:11.655464+0000","last_unstale":"2024-03-17T23:49:11.659651+0000","last_undegraded":"2024-03-17T23:49:11.659651+0000","last_fullsized":"2024-03-17T23:49:11.659651+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T23:59:25.825110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"58'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.658418+0000","last_change":"2024-03-17T23:49:10.649196+0000","last_active":"2024-03-17T23:49:12.658418+0000","last_peered":"2024-03-17T23:49:12.658418+0000","last_clean":"2024-03-17T23:49:12.658418+0000","last_became_active":"2024-03-17T23:49:10.648830+0000","last_became_peered":"2024-03-17T23:49:10.648830+0000","last_unstale":"2024-03-17T23:49:12.658418+0000","last_undegraded":"2024-03-17T23:49:12.658418+0000","last_fullsized":"2024-03-17T23:49:12.658418+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:55:31.438395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.652228+0000","last_change":"2024-03-17T23:49:11.649056+0000","last_active":"2024-03-17T23:49:11.652228+0000","last_peered":"2024-03-17T23:49:11.652228+0000","last_clean":"2024-03-17T23:49:11.652228+0000","last_became_active":"2024-03-17T23:49:11.648791+0000","last_became_peered":"2024-03-17T23:49:11.648791+0000","last_unstale":"2024-03-17T23:49:11.652228+0000","last_undegraded":"2024-03-17T23:49:11.652228+0000","last_fullsized":"2024-03-17T23:49:11.652228+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:09:15.871249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.642103+0000","last_change":"2024-03-17T23:49:10.648679+0000","last_active":"2024-03-17T23:49:11.642103+0000","last_peered":"2024-03-17T23:49:11.642103+0000","last_clean":"2024-03-17T23:49:11.642103+0000","last_became_active":"2024-03-17T23:49:10.648407+0000","last_became_peered":"2024-03-17T23:49:10.648407+0000","last_unstale":"2024-03-17T23:49:11.642103+0000","last_undegraded":"2024-03-17T23:49:11.642103+0000","last_fullsized":"2024-03-17T23:49:11.642103+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:38:18.229949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.657294+0000","last_change":"2024-03-17T23:49:11.659172+0000","last_active":"2024-03-17T23:49:12.657294+0000","last_peered":"2024-03-17T23:49:12.657294+0000","last_clean":"2024-03-17T23:49:12.657294+0000","last_became_active":"2024-03-17T23:49:11.658577+0000","last_became_peered":"2024-03-17T23:49:11.658577+0000","last_unstale":"2024-03-17T23:49:12.657294+0000","last_undegraded":"2024-03-17T23:49:12.657294+0000","last_fullsized":"2024-03-17T23:49:12.657294+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:05:24.563191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.649926+0000","last_change":"2024-03-17T23:49:10.655937+0000","last_active":"2024-03-17T23:49:11.649926+0000","last_peered":"2024-03-17T23:49:11.649926+0000","last_clean":"2024-03-17T23:49:11.649926+0000","last_became_active":"2024-03-17T23:49:10.654317+0000","last_became_peered":"2024-03-17T23:49:10.654317+0000","last_unstale":"2024-03-17T23:49:11.649926+0000","last_undegraded":"2024-03-17T23:49:11.649926+0000","last_fullsized":"2024-03-17T23:49:11.649926+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:34.750762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531239+0000","last_change":"2024-03-17T23:49:11.660013+0000","last_active":"2024-03-17T23:49:13.531239+0000","last_peered":"2024-03-17T23:49:13.531239+0000","last_clean":"2024-03-17T23:49:13.531239+0000","last_became_active":"2024-03-17T23:49:11.659686+0000","last_became_peered":"2024-03-17T23:49:11.659686+0000","last_unstale":"2024-03-17T23:49:13.531239+0000","last_undegraded":"2024-03-17T23:49:13.531239+0000","last_fullsized":"2024-03-17T23:49:13.531239+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:50:38.600565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.639931+0000","last_change":"2024-03-17T23:49:10.655613+0000","last_active":"2024-03-17T23:49:11.639931+0000","last_peered":"2024-03-17T23:49:11.639931+0000","last_clean":"2024-03-17T23:49:11.639931+0000","last_became_active":"2024-03-17T23:49:10.655134+0000","last_became_peered":"2024-03-17T23:49:10.655134+0000","last_unstale":"2024-03-17T23:49:11.639931+0000","last_undegraded":"2024-03-17T23:49:11.639931+0000","last_fullsized":"2024-03-17T23:49:11.639931+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:48:38.320666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.660923+0000","last_change":"2024-03-17T23:49:11.656942+0000","last_active":"2024-03-17T23:49:11.660923+0000","last_peered":"2024-03-17T23:49:11.660923+0000","last_clean":"2024-03-17T23:49:11.660923+0000","last_became_active":"2024-03-17T23:49:11.656612+0000","last_became_peered":"2024-03-17T23:49:11.656612+0000","last_unstale":"2024-03-17T23:49:11.660923+0000","last_undegraded":"2024-03-17T23:49:11.660923+0000","last_fullsized":"2024-03-17T23:49:11.660923+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:21:49.823973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.643561+0000","last_change":"2024-03-17T23:49:10.647432+0000","last_active":"2024-03-17T23:49:11.643561+0000","last_peered":"2024-03-17T23:49:11.643561+0000","last_clean":"2024-03-17T23:49:11.643561+0000","last_became_active":"2024-03-17T23:49:10.647160+0000","last_became_peered":"2024-03-17T23:49:10.647160+0000","last_unstale":"2024-03-17T23:49:11.643561+0000","last_undegraded":"2024-03-17T23:49:11.643561+0000","last_fullsized":"2024-03-17T23:49:11.643561+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:05:19.719830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.651916+0000","last_change":"2024-03-17T23:49:11.645350+0000","last_active":"2024-03-17T23:49:11.651916+0000","last_peered":"2024-03-17T23:49:11.651916+0000","last_clean":"2024-03-17T23:49:11.651916+0000","last_became_active":"2024-03-17T23:49:11.645101+0000","last_became_peered":"2024-03-17T23:49:11.645101+0000","last_unstale":"2024-03-17T23:49:11.651916+0000","last_undegraded":"2024-03-17T23:49:11.651916+0000","last_fullsized":"2024-03-17T23:49:11.651916+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:57:36.445652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.639772+0000","last_change":"2024-03-17T23:49:10.655236+0000","last_active":"2024-03-17T23:49:11.639772+0000","last_peered":"2024-03-17T23:49:11.639772+0000","last_clean":"2024-03-17T23:49:11.639772+0000","last_became_active":"2024-03-17T23:49:10.654952+0000","last_became_peered":"2024-03-17T23:49:10.654952+0000","last_unstale":"2024-03-17T23:49:11.639772+0000","last_undegraded":"2024-03-17T23:49:11.639772+0000","last_fullsized":"2024-03-17T23:49:11.639772+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:59:36.836697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531791+0000","last_change":"2024-03-17T23:49:11.652883+0000","last_active":"2024-03-17T23:49:13.531791+0000","last_peered":"2024-03-17T23:49:13.531791+0000","last_clean":"2024-03-17T23:49:13.531791+0000","last_became_active":"2024-03-17T23:49:11.652606+0000","last_became_peered":"2024-03-17T23:49:11.652606+0000","last_unstale":"2024-03-17T23:49:13.531791+0000","last_undegraded":"2024-03-17T23:49:13.531791+0000","last_fullsized":"2024-03-17T23:49:13.531791+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:35:26.662410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531549+0000","last_change":"2024-03-17T23:49:10.654224+0000","last_active":"2024-03-17T23:49:13.531549+0000","last_peered":"2024-03-17T23:49:13.531549+0000","last_clean":"2024-03-17T23:49:13.531549+0000","last_became_active":"2024-03-17T23:49:10.653909+0000","last_became_peered":"2024-03-17T23:49:10.653909+0000","last_unstale":"2024-03-17T23:49:13.531549+0000","last_undegraded":"2024-03-17T23:49:13.531549+0000","last_fullsized":"2024-03-17T23:49:13.531549+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:37:55.892299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531518+0000","last_change":"2024-03-17T23:49:11.658312+0000","last_active":"2024-03-17T23:49:13.531518+0000","last_peered":"2024-03-17T23:49:13.531518+0000","last_clean":"2024-03-17T23:49:13.531518+0000","last_became_active":"2024-03-17T23:49:11.657765+0000","last_became_peered":"2024-03-17T23:49:11.657765+0000","last_unstale":"2024-03-17T23:49:13.531518+0000","last_undegraded":"2024-03-17T23:49:13.531518+0000","last_fullsized":"2024-03-17T23:49:13.531518+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:55:22.554224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.641904+0000","last_change":"2024-03-17T23:49:10.653100+0000","last_active":"2024-03-17T23:49:11.641904+0000","last_peered":"2024-03-17T23:49:11.641904+0000","last_clean":"2024-03-17T23:49:11.641904+0000","last_became_active":"2024-03-17T23:49:10.652809+0000","last_became_peered":"2024-03-17T23:49:10.652809+0000","last_unstale":"2024-03-17T23:49:11.641904+0000","last_undegraded":"2024-03-17T23:49:11.641904+0000","last_fullsized":"2024-03-17T23:49:11.641904+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:51:40.737241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":238,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.645564+0000","last_change":"2024-03-17T23:46:44.768668+0000","last_active":"2024-03-17T23:49:11.645564+0000","last_peered":"2024-03-17T23:49:11.645564+0000","last_clean":"2024-03-17T23:49:11.645564+0000","last_became_active":"2024-03-17T23:46:44.767826+0000","last_became_peered":"2024-03-17T23:46:44.767826+0000","last_unstale":"2024-03-17T23:49:11.645564+0000","last_undegraded":"2024-03-17T23:49:11.645564+0000","last_fullsized":"2024-03-17T23:49:11.645564+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_clean_scrub_stamp":"2024-03-17T23:46:43.744878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:54:22.286155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":14,"seq":60129542178,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":688128,"data_stored":309517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542178,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27708,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344132,"statfs":{"total":96636764160,"available":96608391168,"internally_reserved":0,"allocated":700416,"data_stored":316209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542178,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28296,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343544,"statfs":{"total":96636764160,"available":96607789056,"internally_reserved":0,"allocated":1302528,"data_stored":908715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574881,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32432,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339408,"statfs":{"total":96636764160,"available":96603553792,"internally_reserved":0,"allocated":1282048,"data_stored":899989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31844,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339996,"statfs":{"total":96636764160,"available":96604155904,"internally_reserved":0,"allocated":679936,"data_stored":309461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574881,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":688128,"data_stored":310113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574881,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":692224,"data_stored":311073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":684032,"data_stored":309551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-17T23:49:16.631 INFO:tasks.ceph.mds.b.smithi157.stderr:2024-03-17T23:49:16.631+0000 7fa205d37640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-17T23:49:19.019 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-17T23:49:19.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.130+0000 7f449bca0640 1 Processor -- start 2024-03-17T23:49:19.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.131+0000 7f449bca0640 1 -- start start 2024-03-17T23:49:19.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.132+0000 7f449bca0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f449406f400 0x7f44940cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:19.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.132+0000 7f449bca0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f44940ce4d0 0x7f4494165a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:19.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.132+0000 7f449bca0640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4494145170 con 0x7f4494144bb0 2024-03-17T23:49:19.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.132+0000 7f449bca0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f44940ce920 con 0x7f44940ce4d0 2024-03-17T23:49:19.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.132+0000 7f449bca0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4494145450 con 0x7f449406f400 2024-03-17T23:49:19.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.132+0000 7f449a23d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f44940ce4d0 0x7f4494165a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:19.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.132+0000 7f449a23d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f44940ce4d0 0x7f4494165a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45526/0 (socket says 172.21.15.150:45526) 2024-03-17T23:49:19.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.132+0000 7f449a23d640 1 -- 172.21.15.150:0/1267313417 learned_addr learned my addr 172.21.15.150:0/1267313417 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:19.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.132+0000 7f449923b640 1 --2- 172.21.15.150:0/1267313417 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f449406f400 0x7f44940cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:19.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.132+0000 7f449a23d640 1 -- 172.21.15.150:0/1267313417 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f449406f400 msgr2=0x7f44940cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:19.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.132+0000 7f449a23d640 1 --2- 172.21.15.150:0/1267313417 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f449406f400 0x7f44940cdf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:19.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.132+0000 7f449a23d640 1 -- 172.21.15.150:0/1267313417 >> v1:172.21.15.157:6789/0 conn(0x7f4494144bb0 legacy=0x7f449415ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:19.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.132+0000 7f449a23d640 1 -- 172.21.15.150:0/1267313417 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4494145730 con 0x7f44940ce4d0 2024-03-17T23:49:19.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.132+0000 7f4498a3a640 1 -- 172.21.15.150:0/1267313417 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1742815102 0 0) 0x7f4494145170 con 0x7f4494144bb0 2024-03-17T23:49:19.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.133+0000 7f449a23d640 1 --2- 172.21.15.150:0/1267313417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44940ce4d0 0x7f4494165a30 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f4484009870 tx=0x7f448402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce445fb69cc17252 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:19.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.133+0000 7f4498a3a640 1 -- 172.21.15.150:0/1267313417 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4484035020 con 0x7f44940ce4d0 2024-03-17T23:49:19.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.133+0000 7f4498a3a640 1 -- 172.21.15.150:0/1267313417 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4484031640 con 0x7f44940ce4d0 2024-03-17T23:49:19.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.133+0000 7f4498a3a640 1 -- 172.21.15.150:0/1267313417 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f448402d040 con 0x7f44940ce4d0 2024-03-17T23:49:19.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.134+0000 7f449bca0640 1 -- 172.21.15.150:0/1267313417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44940ce4d0 msgr2=0x7f4494165a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:19.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.134+0000 7f449bca0640 1 --2- 172.21.15.150:0/1267313417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44940ce4d0 0x7f4494165a30 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f4484009870 tx=0x7f448402b8f0 comp rx=0 tx=0).stop 2024-03-17T23:49:19.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.134+0000 7f449bca0640 1 -- 172.21.15.150:0/1267313417 shutdown_connections 2024-03-17T23:49:19.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.134+0000 7f449bca0640 1 --2- 172.21.15.150:0/1267313417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f44940ce4d0 0x7f4494165a30 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:19.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.134+0000 7f449bca0640 1 --2- 172.21.15.150:0/1267313417 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f449406f400 0x7f44940cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:19.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.134+0000 7f449bca0640 1 -- 172.21.15.150:0/1267313417 >> 172.21.15.150:0/1267313417 conn(0x7f449406b190 msgr2=0x7f449406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:19.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.134+0000 7f449bca0640 1 -- 172.21.15.150:0/1267313417 shutdown_connections 2024-03-17T23:49:19.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.134+0000 7f449bca0640 1 -- 172.21.15.150:0/1267313417 wait complete. 2024-03-17T23:49:19.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.135+0000 7f449bca0640 1 Processor -- start 2024-03-17T23:49:19.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.135+0000 7f449bca0640 1 -- start start 2024-03-17T23:49:19.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.136+0000 7f449bca0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f449406f400 0x7f44940ae700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:19.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.136+0000 7f449bca0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44940ce4d0 0x7f44940ad070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:19.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.136+0000 7f449bca0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4494144bb0 0x7f4494139e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:19.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.136+0000 7f449bca0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f449406bee0 con 0x7f4494144bb0 2024-03-17T23:49:19.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.136+0000 7f449bca0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f449406bd60 con 0x7f449406f400 2024-03-17T23:49:19.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.136+0000 7f449bca0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4494165f70 con 0x7f44940ce4d0 2024-03-17T23:49:19.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.136+0000 7f449a23d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4494144bb0 0x7f4494139e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:19.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.136+0000 7f449a23d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4494144bb0 0x7f4494139e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45542/0 (socket says 172.21.15.150:45542) 2024-03-17T23:49:19.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.136+0000 7f449923b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44940ce4d0 0x7f44940ad070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:19.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.136+0000 7f449a23d640 1 -- 172.21.15.150:0/3093007108 learned_addr learned my addr 172.21.15.150:0/3093007108 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:19.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.136+0000 7f4499a3c640 1 --2- 172.21.15.150:0/3093007108 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f449406f400 0x7f44940ae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:19.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.136+0000 7f449923b640 1 -- 172.21.15.150:0/3093007108 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f449406f400 msgr2=0x7f44940ae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:19.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.136+0000 7f449923b640 1 --2- 172.21.15.150:0/3093007108 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f449406f400 0x7f44940ae700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:19.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.136+0000 7f449923b640 1 -- 172.21.15.150:0/3093007108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4494144bb0 msgr2=0x7f4494139e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:19.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.136+0000 7f449923b640 1 --2- 172.21.15.150:0/3093007108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4494144bb0 0x7f4494139e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:19.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.136+0000 7f449923b640 1 -- 172.21.15.150:0/3093007108 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44940ad7c0 con 0x7f44940ce4d0 2024-03-17T23:49:19.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.136+0000 7f449a23d640 1 --2- 172.21.15.150:0/3093007108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4494144bb0 0x7f4494139e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:19.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.136+0000 7f449923b640 1 --2- 172.21.15.150:0/3093007108 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44940ce4d0 0x7f44940ad070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f448800e850 tx=0x7f448800ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:19.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.136+0000 7f44927fc640 1 -- 172.21.15.150:0/3093007108 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4488017070 con 0x7f44940ce4d0 2024-03-17T23:49:19.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.136+0000 7f44927fc640 1 -- 172.21.15.150:0/3093007108 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f448800c940 con 0x7f44940ce4d0 2024-03-17T23:49:19.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.136+0000 7f44927fc640 1 -- 172.21.15.150:0/3093007108 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f448800cd10 con 0x7f44940ce4d0 2024-03-17T23:49:19.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.136+0000 7f449bca0640 1 -- 172.21.15.150:0/3093007108 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44940adaa0 con 0x7f44940ce4d0 2024-03-17T23:49:19.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.136+0000 7f449bca0640 1 -- 172.21.15.150:0/3093007108 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f44940b2d60 con 0x7f44940ce4d0 2024-03-17T23:49:19.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.137+0000 7f449bca0640 1 -- 172.21.15.150:0/3093007108 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44940681a0 con 0x7f44940ce4d0 2024-03-17T23:49:19.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.138+0000 7f44927fc640 1 -- 172.21.15.150:0/3093007108 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f448801a070 con 0x7f44940ce4d0 2024-03-17T23:49:19.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.138+0000 7f44927fc640 1 --2- 172.21.15.150:0/3093007108 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f448006bbd0 0x7f448006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:19.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.138+0000 7f44927fc640 1 -- 172.21.15.150:0/3093007108 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f448808add0 con 0x7f44940ce4d0 2024-03-17T23:49:19.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.138+0000 7f4499a3c640 1 --2- 172.21.15.150:0/3093007108 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f448006bbd0 0x7f448006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:19.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.139+0000 7f4499a3c640 1 --2- 172.21.15.150:0/3093007108 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f448006bbd0 0x7f448006e090 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f448c0012e0 tx=0x7f448c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:19.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.143+0000 7f44927fc640 1 -- 172.21.15.150:0/3093007108 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4488059170 con 0x7f44940ce4d0 2024-03-17T23:49:19.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.312+0000 7f449bca0640 1 -- 172.21.15.150:0/3093007108 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f449413a370 con 0x7f448006bbd0 2024-03-17T23:49:19.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.318+0000 7f44927fc640 1 -- 172.21.15.150:0/3093007108 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251901 (secure 0 0 0) 0x7f449413a370 con 0x7f448006bbd0 2024-03-17T23:49:19.321 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:19.323 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-03-17T23:49:19.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.321+0000 7f449bca0640 1 -- 172.21.15.150:0/3093007108 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f448006bbd0 msgr2=0x7f448006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:19.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.321+0000 7f449bca0640 1 --2- 172.21.15.150:0/3093007108 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f448006bbd0 0x7f448006e090 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f448c0012e0 tx=0x7f448c009040 comp rx=0 tx=0).stop 2024-03-17T23:49:19.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.321+0000 7f449bca0640 1 -- 172.21.15.150:0/3093007108 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44940ce4d0 msgr2=0x7f44940ad070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:19.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.321+0000 7f449bca0640 1 --2- 172.21.15.150:0/3093007108 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44940ce4d0 0x7f44940ad070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f448800e850 tx=0x7f448800ed10 comp rx=0 tx=0).stop 2024-03-17T23:49:19.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.321+0000 7f449bca0640 1 -- 172.21.15.150:0/3093007108 shutdown_connections 2024-03-17T23:49:19.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.321+0000 7f449bca0640 1 --2- 172.21.15.150:0/3093007108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4494144bb0 0x7f4494139e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:19.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.321+0000 7f449bca0640 1 --2- 172.21.15.150:0/3093007108 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f44940ce4d0 0x7f44940ad070 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:19.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.321+0000 7f449bca0640 1 --2- 172.21.15.150:0/3093007108 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f448006bbd0 0x7f448006e090 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:19.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.321+0000 7f449bca0640 1 --2- 172.21.15.150:0/3093007108 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f449406f400 0x7f44940ae700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:19.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.321+0000 7f449bca0640 1 -- 172.21.15.150:0/3093007108 >> 172.21.15.150:0/3093007108 conn(0x7f449406b190 msgr2=0x7f44940b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:19.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.321+0000 7f449bca0640 1 -- 172.21.15.150:0/3093007108 shutdown_connections 2024-03-17T23:49:19.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.321+0000 7f449bca0640 1 -- 172.21.15.150:0/3093007108 wait complete. 2024-03-17T23:49:19.333 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-03-17T23:49:17.625168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":174,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239416,"kb_used_data":6596,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735304,"statfs":{"total":773094113280,"available":772848951296,"internally_reserved":0,"allocated":6754304,"data_stored":3694704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.988124"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.652693+0000","last_change":"2024-03-17T23:49:11.657789+0000","last_active":"2024-03-17T23:49:12.652693+0000","last_peered":"2024-03-17T23:49:12.652693+0000","last_clean":"2024-03-17T23:49:12.652693+0000","last_became_active":"2024-03-17T23:49:11.657538+0000","last_became_peered":"2024-03-17T23:49:11.657538+0000","last_unstale":"2024-03-17T23:49:12.652693+0000","last_undegraded":"2024-03-17T23:49:12.652693+0000","last_fullsized":"2024-03-17T23:49:12.652693+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:57:19.872599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531008+0000","last_change":"2024-03-17T23:49:11.653495+0000","last_active":"2024-03-17T23:49:13.531008+0000","last_peered":"2024-03-17T23:49:13.531008+0000","last_clean":"2024-03-17T23:49:13.531008+0000","last_became_active":"2024-03-17T23:49:11.653244+0000","last_became_peered":"2024-03-17T23:49:11.653244+0000","last_unstale":"2024-03-17T23:49:13.531008+0000","last_undegraded":"2024-03-17T23:49:13.531008+0000","last_fullsized":"2024-03-17T23:49:13.531008+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:11:54.641346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.670271+0000","last_change":"2024-03-17T23:49:11.667507+0000","last_active":"2024-03-17T23:49:11.670271+0000","last_peered":"2024-03-17T23:49:11.670271+0000","last_clean":"2024-03-17T23:49:11.670271+0000","last_became_active":"2024-03-17T23:49:11.666238+0000","last_became_peered":"2024-03-17T23:49:11.666238+0000","last_unstale":"2024-03-17T23:49:11.670271+0000","last_undegraded":"2024-03-17T23:49:11.670271+0000","last_fullsized":"2024-03-17T23:49:11.670271+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:15.111002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531305+0000","last_change":"2024-03-17T23:49:11.660542+0000","last_active":"2024-03-17T23:49:13.531305+0000","last_peered":"2024-03-17T23:49:13.531305+0000","last_clean":"2024-03-17T23:49:13.531305+0000","last_became_active":"2024-03-17T23:49:11.660272+0000","last_became_peered":"2024-03-17T23:49:11.660272+0000","last_unstale":"2024-03-17T23:49:13.531305+0000","last_undegraded":"2024-03-17T23:49:13.531305+0000","last_fullsized":"2024-03-17T23:49:13.531305+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:53:54.584217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.652530+0000","last_change":"2024-03-17T23:49:11.648970+0000","last_active":"2024-03-17T23:49:12.652530+0000","last_peered":"2024-03-17T23:49:12.652530+0000","last_clean":"2024-03-17T23:49:12.652530+0000","last_became_active":"2024-03-17T23:49:11.648699+0000","last_became_peered":"2024-03-17T23:49:11.648699+0000","last_unstale":"2024-03-17T23:49:12.652530+0000","last_undegraded":"2024-03-17T23:49:12.652530+0000","last_fullsized":"2024-03-17T23:49:12.652530+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:24:41.316664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.656797+0000","last_change":"2024-03-17T23:49:11.665421+0000","last_active":"2024-03-17T23:49:12.656797+0000","last_peered":"2024-03-17T23:49:12.656797+0000","last_clean":"2024-03-17T23:49:12.656797+0000","last_became_active":"2024-03-17T23:49:11.664791+0000","last_became_peered":"2024-03-17T23:49:11.664791+0000","last_unstale":"2024-03-17T23:49:12.656797+0000","last_undegraded":"2024-03-17T23:49:12.656797+0000","last_fullsized":"2024-03-17T23:49:12.656797+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:38:28.202171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.653260+0000","last_change":"2024-03-17T23:49:11.655317+0000","last_active":"2024-03-17T23:49:12.653260+0000","last_peered":"2024-03-17T23:49:12.653260+0000","last_clean":"2024-03-17T23:49:12.653260+0000","last_became_active":"2024-03-17T23:49:11.655056+0000","last_became_peered":"2024-03-17T23:49:11.655056+0000","last_unstale":"2024-03-17T23:49:12.653260+0000","last_undegraded":"2024-03-17T23:49:12.653260+0000","last_fullsized":"2024-03-17T23:49:12.653260+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:48:27.835084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531333+0000","last_change":"2024-03-17T23:49:11.658083+0000","last_active":"2024-03-17T23:49:13.531333+0000","last_peered":"2024-03-17T23:49:13.531333+0000","last_clean":"2024-03-17T23:49:13.531333+0000","last_became_active":"2024-03-17T23:49:11.657418+0000","last_became_peered":"2024-03-17T23:49:11.657418+0000","last_unstale":"2024-03-17T23:49:13.531333+0000","last_undegraded":"2024-03-17T23:49:13.531333+0000","last_fullsized":"2024-03-17T23:49:13.531333+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:16:16.395414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.653993+0000","last_change":"2024-03-17T23:49:11.657708+0000","last_active":"2024-03-17T23:49:12.653993+0000","last_peered":"2024-03-17T23:49:12.653993+0000","last_clean":"2024-03-17T23:49:12.653993+0000","last_became_active":"2024-03-17T23:49:11.656418+0000","last_became_peered":"2024-03-17T23:49:11.656418+0000","last_unstale":"2024-03-17T23:49:12.653993+0000","last_undegraded":"2024-03-17T23:49:12.653993+0000","last_fullsized":"2024-03-17T23:49:12.653993+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:31:07.433374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.652513+0000","last_change":"2024-03-17T23:49:11.645251+0000","last_active":"2024-03-17T23:49:11.652513+0000","last_peered":"2024-03-17T23:49:11.652513+0000","last_clean":"2024-03-17T23:49:11.652513+0000","last_became_active":"2024-03-17T23:49:11.644999+0000","last_became_peered":"2024-03-17T23:49:11.644999+0000","last_unstale":"2024-03-17T23:49:11.652513+0000","last_undegraded":"2024-03-17T23:49:11.652513+0000","last_fullsized":"2024-03-17T23:49:11.652513+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:10:05.668007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.652609+0000","last_change":"2024-03-17T23:49:11.648149+0000","last_active":"2024-03-17T23:49:11.652609+0000","last_peered":"2024-03-17T23:49:11.652609+0000","last_clean":"2024-03-17T23:49:11.652609+0000","last_became_active":"2024-03-17T23:49:11.647887+0000","last_became_peered":"2024-03-17T23:49:11.647887+0000","last_unstale":"2024-03-17T23:49:11.652609+0000","last_undegraded":"2024-03-17T23:49:11.652609+0000","last_fullsized":"2024-03-17T23:49:11.652609+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:16:38.296758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531440+0000","last_change":"2024-03-17T23:49:11.655458+0000","last_active":"2024-03-17T23:49:13.531440+0000","last_peered":"2024-03-17T23:49:13.531440+0000","last_clean":"2024-03-17T23:49:13.531440+0000","last_became_active":"2024-03-17T23:49:11.655181+0000","last_became_peered":"2024-03-17T23:49:11.655181+0000","last_unstale":"2024-03-17T23:49:13.531440+0000","last_undegraded":"2024-03-17T23:49:13.531440+0000","last_fullsized":"2024-03-17T23:49:13.531440+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:42:40.988689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.653973+0000","last_change":"2024-03-17T23:49:11.656693+0000","last_active":"2024-03-17T23:49:12.653973+0000","last_peered":"2024-03-17T23:49:12.653973+0000","last_clean":"2024-03-17T23:49:12.653973+0000","last_became_active":"2024-03-17T23:49:11.656400+0000","last_became_peered":"2024-03-17T23:49:11.656400+0000","last_unstale":"2024-03-17T23:49:12.653973+0000","last_undegraded":"2024-03-17T23:49:12.653973+0000","last_fullsized":"2024-03-17T23:49:12.653973+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:57:36.445652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.653776+0000","last_change":"2024-03-17T23:49:11.652508+0000","last_active":"2024-03-17T23:49:12.653776+0000","last_peered":"2024-03-17T23:49:12.653776+0000","last_clean":"2024-03-17T23:49:12.653776+0000","last_became_active":"2024-03-17T23:49:11.652153+0000","last_became_peered":"2024-03-17T23:49:11.652153+0000","last_unstale":"2024-03-17T23:49:12.653776+0000","last_undegraded":"2024-03-17T23:49:12.653776+0000","last_fullsized":"2024-03-17T23:49:12.653776+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:52:42.007552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.653519+0000","last_change":"2024-03-17T23:49:11.647575+0000","last_active":"2024-03-17T23:49:12.653519+0000","last_peered":"2024-03-17T23:49:12.653519+0000","last_clean":"2024-03-17T23:49:12.653519+0000","last_became_active":"2024-03-17T23:49:11.647313+0000","last_became_peered":"2024-03-17T23:49:11.647313+0000","last_unstale":"2024-03-17T23:49:12.653519+0000","last_undegraded":"2024-03-17T23:49:12.653519+0000","last_fullsized":"2024-03-17T23:49:12.653519+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:23:45.083399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.656786+0000","last_change":"2024-03-17T23:49:11.665738+0000","last_active":"2024-03-17T23:49:12.656786+0000","last_peered":"2024-03-17T23:49:12.656786+0000","last_clean":"2024-03-17T23:49:12.656786+0000","last_became_active":"2024-03-17T23:49:11.665255+0000","last_became_peered":"2024-03-17T23:49:11.665255+0000","last_unstale":"2024-03-17T23:49:12.656786+0000","last_undegraded":"2024-03-17T23:49:12.656786+0000","last_fullsized":"2024-03-17T23:49:12.656786+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:42:40.988689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531384+0000","last_change":"2024-03-17T23:49:11.654570+0000","last_active":"2024-03-17T23:49:13.531384+0000","last_peered":"2024-03-17T23:49:13.531384+0000","last_clean":"2024-03-17T23:49:13.531384+0000","last_became_active":"2024-03-17T23:49:11.654322+0000","last_became_peered":"2024-03-17T23:49:11.654322+0000","last_unstale":"2024-03-17T23:49:13.531384+0000","last_undegraded":"2024-03-17T23:49:13.531384+0000","last_fullsized":"2024-03-17T23:49:13.531384+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:01:11.561015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.670209+0000","last_change":"2024-03-17T23:49:11.668183+0000","last_active":"2024-03-17T23:49:11.670209+0000","last_peered":"2024-03-17T23:49:11.670209+0000","last_clean":"2024-03-17T23:49:11.670209+0000","last_became_active":"2024-03-17T23:49:11.667577+0000","last_became_peered":"2024-03-17T23:49:11.667577+0000","last_unstale":"2024-03-17T23:49:11.670209+0000","last_undegraded":"2024-03-17T23:49:11.670209+0000","last_fullsized":"2024-03-17T23:49:11.670209+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T23:59:25.825110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.652561+0000","last_change":"2024-03-17T23:49:11.655445+0000","last_active":"2024-03-17T23:49:12.652561+0000","last_peered":"2024-03-17T23:49:12.652561+0000","last_clean":"2024-03-17T23:49:12.652561+0000","last_became_active":"2024-03-17T23:49:11.655177+0000","last_became_peered":"2024-03-17T23:49:11.655177+0000","last_unstale":"2024-03-17T23:49:12.652561+0000","last_undegraded":"2024-03-17T23:49:12.652561+0000","last_fullsized":"2024-03-17T23:49:12.652561+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:36:05.463772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531709+0000","last_change":"2024-03-17T23:49:11.659640+0000","last_active":"2024-03-17T23:49:13.531709+0000","last_peered":"2024-03-17T23:49:13.531709+0000","last_clean":"2024-03-17T23:49:13.531709+0000","last_became_active":"2024-03-17T23:49:11.659386+0000","last_became_peered":"2024-03-17T23:49:11.659386+0000","last_unstale":"2024-03-17T23:49:13.531709+0000","last_undegraded":"2024-03-17T23:49:13.531709+0000","last_fullsized":"2024-03-17T23:49:13.531709+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:04:57.089593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.657425+0000","last_change":"2024-03-17T23:49:11.656210+0000","last_active":"2024-03-17T23:49:11.657425+0000","last_peered":"2024-03-17T23:49:11.657425+0000","last_clean":"2024-03-17T23:49:11.657425+0000","last_became_active":"2024-03-17T23:49:11.655629+0000","last_became_peered":"2024-03-17T23:49:11.655629+0000","last_unstale":"2024-03-17T23:49:11.657425+0000","last_undegraded":"2024-03-17T23:49:11.657425+0000","last_fullsized":"2024-03-17T23:49:11.657425+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:09:15.871249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.530914+0000","last_change":"2024-03-17T23:49:11.657524+0000","last_active":"2024-03-17T23:49:13.530914+0000","last_peered":"2024-03-17T23:49:13.530914+0000","last_clean":"2024-03-17T23:49:13.530914+0000","last_became_active":"2024-03-17T23:49:11.657227+0000","last_became_peered":"2024-03-17T23:49:11.657227+0000","last_unstale":"2024-03-17T23:49:13.530914+0000","last_undegraded":"2024-03-17T23:49:13.530914+0000","last_fullsized":"2024-03-17T23:49:13.530914+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:05:09.392406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.660884+0000","last_change":"2024-03-17T23:49:11.658319+0000","last_active":"2024-03-17T23:49:11.660884+0000","last_peered":"2024-03-17T23:49:11.660884+0000","last_clean":"2024-03-17T23:49:11.660884+0000","last_became_active":"2024-03-17T23:49:11.656866+0000","last_became_peered":"2024-03-17T23:49:11.656866+0000","last_unstale":"2024-03-17T23:49:11.660884+0000","last_undegraded":"2024-03-17T23:49:11.660884+0000","last_fullsized":"2024-03-17T23:49:11.660884+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:26:49.763854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.653370+0000","last_change":"2024-03-17T23:49:11.658960+0000","last_active":"2024-03-17T23:49:12.653370+0000","last_peered":"2024-03-17T23:49:12.653370+0000","last_clean":"2024-03-17T23:49:12.653370+0000","last_became_active":"2024-03-17T23:49:11.658695+0000","last_became_peered":"2024-03-17T23:49:11.658695+0000","last_unstale":"2024-03-17T23:49:12.653370+0000","last_undegraded":"2024-03-17T23:49:12.653370+0000","last_fullsized":"2024-03-17T23:49:12.653370+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:48:39.325339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.652724+0000","last_change":"2024-03-17T23:49:11.661827+0000","last_active":"2024-03-17T23:49:12.652724+0000","last_peered":"2024-03-17T23:49:12.652724+0000","last_clean":"2024-03-17T23:49:12.652724+0000","last_became_active":"2024-03-17T23:49:11.661536+0000","last_became_peered":"2024-03-17T23:49:11.661536+0000","last_unstale":"2024-03-17T23:49:12.652724+0000","last_undegraded":"2024-03-17T23:49:12.652724+0000","last_fullsized":"2024-03-17T23:49:12.652724+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:42:06.022555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.530837+0000","last_change":"2024-03-17T23:49:11.654139+0000","last_active":"2024-03-17T23:49:13.530837+0000","last_peered":"2024-03-17T23:49:13.530837+0000","last_clean":"2024-03-17T23:49:13.530837+0000","last_became_active":"2024-03-17T23:49:11.653838+0000","last_became_peered":"2024-03-17T23:49:11.653838+0000","last_unstale":"2024-03-17T23:49:13.530837+0000","last_undegraded":"2024-03-17T23:49:13.530837+0000","last_fullsized":"2024-03-17T23:49:13.530837+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:35:47.215326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.657337+0000","last_change":"2024-03-17T23:49:11.655872+0000","last_active":"2024-03-17T23:49:12.657337+0000","last_peered":"2024-03-17T23:49:12.657337+0000","last_clean":"2024-03-17T23:49:12.657337+0000","last_became_active":"2024-03-17T23:49:11.654787+0000","last_became_peered":"2024-03-17T23:49:11.654787+0000","last_unstale":"2024-03-17T23:49:12.657337+0000","last_undegraded":"2024-03-17T23:49:12.657337+0000","last_fullsized":"2024-03-17T23:49:12.657337+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:41:43.506586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.652643+0000","last_change":"2024-03-17T23:49:11.657164+0000","last_active":"2024-03-17T23:49:12.652643+0000","last_peered":"2024-03-17T23:49:12.652643+0000","last_clean":"2024-03-17T23:49:12.652643+0000","last_became_active":"2024-03-17T23:49:11.656855+0000","last_became_peered":"2024-03-17T23:49:11.656855+0000","last_unstale":"2024-03-17T23:49:12.652643+0000","last_undegraded":"2024-03-17T23:49:12.652643+0000","last_fullsized":"2024-03-17T23:49:12.652643+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:07:00.700235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531614+0000","last_change":"2024-03-17T23:49:11.653476+0000","last_active":"2024-03-17T23:49:13.531614+0000","last_peered":"2024-03-17T23:49:13.531614+0000","last_clean":"2024-03-17T23:49:13.531614+0000","last_became_active":"2024-03-17T23:49:11.653189+0000","last_became_peered":"2024-03-17T23:49:11.653189+0000","last_unstale":"2024-03-17T23:49:13.531614+0000","last_undegraded":"2024-03-17T23:49:13.531614+0000","last_fullsized":"2024-03-17T23:49:13.531614+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T23:50:29.132627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.670150+0000","last_change":"2024-03-17T23:49:11.667572+0000","last_active":"2024-03-17T23:49:11.670150+0000","last_peered":"2024-03-17T23:49:11.670150+0000","last_clean":"2024-03-17T23:49:11.670150+0000","last_became_active":"2024-03-17T23:49:11.666249+0000","last_became_peered":"2024-03-17T23:49:11.666249+0000","last_unstale":"2024-03-17T23:49:11.670150+0000","last_undegraded":"2024-03-17T23:49:11.670150+0000","last_fullsized":"2024-03-17T23:49:11.670150+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:00:05.868962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.652416+0000","last_change":"2024-03-17T23:49:11.661039+0000","last_active":"2024-03-17T23:49:12.652416+0000","last_peered":"2024-03-17T23:49:12.652416+0000","last_clean":"2024-03-17T23:49:12.652416+0000","last_became_active":"2024-03-17T23:49:11.660765+0000","last_became_peered":"2024-03-17T23:49:11.660765+0000","last_unstale":"2024-03-17T23:49:12.652416+0000","last_undegraded":"2024-03-17T23:49:12.652416+0000","last_fullsized":"2024-03-17T23:49:12.652416+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:25:25.661450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.656858+0000","last_change":"2024-03-17T23:49:11.665506+0000","last_active":"2024-03-17T23:49:12.656858+0000","last_peered":"2024-03-17T23:49:12.656858+0000","last_clean":"2024-03-17T23:49:12.656858+0000","last_became_active":"2024-03-17T23:49:11.664919+0000","last_became_peered":"2024-03-17T23:49:11.664919+0000","last_unstale":"2024-03-17T23:49:12.656858+0000","last_undegraded":"2024-03-17T23:49:12.656858+0000","last_fullsized":"2024-03-17T23:49:12.656858+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:00:38.498038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.657299+0000","last_change":"2024-03-17T23:49:11.656201+0000","last_active":"2024-03-17T23:49:12.657299+0000","last_peered":"2024-03-17T23:49:12.657299+0000","last_clean":"2024-03-17T23:49:12.657299+0000","last_became_active":"2024-03-17T23:49:11.655858+0000","last_became_peered":"2024-03-17T23:49:11.655858+0000","last_unstale":"2024-03-17T23:49:12.657299+0000","last_undegraded":"2024-03-17T23:49:12.657299+0000","last_fullsized":"2024-03-17T23:49:12.657299+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:39:45.444066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"57'4","reported_seq":18,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.657280+0000","last_change":"2024-03-17T23:49:10.652792+0000","last_active":"2024-03-17T23:49:12.657280+0000","last_peered":"2024-03-17T23:49:12.657280+0000","last_clean":"2024-03-17T23:49:12.657280+0000","last_became_active":"2024-03-17T23:49:10.652498+0000","last_became_peered":"2024-03-17T23:49:10.652498+0000","last_unstale":"2024-03-17T23:49:12.657280+0000","last_undegraded":"2024-03-17T23:49:12.657280+0000","last_fullsized":"2024-03-17T23:49:12.657280+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:14.106330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.661421+0000","last_change":"2024-03-17T23:49:11.656653+0000","last_active":"2024-03-17T23:49:11.661421+0000","last_peered":"2024-03-17T23:49:11.661421+0000","last_clean":"2024-03-17T23:49:11.661421+0000","last_became_active":"2024-03-17T23:49:11.656335+0000","last_became_peered":"2024-03-17T23:49:11.656335+0000","last_unstale":"2024-03-17T23:49:11.661421+0000","last_undegraded":"2024-03-17T23:49:11.661421+0000","last_fullsized":"2024-03-17T23:49:11.661421+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:21:49.823973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531085+0000","last_change":"2024-03-17T23:49:10.652668+0000","last_active":"2024-03-17T23:49:13.531085+0000","last_peered":"2024-03-17T23:49:13.531085+0000","last_clean":"2024-03-17T23:49:13.531085+0000","last_became_active":"2024-03-17T23:49:10.652257+0000","last_became_peered":"2024-03-17T23:49:10.652257+0000","last_unstale":"2024-03-17T23:49:13.531085+0000","last_undegraded":"2024-03-17T23:49:13.531085+0000","last_fullsized":"2024-03-17T23:49:13.531085+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:39:44.439394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.657074+0000","last_change":"2024-03-17T23:49:11.661266+0000","last_active":"2024-03-17T23:49:12.657074+0000","last_peered":"2024-03-17T23:49:12.657074+0000","last_clean":"2024-03-17T23:49:12.657074+0000","last_became_active":"2024-03-17T23:49:11.660990+0000","last_became_peered":"2024-03-17T23:49:11.660990+0000","last_unstale":"2024-03-17T23:49:12.657074+0000","last_undegraded":"2024-03-17T23:49:12.657074+0000","last_fullsized":"2024-03-17T23:49:12.657074+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:37:56.896972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.657389+0000","last_change":"2024-03-17T23:49:10.653477+0000","last_active":"2024-03-17T23:49:12.657389+0000","last_peered":"2024-03-17T23:49:12.657389+0000","last_clean":"2024-03-17T23:49:12.657389+0000","last_became_active":"2024-03-17T23:49:10.653143+0000","last_became_peered":"2024-03-17T23:49:10.653143+0000","last_unstale":"2024-03-17T23:49:12.657389+0000","last_undegraded":"2024-03-17T23:49:12.657389+0000","last_fullsized":"2024-03-17T23:49:12.657389+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T23:59:24.820437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.657139+0000","last_change":"2024-03-17T23:49:11.659380+0000","last_active":"2024-03-17T23:49:12.657139+0000","last_peered":"2024-03-17T23:49:12.657139+0000","last_clean":"2024-03-17T23:49:12.657139+0000","last_became_active":"2024-03-17T23:49:11.658920+0000","last_became_peered":"2024-03-17T23:49:11.658920+0000","last_unstale":"2024-03-17T23:49:12.657139+0000","last_undegraded":"2024-03-17T23:49:12.657139+0000","last_fullsized":"2024-03-17T23:49:12.657139+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:09:04.313151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531521+0000","last_change":"2024-03-17T23:49:10.654468+0000","last_active":"2024-03-17T23:49:13.531521+0000","last_peered":"2024-03-17T23:49:13.531521+0000","last_clean":"2024-03-17T23:49:13.531521+0000","last_became_active":"2024-03-17T23:49:10.652810+0000","last_became_peered":"2024-03-17T23:49:10.652810+0000","last_unstale":"2024-03-17T23:49:13.531521+0000","last_undegraded":"2024-03-17T23:49:13.531521+0000","last_fullsized":"2024-03-17T23:49:13.531521+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:36.605162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531022+0000","last_change":"2024-03-17T23:49:11.656823+0000","last_active":"2024-03-17T23:49:13.531022+0000","last_peered":"2024-03-17T23:49:13.531022+0000","last_clean":"2024-03-17T23:49:13.531022+0000","last_became_active":"2024-03-17T23:49:11.656546+0000","last_became_peered":"2024-03-17T23:49:11.656546+0000","last_unstale":"2024-03-17T23:49:13.531022+0000","last_undegraded":"2024-03-17T23:49:13.531022+0000","last_fullsized":"2024-03-17T23:49:13.531022+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:22:21.200213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.653643+0000","last_change":"2024-03-17T23:49:10.653909+0000","last_active":"2024-03-17T23:49:12.653643+0000","last_peered":"2024-03-17T23:49:12.653643+0000","last_clean":"2024-03-17T23:49:12.653643+0000","last_became_active":"2024-03-17T23:49:10.653648+0000","last_became_peered":"2024-03-17T23:49:10.653648+0000","last_unstale":"2024-03-17T23:49:12.653643+0000","last_undegraded":"2024-03-17T23:49:12.653643+0000","last_fullsized":"2024-03-17T23:49:12.653643+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:04.395229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531539+0000","last_change":"2024-03-17T23:49:11.659042+0000","last_active":"2024-03-17T23:49:13.531539+0000","last_peered":"2024-03-17T23:49:13.531539+0000","last_clean":"2024-03-17T23:49:13.531539+0000","last_became_active":"2024-03-17T23:49:11.658786+0000","last_became_peered":"2024-03-17T23:49:11.658786+0000","last_unstale":"2024-03-17T23:49:13.531539+0000","last_undegraded":"2024-03-17T23:49:13.531539+0000","last_fullsized":"2024-03-17T23:49:13.531539+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:09:46.753563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.656898+0000","last_change":"2024-03-17T23:49:10.645546+0000","last_active":"2024-03-17T23:49:12.656898+0000","last_peered":"2024-03-17T23:49:12.656898+0000","last_clean":"2024-03-17T23:49:12.656898+0000","last_became_active":"2024-03-17T23:49:10.645217+0000","last_became_peered":"2024-03-17T23:49:10.645217+0000","last_unstale":"2024-03-17T23:49:12.656898+0000","last_undegraded":"2024-03-17T23:49:12.656898+0000","last_fullsized":"2024-03-17T23:49:12.656898+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:48:26.830412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.653544+0000","last_change":"2024-03-17T23:49:11.655809+0000","last_active":"2024-03-17T23:49:12.653544+0000","last_peered":"2024-03-17T23:49:12.653544+0000","last_clean":"2024-03-17T23:49:12.653544+0000","last_became_active":"2024-03-17T23:49:11.655510+0000","last_became_peered":"2024-03-17T23:49:11.655510+0000","last_unstale":"2024-03-17T23:49:12.653544+0000","last_undegraded":"2024-03-17T23:49:12.653544+0000","last_fullsized":"2024-03-17T23:49:12.653544+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:10:05.668007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.530690+0000","last_change":"2024-03-17T23:49:10.656254+0000","last_active":"2024-03-17T23:49:13.530690+0000","last_peered":"2024-03-17T23:49:13.530690+0000","last_clean":"2024-03-17T23:49:13.530690+0000","last_became_active":"2024-03-17T23:49:10.655915+0000","last_became_peered":"2024-03-17T23:49:10.655915+0000","last_unstale":"2024-03-17T23:49:13.530690+0000","last_undegraded":"2024-03-17T23:49:13.530690+0000","last_fullsized":"2024-03-17T23:49:13.530690+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:26:48.759181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531727+0000","last_change":"2024-03-17T23:49:11.654251+0000","last_active":"2024-03-17T23:49:13.531727+0000","last_peered":"2024-03-17T23:49:13.531727+0000","last_clean":"2024-03-17T23:49:13.531727+0000","last_became_active":"2024-03-17T23:49:11.654004+0000","last_became_peered":"2024-03-17T23:49:11.654004+0000","last_unstale":"2024-03-17T23:49:13.531727+0000","last_undegraded":"2024-03-17T23:49:13.531727+0000","last_fullsized":"2024-03-17T23:49:13.531727+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:39:15.399081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.653324+0000","last_change":"2024-03-17T23:49:10.649005+0000","last_active":"2024-03-17T23:49:12.653324+0000","last_peered":"2024-03-17T23:49:12.653324+0000","last_clean":"2024-03-17T23:49:12.653324+0000","last_became_active":"2024-03-17T23:49:10.648717+0000","last_became_peered":"2024-03-17T23:49:10.648717+0000","last_unstale":"2024-03-17T23:49:12.653324+0000","last_undegraded":"2024-03-17T23:49:12.653324+0000","last_fullsized":"2024-03-17T23:49:12.653324+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:26:48.759181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.654190+0000","last_change":"2024-03-17T23:49:11.658818+0000","last_active":"2024-03-17T23:49:12.654190+0000","last_peered":"2024-03-17T23:49:12.654190+0000","last_clean":"2024-03-17T23:49:12.654190+0000","last_became_active":"2024-03-17T23:49:11.658546+0000","last_became_peered":"2024-03-17T23:49:11.658546+0000","last_unstale":"2024-03-17T23:49:12.654190+0000","last_undegraded":"2024-03-17T23:49:12.654190+0000","last_fullsized":"2024-03-17T23:49:12.654190+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:35:47.215326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.648333+0000","last_change":"2024-03-17T23:49:10.658240+0000","last_active":"2024-03-17T23:49:11.648333+0000","last_peered":"2024-03-17T23:49:11.648333+0000","last_clean":"2024-03-17T23:49:11.648333+0000","last_became_active":"2024-03-17T23:49:10.657882+0000","last_became_peered":"2024-03-17T23:49:10.657882+0000","last_unstale":"2024-03-17T23:49:11.648333+0000","last_undegraded":"2024-03-17T23:49:11.648333+0000","last_fullsized":"2024-03-17T23:49:11.648333+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:16:37.292086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.665442+0000","last_change":"2024-03-17T23:49:11.660908+0000","last_active":"2024-03-17T23:49:11.665442+0000","last_peered":"2024-03-17T23:49:11.665442+0000","last_clean":"2024-03-17T23:49:11.665442+0000","last_became_active":"2024-03-17T23:49:11.660644+0000","last_became_peered":"2024-03-17T23:49:11.660644+0000","last_unstale":"2024-03-17T23:49:11.665442+0000","last_undegraded":"2024-03-17T23:49:11.665442+0000","last_fullsized":"2024-03-17T23:49:11.665442+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:24:35.182836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531096+0000","last_change":"2024-03-17T23:49:10.655338+0000","last_active":"2024-03-17T23:49:13.531096+0000","last_peered":"2024-03-17T23:49:13.531096+0000","last_clean":"2024-03-17T23:49:13.531096+0000","last_became_active":"2024-03-17T23:49:10.655070+0000","last_became_peered":"2024-03-17T23:49:10.655070+0000","last_unstale":"2024-03-17T23:49:13.531096+0000","last_undegraded":"2024-03-17T23:49:13.531096+0000","last_fullsized":"2024-03-17T23:49:13.531096+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:41:42.501913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659276+0000","last_change":"2024-03-17T23:49:11.657785+0000","last_active":"2024-03-17T23:49:11.659276+0000","last_peered":"2024-03-17T23:49:11.659276+0000","last_clean":"2024-03-17T23:49:11.659276+0000","last_became_active":"2024-03-17T23:49:11.657040+0000","last_became_peered":"2024-03-17T23:49:11.657040+0000","last_unstale":"2024-03-17T23:49:11.659276+0000","last_undegraded":"2024-03-17T23:49:11.659276+0000","last_fullsized":"2024-03-17T23:49:11.659276+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:05.399901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"57'3","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.672409+0000","last_change":"2024-03-17T23:49:10.656304+0000","last_active":"2024-03-17T23:49:11.672409+0000","last_peered":"2024-03-17T23:49:11.672409+0000","last_clean":"2024-03-17T23:49:11.672409+0000","last_became_active":"2024-03-17T23:49:10.655795+0000","last_became_peered":"2024-03-17T23:49:10.655795+0000","last_unstale":"2024-03-17T23:49:11.672409+0000","last_undegraded":"2024-03-17T23:49:11.672409+0000","last_fullsized":"2024-03-17T23:49:11.672409+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:57:35.440980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.663919+0000","last_change":"2024-03-17T23:49:11.660244+0000","last_active":"2024-03-17T23:49:11.663919+0000","last_peered":"2024-03-17T23:49:11.663919+0000","last_clean":"2024-03-17T23:49:11.663919+0000","last_became_active":"2024-03-17T23:49:11.659982+0000","last_became_peered":"2024-03-17T23:49:11.659982+0000","last_unstale":"2024-03-17T23:49:11.663919+0000","last_undegraded":"2024-03-17T23:49:11.663919+0000","last_fullsized":"2024-03-17T23:49:11.663919+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:05:20.724503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"57'6","reported_seq":20,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.654156+0000","last_change":"2024-03-17T23:49:10.650050+0000","last_active":"2024-03-17T23:49:12.654156+0000","last_peered":"2024-03-17T23:49:12.654156+0000","last_clean":"2024-03-17T23:49:12.654156+0000","last_became_active":"2024-03-17T23:49:10.649505+0000","last_became_peered":"2024-03-17T23:49:10.649505+0000","last_unstale":"2024-03-17T23:49:12.654156+0000","last_undegraded":"2024-03-17T23:49:12.654156+0000","last_fullsized":"2024-03-17T23:49:12.654156+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:24:34.178164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.530796+0000","last_change":"2024-03-17T23:49:11.659236+0000","last_active":"2024-03-17T23:49:13.530796+0000","last_peered":"2024-03-17T23:49:13.530796+0000","last_clean":"2024-03-17T23:49:13.530796+0000","last_became_active":"2024-03-17T23:49:11.658962+0000","last_became_peered":"2024-03-17T23:49:11.658962+0000","last_unstale":"2024-03-17T23:49:13.530796+0000","last_undegraded":"2024-03-17T23:49:13.530796+0000","last_fullsized":"2024-03-17T23:49:13.530796+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:31:07.433374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531494+0000","last_change":"2024-03-17T23:49:10.657312+0000","last_active":"2024-03-17T23:49:13.531494+0000","last_peered":"2024-03-17T23:49:13.531494+0000","last_clean":"2024-03-17T23:49:13.531494+0000","last_became_active":"2024-03-17T23:49:10.657002+0000","last_became_peered":"2024-03-17T23:49:10.657002+0000","last_unstale":"2024-03-17T23:49:13.531494+0000","last_undegraded":"2024-03-17T23:49:13.531494+0000","last_fullsized":"2024-03-17T23:49:13.531494+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:19:39.198683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.670115+0000","last_change":"2024-03-17T23:49:11.667510+0000","last_active":"2024-03-17T23:49:11.670115+0000","last_peered":"2024-03-17T23:49:11.670115+0000","last_clean":"2024-03-17T23:49:11.670115+0000","last_became_active":"2024-03-17T23:49:11.665443+0000","last_became_peered":"2024-03-17T23:49:11.665443+0000","last_unstale":"2024-03-17T23:49:11.670115+0000","last_undegraded":"2024-03-17T23:49:11.670115+0000","last_fullsized":"2024-03-17T23:49:11.670115+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:35.755434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.653698+0000","last_change":"2024-03-17T23:49:10.650246+0000","last_active":"2024-03-17T23:49:12.653698+0000","last_peered":"2024-03-17T23:49:12.653698+0000","last_clean":"2024-03-17T23:49:12.653698+0000","last_became_active":"2024-03-17T23:49:10.649837+0000","last_became_peered":"2024-03-17T23:49:10.649837+0000","last_unstale":"2024-03-17T23:49:12.653698+0000","last_undegraded":"2024-03-17T23:49:12.653698+0000","last_fullsized":"2024-03-17T23:49:12.653698+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:08:24.371365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.653338+0000","last_change":"2024-03-17T23:49:11.655903+0000","last_active":"2024-03-17T23:49:12.653338+0000","last_peered":"2024-03-17T23:49:12.653338+0000","last_clean":"2024-03-17T23:49:12.653338+0000","last_became_active":"2024-03-17T23:49:11.655521+0000","last_became_peered":"2024-03-17T23:49:11.655521+0000","last_unstale":"2024-03-17T23:49:12.653338+0000","last_undegraded":"2024-03-17T23:49:12.653338+0000","last_fullsized":"2024-03-17T23:49:12.653338+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:11:49.300093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.530663+0000","last_change":"2024-03-17T23:49:10.651590+0000","last_active":"2024-03-17T23:49:13.530663+0000","last_peered":"2024-03-17T23:49:13.530663+0000","last_clean":"2024-03-17T23:49:13.530663+0000","last_became_active":"2024-03-17T23:49:10.651300+0000","last_became_peered":"2024-03-17T23:49:10.651300+0000","last_unstale":"2024-03-17T23:49:13.530663+0000","last_undegraded":"2024-03-17T23:49:13.530663+0000","last_fullsized":"2024-03-17T23:49:13.530663+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:05:19.719830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.659701+0000","last_change":"2024-03-17T23:49:11.657665+0000","last_active":"2024-03-17T23:49:11.659701+0000","last_peered":"2024-03-17T23:49:11.659701+0000","last_clean":"2024-03-17T23:49:11.659701+0000","last_became_active":"2024-03-17T23:49:11.657348+0000","last_became_peered":"2024-03-17T23:49:11.657348+0000","last_unstale":"2024-03-17T23:49:11.659701+0000","last_undegraded":"2024-03-17T23:49:11.659701+0000","last_fullsized":"2024-03-17T23:49:11.659701+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:05.399901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"57'8","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.672231+0000","last_change":"2024-03-17T23:49:10.654069+0000","last_active":"2024-03-17T23:49:11.672231+0000","last_peered":"2024-03-17T23:49:11.672231+0000","last_clean":"2024-03-17T23:49:11.672231+0000","last_became_active":"2024-03-17T23:49:10.653800+0000","last_became_peered":"2024-03-17T23:49:10.653800+0000","last_unstale":"2024-03-17T23:49:11.672231+0000","last_undegraded":"2024-03-17T23:49:11.672231+0000","last_fullsized":"2024-03-17T23:49:11.672231+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:10:04.663334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.670182+0000","last_change":"2024-03-17T23:49:11.667630+0000","last_active":"2024-03-17T23:49:11.670182+0000","last_peered":"2024-03-17T23:49:11.670182+0000","last_clean":"2024-03-17T23:49:11.670182+0000","last_became_active":"2024-03-17T23:49:11.665756+0000","last_became_peered":"2024-03-17T23:49:11.665756+0000","last_unstale":"2024-03-17T23:49:11.670182+0000","last_undegraded":"2024-03-17T23:49:11.670182+0000","last_fullsized":"2024-03-17T23:49:11.670182+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:55:32.443067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531371+0000","last_change":"2024-03-17T23:49:10.655436+0000","last_active":"2024-03-17T23:49:13.531371+0000","last_peered":"2024-03-17T23:49:13.531371+0000","last_clean":"2024-03-17T23:49:13.531371+0000","last_became_active":"2024-03-17T23:49:10.655114+0000","last_became_peered":"2024-03-17T23:49:10.655114+0000","last_unstale":"2024-03-17T23:49:13.531371+0000","last_undegraded":"2024-03-17T23:49:13.531371+0000","last_fullsized":"2024-03-17T23:49:13.531371+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:05:23.558519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.530864+0000","last_change":"2024-03-17T23:49:11.657629+0000","last_active":"2024-03-17T23:49:13.530864+0000","last_peered":"2024-03-17T23:49:13.530864+0000","last_clean":"2024-03-17T23:49:13.530864+0000","last_became_active":"2024-03-17T23:49:11.657356+0000","last_became_peered":"2024-03-17T23:49:11.657356+0000","last_unstale":"2024-03-17T23:49:13.530864+0000","last_undegraded":"2024-03-17T23:49:13.530864+0000","last_fullsized":"2024-03-17T23:49:13.530864+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:25:29.880006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.657242+0000","last_change":"2024-03-17T23:49:10.653706+0000","last_active":"2024-03-17T23:49:12.657242+0000","last_peered":"2024-03-17T23:49:12.657242+0000","last_clean":"2024-03-17T23:49:12.657242+0000","last_became_active":"2024-03-17T23:49:10.653133+0000","last_became_peered":"2024-03-17T23:49:10.653133+0000","last_unstale":"2024-03-17T23:49:12.657242+0000","last_undegraded":"2024-03-17T23:49:12.657242+0000","last_fullsized":"2024-03-17T23:49:12.657242+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:00:04.864289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531727+0000","last_change":"2024-03-17T23:49:11.649960+0000","last_active":"2024-03-17T23:49:13.531727+0000","last_peered":"2024-03-17T23:49:13.531727+0000","last_clean":"2024-03-17T23:49:13.531727+0000","last_became_active":"2024-03-17T23:49:11.649688+0000","last_became_peered":"2024-03-17T23:49:11.649688+0000","last_unstale":"2024-03-17T23:49:13.531727+0000","last_undegraded":"2024-03-17T23:49:13.531727+0000","last_fullsized":"2024-03-17T23:49:13.531727+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:21:14.621063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"57'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531096+0000","last_change":"2024-03-17T23:49:10.652457+0000","last_active":"2024-03-17T23:49:13.531096+0000","last_peered":"2024-03-17T23:49:13.531096+0000","last_clean":"2024-03-17T23:49:13.531096+0000","last_became_active":"2024-03-17T23:49:10.652105+0000","last_became_peered":"2024-03-17T23:49:10.652105+0000","last_unstale":"2024-03-17T23:49:13.531096+0000","last_undegraded":"2024-03-17T23:49:13.531096+0000","last_fullsized":"2024-03-17T23:49:13.531096+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:24:34.178164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.665510+0000","last_change":"2024-03-17T23:49:11.661804+0000","last_active":"2024-03-17T23:49:11.665510+0000","last_peered":"2024-03-17T23:49:11.665510+0000","last_clean":"2024-03-17T23:49:11.665510+0000","last_became_active":"2024-03-17T23:49:11.661518+0000","last_became_peered":"2024-03-17T23:49:11.661518+0000","last_unstale":"2024-03-17T23:49:11.665510+0000","last_undegraded":"2024-03-17T23:49:11.665510+0000","last_fullsized":"2024-03-17T23:49:11.665510+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:08:37.609834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.657185+0000","last_change":"2024-03-17T23:49:10.652713+0000","last_active":"2024-03-17T23:49:12.657185+0000","last_peered":"2024-03-17T23:49:12.657185+0000","last_clean":"2024-03-17T23:49:12.657185+0000","last_became_active":"2024-03-17T23:49:10.652449+0000","last_became_peered":"2024-03-17T23:49:10.652449+0000","last_unstale":"2024-03-17T23:49:12.657185+0000","last_undegraded":"2024-03-17T23:49:12.657185+0000","last_fullsized":"2024-03-17T23:49:12.657185+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:34.750762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.530926+0000","last_change":"2024-03-17T23:49:11.659323+0000","last_active":"2024-03-17T23:49:13.530926+0000","last_peered":"2024-03-17T23:49:13.530926+0000","last_clean":"2024-03-17T23:49:13.530926+0000","last_became_active":"2024-03-17T23:49:11.659047+0000","last_became_peered":"2024-03-17T23:49:11.659047+0000","last_unstale":"2024-03-17T23:49:13.530926+0000","last_undegraded":"2024-03-17T23:49:13.530926+0000","last_fullsized":"2024-03-17T23:49:13.530926+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:34:39.838698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"57'2","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531663+0000","last_change":"2024-03-17T23:49:10.653875+0000","last_active":"2024-03-17T23:49:13.531663+0000","last_peered":"2024-03-17T23:49:13.531663+0000","last_clean":"2024-03-17T23:49:13.531663+0000","last_became_active":"2024-03-17T23:49:10.653617+0000","last_became_peered":"2024-03-17T23:49:10.653617+0000","last_unstale":"2024-03-17T23:49:13.531663+0000","last_undegraded":"2024-03-17T23:49:13.531663+0000","last_fullsized":"2024-03-17T23:49:13.531663+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:09:03.308479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.657098+0000","last_change":"2024-03-17T23:49:11.655254+0000","last_active":"2024-03-17T23:49:12.657098+0000","last_peered":"2024-03-17T23:49:12.657098+0000","last_clean":"2024-03-17T23:49:12.657098+0000","last_became_active":"2024-03-17T23:49:11.654992+0000","last_became_peered":"2024-03-17T23:49:11.654992+0000","last_unstale":"2024-03-17T23:49:12.657098+0000","last_undegraded":"2024-03-17T23:49:12.657098+0000","last_fullsized":"2024-03-17T23:49:12.657098+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:19:40.203356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.639827+0000","last_change":"2024-03-17T23:49:10.655445+0000","last_active":"2024-03-17T23:49:11.639827+0000","last_peered":"2024-03-17T23:49:11.639827+0000","last_clean":"2024-03-17T23:49:11.639827+0000","last_became_active":"2024-03-17T23:49:10.654632+0000","last_became_peered":"2024-03-17T23:49:10.654632+0000","last_unstale":"2024-03-17T23:49:11.639827+0000","last_undegraded":"2024-03-17T23:49:11.639827+0000","last_fullsized":"2024-03-17T23:49:11.639827+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:54:00.207576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.656848+0000","last_change":"2024-03-17T23:49:11.666849+0000","last_active":"2024-03-17T23:49:12.656848+0000","last_peered":"2024-03-17T23:49:12.656848+0000","last_clean":"2024-03-17T23:49:12.656848+0000","last_became_active":"2024-03-17T23:49:11.666568+0000","last_became_peered":"2024-03-17T23:49:11.666568+0000","last_unstale":"2024-03-17T23:49:12.656848+0000","last_undegraded":"2024-03-17T23:49:12.656848+0000","last_fullsized":"2024-03-17T23:49:12.656848+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:58:45.165907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.530596+0000","last_change":"2024-03-17T23:49:10.654321+0000","last_active":"2024-03-17T23:49:13.530596+0000","last_peered":"2024-03-17T23:49:13.530596+0000","last_clean":"2024-03-17T23:49:13.530596+0000","last_became_active":"2024-03-17T23:49:10.653691+0000","last_became_peered":"2024-03-17T23:49:10.653691+0000","last_unstale":"2024-03-17T23:49:13.530596+0000","last_undegraded":"2024-03-17T23:49:13.530596+0000","last_fullsized":"2024-03-17T23:49:13.530596+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:29:04.395229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.530970+0000","last_change":"2024-03-17T23:49:11.660090+0000","last_active":"2024-03-17T23:49:13.530970+0000","last_peered":"2024-03-17T23:49:13.530970+0000","last_clean":"2024-03-17T23:49:13.530970+0000","last_became_active":"2024-03-17T23:49:11.659121+0000","last_became_peered":"2024-03-17T23:49:11.659121+0000","last_unstale":"2024-03-17T23:49:13.530970+0000","last_undegraded":"2024-03-17T23:49:13.530970+0000","last_fullsized":"2024-03-17T23:49:13.530970+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:53:32.384606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"57'2","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.675037+0000","last_change":"2024-03-17T23:49:10.655569+0000","last_active":"2024-03-17T23:49:11.675037+0000","last_peered":"2024-03-17T23:49:11.675037+0000","last_clean":"2024-03-17T23:49:11.675037+0000","last_became_active":"2024-03-17T23:49:10.655284+0000","last_became_peered":"2024-03-17T23:49:10.655284+0000","last_unstale":"2024-03-17T23:49:11.675037+0000","last_undegraded":"2024-03-17T23:49:11.675037+0000","last_fullsized":"2024-03-17T23:49:11.675037+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:07:07.135215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531111+0000","last_change":"2024-03-17T23:49:11.655737+0000","last_active":"2024-03-17T23:49:13.531111+0000","last_peered":"2024-03-17T23:49:13.531111+0000","last_clean":"2024-03-17T23:49:13.531111+0000","last_became_active":"2024-03-17T23:49:11.655464+0000","last_became_peered":"2024-03-17T23:49:11.655464+0000","last_unstale":"2024-03-17T23:49:13.531111+0000","last_undegraded":"2024-03-17T23:49:13.531111+0000","last_fullsized":"2024-03-17T23:49:13.531111+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:28:00.991361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"58'1","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.658418+0000","last_change":"2024-03-17T23:49:10.649196+0000","last_active":"2024-03-17T23:49:12.658418+0000","last_peered":"2024-03-17T23:49:12.658418+0000","last_clean":"2024-03-17T23:49:12.658418+0000","last_became_active":"2024-03-17T23:49:10.648830+0000","last_became_peered":"2024-03-17T23:49:10.648830+0000","last_unstale":"2024-03-17T23:49:12.658418+0000","last_undegraded":"2024-03-17T23:49:12.658418+0000","last_fullsized":"2024-03-17T23:49:12.658418+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:55:31.438395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.652228+0000","last_change":"2024-03-17T23:49:11.649056+0000","last_active":"2024-03-17T23:49:11.652228+0000","last_peered":"2024-03-17T23:49:11.652228+0000","last_clean":"2024-03-17T23:49:11.652228+0000","last_became_active":"2024-03-17T23:49:11.648791+0000","last_became_peered":"2024-03-17T23:49:11.648791+0000","last_unstale":"2024-03-17T23:49:11.652228+0000","last_undegraded":"2024-03-17T23:49:11.652228+0000","last_fullsized":"2024-03-17T23:49:11.652228+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:09:15.871249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.653821+0000","last_change":"2024-03-17T23:49:10.648679+0000","last_active":"2024-03-17T23:49:12.653821+0000","last_peered":"2024-03-17T23:49:12.653821+0000","last_clean":"2024-03-17T23:49:12.653821+0000","last_became_active":"2024-03-17T23:49:10.648407+0000","last_became_peered":"2024-03-17T23:49:10.648407+0000","last_unstale":"2024-03-17T23:49:12.653821+0000","last_undegraded":"2024-03-17T23:49:12.653821+0000","last_fullsized":"2024-03-17T23:49:12.653821+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:38:18.229949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.657294+0000","last_change":"2024-03-17T23:49:11.659172+0000","last_active":"2024-03-17T23:49:12.657294+0000","last_peered":"2024-03-17T23:49:12.657294+0000","last_clean":"2024-03-17T23:49:12.657294+0000","last_became_active":"2024-03-17T23:49:11.658577+0000","last_became_peered":"2024-03-17T23:49:11.658577+0000","last_unstale":"2024-03-17T23:49:12.657294+0000","last_undegraded":"2024-03-17T23:49:12.657294+0000","last_fullsized":"2024-03-17T23:49:12.657294+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:05:24.563191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531611+0000","last_change":"2024-03-17T23:49:10.655937+0000","last_active":"2024-03-17T23:49:13.531611+0000","last_peered":"2024-03-17T23:49:13.531611+0000","last_clean":"2024-03-17T23:49:13.531611+0000","last_became_active":"2024-03-17T23:49:10.654317+0000","last_became_peered":"2024-03-17T23:49:10.654317+0000","last_unstale":"2024-03-17T23:49:13.531611+0000","last_undegraded":"2024-03-17T23:49:13.531611+0000","last_fullsized":"2024-03-17T23:49:13.531611+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:41:34.750762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531239+0000","last_change":"2024-03-17T23:49:11.660013+0000","last_active":"2024-03-17T23:49:13.531239+0000","last_peered":"2024-03-17T23:49:13.531239+0000","last_clean":"2024-03-17T23:49:13.531239+0000","last_became_active":"2024-03-17T23:49:11.659686+0000","last_became_peered":"2024-03-17T23:49:11.659686+0000","last_unstale":"2024-03-17T23:49:13.531239+0000","last_undegraded":"2024-03-17T23:49:13.531239+0000","last_fullsized":"2024-03-17T23:49:13.531239+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:50:38.600565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.639931+0000","last_change":"2024-03-17T23:49:10.655613+0000","last_active":"2024-03-17T23:49:11.639931+0000","last_peered":"2024-03-17T23:49:11.639931+0000","last_clean":"2024-03-17T23:49:11.639931+0000","last_became_active":"2024-03-17T23:49:10.655134+0000","last_became_peered":"2024-03-17T23:49:10.655134+0000","last_unstale":"2024-03-17T23:49:11.639931+0000","last_undegraded":"2024-03-17T23:49:11.639931+0000","last_fullsized":"2024-03-17T23:49:11.639931+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:48:38.320666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.660923+0000","last_change":"2024-03-17T23:49:11.656942+0000","last_active":"2024-03-17T23:49:11.660923+0000","last_peered":"2024-03-17T23:49:11.660923+0000","last_clean":"2024-03-17T23:49:11.660923+0000","last_became_active":"2024-03-17T23:49:11.656612+0000","last_became_peered":"2024-03-17T23:49:11.656612+0000","last_unstale":"2024-03-17T23:49:11.660923+0000","last_undegraded":"2024-03-17T23:49:11.660923+0000","last_fullsized":"2024-03-17T23:49:11.660923+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:21:49.823973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.653877+0000","last_change":"2024-03-17T23:49:10.647432+0000","last_active":"2024-03-17T23:49:12.653877+0000","last_peered":"2024-03-17T23:49:12.653877+0000","last_clean":"2024-03-17T23:49:12.653877+0000","last_became_active":"2024-03-17T23:49:10.647160+0000","last_became_peered":"2024-03-17T23:49:10.647160+0000","last_unstale":"2024-03-17T23:49:12.653877+0000","last_undegraded":"2024-03-17T23:49:12.653877+0000","last_fullsized":"2024-03-17T23:49:12.653877+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:05:19.719830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.651916+0000","last_change":"2024-03-17T23:49:11.645350+0000","last_active":"2024-03-17T23:49:11.651916+0000","last_peered":"2024-03-17T23:49:11.651916+0000","last_clean":"2024-03-17T23:49:11.651916+0000","last_became_active":"2024-03-17T23:49:11.645101+0000","last_became_peered":"2024-03-17T23:49:11.645101+0000","last_unstale":"2024-03-17T23:49:11.651916+0000","last_undegraded":"2024-03-17T23:49:11.651916+0000","last_fullsized":"2024-03-17T23:49:11.651916+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:57:36.445652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.639772+0000","last_change":"2024-03-17T23:49:10.655236+0000","last_active":"2024-03-17T23:49:11.639772+0000","last_peered":"2024-03-17T23:49:11.639772+0000","last_clean":"2024-03-17T23:49:11.639772+0000","last_became_active":"2024-03-17T23:49:10.654952+0000","last_became_peered":"2024-03-17T23:49:10.654952+0000","last_unstale":"2024-03-17T23:49:11.639772+0000","last_undegraded":"2024-03-17T23:49:11.639772+0000","last_fullsized":"2024-03-17T23:49:11.639772+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:59:36.836697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531791+0000","last_change":"2024-03-17T23:49:11.652883+0000","last_active":"2024-03-17T23:49:13.531791+0000","last_peered":"2024-03-17T23:49:13.531791+0000","last_clean":"2024-03-17T23:49:13.531791+0000","last_became_active":"2024-03-17T23:49:11.652606+0000","last_became_peered":"2024-03-17T23:49:11.652606+0000","last_unstale":"2024-03-17T23:49:13.531791+0000","last_undegraded":"2024-03-17T23:49:13.531791+0000","last_fullsized":"2024-03-17T23:49:13.531791+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:35:26.662410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531549+0000","last_change":"2024-03-17T23:49:10.654224+0000","last_active":"2024-03-17T23:49:13.531549+0000","last_peered":"2024-03-17T23:49:13.531549+0000","last_clean":"2024-03-17T23:49:13.531549+0000","last_became_active":"2024-03-17T23:49:10.653909+0000","last_became_peered":"2024-03-17T23:49:10.653909+0000","last_unstale":"2024-03-17T23:49:13.531549+0000","last_undegraded":"2024-03-17T23:49:13.531549+0000","last_fullsized":"2024-03-17T23:49:13.531549+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:37:55.892299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:13.531518+0000","last_change":"2024-03-17T23:49:11.658312+0000","last_active":"2024-03-17T23:49:13.531518+0000","last_peered":"2024-03-17T23:49:13.531518+0000","last_clean":"2024-03-17T23:49:13.531518+0000","last_became_active":"2024-03-17T23:49:11.657765+0000","last_became_peered":"2024-03-17T23:49:11.657765+0000","last_unstale":"2024-03-17T23:49:13.531518+0000","last_undegraded":"2024-03-17T23:49:13.531518+0000","last_fullsized":"2024-03-17T23:49:13.531518+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:10.630539+0000","last_clean_scrub_stamp":"2024-03-17T23:49:10.630539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:55:22.554224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-17T23:49:12.653581+0000","last_change":"2024-03-17T23:49:10.653100+0000","last_active":"2024-03-17T23:49:12.653581+0000","last_peered":"2024-03-17T23:49:12.653581+0000","last_clean":"2024-03-17T23:49:12.653581+0000","last_became_active":"2024-03-17T23:49:10.652809+0000","last_became_peered":"2024-03-17T23:49:10.652809+0000","last_unstale":"2024-03-17T23:49:12.653581+0000","last_undegraded":"2024-03-17T23:49:12.653581+0000","last_fullsized":"2024-03-17T23:49:12.653581+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:49:09.625866+0000","last_clean_scrub_stamp":"2024-03-17T23:49:09.625866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:51:40.737241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":238,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-03-17T23:49:11.645564+0000","last_change":"2024-03-17T23:46:44.768668+0000","last_active":"2024-03-17T23:49:11.645564+0000","last_peered":"2024-03-17T23:49:11.645564+0000","last_clean":"2024-03-17T23:49:11.645564+0000","last_became_active":"2024-03-17T23:46:44.767826+0000","last_became_peered":"2024-03-17T23:46:44.767826+0000","last_unstale":"2024-03-17T23:49:11.645564+0000","last_undegraded":"2024-03-17T23:49:11.645564+0000","last_fullsized":"2024-03-17T23:49:11.645564+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_clean_scrub_stamp":"2024-03-17T23:46:43.744878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:54:22.286155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":14,"seq":60129542178,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":688128,"data_stored":309517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542178,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27708,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344132,"statfs":{"total":96636764160,"available":96608391168,"internally_reserved":0,"allocated":700416,"data_stored":316209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542178,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28296,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343544,"statfs":{"total":96636764160,"available":96607789056,"internally_reserved":0,"allocated":1302528,"data_stored":908715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574881,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32432,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339408,"statfs":{"total":96636764160,"available":96603553792,"internally_reserved":0,"allocated":1282048,"data_stored":899989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574882,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":692224,"data_stored":316153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574882,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":700416,"data_stored":316805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574882,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":704512,"data_stored":317765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344152,"statfs":{"total":96636764160,"available":96608411648,"internally_reserved":0,"allocated":684032,"data_stored":309551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-17T23:49:19.337 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-17T23:49:19.338 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-17T23:49:19.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.480+0000 7f77e5e44640 1 Processor -- start 2024-03-17T23:49:19.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.480+0000 7f77e5e44640 1 -- start start 2024-03-17T23:49:19.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.481+0000 7f77e5e44640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f77e006bae0 0x7f77e0165a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:19.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.481+0000 7f77e5e44640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77e00cc820 con 0x7f77e00cc440 2024-03-17T23:49:19.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.481+0000 7f77e5e44640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77e006e200 con 0x7f77e00cdce0 2024-03-17T23:49:19.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.481+0000 7f77e5e44640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f77e006e4e0 con 0x7f77e006bae0 2024-03-17T23:49:19.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.481+0000 7f77df7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f77e00cc440 0x7f77e015bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:51766/0 (socket says 172.21.15.150:51766) 2024-03-17T23:49:19.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.481+0000 7f77df7fe640 1 -- 172.21.15.150:0/2828800171 learned_addr learned my addr 172.21.15.150:0/2828800171 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:19.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.481+0000 7f77dffff640 1 --2- 172.21.15.150:0/2828800171 >> v2:172.21.15.157:3300/0 conn(0x7f77e006bae0 0x7f77e0165a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:19.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.481+0000 7f77de7fc640 1 -- 172.21.15.150:0/2828800171 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3106711090 0 0) 0x7f77e00cc820 con 0x7f77e00cc440 2024-03-17T23:49:19.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.481+0000 7f77de7fc640 1 -- 172.21.15.150:0/2828800171 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77c8003520 con 0x7f77e00cc440 2024-03-17T23:49:19.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.481+0000 7f77dffff640 1 -- 172.21.15.150:0/2828800171 >> v1:172.21.15.157:6789/0 conn(0x7f77e00cdce0 legacy=0x7f77e006b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:19.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.481+0000 7f77dffff640 1 -- 172.21.15.150:0/2828800171 >> v1:172.21.15.150:6789/0 conn(0x7f77e00cc440 legacy=0x7f77e015bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:19.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.481+0000 7f77dffff640 1 -- 172.21.15.150:0/2828800171 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77e006e7c0 con 0x7f77e006bae0 2024-03-17T23:49:19.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.482+0000 7f77dffff640 1 --2- 172.21.15.150:0/2828800171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77e006bae0 0x7f77e0165a60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f77d8009d90 tx=0x7f77d802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3725d8ea4f26012 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:19.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.482+0000 7f77de7fc640 1 -- 172.21.15.150:0/2828800171 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d8039070 con 0x7f77e006bae0 2024-03-17T23:49:19.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.482+0000 7f77de7fc640 1 -- 172.21.15.150:0/2828800171 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77d802cc60 con 0x7f77e006bae0 2024-03-17T23:49:19.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.482+0000 7f77de7fc640 1 -- 172.21.15.150:0/2828800171 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d8034640 con 0x7f77e006bae0 2024-03-17T23:49:19.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.483+0000 7f77e5e44640 1 -- 172.21.15.150:0/2828800171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77e006bae0 msgr2=0x7f77e0165a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:19.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.483+0000 7f77e5e44640 1 --2- 172.21.15.150:0/2828800171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77e006bae0 0x7f77e0165a60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f77d8009d90 tx=0x7f77d802beb0 comp rx=0 tx=0).stop 2024-03-17T23:49:19.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.483+0000 7f77e5e44640 1 -- 172.21.15.150:0/2828800171 shutdown_connections 2024-03-17T23:49:19.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.483+0000 7f77e5e44640 1 --2- 172.21.15.150:0/2828800171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77e006bae0 0x7f77e0165a60 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:19.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.483+0000 7f77e5e44640 1 -- 172.21.15.150:0/2828800171 >> 172.21.15.150:0/2828800171 conn(0x7f77e00b18c0 msgr2=0x7f77e00b1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:19.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.483+0000 7f77e5e44640 1 -- 172.21.15.150:0/2828800171 shutdown_connections 2024-03-17T23:49:19.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.483+0000 7f77e5e44640 1 -- 172.21.15.150:0/2828800171 wait complete. 2024-03-17T23:49:19.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.484+0000 7f77e5e44640 1 Processor -- start 2024-03-17T23:49:19.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.484+0000 7f77e5e44640 1 -- start start 2024-03-17T23:49:19.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.485+0000 7f77e5e44640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77e006bae0 0x7f77e00cb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.485+0000 7f77e5e44640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77e00cc440 0x7f77e014b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.485+0000 7f77e5e44640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77e00cdce0 0x7f77e014b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.485+0000 7f77e5e44640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f77e006dde0 con 0x7f77e006bae0 2024-03-17T23:49:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.485+0000 7f77e5e44640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f77e006dc60 con 0x7f77e00cc440 2024-03-17T23:49:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.485+0000 7f77e5e44640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f77e006df60 con 0x7f77e00cdce0 2024-03-17T23:49:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.485+0000 7f77df7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77e006bae0 0x7f77e00cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.485+0000 7f77df7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77e006bae0 0x7f77e00cb720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45544/0 (socket says 172.21.15.150:45544) 2024-03-17T23:49:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.485+0000 7f77deffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77e00cc440 0x7f77e014b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.485+0000 7f77df7fe640 1 -- 172.21.15.150:0/691836654 learned_addr learned my addr 172.21.15.150:0/691836654 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.485+0000 7f77dffff640 1 --2- 172.21.15.150:0/691836654 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77e00cdce0 0x7f77e014b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.485+0000 7f77df7fe640 1 -- 172.21.15.150:0/691836654 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77e00cdce0 msgr2=0x7f77e014b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.485+0000 7f77df7fe640 1 --2- 172.21.15.150:0/691836654 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77e00cdce0 0x7f77e014b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.485+0000 7f77df7fe640 1 -- 172.21.15.150:0/691836654 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77e00cc440 msgr2=0x7f77e014b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.485+0000 7f77df7fe640 1 --2- 172.21.15.150:0/691836654 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77e00cc440 0x7f77e014b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.485+0000 7f77df7fe640 1 -- 172.21.15.150:0/691836654 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77e00cbc60 con 0x7f77e006bae0 2024-03-17T23:49:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.485+0000 7f77dffff640 1 --2- 172.21.15.150:0/691836654 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77e00cdce0 0x7f77e014b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.485+0000 7f77deffd640 1 --2- 172.21.15.150:0/691836654 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77e00cc440 0x7f77e014b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.485+0000 7f77df7fe640 1 --2- 172.21.15.150:0/691836654 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77e006bae0 0x7f77e00cb720 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f77d000c500 tx=0x7f77d000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.485+0000 7f77c7fff640 1 -- 172.21.15.150:0/691836654 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d0017070 con 0x7f77e006bae0 2024-03-17T23:49:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.486+0000 7f77c7fff640 1 -- 172.21.15.150:0/691836654 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77d0005ce0 con 0x7f77e006bae0 2024-03-17T23:49:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.486+0000 7f77e5e44640 1 -- 172.21.15.150:0/691836654 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77e014bd10 con 0x7f77e006bae0 2024-03-17T23:49:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.486+0000 7f77c7fff640 1 -- 172.21.15.150:0/691836654 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d0007330 con 0x7f77e006bae0 2024-03-17T23:49:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.486+0000 7f77e5e44640 1 -- 172.21.15.150:0/691836654 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77e014c2d0 con 0x7f77e006bae0 2024-03-17T23:49:19.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.486+0000 7f77e5e44640 1 -- 172.21.15.150:0/691836654 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77e00681a0 con 0x7f77e006bae0 2024-03-17T23:49:19.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.487+0000 7f77c7fff640 1 -- 172.21.15.150:0/691836654 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f77d001a070 con 0x7f77e006bae0 2024-03-17T23:49:19.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.487+0000 7f77c7fff640 1 --2- 172.21.15.150:0/691836654 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f77cc06bbd0 0x7f77cc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:19.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.487+0000 7f77c7fff640 1 -- 172.21.15.150:0/691836654 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f77d0089f40 con 0x7f77e006bae0 2024-03-17T23:49:19.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.487+0000 7f77deffd640 1 --2- 172.21.15.150:0/691836654 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f77cc06bbd0 0x7f77cc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:19.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.488+0000 7f77deffd640 1 --2- 172.21.15.150:0/691836654 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f77cc06bbd0 0x7f77cc06e090 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f77e00c4a00 tx=0x7f77d4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:19.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.492+0000 7f77c7fff640 1 -- 172.21.15.150:0/691836654 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f77d00583a0 con 0x7f77e006bae0 2024-03-17T23:49:19.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.735+0000 7f77e5e44640 1 -- 172.21.15.150:0/691836654 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f77e006f7a0 con 0x7f77e006bae0 2024-03-17T23:49:19.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.737+0000 7f77c7fff640 1 -- 172.21.15.150:0/691836654 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) ==== 190+0+147 (secure 0 0 0) 0x7f77d005c440 con 0x7f77e006bae0 2024-03-17T23:49:19.738 INFO:teuthology.orchestra.run.smithi150.stdout:[client.0] 2024-03-17T23:49:19.738 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQC3gPdlvlPJNRAAkUfetPJK4hZQG5rNiSDozw== 2024-03-17T23:49:19.738 INFO:teuthology.orchestra.run.smithi150.stdout: caps mds = "allow" 2024-03-17T23:49:19.738 INFO:teuthology.orchestra.run.smithi150.stdout: caps mon = "allow r" 2024-03-17T23:49:19.738 INFO:teuthology.orchestra.run.smithi150.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-17T23:49:19.738 INFO:teuthology.orchestra.run.smithi150.stderr:updated caps for client.0 2024-03-17T23:49:19.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.739+0000 7f77e5e44640 1 -- 172.21.15.150:0/691836654 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f77cc06bbd0 msgr2=0x7f77cc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:19.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.739+0000 7f77e5e44640 1 --2- 172.21.15.150:0/691836654 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f77cc06bbd0 0x7f77cc06e090 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f77e00c4a00 tx=0x7f77d4009040 comp rx=0 tx=0).stop 2024-03-17T23:49:19.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.739+0000 7f77e5e44640 1 -- 172.21.15.150:0/691836654 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77e006bae0 msgr2=0x7f77e00cb720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:19.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.739+0000 7f77e5e44640 1 --2- 172.21.15.150:0/691836654 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77e006bae0 0x7f77e00cb720 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f77d000c500 tx=0x7f77d000c9c0 comp rx=0 tx=0).stop 2024-03-17T23:49:19.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.739+0000 7f77e5e44640 1 -- 172.21.15.150:0/691836654 shutdown_connections 2024-03-17T23:49:19.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.739+0000 7f77e5e44640 1 --2- 172.21.15.150:0/691836654 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f77e00cdce0 0x7f77e014b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:19.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.739+0000 7f77e5e44640 1 --2- 172.21.15.150:0/691836654 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f77cc06bbd0 0x7f77cc06e090 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:19.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.739+0000 7f77e5e44640 1 --2- 172.21.15.150:0/691836654 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77e00cc440 0x7f77e014b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:19.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.739+0000 7f77e5e44640 1 --2- 172.21.15.150:0/691836654 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77e006bae0 0x7f77e00cb720 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:19.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.739+0000 7f77e5e44640 1 -- 172.21.15.150:0/691836654 >> 172.21.15.150:0/691836654 conn(0x7f77e00b18c0 msgr2=0x7f77e00ce140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:19.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.740+0000 7f77e5e44640 1 -- 172.21.15.150:0/691836654 shutdown_connections 2024-03-17T23:49:19.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.740+0000 7f77e5e44640 1 -- 172.21.15.150:0/691836654 wait complete. 2024-03-17T23:49:19.751 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:49:19.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.851+0000 7f33adfdf640 1 Processor -- start 2024-03-17T23:49:19.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.851+0000 7f33adfdf640 1 -- start start 2024-03-17T23:49:19.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.852+0000 7f33adfdf640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f33a815de30 0x7f33a81619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:19.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.852+0000 7f33adfdf640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f33a80cf120 0x7f33a80d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:19.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.852+0000 7f33adfdf640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33a80b9cf0 con 0x7f33a80b9860 2024-03-17T23:49:19.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.852+0000 7f33adfdf640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f33a8162940 con 0x7f33a815de30 2024-03-17T23:49:19.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.852+0000 7f33adfdf640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f33a81620a0 con 0x7f33a80cf120 2024-03-17T23:49:19.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.852+0000 7f33a77fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f33a815de30 0x7f33a81619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:19.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.852+0000 7f33a77fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f33a815de30 0x7f33a81619a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45554/0 (socket says 172.21.15.150:45554) 2024-03-17T23:49:19.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.852+0000 7f33a77fe640 1 -- 172.21.15.150:0/728333022 learned_addr learned my addr 172.21.15.150:0/728333022 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:19.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.852+0000 7f33a7fff640 1 --2- 172.21.15.150:0/728333022 >> v2:172.21.15.157:3300/0 conn(0x7f33a80cf120 0x7f33a80d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:19.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.853+0000 7f33a77fe640 1 -- 172.21.15.150:0/728333022 >> v2:172.21.15.157:3300/0 conn(0x7f33a80cf120 msgr2=0x7f33a80d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:19.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.853+0000 7f33a77fe640 1 --2- 172.21.15.150:0/728333022 >> v2:172.21.15.157:3300/0 conn(0x7f33a80cf120 0x7f33a80d2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:19.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.853+0000 7f33a77fe640 1 -- 172.21.15.150:0/728333022 >> v1:172.21.15.157:6789/0 conn(0x7f33a80b9860 legacy=0x7f33a80cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:19.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.853+0000 7f33a77fe640 1 -- 172.21.15.150:0/728333022 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33a8162380 con 0x7f33a815de30 2024-03-17T23:49:19.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.853+0000 7f33a77fe640 1 --2- 172.21.15.150:0/728333022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33a815de30 0x7f33a81619a0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f33a81618d0 tx=0x7f339802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a29141015243013b server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:19.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.854+0000 7f33a6ffd640 1 -- 172.21.15.150:0/728333022 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3398035020 con 0x7f33a815de30 2024-03-17T23:49:19.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.854+0000 7f33a6ffd640 1 -- 172.21.15.150:0/728333022 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3398031870 con 0x7f33a815de30 2024-03-17T23:49:19.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.854+0000 7f33a6ffd640 1 -- 172.21.15.150:0/728333022 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3398031c40 con 0x7f33a815de30 2024-03-17T23:49:19.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.854+0000 7f33adfdf640 1 -- 172.21.15.150:0/728333022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33a815de30 msgr2=0x7f33a81619a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:19.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.854+0000 7f33adfdf640 1 --2- 172.21.15.150:0/728333022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33a815de30 0x7f33a81619a0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f33a81618d0 tx=0x7f339802b910 comp rx=0 tx=0).stop 2024-03-17T23:49:19.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.854+0000 7f33adfdf640 1 -- 172.21.15.150:0/728333022 shutdown_connections 2024-03-17T23:49:19.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.854+0000 7f33adfdf640 1 --2- 172.21.15.150:0/728333022 >> v2:172.21.15.157:3300/0 conn(0x7f33a80cf120 0x7f33a80d2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:19.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.854+0000 7f33adfdf640 1 --2- 172.21.15.150:0/728333022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33a815de30 0x7f33a81619a0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:19.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.854+0000 7f33adfdf640 1 -- 172.21.15.150:0/728333022 >> 172.21.15.150:0/728333022 conn(0x7f33a80730b0 msgr2=0x7f33a80d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:19.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.854+0000 7f33adfdf640 1 -- 172.21.15.150:0/728333022 shutdown_connections 2024-03-17T23:49:19.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.854+0000 7f33adfdf640 1 -- 172.21.15.150:0/728333022 wait complete. 2024-03-17T23:49:19.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.855+0000 7f33adfdf640 1 Processor -- start 2024-03-17T23:49:19.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.856+0000 7f33adfdf640 1 -- start start 2024-03-17T23:49:19.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.856+0000 7f33adfdf640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f33a80b9860 0x7f33a81f4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:19.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.856+0000 7f33adfdf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f33a80cf120 0x7f33a8157380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:19.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.856+0000 7f33adfdf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33a815de30 0x7f33a80cc320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.856+0000 7f33adfdf640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f33a815e210 con 0x7f33a815de30 2024-03-17T23:49:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.856+0000 7f33adfdf640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f33a80d2170 con 0x7f33a80b9860 2024-03-17T23:49:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.856+0000 7f33adfdf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f33a80d2bd0 con 0x7f33a80cf120 2024-03-17T23:49:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.856+0000 7f33a7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33a815de30 0x7f33a80cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.856+0000 7f33a7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33a815de30 0x7f33a80cc320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45564/0 (socket says 172.21.15.150:45564) 2024-03-17T23:49:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.856+0000 7f33a7fff640 1 -- 172.21.15.150:0/2523546083 learned_addr learned my addr 172.21.15.150:0/2523546083 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.856+0000 7f33a77fe640 1 --2- 172.21.15.150:0/2523546083 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f33a80b9860 0x7f33a81f4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.856+0000 7f339effd640 1 --2- 172.21.15.150:0/2523546083 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f33a80cf120 0x7f33a8157380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.857+0000 7f33a7fff640 1 -- 172.21.15.150:0/2523546083 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f33a80cf120 msgr2=0x7f33a8157380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.857+0000 7f33a7fff640 1 --2- 172.21.15.150:0/2523546083 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f33a80cf120 0x7f33a8157380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.857+0000 7f33a7fff640 1 -- 172.21.15.150:0/2523546083 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f33a80b9860 msgr2=0x7f33a81f4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.857+0000 7f33a7fff640 1 --2- 172.21.15.150:0/2523546083 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f33a80b9860 0x7f33a81f4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.857+0000 7f33a7fff640 1 -- 172.21.15.150:0/2523546083 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33a81f54b0 con 0x7f33a815de30 2024-03-17T23:49:19.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.857+0000 7f33a7fff640 1 --2- 172.21.15.150:0/2523546083 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33a815de30 0x7f33a80cc320 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f33a000efc0 tx=0x7f33a000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:19.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.857+0000 7f33a4ff9640 1 -- 172.21.15.150:0/2523546083 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a0018020 con 0x7f33a815de30 2024-03-17T23:49:19.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.857+0000 7f33a4ff9640 1 -- 172.21.15.150:0/2523546083 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33a0014480 con 0x7f33a815de30 2024-03-17T23:49:19.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.857+0000 7f33adfdf640 1 -- 172.21.15.150:0/2523546083 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33a815ab90 con 0x7f33a815de30 2024-03-17T23:49:19.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.857+0000 7f33a4ff9640 1 -- 172.21.15.150:0/2523546083 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a0014870 con 0x7f33a815de30 2024-03-17T23:49:19.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.857+0000 7f33adfdf640 1 -- 172.21.15.150:0/2523546083 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33a8158aa0 con 0x7f33a815de30 2024-03-17T23:49:19.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.858+0000 7f33adfdf640 1 -- 172.21.15.150:0/2523546083 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33a80d4920 con 0x7f33a815de30 2024-03-17T23:49:19.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.858+0000 7f33a4ff9640 1 -- 172.21.15.150:0/2523546083 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f33a0005d10 con 0x7f33a815de30 2024-03-17T23:49:19.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.859+0000 7f33a4ff9640 1 --2- 172.21.15.150:0/2523546083 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f338806bbd0 0x7f338806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:19.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.859+0000 7f33a4ff9640 1 -- 172.21.15.150:0/2523546083 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f33a008b4f0 con 0x7f33a815de30 2024-03-17T23:49:19.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.859+0000 7f33a77fe640 1 --2- 172.21.15.150:0/2523546083 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f338806bbd0 0x7f338806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:19.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.859+0000 7f33a77fe640 1 --2- 172.21.15.150:0/2523546083 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f338806bbd0 0x7f338806e090 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3398036000 tx=0x7f339803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:19.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:19.863+0000 7f33a4ff9640 1 -- 172.21.15.150:0/2523546083 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f33a80d4920 con 0x7f33a815de30 2024-03-17T23:49:20.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.072+0000 7f33adfdf640 1 -- 172.21.15.150:0/2523546083 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33a80681a0 con 0x7f33a815de30 2024-03-17T23:49:20.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.073+0000 7f33a4ff9640 1 -- 172.21.15.150:0/2523546083 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+5187 (secure 0 0 0) 0x7f33a0022070 con 0x7f33a815de30 2024-03-17T23:49:20.074 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:20.074 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":22,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/2082413564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2082413564},{"type":"v1","addr":"172.21.15.150:6836","nonce":2082413564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":6043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/2350077845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":2350077845},{"type":"v1","addr":"172.21.15.150:6837","nonce":2350077845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":6052,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6834/137606728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":137606728},{"type":"v1","addr":"172.21.15.157:6834","nonce":137606728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-17T23:49:10.996124+0000","modified":"2024-03-17T23:49:12.994518+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6835/479821580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6833","nonce":479821580},{"type":"v1","addr":"172.21.15.157:6835","nonce":479821580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6204,"qdb_cluster":[6204]},"id":3}]} 2024-03-17T23:49:20.074 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 22 2024-03-17T23:49:20.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.075+0000 7f33adfdf640 1 -- 172.21.15.150:0/2523546083 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f338806bbd0 msgr2=0x7f338806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:20.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.075+0000 7f33adfdf640 1 --2- 172.21.15.150:0/2523546083 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f338806bbd0 0x7f338806e090 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3398036000 tx=0x7f339803c040 comp rx=0 tx=0).stop 2024-03-17T23:49:20.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.075+0000 7f33adfdf640 1 -- 172.21.15.150:0/2523546083 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33a815de30 msgr2=0x7f33a80cc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:20.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.075+0000 7f33adfdf640 1 --2- 172.21.15.150:0/2523546083 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33a815de30 0x7f33a80cc320 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f33a000efc0 tx=0x7f33a000c490 comp rx=0 tx=0).stop 2024-03-17T23:49:20.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.075+0000 7f33adfdf640 1 -- 172.21.15.150:0/2523546083 shutdown_connections 2024-03-17T23:49:20.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.075+0000 7f33adfdf640 1 --2- 172.21.15.150:0/2523546083 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33a815de30 0x7f33a80cc320 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.075+0000 7f33adfdf640 1 --2- 172.21.15.150:0/2523546083 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f33a80cf120 0x7f33a8157380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.075+0000 7f33adfdf640 1 --2- 172.21.15.150:0/2523546083 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f338806bbd0 0x7f338806e090 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.075+0000 7f33adfdf640 1 --2- 172.21.15.150:0/2523546083 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f33a80b9860 0x7f33a81f4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.075+0000 7f33adfdf640 1 -- 172.21.15.150:0/2523546083 >> 172.21.15.150:0/2523546083 conn(0x7f33a80730b0 msgr2=0x7f33a80d2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:20.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.075+0000 7f33adfdf640 1 -- 172.21.15.150:0/2523546083 shutdown_connections 2024-03-17T23:49:20.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.075+0000 7f33adfdf640 1 -- 172.21.15.150:0/2523546083 wait complete. 2024-03-17T23:49:20.489 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-17T23:49:10.996124+0000', 'modified': '2024-03-17T23:49:12.994518+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6204}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6204': {'gid': 6204, 'name': 'b', 'rank': 0, 'incarnation': 20, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.157:6835/479821580', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6833', 'nonce': 479821580}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 479821580}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6204, 'qdb_cluster': [6204]} 2024-03-17T23:49:20.489 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-17T23:49:20.490 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-17T23:49:20.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.591+0000 7f5dd6ab5640 1 Processor -- start 2024-03-17T23:49:20.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.592+0000 7f5dd6ab5640 1 -- start start 2024-03-17T23:49:20.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.592+0000 7f5dd6ab5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5dd00b97f0 0x7f5dd0162160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:20.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.592+0000 7f5dd6ab5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5dd00b9c70 con 0x7f5dd015df40 2024-03-17T23:49:20.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.592+0000 7f5dd6ab5640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5dd015ec90 con 0x7f5dd01626a0 2024-03-17T23:49:20.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.592+0000 7f5dd6ab5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5dd00b6d40 con 0x7f5dd00b97f0 2024-03-17T23:49:20.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.592+0000 7f5dd4851640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f5dd015df40 0x7f5dd015e320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:51770/0 (socket says 172.21.15.150:51770) 2024-03-17T23:49:20.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.592+0000 7f5dd4851640 1 -- 172.21.15.150:0/2448181688 learned_addr learned my addr 172.21.15.150:0/2448181688 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:20.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.593+0000 7f5dcffff640 1 --2- 172.21.15.150:0/2448181688 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5dd00b97f0 0x7f5dd0162160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:20.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.593+0000 7f5dcf7fe640 1 -- 172.21.15.150:0/2448181688 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3288325612 0 0) 0x7f5dd00b9c70 con 0x7f5dd015df40 2024-03-17T23:49:20.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.593+0000 7f5dcf7fe640 1 -- 172.21.15.150:0/2448181688 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5dc0003520 con 0x7f5dd015df40 2024-03-17T23:49:20.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.593+0000 7f5dcf7fe640 1 -- 172.21.15.150:0/2448181688 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2729000282 0 0) 0x7f5dd015ec90 con 0x7f5dd01626a0 2024-03-17T23:49:20.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.593+0000 7f5dcf7fe640 1 -- 172.21.15.150:0/2448181688 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5dd00b9c70 con 0x7f5dd01626a0 2024-03-17T23:49:20.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.593+0000 7f5dcffff640 1 -- 172.21.15.150:0/2448181688 >> v1:172.21.15.157:6789/0 conn(0x7f5dd01626a0 legacy=0x7f5dd00d2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:20.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.593+0000 7f5dcffff640 1 -- 172.21.15.150:0/2448181688 >> v1:172.21.15.150:6789/0 conn(0x7f5dd015df40 legacy=0x7f5dd015e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:20.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.593+0000 7f5dcffff640 1 -- 172.21.15.150:0/2448181688 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd015ef70 con 0x7f5dd00b97f0 2024-03-17T23:49:20.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.594+0000 7f5dcffff640 1 --2- 172.21.15.150:0/2448181688 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5dd00b97f0 0x7f5dd0162160 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc0097d0 tx=0x7f5dbc02b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b901a98f544bb9f server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:20.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.594+0000 7f5dcf7fe640 1 -- 172.21.15.150:0/2448181688 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc034070 con 0x7f5dd00b97f0 2024-03-17T23:49:20.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.594+0000 7f5dcf7fe640 1 -- 172.21.15.150:0/2448181688 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dbc030880 con 0x7f5dd00b97f0 2024-03-17T23:49:20.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.594+0000 7f5dcf7fe640 1 -- 172.21.15.150:0/2448181688 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc030c30 con 0x7f5dd00b97f0 2024-03-17T23:49:20.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.594+0000 7f5dd6ab5640 1 -- 172.21.15.150:0/2448181688 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5dd00b97f0 msgr2=0x7f5dd0162160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:20.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.594+0000 7f5dd6ab5640 1 --2- 172.21.15.150:0/2448181688 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5dd00b97f0 0x7f5dd0162160 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc0097d0 tx=0x7f5dbc02b9c0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.594+0000 7f5dd6ab5640 1 -- 172.21.15.150:0/2448181688 shutdown_connections 2024-03-17T23:49:20.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.594+0000 7f5dd6ab5640 1 --2- 172.21.15.150:0/2448181688 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5dd00b97f0 0x7f5dd0162160 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.594+0000 7f5dd6ab5640 1 -- 172.21.15.150:0/2448181688 >> 172.21.15.150:0/2448181688 conn(0x7f5dd0072ef0 msgr2=0x7f5dd00d3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:20.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.595+0000 7f5dd6ab5640 1 -- 172.21.15.150:0/2448181688 shutdown_connections 2024-03-17T23:49:20.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.595+0000 7f5dd6ab5640 1 -- 172.21.15.150:0/2448181688 wait complete. 2024-03-17T23:49:20.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.596+0000 7f5dd6ab5640 1 Processor -- start 2024-03-17T23:49:20.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.596+0000 7f5dd6ab5640 1 -- start start 2024-03-17T23:49:20.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.596+0000 7f5dd6ab5640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5dd00b97f0 0x7f5dd00cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:20.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.596+0000 7f5dd6ab5640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5dd015df40 0x7f5dd00c8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:20.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.597+0000 7f5dd6ab5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd01626a0 0x7f5dd00c9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:20.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.597+0000 7f5dd6ab5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd0161c40 con 0x7f5dd01626a0 2024-03-17T23:49:20.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.597+0000 7f5dd6ab5640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd015ea30 con 0x7f5dd015df40 2024-03-17T23:49:20.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.597+0000 7f5dd6ab5640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5dd00d2810 con 0x7f5dd00b97f0 2024-03-17T23:49:20.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.597+0000 7f5dd5052640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd01626a0 0x7f5dd00c9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:20.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.597+0000 7f5dd5052640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd01626a0 0x7f5dd00c9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45576/0 (socket says 172.21.15.150:45576) 2024-03-17T23:49:20.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.597+0000 7f5dd4851640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5dd00b97f0 0x7f5dd00cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:20.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.597+0000 7f5dd5052640 1 -- 172.21.15.150:0/1378705267 learned_addr learned my addr 172.21.15.150:0/1378705267 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:20.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.597+0000 7f5dcffff640 1 --2- 172.21.15.150:0/1378705267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5dd015df40 0x7f5dd00c8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:20.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.597+0000 7f5dd5052640 1 -- 172.21.15.150:0/1378705267 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5dd00b97f0 msgr2=0x7f5dd00cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:20.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.597+0000 7f5dd5052640 1 --2- 172.21.15.150:0/1378705267 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5dd00b97f0 0x7f5dd00cc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.597+0000 7f5dd5052640 1 -- 172.21.15.150:0/1378705267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5dd015df40 msgr2=0x7f5dd00c8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:20.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.597+0000 7f5dd5052640 1 --2- 172.21.15.150:0/1378705267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5dd015df40 0x7f5dd00c8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.597+0000 7f5dd5052640 1 -- 172.21.15.150:0/1378705267 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd0151cc0 con 0x7f5dd01626a0 2024-03-17T23:49:20.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.597+0000 7f5dd5052640 1 --2- 172.21.15.150:0/1378705267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd01626a0 0x7f5dd00c9490 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f5dc800c790 tx=0x7f5dc800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:20.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.597+0000 7f5dcd7fa640 1 -- 172.21.15.150:0/1378705267 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc8003b70 con 0x7f5dd01626a0 2024-03-17T23:49:20.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.597+0000 7f5dcd7fa640 1 -- 172.21.15.150:0/1378705267 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dc8003d10 con 0x7f5dd01626a0 2024-03-17T23:49:20.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.597+0000 7f5dd6ab5640 1 -- 172.21.15.150:0/1378705267 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dd0151fa0 con 0x7f5dd01626a0 2024-03-17T23:49:20.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.597+0000 7f5dcd7fa640 1 -- 172.21.15.150:0/1378705267 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc800aa50 con 0x7f5dd01626a0 2024-03-17T23:49:20.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.597+0000 7f5dd6ab5640 1 -- 172.21.15.150:0/1378705267 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dd0154bd0 con 0x7f5dd01626a0 2024-03-17T23:49:20.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.599+0000 7f5dcd7fa640 1 -- 172.21.15.150:0/1378705267 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5dc800abf0 con 0x7f5dd01626a0 2024-03-17T23:49:20.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.599+0000 7f5dcd7fa640 1 --2- 172.21.15.150:0/1378705267 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5db406bbd0 0x7f5db406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:20.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.599+0000 7f5dcd7fa640 1 -- 172.21.15.150:0/1378705267 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f5dc808aa30 con 0x7f5dd01626a0 2024-03-17T23:49:20.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.599+0000 7f5dd4851640 1 --2- 172.21.15.150:0/1378705267 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5db406bbd0 0x7f5db406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:20.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.600+0000 7f5dd4851640 1 --2- 172.21.15.150:0/1378705267 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5db406bbd0 0x7f5db406e090 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4000f60 tx=0x7f5dc4007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:20.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.600+0000 7f5dd9777740 1 Processor -- start 2024-03-17T23:49:20.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.600+0000 7f5dd9777740 1 -- start start 2024-03-17T23:49:20.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.601+0000 7f5dd9777740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281ac80 0x5619727edc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:20.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.601+0000 7f5dd9777740 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x56197281b060 0x56197281da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:20.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.601+0000 7f5dd9777740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5619727ee160 0x56197281df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:20.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.601+0000 7f5dd9777740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x561972756c80 con 0x56197281ac80 2024-03-17T23:49:20.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.601+0000 7f5dd9777740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x5619726ffaa0 con 0x5619727ee160 2024-03-17T23:49:20.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.601+0000 7f5dd9777740 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x561972752d20 con 0x56197281b060 2024-03-17T23:49:20.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.601+0000 7f5dd4851640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281ac80 0x5619727edc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:20.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.601+0000 7f5dd4851640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281ac80 0x5619727edc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45580/0 (socket says 172.21.15.150:45580) 2024-03-17T23:49:20.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.601+0000 7f5dd4851640 1 -- 172.21.15.150:0/4290816737 learned_addr learned my addr 172.21.15.150:0/4290816737 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:20.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.601+0000 7f5dd5052640 1 --2- 172.21.15.150:0/4290816737 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5619727ee160 0x56197281df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:20.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.601+0000 7f5dcffff640 1 --2- 172.21.15.150:0/4290816737 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x56197281b060 0x56197281da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:20.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.601+0000 7f5dd4851640 1 -- 172.21.15.150:0/4290816737 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x56197281b060 msgr2=0x56197281da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:20.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.601+0000 7f5dd4851640 1 --2- 172.21.15.150:0/4290816737 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x56197281b060 0x56197281da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.601+0000 7f5dd4851640 1 -- 172.21.15.150:0/4290816737 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5619727ee160 msgr2=0x56197281df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:20.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.601+0000 7f5dd4851640 1 --2- 172.21.15.150:0/4290816737 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5619727ee160 0x56197281df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.601+0000 7f5dd4851640 1 -- 172.21.15.150:0/4290816737 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561972721ff0 con 0x56197281ac80 2024-03-17T23:49:20.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.601+0000 7f5dd4851640 1 --2- 172.21.15.150:0/4290816737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281ac80 0x5619727edc20 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4001840 tx=0x7f5dc4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b174b5be04c8c18 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:20.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.602+0000 7f5dba7fc640 1 -- 172.21.15.150:0/4290816737 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4025070 con 0x56197281ac80 2024-03-17T23:49:20.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.602+0000 7f5dba7fc640 1 -- 172.21.15.150:0/4290816737 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dc401b050 con 0x56197281ac80 2024-03-17T23:49:20.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.602+0000 7f5dba7fc640 1 -- 172.21.15.150:0/4290816737 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4014de0 con 0x56197281ac80 2024-03-17T23:49:20.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.602+0000 7f5dd9777740 1 -- 172.21.15.150:0/4290816737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281ac80 msgr2=0x5619727edc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:20.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.602+0000 7f5dd9777740 1 --2- 172.21.15.150:0/4290816737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281ac80 0x5619727edc20 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4001840 tx=0x7f5dc4012000 comp rx=0 tx=0).stop 2024-03-17T23:49:20.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.602+0000 7f5dd9777740 1 -- 172.21.15.150:0/4290816737 shutdown_connections 2024-03-17T23:49:20.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.602+0000 7f5dd9777740 1 --2- 172.21.15.150:0/4290816737 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5619727ee160 0x56197281df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.602+0000 7f5dd9777740 1 --2- 172.21.15.150:0/4290816737 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x56197281b060 0x56197281da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.602+0000 7f5dd9777740 1 --2- 172.21.15.150:0/4290816737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281ac80 0x5619727edc20 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.602+0000 7f5dd9777740 1 -- 172.21.15.150:0/4290816737 >> 172.21.15.150:0/4290816737 conn(0x5619727efe90 msgr2=0x561972838a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:20.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.602+0000 7f5dd9777740 1 -- 172.21.15.150:0/4290816737 shutdown_connections 2024-03-17T23:49:20.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.602+0000 7f5dd9777740 1 -- 172.21.15.150:0/4290816737 wait complete. 2024-03-17T23:49:20.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.603+0000 7f5dd9777740 1 -- start start 2024-03-17T23:49:20.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.603+0000 7f5dd9777740 1 Processor -- start 2024-03-17T23:49:20.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.604+0000 7f5dd9777740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281ac80 0x561972815950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:20.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.604+0000 7f5dd9777740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56197281b060 0x561972813cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:20.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.604+0000 7f5dd9777740 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x561972801520 0x5619728019a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:20.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.604+0000 7f5dd9777740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5619727f2530 con 0x56197281ac80 2024-03-17T23:49:20.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.604+0000 7f5dd9777740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x561972712ef0 con 0x56197281b060 2024-03-17T23:49:20.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.604+0000 7f5dd9777740 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x5619727f4d70 con 0x561972801520 2024-03-17T23:49:20.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.604+0000 7f5dd9777740 10 client.0 _ll_register_callbacks cb 0x56197270ddc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:49:20.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.604+0000 7f5dd4851640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281ac80 0x561972815950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:20.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.604+0000 7f5dd4851640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281ac80 0x561972815950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45596/0 (socket says 172.21.15.150:45596) 2024-03-17T23:49:20.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.604+0000 7f5dd4851640 1 -- 172.21.15.150:0/2186793042 learned_addr learned my addr 172.21.15.150:0/2186793042 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:20.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.604+0000 7f5dd5052640 1 --2- 172.21.15.150:0/2186793042 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x561972801520 0x5619728019a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:20.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.604+0000 7f5dcffff640 1 --2- 172.21.15.150:0/2186793042 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56197281b060 0x561972813cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:20.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.604+0000 7f5dd4851640 1 -- 172.21.15.150:0/2186793042 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x561972801520 msgr2=0x5619728019a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:20.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.604+0000 7f5dd4851640 1 --2- 172.21.15.150:0/2186793042 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x561972801520 0x5619728019a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.604+0000 7f5dd4851640 1 -- 172.21.15.150:0/2186793042 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56197281b060 msgr2=0x561972813cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:20.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.604+0000 7f5dd4851640 1 --2- 172.21.15.150:0/2186793042 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56197281b060 0x561972813cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.604+0000 7f5dd4851640 1 -- 172.21.15.150:0/2186793042 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56197271fa30 con 0x56197281ac80 2024-03-17T23:49:20.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.604+0000 7f5dd5052640 1 --2- 172.21.15.150:0/2186793042 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x561972801520 0x5619728019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:20.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.604+0000 7f5dd4851640 1 --2- 172.21.15.150:0/2186793042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281ac80 0x561972815950 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f5dc40206c0 tx=0x7f5dc4000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39fd79e178a45e77 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:20.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.604+0000 7f5daf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-03-17T23:49:20.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.604+0000 7f5daf7fe640 1 -- 172.21.15.150:0/2186793042 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4032060 con 0x56197281ac80 2024-03-17T23:49:20.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.604+0000 7f5daf7fe640 1 -- 172.21.15.150:0/2186793042 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f5dc402dc70 con 0x56197281ac80 2024-03-17T23:49:20.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.604+0000 7f5dd9777740 1 -- 172.21.15.150:0/2186793042 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5dc401b050 con 0x56197281ac80 2024-03-17T23:49:20.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.605+0000 7f5daf7fe640 1 -- 172.21.15.150:0/2186793042 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f5dc4025070 con 0x56197281ac80 2024-03-17T23:49:20.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.605+0000 7f5dd9777740 10 client.6363 fetch_fsmap learned FSMap version 22 2024-03-17T23:49:20.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.605+0000 7f5dd9777740 1 -- 172.21.15.150:0/2186793042 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=2+}) -- 0x56197246b2d0 con 0x56197281ac80 2024-03-17T23:49:20.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.605+0000 7f5daf7fe640 1 -- 172.21.15.150:0/2186793042 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dc401b050 con 0x56197281ac80 2024-03-17T23:49:20.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.605+0000 7f5daf7fe640 1 -- 172.21.15.150:0/2186793042 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 22) ==== 3494+0+0 (secure 0 0 0) 0x7f5dc402b070 con 0x56197281ac80 2024-03-17T23:49:20.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.605+0000 7f5dd9777740 10 client.6363 fetch_fsmap finished waiting for FSMap version 22 2024-03-17T23:49:20.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.605+0000 7f5dd9777740 10 client.6363 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-17T23:49:20.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.605+0000 7f5dd9777740 20 client.6363 populate_metadata read hostname 'smithi150' 2024-03-17T23:49:20.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.605+0000 7f5dd9777740 1 --2- 172.21.15.150:0/2186793042 >> [v2:172.21.15.157:6833/479821580,v1:172.21.15.157:6835/479821580] conn(0x56197281c150 0x56197281c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:20.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.605+0000 7f5dd9777740 4 client.6363 mds_command: new command op to 6204 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-17T23:49:20.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.605+0000 7f5dd9777740 1 -- 172.21.15.150:0/2186793042 --> [v2:172.21.15.157:6833/479821580,v1:172.21.15.157:6835/479821580] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5619727641c0 con 0x56197281c150 2024-03-17T23:49:20.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.605+0000 7f5dcffff640 1 --2- 172.21.15.150:0/2186793042 >> [v2:172.21.15.157:6833/479821580,v1:172.21.15.157:6835/479821580] conn(0x56197281c150 0x56197281c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:20.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.605+0000 7f5dcffff640 1 --2- 172.21.15.150:0/2186793042 >> [v2:172.21.15.157:6833/479821580,v1:172.21.15.157:6835/479821580] conn(0x56197281c150 0x56197281c530 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29d0a861dc7e111 server_cookie=fd20e289ff71efac in_seq=0 out_seq=0 2024-03-17T23:49:20.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.605+0000 7f5daf7fe640 10 client.6363 ms_handle_connect on v2:172.21.15.157:6833/479821580 2024-03-17T23:49:20.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.606+0000 7f5daf7fe640 1 -- 172.21.15.150:0/2186793042 <== mds.0 v2:172.21.15.157:6833/479821580 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5619727641c0 con 0x56197281c150 2024-03-17T23:49:20.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.606+0000 7f5daf7fe640 10 client.6363 handle_command_reply: tid=0 2024-03-17T23:49:20.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.606+0000 7f5dd9777740 1 client.6363 shutdown 2024-03-17T23:49:20.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.606+0000 7f5dd9777740 1 -- 172.21.15.150:0/2186793042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281ac80 msgr2=0x561972815950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:20.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.606+0000 7f5dd9777740 1 --2- 172.21.15.150:0/2186793042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281ac80 0x561972815950 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f5dc40206c0 tx=0x7f5dc4000ed0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.606+0000 7f5dd9777740 1 -- 172.21.15.150:0/2186793042 shutdown_connections 2024-03-17T23:49:20.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.606+0000 7f5dd9777740 1 --2- 172.21.15.150:0/2186793042 >> [v2:172.21.15.157:6833/479821580,v1:172.21.15.157:6835/479821580] conn(0x56197281c150 0x56197281c530 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.606+0000 7f5dd9777740 1 --2- 172.21.15.150:0/2186793042 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x561972801520 0x5619728019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.606+0000 7f5dd9777740 1 --2- 172.21.15.150:0/2186793042 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56197281b060 0x561972813cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.606+0000 7f5daf7fe640 0 client.6363 ms_handle_reset on v2:172.21.15.157:6833/479821580 2024-03-17T23:49:20.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.606+0000 7f5dd9777740 1 --2- 172.21.15.150:0/2186793042 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281ac80 0x561972815950 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.606+0000 7f5dd9777740 1 -- 172.21.15.150:0/2186793042 >> 172.21.15.150:0/2186793042 conn(0x5619727ee160 msgr2=0x5619727f1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:20.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.606+0000 7f5dd9777740 1 -- 172.21.15.150:0/2186793042 shutdown_connections 2024-03-17T23:49:20.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.606+0000 7f5dd9777740 1 -- 172.21.15.150:0/2186793042 wait complete. 2024-03-17T23:49:20.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.607+0000 7f5dd9777740 20 client.6363 trim_cache size 0 max 16384 2024-03-17T23:49:20.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.624+0000 7f5dd9777740 1 Processor -- start 2024-03-17T23:49:20.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.624+0000 7f5dd9777740 1 -- start start 2024-03-17T23:49:20.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.625+0000 7f5dd9777740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561972801520 0x5619727f4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:20.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.625+0000 7f5dd9777740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281ac80 0x5619727f4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:20.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.625+0000 7f5dd9777740 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x56197281b060 0x56197284b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:20.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.625+0000 7f5dd9777740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd01ffaf0 con 0x56197281ac80 2024-03-17T23:49:20.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.625+0000 7f5dd9777740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd0157100 con 0x561972801520 2024-03-17T23:49:20.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.625+0000 7f5dd9777740 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5dd0157750 con 0x56197281b060 2024-03-17T23:49:20.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.625+0000 7f5dcffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281ac80 0x5619727f4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:20.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.625+0000 7f5dcffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281ac80 0x5619727f4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45600/0 (socket says 172.21.15.150:45600) 2024-03-17T23:49:20.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.625+0000 7f5dcffff640 1 -- 172.21.15.150:0/3397083583 learned_addr learned my addr 172.21.15.150:0/3397083583 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:20.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.625+0000 7f5dd5052640 1 --2- 172.21.15.150:0/3397083583 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x56197281b060 0x56197284b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:20.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.625+0000 7f5dd4851640 1 --2- 172.21.15.150:0/3397083583 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561972801520 0x5619727f4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:20.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.625+0000 7f5dcffff640 1 -- 172.21.15.150:0/3397083583 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x56197281b060 msgr2=0x56197284b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:20.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.625+0000 7f5dcffff640 1 --2- 172.21.15.150:0/3397083583 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x56197281b060 0x56197284b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.625+0000 7f5dcffff640 1 -- 172.21.15.150:0/3397083583 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561972801520 msgr2=0x5619727f4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:20.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.625+0000 7f5dcffff640 1 --2- 172.21.15.150:0/3397083583 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561972801520 0x5619727f4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.625+0000 7f5dcffff640 1 -- 172.21.15.150:0/3397083583 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619725356b0 con 0x56197281ac80 2024-03-17T23:49:20.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.626+0000 7f5dcffff640 1 --2- 172.21.15.150:0/3397083583 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281ac80 0x5619727f4ca0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc0097d0 tx=0x7f5dbc030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9967b4eee99302fc server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:20.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.626+0000 7f5db9ffb640 1 -- 172.21.15.150:0/3397083583 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc051070 con 0x56197281ac80 2024-03-17T23:49:20.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.626+0000 7f5db9ffb640 1 -- 172.21.15.150:0/3397083583 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dbc030a40 con 0x56197281ac80 2024-03-17T23:49:20.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.626+0000 7f5db9ffb640 1 -- 172.21.15.150:0/3397083583 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc034040 con 0x56197281ac80 2024-03-17T23:49:20.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.626+0000 7f5dd9777740 1 -- 172.21.15.150:0/3397083583 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281ac80 msgr2=0x5619727f4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:20.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.626+0000 7f5dd9777740 1 --2- 172.21.15.150:0/3397083583 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281ac80 0x5619727f4ca0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc0097d0 tx=0x7f5dbc030000 comp rx=0 tx=0).stop 2024-03-17T23:49:20.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.626+0000 7f5dd9777740 1 -- 172.21.15.150:0/3397083583 shutdown_connections 2024-03-17T23:49:20.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.626+0000 7f5dd9777740 1 --2- 172.21.15.150:0/3397083583 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x56197281b060 0x56197284b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.626+0000 7f5dd9777740 1 --2- 172.21.15.150:0/3397083583 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281ac80 0x5619727f4ca0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.626+0000 7f5dd9777740 1 --2- 172.21.15.150:0/3397083583 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561972801520 0x5619727f4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.626+0000 7f5dd9777740 1 -- 172.21.15.150:0/3397083583 >> 172.21.15.150:0/3397083583 conn(0x5619727ee160 msgr2=0x56197283c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:20.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.626+0000 7f5dd9777740 1 -- 172.21.15.150:0/3397083583 shutdown_connections 2024-03-17T23:49:20.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.626+0000 7f5dd9777740 1 -- 172.21.15.150:0/3397083583 wait complete. 2024-03-17T23:49:20.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.627+0000 7f5dd9777740 1 -- start start 2024-03-17T23:49:20.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.627+0000 7f5dd9777740 1 Processor -- start 2024-03-17T23:49:20.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.628+0000 7f5dd9777740 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561972801520 0x56197289a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:20.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.628+0000 7f5dd9777740 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x56197281ac80 0x56197281c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:20.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.628+0000 7f5dd9777740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281b060 0x56197284b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:20.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.628+0000 7f5dd9777740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x561972801ee0 con 0x56197281b060 2024-03-17T23:49:20.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.628+0000 7f5dd9777740 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x561972816a40 con 0x561972801520 2024-03-17T23:49:20.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.628+0000 7f5dd9777740 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x56197284c780 con 0x56197281ac80 2024-03-17T23:49:20.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.628+0000 7f5dd9777740 10 client.0 _ll_register_callbacks cb 0x7f5dc4023070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-17T23:49:20.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.628+0000 7f5dd4851640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561972801520 0x56197289a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:20.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.628+0000 7f5dd5052640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281b060 0x56197284b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:20.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.628+0000 7f5dcffff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x56197281ac80 0x56197281c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:20.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.628+0000 7f5dd4851640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561972801520 0x56197289a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:58464/0 (socket says 172.21.15.150:58464) 2024-03-17T23:49:20.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.628+0000 7f5dd4851640 1 -- 172.21.15.150:0/501977764 learned_addr learned my addr 172.21.15.150:0/501977764 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:20.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.628+0000 7f5dcffff640 1 -- 172.21.15.150:0/501977764 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561972801520 msgr2=0x56197289a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:20.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.628+0000 7f5dcffff640 1 --2- 172.21.15.150:0/501977764 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561972801520 0x56197289a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.628+0000 7f5dcffff640 1 -- 172.21.15.150:0/501977764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281b060 msgr2=0x56197284b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:20.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.628+0000 7f5dcffff640 1 --2- 172.21.15.150:0/501977764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281b060 0x56197284b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.628+0000 7f5dcffff640 1 -- 172.21.15.150:0/501977764 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x5619728b2670 con 0x56197281ac80 2024-03-17T23:49:20.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.628+0000 7f5dd5052640 1 --2- 172.21.15.150:0/501977764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281b060 0x56197284b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:20.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.628+0000 7f5dcffff640 1 --2- 172.21.15.150:0/501977764 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x56197281ac80 0x56197281c7d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc02bfa0 tx=0x7f5dbc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce61c513b01f5652 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:20.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.628+0000 7f5daf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.157:3301/0 2024-03-17T23:49:20.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.628+0000 7f5daf7fe640 1 -- 172.21.15.150:0/501977764 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc051070 con 0x56197281ac80 2024-03-17T23:49:20.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.628+0000 7f5daf7fe640 1 -- 172.21.15.150:0/501977764 <== mon.2 v2:172.21.15.157:3301/0 2 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f5dbc005990 con 0x56197281ac80 2024-03-17T23:49:20.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.628+0000 7f5dd9777740 1 -- 172.21.15.150:0/501977764 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5dbc030a40 con 0x56197281ac80 2024-03-17T23:49:20.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.629+0000 7f5daf7fe640 1 -- 172.21.15.150:0/501977764 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f5dbc034040 con 0x56197281ac80 2024-03-17T23:49:20.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.629+0000 7f5dd9777740 10 client.5666 fetch_fsmap learned FSMap version 22 2024-03-17T23:49:20.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.629+0000 7f5dd9777740 1 -- 172.21.15.150:0/501977764 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,fsmap=22,monmap=2+}) -- 0x5619728b2190 con 0x56197281ac80 2024-03-17T23:49:20.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.629+0000 7f5daf7fe640 1 -- 172.21.15.150:0/501977764 <== mon.2 v2:172.21.15.157:3301/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dbc030a40 con 0x56197281ac80 2024-03-17T23:49:20.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.629+0000 7f5daf7fe640 1 -- 172.21.15.150:0/501977764 <== mon.2 v2:172.21.15.157:3301/0 5 ==== fsmap(e 22) ==== 3494+0+0 (secure 0 0 0) 0x7f5dbc005b90 con 0x56197281ac80 2024-03-17T23:49:20.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.629+0000 7f5dd9777740 10 client.5666 fetch_fsmap finished waiting for FSMap version 22 2024-03-17T23:49:20.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.629+0000 7f5dd9777740 10 client.5666 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-17T23:49:20.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.629+0000 7f5dd9777740 20 client.5666 populate_metadata read hostname 'smithi150' 2024-03-17T23:49:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.629+0000 7f5dd9777740 1 --2- 172.21.15.150:0/501977764 >> [v2:172.21.15.157:6833/479821580,v1:172.21.15.157:6835/479821580] conn(0x561972821c60 0x5619728531d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.629+0000 7f5dd9777740 4 client.5666 mds_command: new command op to 6204 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-17T23:49:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.629+0000 7f5dd9777740 1 -- 172.21.15.150:0/501977764 --> [v2:172.21.15.157:6833/479821580,v1:172.21.15.157:6835/479821580] -- command(tid 0: {"prefix": "status"}) -- 0x56197275e510 con 0x561972821c60 2024-03-17T23:49:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.629+0000 7f5dd4851640 1 --2- 172.21.15.150:0/501977764 >> [v2:172.21.15.157:6833/479821580,v1:172.21.15.157:6835/479821580] conn(0x561972821c60 0x5619728531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.629+0000 7f5daf7fe640 10 client.5666 ms_handle_connect on v2:172.21.15.157:6833/479821580 2024-03-17T23:49:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.629+0000 7f5dd4851640 1 --2- 172.21.15.150:0/501977764 >> [v2:172.21.15.157:6833/479821580,v1:172.21.15.157:6835/479821580] conn(0x561972821c60 0x5619728531d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=247ef534842d6769 server_cookie=b85d262775760838 in_seq=0 out_seq=0 2024-03-17T23:49:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.630+0000 7f5daf7fe640 1 -- 172.21.15.150:0/501977764 <== mds.0 v2:172.21.15.157:6833/479821580 1 ==== command_reply(tid 0: 0 ) ==== 8+0+318 (crc 0 0 0) 0x56197275e510 con 0x561972821c60 2024-03-17T23:49:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.630+0000 7f5daf7fe640 10 client.5666 handle_command_reply: tid=0 2024-03-17T23:49:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.630+0000 7f5dd9777740 1 client.5666 shutdown 2024-03-17T23:49:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.630+0000 7f5dd9777740 1 -- 172.21.15.150:0/501977764 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x56197281ac80 msgr2=0x56197281c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.630+0000 7f5dd9777740 1 --2- 172.21.15.150:0/501977764 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x56197281ac80 0x56197281c7d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc02bfa0 tx=0x7f5dbc004730 comp rx=0 tx=0).stop 2024-03-17T23:49:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.630+0000 7f5dd9777740 1 -- 172.21.15.150:0/501977764 shutdown_connections 2024-03-17T23:49:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.630+0000 7f5dd9777740 1 --2- 172.21.15.150:0/501977764 >> [v2:172.21.15.157:6833/479821580,v1:172.21.15.157:6835/479821580] conn(0x561972821c60 0x5619728531d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.630+0000 7f5dd9777740 1 --2- 172.21.15.150:0/501977764 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x56197281b060 0x56197284b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.630+0000 7f5dd9777740 1 --2- 172.21.15.150:0/501977764 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x56197281ac80 0x56197281c7d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.630+0000 7f5dd9777740 1 --2- 172.21.15.150:0/501977764 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561972801520 0x56197289a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.630+0000 7f5daf7fe640 0 client.5666 ms_handle_reset on v2:172.21.15.157:6833/479821580 2024-03-17T23:49:20.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.630+0000 7f5dd9777740 1 -- 172.21.15.150:0/501977764 >> 172.21.15.150:0/501977764 conn(0x5619727ee160 msgr2=0x561972807fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:20.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.630+0000 7f5dd9777740 1 -- 172.21.15.150:0/501977764 shutdown_connections 2024-03-17T23:49:20.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.630+0000 7f5dd9777740 1 -- 172.21.15.150:0/501977764 wait complete. 2024-03-17T23:49:20.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.630+0000 7f5dd9777740 20 client.5666 trim_cache size 0 max 16384 2024-03-17T23:49:20.637 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-17T23:49:20.637 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_fsid": "a1c5acd3-e97f-470b-8226-335736a7795b", 2024-03-17T23:49:20.637 INFO:teuthology.orchestra.run.smithi150.stdout: "whoami": 0, 2024-03-17T23:49:20.638 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 6204, 2024-03-17T23:49:20.638 INFO:teuthology.orchestra.run.smithi150.stdout: "want_state": "up:active", 2024-03-17T23:49:20.638 INFO:teuthology.orchestra.run.smithi150.stdout: "state": "up:active", 2024-03-17T23:49:20.638 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "cephfs", 2024-03-17T23:49:20.638 INFO:teuthology.orchestra.run.smithi150.stdout: "rank_uptime": 8.983523602, 2024-03-17T23:49:20.638 INFO:teuthology.orchestra.run.smithi150.stdout: "mdsmap_epoch": 22, 2024-03-17T23:49:20.638 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch": 58, 2024-03-17T23:49:20.638 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch_barrier": 58, 2024-03-17T23:49:20.638 INFO:teuthology.orchestra.run.smithi150.stdout: "uptime": 14.312386425 2024-03-17T23:49:20.638 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-17T23:49:20.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.630+0000 7f5dd6ab5640 1 -- 172.21.15.150:0/1378705267 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5db406bbd0 msgr2=0x7f5db406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:20.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.630+0000 7f5dd6ab5640 1 --2- 172.21.15.150:0/1378705267 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5db406bbd0 0x7f5db406e090 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4000f60 tx=0x7f5dc4007d50 comp rx=0 tx=0).stop 2024-03-17T23:49:20.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.630+0000 7f5dd6ab5640 1 -- 172.21.15.150:0/1378705267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd01626a0 msgr2=0x7f5dd00c9490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:20.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.630+0000 7f5dd6ab5640 1 --2- 172.21.15.150:0/1378705267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd01626a0 0x7f5dd00c9490 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f5dc800c790 tx=0x7f5dc800cc50 comp rx=0 tx=0).stop 2024-03-17T23:49:20.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.631+0000 7f5dd6ab5640 1 -- 172.21.15.150:0/1378705267 shutdown_connections 2024-03-17T23:49:20.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.631+0000 7f5dd6ab5640 1 --2- 172.21.15.150:0/1378705267 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5dd01626a0 0x7f5dd00c9490 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.631+0000 7f5dd6ab5640 1 --2- 172.21.15.150:0/1378705267 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5db406bbd0 0x7f5db406e090 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.631+0000 7f5dd6ab5640 1 --2- 172.21.15.150:0/1378705267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5dd015df40 0x7f5dd00c8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.631+0000 7f5dd6ab5640 1 --2- 172.21.15.150:0/1378705267 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5dd00b97f0 0x7f5dd00cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.631+0000 7f5dd6ab5640 1 -- 172.21.15.150:0/1378705267 >> 172.21.15.150:0/1378705267 conn(0x7f5dd0072ef0 msgr2=0x7f5dd00d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:20.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.631+0000 7f5dd6ab5640 1 -- 172.21.15.150:0/1378705267 shutdown_connections 2024-03-17T23:49:20.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.631+0000 7f5dd6ab5640 1 -- 172.21.15.150:0/1378705267 wait complete. 2024-03-17T23:49:20.642 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-17T23:49:20.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.742+0000 7fa66d159640 1 Processor -- start 2024-03-17T23:49:20.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.742+0000 7fa66d159640 1 -- start start 2024-03-17T23:49:20.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.743+0000 7fa66d159640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fa66815de30 0x7fa6681619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:20.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.743+0000 7fa66d159640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa6680cf120 0x7fa6680d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:20.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.743+0000 7fa66d159640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6680b9cf0 con 0x7fa6680b9860 2024-03-17T23:49:20.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.743+0000 7fa66d159640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fa668162940 con 0x7fa6680cf120 2024-03-17T23:49:20.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.743+0000 7fa66d159640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fa6681620a0 con 0x7fa66815de30 2024-03-17T23:49:20.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.743+0000 7fa66759e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa6680cf120 0x7fa6680d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:20.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.743+0000 7fa66759e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa6680cf120 0x7fa6680d2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45624/0 (socket says 172.21.15.150:45624) 2024-03-17T23:49:20.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.743+0000 7fa66759e640 1 -- 172.21.15.150:0/3293737672 learned_addr learned my addr 172.21.15.150:0/3293737672 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:20.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.743+0000 7fa666d9d640 1 --2- 172.21.15.150:0/3293737672 >> v2:172.21.15.157:3300/0 conn(0x7fa66815de30 0x7fa6681619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:20.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.744+0000 7fa66759e640 1 -- 172.21.15.150:0/3293737672 >> v2:172.21.15.157:3300/0 conn(0x7fa66815de30 msgr2=0x7fa6681619a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:20.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.744+0000 7fa66759e640 1 --2- 172.21.15.150:0/3293737672 >> v2:172.21.15.157:3300/0 conn(0x7fa66815de30 0x7fa6681619a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.744+0000 7fa66759e640 1 -- 172.21.15.150:0/3293737672 >> v1:172.21.15.157:6789/0 conn(0x7fa6680b9860 legacy=0x7fa6680cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:20.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.744+0000 7fa66759e640 1 -- 172.21.15.150:0/3293737672 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa668162380 con 0x7fa6680cf120 2024-03-17T23:49:20.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.744+0000 7fa66759e640 1 --2- 172.21.15.150:0/3293737672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6680cf120 0x7fa6680d2690 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fa66806d7c0 tx=0x7fa66002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52a615bae122546e server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:20.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.744+0000 7fa665d9b640 1 -- 172.21.15.150:0/3293737672 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa660035020 con 0x7fa6680cf120 2024-03-17T23:49:20.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.744+0000 7fa665d9b640 1 -- 172.21.15.150:0/3293737672 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa660031870 con 0x7fa6680cf120 2024-03-17T23:49:20.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.744+0000 7fa665d9b640 1 -- 172.21.15.150:0/3293737672 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa660031c40 con 0x7fa6680cf120 2024-03-17T23:49:20.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.745+0000 7fa66d159640 1 -- 172.21.15.150:0/3293737672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6680cf120 msgr2=0x7fa6680d2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:20.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.745+0000 7fa66d159640 1 --2- 172.21.15.150:0/3293737672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6680cf120 0x7fa6680d2690 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fa66806d7c0 tx=0x7fa66002b910 comp rx=0 tx=0).stop 2024-03-17T23:49:20.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.745+0000 7fa66d159640 1 -- 172.21.15.150:0/3293737672 shutdown_connections 2024-03-17T23:49:20.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.745+0000 7fa66d159640 1 --2- 172.21.15.150:0/3293737672 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6680cf120 0x7fa6680d2690 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.745+0000 7fa66d159640 1 --2- 172.21.15.150:0/3293737672 >> v2:172.21.15.157:3300/0 conn(0x7fa66815de30 0x7fa6681619a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.745+0000 7fa66d159640 1 -- 172.21.15.150:0/3293737672 >> 172.21.15.150:0/3293737672 conn(0x7fa6680730b0 msgr2=0x7fa6680d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:20.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.745+0000 7fa66d159640 1 -- 172.21.15.150:0/3293737672 shutdown_connections 2024-03-17T23:49:20.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.745+0000 7fa66d159640 1 -- 172.21.15.150:0/3293737672 wait complete. 2024-03-17T23:49:20.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.746+0000 7fa66d159640 1 Processor -- start 2024-03-17T23:49:20.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.746+0000 7fa66d159640 1 -- start start 2024-03-17T23:49:20.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.747+0000 7fa66d159640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6680b9860 0x7fa6680cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:20.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.747+0000 7fa66d159640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6680cf120 0x7fa668158bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:20.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.747+0000 7fa66d159640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa66815de30 0x7fa6680b5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:20.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.747+0000 7fa66d159640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa66815e210 con 0x7fa6680cf120 2024-03-17T23:49:20.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.747+0000 7fa66d159640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa6680d2170 con 0x7fa6680b9860 2024-03-17T23:49:20.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.747+0000 7fa66d159640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa6680d2bd0 con 0x7fa66815de30 2024-03-17T23:49:20.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.747+0000 7fa66659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6680cf120 0x7fa668158bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:20.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.747+0000 7fa66659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6680cf120 0x7fa668158bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45640/0 (socket says 172.21.15.150:45640) 2024-03-17T23:49:20.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.747+0000 7fa66659c640 1 -- 172.21.15.150:0/1215089481 learned_addr learned my addr 172.21.15.150:0/1215089481 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:20.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.747+0000 7fa66759e640 1 --2- 172.21.15.150:0/1215089481 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa66815de30 0x7fa6680b5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:20.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.747+0000 7fa666d9d640 1 --2- 172.21.15.150:0/1215089481 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6680b9860 0x7fa6680cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:20.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.748+0000 7fa66659c640 1 -- 172.21.15.150:0/1215089481 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa66815de30 msgr2=0x7fa6680b5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:20.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.748+0000 7fa66659c640 1 --2- 172.21.15.150:0/1215089481 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa66815de30 0x7fa6680b5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.748+0000 7fa66659c640 1 -- 172.21.15.150:0/1215089481 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6680b9860 msgr2=0x7fa6680cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:20.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.748+0000 7fa66659c640 1 --2- 172.21.15.150:0/1215089481 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6680b9860 0x7fa6680cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.748+0000 7fa66659c640 1 -- 172.21.15.150:0/1215089481 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa66815f680 con 0x7fa6680cf120 2024-03-17T23:49:20.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.748+0000 7fa66659c640 1 --2- 172.21.15.150:0/1215089481 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6680cf120 0x7fa668158bb0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fa65c00c450 tx=0x7fa65c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:20.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.748+0000 7fa64f7fe640 1 -- 172.21.15.150:0/1215089481 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c0052d0 con 0x7fa6680cf120 2024-03-17T23:49:20.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.748+0000 7fa64f7fe640 1 -- 172.21.15.150:0/1215089481 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa65c005470 con 0x7fa6680cf120 2024-03-17T23:49:20.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.748+0000 7fa66d159640 1 -- 172.21.15.150:0/1215089481 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa66815e5e0 con 0x7fa6680cf120 2024-03-17T23:49:20.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.748+0000 7fa64f7fe640 1 -- 172.21.15.150:0/1215089481 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c012870 con 0x7fa6680cf120 2024-03-17T23:49:20.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.748+0000 7fa66d159640 1 -- 172.21.15.150:0/1215089481 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa66815eba0 con 0x7fa6680cf120 2024-03-17T23:49:20.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.749+0000 7fa66d159640 1 -- 172.21.15.150:0/1215089481 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6680b2ab0 con 0x7fa6680cf120 2024-03-17T23:49:20.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.749+0000 7fa64f7fe640 1 -- 172.21.15.150:0/1215089481 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa65c012a10 con 0x7fa6680cf120 2024-03-17T23:49:20.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.750+0000 7fa64f7fe640 1 --2- 172.21.15.150:0/1215089481 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa65406bbd0 0x7fa65406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:20.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.750+0000 7fa64f7fe640 1 -- 172.21.15.150:0/1215089481 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7fa65c08a6e0 con 0x7fa6680cf120 2024-03-17T23:49:20.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.750+0000 7fa666d9d640 1 --2- 172.21.15.150:0/1215089481 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa65406bbd0 0x7fa65406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:20.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.750+0000 7fa666d9d640 1 --2- 172.21.15.150:0/1215089481 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa65406bbd0 0x7fa65406e090 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa6680b3070 tx=0x7fa65800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:20.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.754+0000 7fa64f7fe640 1 -- 172.21.15.150:0/1215089481 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fa6680b2ab0 con 0x7fa6680cf120 2024-03-17T23:49:20.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.963+0000 7fa66d159640 1 -- 172.21.15.150:0/1215089481 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa6680681a0 con 0x7fa6680cf120 2024-03-17T23:49:20.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.963+0000 7fa64f7fe640 1 -- 172.21.15.150:0/1215089481 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v22) ==== 53+0+73 (secure 0 0 0) 0x7fa65c0589f0 con 0x7fa6680cf120 2024-03-17T23:49:20.964 INFO:teuthology.orchestra.run.smithi150.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-17T23:49:20.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.965+0000 7fa66d159640 1 -- 172.21.15.150:0/1215089481 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa65406bbd0 msgr2=0x7fa65406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:20.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.965+0000 7fa66d159640 1 --2- 172.21.15.150:0/1215089481 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa65406bbd0 0x7fa65406e090 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa6680b3070 tx=0x7fa65800b040 comp rx=0 tx=0).stop 2024-03-17T23:49:20.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.965+0000 7fa66d159640 1 -- 172.21.15.150:0/1215089481 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6680cf120 msgr2=0x7fa668158bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:20.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.965+0000 7fa66d159640 1 --2- 172.21.15.150:0/1215089481 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6680cf120 0x7fa668158bb0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fa65c00c450 tx=0x7fa65c00c910 comp rx=0 tx=0).stop 2024-03-17T23:49:20.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.965+0000 7fa66d159640 1 -- 172.21.15.150:0/1215089481 shutdown_connections 2024-03-17T23:49:20.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.965+0000 7fa66d159640 1 --2- 172.21.15.150:0/1215089481 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa66815de30 0x7fa6680b5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.965+0000 7fa66d159640 1 --2- 172.21.15.150:0/1215089481 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6680cf120 0x7fa668158bb0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.965+0000 7fa66d159640 1 --2- 172.21.15.150:0/1215089481 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa65406bbd0 0x7fa65406e090 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.965+0000 7fa66d159640 1 --2- 172.21.15.150:0/1215089481 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa6680b9860 0x7fa6680cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:20.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.965+0000 7fa66d159640 1 -- 172.21.15.150:0/1215089481 >> 172.21.15.150:0/1215089481 conn(0x7fa6680730b0 msgr2=0x7fa6680b8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:20.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.965+0000 7fa66d159640 1 -- 172.21.15.150:0/1215089481 shutdown_connections 2024-03-17T23:49:20.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:20.965+0000 7fa66d159640 1 -- 172.21.15.150:0/1215089481 wait complete. 2024-03-17T23:49:20.977 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-17T23:49:20.977 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-17T23:49:20.977 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi150.front.sepia.ceph.com 2024-03-17T23:49:20.977 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-17T23:49:20.977 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-17T23:49:20.977 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-17T23:49:20.977 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-17T23:49:20.977 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-17T23:49:20.977 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-17T23:49:20.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:20.979 DEBUG:teuthology.orchestra.run.smithi150:> ip addr 2024-03-17T23:49:20.995 INFO:teuthology.orchestra.run.smithi150.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-17T23:49:20.995 INFO:teuthology.orchestra.run.smithi150.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-17T23:49:20.995 INFO:teuthology.orchestra.run.smithi150.stdout: inet 127.0.0.1/8 scope host lo 2024-03-17T23:49:20.995 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:49:20.995 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 ::1/128 scope host 2024-03-17T23:49:20.995 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:49:20.995 INFO:teuthology.orchestra.run.smithi150.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:49:20.995 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bc brd ff:ff:ff:ff:ff:ff 2024-03-17T23:49:20.995 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f0 2024-03-17T23:49:20.995 INFO:teuthology.orchestra.run.smithi150.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:49:20.995 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bd brd ff:ff:ff:ff:ff:ff 2024-03-17T23:49:20.996 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f1 2024-03-17T23:49:20.996 INFO:teuthology.orchestra.run.smithi150.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:49:20.996 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:34 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:49:20.996 INFO:teuthology.orchestra.run.smithi150.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-17T23:49:20.996 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:35 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:49:20.996 INFO:teuthology.orchestra.run.smithi150.stdout: inet 172.21.15.150/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-17T23:49:20.996 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft 42294sec preferred_lft 42294sec 2024-03-17T23:49:20.996 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::ec4:7aff:fe88:6e35/64 scope link 2024-03-17T23:49:20.996 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:49:20.996 INFO:teuthology.orchestra.run.smithi150.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-17T23:49:20.996 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether f6:2d:aa:e3:d6:a3 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:49:20.996 INFO:teuthology.orchestra.run.smithi150.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-17T23:49:20.996 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:49:20.996 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::bc2f:65ff:fe6d:abc3/64 scope link 2024-03-17T23:49:20.997 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:49:20.997 INFO:teuthology.orchestra.run.smithi150.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-17T23:49:20.997 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether f6:2d:aa:e3:d6:a3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:49:20.997 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::f42d:aaff:fee3:d6a3/64 scope link 2024-03-17T23:49:20.997 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:49:20.997 INFO:teuthology.orchestra.run.smithi150.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-17T23:49:20.997 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether fa:95:93:4e:f1:dd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:49:20.997 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::f895:93ff:fe4e:f1dd/64 scope link 2024-03-17T23:49:20.997 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:49:20.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:20.997 DEBUG:teuthology.orchestra.run.smithi150:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-17T23:49:21.063 INFO:teuthology.orchestra.run.smithi150.stdout:1 2024-03-17T23:49:21.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:21.064 DEBUG:teuthology.orchestra.run.smithi150:> route 2024-03-17T23:49:21.120 INFO:teuthology.orchestra.run.smithi150.stdout:Kernel IP routing table 2024-03-17T23:49:21.120 INFO:teuthology.orchestra.run.smithi150.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-17T23:49:21.120 INFO:teuthology.orchestra.run.smithi150.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-17T23:49:21.120 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-17T23:49:21.120 INFO:teuthology.orchestra.run.smithi150.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-17T23:49:21.120 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:49:21.120 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-03-17T23:49:21.120 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-17T23:49:21.121 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-17T23:49:21.121 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-17T23:49:21.121 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-03-17T23:49:21.208 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:49:21.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:49:21.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:21.256 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-03-17T23:49:21.271 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-17T23:49:21.271 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-17T23:49:21.272 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-17T23:49:21.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:21.272 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set brx.0 up 2024-03-17T23:49:21.338 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-17T23:49:21.339 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-17T23:49:21.339 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-17T23:49:21.393 INFO:teuthology.orchestra.run.smithi150.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-17T23:49:21.393 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-17T23:49:21.393 DEBUG:teuthology.orchestra.run.smithi150:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-17T23:49:21.448 DEBUG:teuthology.orchestra.run.smithi150:> sudo modprobe fuse 2024-03-17T23:49:21.516 DEBUG:teuthology.orchestra.run.smithi150:> cat /proc/self/mounts | awk '{print $2}' 2024-03-17T23:49:21.572 INFO:teuthology.orchestra.run.smithi150.stdout:/proc 2024-03-17T23:49:21.572 INFO:teuthology.orchestra.run.smithi150.stdout:/sys 2024-03-17T23:49:21.572 INFO:teuthology.orchestra.run.smithi150.stdout:/dev 2024-03-17T23:49:21.572 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/security 2024-03-17T23:49:21.572 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/shm 2024-03-17T23:49:21.572 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/pts 2024-03-17T23:49:21.572 INFO:teuthology.orchestra.run.smithi150.stdout:/run 2024-03-17T23:49:21.572 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/cgroup 2024-03-17T23:49:21.572 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/pstore 2024-03-17T23:49:21.572 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/bpf 2024-03-17T23:49:21.572 INFO:teuthology.orchestra.run.smithi150.stdout:/ 2024-03-17T23:49:21.573 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/selinux 2024-03-17T23:49:21.573 INFO:teuthology.orchestra.run.smithi150.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:49:21.573 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/hugepages 2024-03-17T23:49:21.573 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/mqueue 2024-03-17T23:49:21.573 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/debug 2024-03-17T23:49:21.573 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/tracing 2024-03-17T23:49:21.573 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-17T23:49:21.573 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/fuse/connections 2024-03-17T23:49:21.573 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/config 2024-03-17T23:49:21.573 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-sysctl.service 2024-03-17T23:49:21.573 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-17T23:49:21.573 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/nfs/rpc_pipefs 2024-03-17T23:49:21.573 INFO:teuthology.orchestra.run.smithi150.stdout:/run/user/1000 2024-03-17T23:49:21.573 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph 2024-03-17T23:49:21.574 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-0 2024-03-17T23:49:21.574 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-1 2024-03-17T23:49:21.574 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-2 2024-03-17T23:49:21.574 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-3 2024-03-17T23:49:21.574 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns 2024-03-17T23:49:21.574 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:49:21.574 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:49:21.574 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:49:21.574 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:49:21.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:21.574 DEBUG:teuthology.orchestra.run.smithi150:> ls /sys/fs/fuse/connections 2024-03-17T23:49:21.627 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-17T23:49:21.627 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-17T23:49:21.670 DEBUG:teuthology.orchestra.run.smithi150:> sudo modprobe fuse 2024-03-17T23:49:21.696 DEBUG:teuthology.orchestra.run.smithi150:> cat /proc/self/mounts | awk '{print $2}' 2024-03-17T23:49:21.752 INFO:teuthology.orchestra.run.smithi150.stdout:/proc 2024-03-17T23:49:21.752 INFO:teuthology.orchestra.run.smithi150.stdout:/sys 2024-03-17T23:49:21.752 INFO:teuthology.orchestra.run.smithi150.stdout:/dev 2024-03-17T23:49:21.752 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/security 2024-03-17T23:49:21.752 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/shm 2024-03-17T23:49:21.752 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/pts 2024-03-17T23:49:21.752 INFO:teuthology.orchestra.run.smithi150.stdout:/run 2024-03-17T23:49:21.753 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/cgroup 2024-03-17T23:49:21.753 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/pstore 2024-03-17T23:49:21.753 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/bpf 2024-03-17T23:49:21.753 INFO:teuthology.orchestra.run.smithi150.stdout:/ 2024-03-17T23:49:21.753 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/selinux 2024-03-17T23:49:21.753 INFO:teuthology.orchestra.run.smithi150.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:49:21.753 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/hugepages 2024-03-17T23:49:21.753 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/mqueue 2024-03-17T23:49:21.753 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/debug 2024-03-17T23:49:21.753 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/tracing 2024-03-17T23:49:21.753 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-17T23:49:21.753 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/fuse/connections 2024-03-17T23:49:21.753 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/config 2024-03-17T23:49:21.753 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-sysctl.service 2024-03-17T23:49:21.754 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-17T23:49:21.754 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/nfs/rpc_pipefs 2024-03-17T23:49:21.754 INFO:teuthology.orchestra.run.smithi150.stdout:/run/user/1000 2024-03-17T23:49:21.754 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph 2024-03-17T23:49:21.754 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-0 2024-03-17T23:49:21.754 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-1 2024-03-17T23:49:21.754 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-2 2024-03-17T23:49:21.754 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-3 2024-03-17T23:49:21.754 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns 2024-03-17T23:49:21.754 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:49:21.754 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:49:21.754 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:49:21.754 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:49:21.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:21.755 DEBUG:teuthology.orchestra.run.smithi150:> ls /sys/fs/fuse/connections 2024-03-17T23:49:21.756 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:2024-03-17T23:49:21.756+0000 7f0880410480 -1 init, newargv = 0x55dc6b52ae80 newargc=15 2024-03-17T23:49:21.756 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:ceph-fuse[60301]: starting ceph client 2024-03-17T23:49:21.761 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:ceph-fuse[60301]: starting fuse 2024-03-17T23:49:21.770 INFO:teuthology.orchestra.run.smithi150.stdout:43 2024-03-17T23:49:21.771 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-17T23:49:21.771 DEBUG:teuthology.orchestra.run.smithi150:> sudo stdin-killer -- python3 -c ' 2024-03-17T23:49:21.771 DEBUG:teuthology.orchestra.run.smithi150:> import glob 2024-03-17T23:49:21.771 DEBUG:teuthology.orchestra.run.smithi150:> import re 2024-03-17T23:49:21.771 DEBUG:teuthology.orchestra.run.smithi150:> import os 2024-03-17T23:49:21.771 DEBUG:teuthology.orchestra.run.smithi150:> import subprocess 2024-03-17T23:49:21.771 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:49:21.771 DEBUG:teuthology.orchestra.run.smithi150:> def _find_admin_socket(client_name): 2024-03-17T23:49:21.771 DEBUG:teuthology.orchestra.run.smithi150:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-17T23:49:21.771 DEBUG:teuthology.orchestra.run.smithi150:> files = glob.glob(asok_path) 2024-03-17T23:49:21.771 DEBUG:teuthology.orchestra.run.smithi150:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-17T23:49:21.771 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:49:21.771 DEBUG:teuthology.orchestra.run.smithi150:> # Given a non-glob path, it better be there 2024-03-17T23:49:21.772 DEBUG:teuthology.orchestra.run.smithi150:> if "*" not in asok_path: 2024-03-17T23:49:21.772 DEBUG:teuthology.orchestra.run.smithi150:> assert(len(files) == 1) 2024-03-17T23:49:21.772 DEBUG:teuthology.orchestra.run.smithi150:> return files[0] 2024-03-17T23:49:21.772 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:49:21.772 DEBUG:teuthology.orchestra.run.smithi150:> for f in files: 2024-03-17T23:49:21.772 DEBUG:teuthology.orchestra.run.smithi150:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-17T23:49:21.772 DEBUG:teuthology.orchestra.run.smithi150:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-17T23:49:21.772 DEBUG:teuthology.orchestra.run.smithi150:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-17T23:49:21.772 DEBUG:teuthology.orchestra.run.smithi150:> contents = proc_f.read() 2024-03-17T23:49:21.772 DEBUG:teuthology.orchestra.run.smithi150:> if mountpoint in contents: 2024-03-17T23:49:21.772 DEBUG:teuthology.orchestra.run.smithi150:> return f 2024-03-17T23:49:21.772 DEBUG:teuthology.orchestra.run.smithi150:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-17T23:49:21.772 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:49:21.772 DEBUG:teuthology.orchestra.run.smithi150:> print(_find_admin_socket("client.0")) 2024-03-17T23:49:21.772 DEBUG:teuthology.orchestra.run.smithi150:> ' 2024-03-17T23:49:21.891 INFO:teuthology.orchestra.run.smithi150.stdout:/var/run/ceph/ceph-client.0.60301.asok 2024-03-17T23:49:21.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:49:21.901 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.60301.asok 2024-03-17T23:49:21.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:21.902 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.60301.asok status 2024-03-17T23:49:22.037 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-17T23:49:22.037 INFO:teuthology.orchestra.run.smithi150.stdout: "metadata": { 2024-03-17T23:49:22.037 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-17T23:49:22.037 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-17T23:49:22.037 INFO:teuthology.orchestra.run.smithi150.stdout: "entity_id": "0", 2024-03-17T23:49:22.037 INFO:teuthology.orchestra.run.smithi150.stdout: "hostname": "smithi150", 2024-03-17T23:49:22.038 INFO:teuthology.orchestra.run.smithi150.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-17T23:49:22.038 INFO:teuthology.orchestra.run.smithi150.stdout: "pid": "60301", 2024-03-17T23:49:22.038 INFO:teuthology.orchestra.run.smithi150.stdout: "root": "/" 2024-03-17T23:49:22.038 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:22.038 INFO:teuthology.orchestra.run.smithi150.stdout: "dentry_count": 0, 2024-03-17T23:49:22.038 INFO:teuthology.orchestra.run.smithi150.stdout: "dentry_pinned_count": 0, 2024-03-17T23:49:22.038 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 6381, 2024-03-17T23:49:22.038 INFO:teuthology.orchestra.run.smithi150.stdout: "inst": { 2024-03-17T23:49:22.038 INFO:teuthology.orchestra.run.smithi150.stdout: "name": { 2024-03-17T23:49:22.038 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "client", 2024-03-17T23:49:22.038 INFO:teuthology.orchestra.run.smithi150.stdout: "num": 6381 2024-03-17T23:49:22.038 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:22.038 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": { 2024-03-17T23:49:22.039 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v1", 2024-03-17T23:49:22.039 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "192.168.0.1:0", 2024-03-17T23:49:22.039 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2197556757 2024-03-17T23:49:22.039 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:49:22.039 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:22.039 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": { 2024-03-17T23:49:22.039 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v1", 2024-03-17T23:49:22.039 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "192.168.0.1:0", 2024-03-17T23:49:22.039 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 2197556757 2024-03-17T23:49:22.039 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:22.039 INFO:teuthology.orchestra.run.smithi150.stdout: "inst_str": "client.6381 192.168.0.1:0/2197556757", 2024-03-17T23:49:22.039 INFO:teuthology.orchestra.run.smithi150.stdout: "addr_str": "192.168.0.1:0/2197556757", 2024-03-17T23:49:22.039 INFO:teuthology.orchestra.run.smithi150.stdout: "inode_count": 1, 2024-03-17T23:49:22.039 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_epoch": 22, 2024-03-17T23:49:22.039 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_epoch": 58, 2024-03-17T23:49:22.040 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_epoch_barrier": 0, 2024-03-17T23:49:22.040 INFO:teuthology.orchestra.run.smithi150.stdout: "blocklisted": false, 2024-03-17T23:49:22.040 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "cephfs" 2024-03-17T23:49:22.040 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-17T23:49:22.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:22.045 DEBUG:teuthology.orchestra.run.smithi150:> stat --file-system '--printf=%T 2024-03-17T23:49:22.045 DEBUG:teuthology.orchestra.run.smithi150:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-17T23:49:22.061 INFO:teuthology.orchestra.run.smithi150.stdout:fuseblk 2024-03-17T23:49:22.061 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-17T23:49:22.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:22.062 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-17T23:49:22.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:22.130 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch test_file) 2024-03-17T23:49:22.191 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow r pool=cephfs_data' 2024-03-17T23:49:22.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.338+0000 7fedabb1b640 1 Processor -- start 2024-03-17T23:49:22.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.338+0000 7fedabb1b640 1 -- start start 2024-03-17T23:49:22.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.339+0000 7fedabb1b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7feda40ad900 0x7feda40cf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:22.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.339+0000 7fedabb1b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7feda40b5e40 0x7feda40cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:22.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.339+0000 7fedabb1b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feda40676d0 con 0x7feda415f220 2024-03-17T23:49:22.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.339+0000 7fedabb1b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7feda40679b0 con 0x7feda40b5e40 2024-03-17T23:49:22.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.339+0000 7fedabb1b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7feda406b230 con 0x7feda40ad900 2024-03-17T23:49:22.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.339+0000 7fedaa0b8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7feda40b5e40 0x7feda40cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:22.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.339+0000 7feda98b7640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7feda415f220 0x7feda40ad1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:51778/0 (socket says 172.21.15.150:51778) 2024-03-17T23:49:22.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.339+0000 7fedaa0b8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7feda40b5e40 0x7feda40cfea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45650/0 (socket says 172.21.15.150:45650) 2024-03-17T23:49:22.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.339+0000 7feda98b7640 1 -- 172.21.15.150:0/3854718880 learned_addr learned my addr 172.21.15.150:0/3854718880 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:22.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.339+0000 7feda90b6640 1 --2- 172.21.15.150:0/3854718880 >> v2:172.21.15.157:3300/0 conn(0x7feda40ad900 0x7feda40cf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:22.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.339+0000 7fedaa0b8640 1 -- 172.21.15.150:0/3854718880 >> v2:172.21.15.157:3300/0 conn(0x7feda40ad900 msgr2=0x7feda40cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:22.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.339+0000 7fedaa0b8640 1 --2- 172.21.15.150:0/3854718880 >> v2:172.21.15.157:3300/0 conn(0x7feda40ad900 0x7feda40cf960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:22.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.339+0000 7fedaa0b8640 1 -- 172.21.15.150:0/3854718880 >> v1:172.21.15.150:6789/0 conn(0x7feda415f220 legacy=0x7feda40ad1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:22.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.339+0000 7fedaa0b8640 1 -- 172.21.15.150:0/3854718880 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feda406b510 con 0x7feda40b5e40 2024-03-17T23:49:22.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.339+0000 7feda88b5640 1 -- 172.21.15.150:0/3854718880 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 528517114 0 0) 0x7feda40676d0 con 0x7feda415f220 2024-03-17T23:49:22.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.340+0000 7fedaa0b8640 1 --2- 172.21.15.150:0/3854718880 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feda40b5e40 0x7feda40cfea0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7feda40cc640 tx=0x7fed9c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eb9dce21dc33926 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:22.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.340+0000 7feda88b5640 1 -- 172.21.15.150:0/3854718880 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed9c039070 con 0x7feda40b5e40 2024-03-17T23:49:22.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.340+0000 7feda88b5640 1 -- 172.21.15.150:0/3854718880 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed9c02cbf0 con 0x7feda40b5e40 2024-03-17T23:49:22.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.340+0000 7feda88b5640 1 -- 172.21.15.150:0/3854718880 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed9c034440 con 0x7feda40b5e40 2024-03-17T23:49:22.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.340+0000 7fedabb1b640 1 -- 172.21.15.150:0/3854718880 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feda40b5e40 msgr2=0x7feda40cfea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:22.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.340+0000 7fedabb1b640 1 --2- 172.21.15.150:0/3854718880 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feda40b5e40 0x7feda40cfea0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7feda40cc640 tx=0x7fed9c02be40 comp rx=0 tx=0).stop 2024-03-17T23:49:22.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.340+0000 7fedabb1b640 1 -- 172.21.15.150:0/3854718880 shutdown_connections 2024-03-17T23:49:22.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.340+0000 7fedabb1b640 1 --2- 172.21.15.150:0/3854718880 >> v2:172.21.15.157:3300/0 conn(0x7feda40ad900 0x7feda40cf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:22.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.340+0000 7fedabb1b640 1 --2- 172.21.15.150:0/3854718880 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feda40b5e40 0x7feda40cfea0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:22.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.341+0000 7fedabb1b640 1 -- 172.21.15.150:0/3854718880 >> 172.21.15.150:0/3854718880 conn(0x7feda4099a70 msgr2=0x7feda40b5730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:22.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.341+0000 7fedabb1b640 1 -- 172.21.15.150:0/3854718880 shutdown_connections 2024-03-17T23:49:22.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.341+0000 7fedabb1b640 1 -- 172.21.15.150:0/3854718880 wait complete. 2024-03-17T23:49:22.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.342+0000 7fedabb1b640 1 Processor -- start 2024-03-17T23:49:22.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.342+0000 7fedabb1b640 1 -- start start 2024-03-17T23:49:22.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.342+0000 7fedabb1b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feda40ad900 0x7feda4159510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:22.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.342+0000 7fedabb1b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feda40b5e40 0x7feda4151830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:22.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.342+0000 7fedabb1b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feda415f220 0x7feda4142810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:22.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.343+0000 7fedabb1b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7feda415f600 con 0x7feda40ad900 2024-03-17T23:49:22.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.343+0000 7fedabb1b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7feda4067360 con 0x7feda415f220 2024-03-17T23:49:22.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.343+0000 7fedabb1b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7feda415f780 con 0x7feda40b5e40 2024-03-17T23:49:22.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.343+0000 7feda98b7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feda40ad900 0x7feda4159510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:22.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.343+0000 7feda98b7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feda40ad900 0x7feda4159510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45658/0 (socket says 172.21.15.150:45658) 2024-03-17T23:49:22.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.343+0000 7feda98b7640 1 -- 172.21.15.150:0/936288146 learned_addr learned my addr 172.21.15.150:0/936288146 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:22.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.343+0000 7fedaa0b8640 1 --2- 172.21.15.150:0/936288146 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feda415f220 0x7feda4142810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:22.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.343+0000 7feda90b6640 1 --2- 172.21.15.150:0/936288146 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feda40b5e40 0x7feda4151830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:22.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.343+0000 7feda98b7640 1 -- 172.21.15.150:0/936288146 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feda40b5e40 msgr2=0x7feda4151830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:22.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.343+0000 7feda98b7640 1 --2- 172.21.15.150:0/936288146 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feda40b5e40 0x7feda4151830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:22.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.343+0000 7feda98b7640 1 -- 172.21.15.150:0/936288146 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feda415f220 msgr2=0x7feda4142810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:22.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.343+0000 7feda98b7640 1 --2- 172.21.15.150:0/936288146 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feda415f220 0x7feda4142810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:22.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.343+0000 7feda98b7640 1 -- 172.21.15.150:0/936288146 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feda4157ba0 con 0x7feda40ad900 2024-03-17T23:49:22.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.343+0000 7feda98b7640 1 --2- 172.21.15.150:0/936288146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feda40ad900 0x7feda4159510 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7feda000c4d0 tx=0x7feda000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:22.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.343+0000 7fed9a7fc640 1 -- 172.21.15.150:0/936288146 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda0018070 con 0x7feda40ad900 2024-03-17T23:49:22.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.343+0000 7fed9a7fc640 1 -- 172.21.15.150:0/936288146 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feda0007500 con 0x7feda40ad900 2024-03-17T23:49:22.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.343+0000 7fedabb1b640 1 -- 172.21.15.150:0/936288146 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feda4143fc0 con 0x7feda40ad900 2024-03-17T23:49:22.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.343+0000 7fed9a7fc640 1 -- 172.21.15.150:0/936288146 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feda0004a20 con 0x7feda40ad900 2024-03-17T23:49:22.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.343+0000 7fedabb1b640 1 -- 172.21.15.150:0/936288146 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feda4144580 con 0x7feda40ad900 2024-03-17T23:49:22.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.344+0000 7fedabb1b640 1 -- 172.21.15.150:0/936288146 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feda40681a0 con 0x7feda40ad900 2024-03-17T23:49:22.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.345+0000 7fed9a7fc640 1 -- 172.21.15.150:0/936288146 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7feda001b070 con 0x7feda40ad900 2024-03-17T23:49:22.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.345+0000 7fed9a7fc640 1 --2- 172.21.15.150:0/936288146 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fed8806bbd0 0x7fed8806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:22.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.345+0000 7fed9a7fc640 1 -- 172.21.15.150:0/936288146 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7feda0089e80 con 0x7feda40ad900 2024-03-17T23:49:22.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.345+0000 7feda90b6640 1 --2- 172.21.15.150:0/936288146 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fed8806bbd0 0x7fed8806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:22.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.346+0000 7feda90b6640 1 --2- 172.21.15.150:0/936288146 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fed8806bbd0 0x7fed8806e090 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7feda4152920 tx=0x7fed94009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:22.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.350+0000 7fed9a7fc640 1 -- 172.21.15.150:0/936288146 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7feda0058190 con 0x7feda40ad900 2024-03-17T23:49:22.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.590+0000 7fedabb1b640 1 -- 172.21.15.150:0/936288146 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow r pool=cephfs_data"]} v 0) -- 0x7feda40afc60 con 0x7feda40ad900 2024-03-17T23:49:22.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.592+0000 7fed9a7fc640 1 -- 172.21.15.150:0/936288146 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow r pool=cephfs_data"]}]=0 updated caps for client.0 v18) ==== 183+0+140 (secure 0 0 0) 0x7feda005c230 con 0x7feda40ad900 2024-03-17T23:49:22.593 INFO:teuthology.orchestra.run.smithi150.stdout:[client.0] 2024-03-17T23:49:22.593 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQC3gPdlvlPJNRAAkUfetPJK4hZQG5rNiSDozw== 2024-03-17T23:49:22.593 INFO:teuthology.orchestra.run.smithi150.stdout: caps mds = "allow" 2024-03-17T23:49:22.594 INFO:teuthology.orchestra.run.smithi150.stdout: caps mon = "allow r" 2024-03-17T23:49:22.594 INFO:teuthology.orchestra.run.smithi150.stdout: caps osd = "allow r pool=cephfs_data" 2024-03-17T23:49:22.594 INFO:teuthology.orchestra.run.smithi150.stderr:updated caps for client.0 2024-03-17T23:49:22.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.594+0000 7fedabb1b640 1 -- 172.21.15.150:0/936288146 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fed8806bbd0 msgr2=0x7fed8806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:22.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.594+0000 7fedabb1b640 1 --2- 172.21.15.150:0/936288146 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fed8806bbd0 0x7fed8806e090 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7feda4152920 tx=0x7fed94009190 comp rx=0 tx=0).stop 2024-03-17T23:49:22.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.594+0000 7fedabb1b640 1 -- 172.21.15.150:0/936288146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feda40ad900 msgr2=0x7feda4159510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:22.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.594+0000 7fedabb1b640 1 --2- 172.21.15.150:0/936288146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feda40ad900 0x7feda4159510 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7feda000c4d0 tx=0x7feda000c990 comp rx=0 tx=0).stop 2024-03-17T23:49:22.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.594+0000 7fedabb1b640 1 -- 172.21.15.150:0/936288146 shutdown_connections 2024-03-17T23:49:22.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.594+0000 7fedabb1b640 1 --2- 172.21.15.150:0/936288146 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fed8806bbd0 0x7fed8806e090 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:22.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.594+0000 7fedabb1b640 1 --2- 172.21.15.150:0/936288146 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7feda415f220 0x7feda4142810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:22.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.594+0000 7fedabb1b640 1 --2- 172.21.15.150:0/936288146 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7feda40b5e40 0x7feda4151830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:22.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.594+0000 7fedabb1b640 1 --2- 172.21.15.150:0/936288146 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feda40ad900 0x7feda4159510 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:22.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.594+0000 7fedabb1b640 1 -- 172.21.15.150:0/936288146 >> 172.21.15.150:0/936288146 conn(0x7feda4099a70 msgr2=0x7feda4153030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:22.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.594+0000 7fedabb1b640 1 -- 172.21.15.150:0/936288146 shutdown_connections 2024-03-17T23:49:22.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.594+0000 7fedabb1b640 1 -- 172.21.15.150:0/936288146 wait complete. 2024-03-17T23:49:22.606 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:49:22.606 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:49:22.623 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:49:22.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.762+0000 7f02fda85640 1 Processor -- start 2024-03-17T23:49:22.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.762+0000 7f02fda85640 1 -- start start 2024-03-17T23:49:22.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.762+0000 7f02fda85640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f02f81457d0 0x7f02f809a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:22.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.763+0000 7f02fda85640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f02f806b8a0 0x7f02f815cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:22.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.763+0000 7f02fda85640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02f80b1e20 con 0x7f02f80d10f0 2024-03-17T23:49:22.764 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.763+0000 7f02fda85640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f02f80d0980 con 0x7f02f806b8a0 2024-03-17T23:49:22.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.763+0000 7f02fda85640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f02f80d0c60 con 0x7f02f81457d0 2024-03-17T23:49:22.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.763+0000 7f02f77fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f02f81457d0 0x7f02f809a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:22.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.763+0000 7f02f77fe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f02f81457d0 0x7f02f809a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:45400/0 (socket says 172.21.15.150:45400) 2024-03-17T23:49:22.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.763+0000 7f02f77fe640 1 -- 172.21.15.150:0/2537945701 learned_addr learned my addr 172.21.15.150:0/2537945701 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:22.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.763+0000 7f02f7fff640 1 --2- 172.21.15.150:0/2537945701 >> v2:172.21.15.157:3300/0 conn(0x7f02f806b8a0 0x7f02f815cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:22.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.763+0000 7f02f77fe640 1 -- 172.21.15.150:0/2537945701 >> v2:172.21.15.157:3300/0 conn(0x7f02f806b8a0 msgr2=0x7f02f815cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:22.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.763+0000 7f02f77fe640 1 --2- 172.21.15.150:0/2537945701 >> v2:172.21.15.157:3300/0 conn(0x7f02f806b8a0 0x7f02f815cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:22.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.763+0000 7f02f77fe640 1 -- 172.21.15.150:0/2537945701 >> v1:172.21.15.157:6789/0 conn(0x7f02f80d10f0 legacy=0x7f02f806b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:22.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.763+0000 7f02f77fe640 1 -- 172.21.15.150:0/2537945701 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02f80b2100 con 0x7f02f81457d0 2024-03-17T23:49:22.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.764+0000 7f02f77fe640 1 --2- 172.21.15.150:0/2537945701 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f02f81457d0 0x7f02f809a180 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f02e8009870 tx=0x7f02e802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7904fa9207aae895 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:22.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.764+0000 7f02f67fc640 1 -- 172.21.15.150:0/2537945701 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e8034070 con 0x7f02f81457d0 2024-03-17T23:49:22.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.764+0000 7f02f67fc640 1 -- 172.21.15.150:0/2537945701 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02e8030840 con 0x7f02f81457d0 2024-03-17T23:49:22.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.764+0000 7f02f67fc640 1 -- 172.21.15.150:0/2537945701 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e8030c10 con 0x7f02f81457d0 2024-03-17T23:49:22.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.764+0000 7f02fda85640 1 -- 172.21.15.150:0/2537945701 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f02f81457d0 msgr2=0x7f02f809a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:22.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.764+0000 7f02fda85640 1 --2- 172.21.15.150:0/2537945701 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f02f81457d0 0x7f02f809a180 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f02e8009870 tx=0x7f02e802b980 comp rx=0 tx=0).stop 2024-03-17T23:49:22.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.765+0000 7f02fda85640 1 -- 172.21.15.150:0/2537945701 shutdown_connections 2024-03-17T23:49:22.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.765+0000 7f02fda85640 1 --2- 172.21.15.150:0/2537945701 >> v2:172.21.15.157:3300/0 conn(0x7f02f806b8a0 0x7f02f815cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:22.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.765+0000 7f02fda85640 1 --2- 172.21.15.150:0/2537945701 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f02f81457d0 0x7f02f809a180 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:22.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.765+0000 7f02fda85640 1 -- 172.21.15.150:0/2537945701 >> 172.21.15.150:0/2537945701 conn(0x7f02f80b6170 msgr2=0x7f02f8099a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:22.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.765+0000 7f02fda85640 1 -- 172.21.15.150:0/2537945701 shutdown_connections 2024-03-17T23:49:22.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.765+0000 7f02fda85640 1 -- 172.21.15.150:0/2537945701 wait complete. 2024-03-17T23:49:22.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.766+0000 7f02fda85640 1 Processor -- start 2024-03-17T23:49:22.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.766+0000 7f02fda85640 1 -- start start 2024-03-17T23:49:22.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.766+0000 7f02fda85640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02f806b8a0 0x7f02f8141be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:22.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.766+0000 7f02fda85640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02f80d10f0 0x7f02f8143540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:22.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.767+0000 7f02fda85640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f02f81457d0 0x7f02f81e66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:22.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.767+0000 7f02fda85640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f02f80d16d0 con 0x7f02f80d10f0 2024-03-17T23:49:22.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.767+0000 7f02fda85640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f02f80d1550 con 0x7f02f806b8a0 2024-03-17T23:49:22.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.767+0000 7f02fda85640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f02f80d1850 con 0x7f02f81457d0 2024-03-17T23:49:22.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.767+0000 7f02f6ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02f80d10f0 0x7f02f8143540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:22.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.767+0000 7f02f77fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02f806b8a0 0x7f02f8141be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:22.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.767+0000 7f02f7fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f02f81457d0 0x7f02f81e66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:22.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.767+0000 7f02f77fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02f806b8a0 0x7f02f8141be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:58510/0 (socket says 172.21.15.150:58510) 2024-03-17T23:49:22.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.767+0000 7f02f6ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02f80d10f0 0x7f02f8143540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45668/0 (socket says 172.21.15.150:45668) 2024-03-17T23:49:22.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.767+0000 7f02f77fe640 1 -- 172.21.15.150:0/204404311 learned_addr learned my addr 172.21.15.150:0/204404311 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:22.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.767+0000 7f02f77fe640 1 -- 172.21.15.150:0/204404311 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f02f81457d0 msgr2=0x7f02f81e66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:22.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.767+0000 7f02f77fe640 1 --2- 172.21.15.150:0/204404311 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f02f81457d0 0x7f02f81e66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:22.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.767+0000 7f02f77fe640 1 -- 172.21.15.150:0/204404311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02f80d10f0 msgr2=0x7f02f8143540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:22.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.767+0000 7f02f77fe640 1 --2- 172.21.15.150:0/204404311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02f80d10f0 0x7f02f8143540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:22.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.767+0000 7f02f77fe640 1 -- 172.21.15.150:0/204404311 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02f8143d70 con 0x7f02f806b8a0 2024-03-17T23:49:22.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.767+0000 7f02f77fe640 1 --2- 172.21.15.150:0/204404311 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02f806b8a0 0x7f02f8141be0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f02e80090b0 tx=0x7f02e80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:22.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.767+0000 7f02dffff640 1 -- 172.21.15.150:0/204404311 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e8034040 con 0x7f02f806b8a0 2024-03-17T23:49:22.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.767+0000 7f02dffff640 1 -- 172.21.15.150:0/204404311 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02e803ada0 con 0x7f02f806b8a0 2024-03-17T23:49:22.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.767+0000 7f02fda85640 1 -- 172.21.15.150:0/204404311 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02f8144050 con 0x7f02f806b8a0 2024-03-17T23:49:22.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.767+0000 7f02dffff640 1 -- 172.21.15.150:0/204404311 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e8044610 con 0x7f02f806b8a0 2024-03-17T23:49:22.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.767+0000 7f02fda85640 1 -- 172.21.15.150:0/204404311 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02f8144610 con 0x7f02f806b8a0 2024-03-17T23:49:22.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.768+0000 7f02fda85640 1 -- 172.21.15.150:0/204404311 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02f80681a0 con 0x7f02f806b8a0 2024-03-17T23:49:22.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.769+0000 7f02dffff640 1 -- 172.21.15.150:0/204404311 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f02e804b060 con 0x7f02f806b8a0 2024-03-17T23:49:22.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.769+0000 7f02dffff640 1 --2- 172.21.15.150:0/204404311 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f02e406bbd0 0x7f02e406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:22.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.769+0000 7f02dffff640 1 -- 172.21.15.150:0/204404311 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f02e80acac0 con 0x7f02f806b8a0 2024-03-17T23:49:22.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.769+0000 7f02f6ffd640 1 --2- 172.21.15.150:0/204404311 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f02e406bbd0 0x7f02e406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:22.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.770+0000 7f02f6ffd640 1 --2- 172.21.15.150:0/204404311 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f02e406bbd0 0x7f02e406e090 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f02f806f990 tx=0x7f02ec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:22.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.774+0000 7f02dffff640 1 -- 172.21.15.150:0/204404311 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f02e803f070 con 0x7f02f806b8a0 2024-03-17T23:49:22.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.946+0000 7f02fda85640 1 -- 172.21.15.150:0/204404311 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f02f806bf20 con 0x7f02f806b8a0 2024-03-17T23:49:22.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.947+0000 7f02dffff640 1 -- 172.21.15.150:0/204404311 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) ==== 74+0+16974 (secure 0 0 0) 0x7f02e807add0 con 0x7f02f806b8a0 2024-03-17T23:49:22.948 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:22.948 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":58,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:49:12.646307+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:49:09.223630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-17T23:49:09.956422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:49:22.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.949+0000 7f02fda85640 1 -- 172.21.15.150:0/204404311 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f02e406bbd0 msgr2=0x7f02e406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:22.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.949+0000 7f02fda85640 1 --2- 172.21.15.150:0/204404311 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f02e406bbd0 0x7f02e406e090 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f02f806f990 tx=0x7f02ec009040 comp rx=0 tx=0).stop 2024-03-17T23:49:22.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.949+0000 7f02fda85640 1 -- 172.21.15.150:0/204404311 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02f806b8a0 msgr2=0x7f02f8141be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:22.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.949+0000 7f02fda85640 1 --2- 172.21.15.150:0/204404311 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02f806b8a0 0x7f02f8141be0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f02e80090b0 tx=0x7f02e80046c0 comp rx=0 tx=0).stop 2024-03-17T23:49:22.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.949+0000 7f02fda85640 1 -- 172.21.15.150:0/204404311 shutdown_connections 2024-03-17T23:49:22.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.949+0000 7f02fda85640 1 --2- 172.21.15.150:0/204404311 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f02f81457d0 0x7f02f81e66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:22.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.949+0000 7f02fda85640 1 --2- 172.21.15.150:0/204404311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f02f80d10f0 0x7f02f8143540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:22.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.949+0000 7f02fda85640 1 --2- 172.21.15.150:0/204404311 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f02e406bbd0 0x7f02e406e090 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:22.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.949+0000 7f02fda85640 1 --2- 172.21.15.150:0/204404311 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02f806b8a0 0x7f02f8141be0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:22.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.949+0000 7f02fda85640 1 -- 172.21.15.150:0/204404311 >> 172.21.15.150:0/204404311 conn(0x7f02f80b6170 msgr2=0x7f02f806ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:22.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.949+0000 7f02fda85640 1 -- 172.21.15.150:0/204404311 shutdown_connections 2024-03-17T23:49:22.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:22.949+0000 7f02fda85640 1 -- 172.21.15.150:0/204404311 wait complete. 2024-03-17T23:49:22.965 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:49:09.223630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:49:09.956422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-17T23:49:22.965 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:49:23.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.068+0000 7f3f37e0b640 1 Processor -- start 2024-03-17T23:49:23.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.068+0000 7f3f37e0b640 1 -- start start 2024-03-17T23:49:23.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.068+0000 7f3f37e0b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3f3006b190 0x7f3f300b2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:23.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.069+0000 7f3f37e0b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f300d1230 con 0x7f3f300b3c30 2024-03-17T23:49:23.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.069+0000 7f3f37e0b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f300b5c10 con 0x7f3f300b5730 2024-03-17T23:49:23.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.069+0000 7f3f37e0b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3f300b2560 con 0x7f3f3006b190 2024-03-17T23:49:23.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.069+0000 7f3f363a8640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f3f300b3c30 0x7f3f300b01a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:51788/0 (socket says 172.21.15.150:51788) 2024-03-17T23:49:23.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.069+0000 7f3f363a8640 1 -- 172.21.15.150:0/3221900268 learned_addr learned my addr 172.21.15.150:0/3221900268 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:23.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.069+0000 7f3f35ba7640 1 --2- 172.21.15.150:0/3221900268 >> v2:172.21.15.157:3300/0 conn(0x7f3f3006b190 0x7f3f300b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:23.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.069+0000 7f3f34ba5640 1 -- 172.21.15.150:0/3221900268 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 390648667 0 0) 0x7f3f300b5c10 con 0x7f3f300b5730 2024-03-17T23:49:23.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.069+0000 7f3f34ba5640 1 -- 172.21.15.150:0/3221900268 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f1c003520 con 0x7f3f300b5730 2024-03-17T23:49:23.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.069+0000 7f3f34ba5640 1 -- 172.21.15.150:0/3221900268 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2250705574 0 0) 0x7f3f300d1230 con 0x7f3f300b3c30 2024-03-17T23:49:23.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.069+0000 7f3f34ba5640 1 -- 172.21.15.150:0/3221900268 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f300b5c10 con 0x7f3f300b3c30 2024-03-17T23:49:23.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.069+0000 7f3f35ba7640 1 -- 172.21.15.150:0/3221900268 >> v1:172.21.15.157:6789/0 conn(0x7f3f300b5730 legacy=0x7f3f3006b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:23.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.069+0000 7f3f35ba7640 1 -- 172.21.15.150:0/3221900268 >> v1:172.21.15.150:6789/0 conn(0x7f3f300b3c30 legacy=0x7f3f300b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:23.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.069+0000 7f3f35ba7640 1 -- 172.21.15.150:0/3221900268 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f300d14d0 con 0x7f3f3006b190 2024-03-17T23:49:23.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.070+0000 7f3f35ba7640 1 --2- 172.21.15.150:0/3221900268 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f3006b190 0x7f3f300b2020 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c0097a0 tx=0x7f3f2c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da58ae767eec8236 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:23.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.070+0000 7f3f34ba5640 1 -- 172.21.15.150:0/3221900268 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c035020 con 0x7f3f3006b190 2024-03-17T23:49:23.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.070+0000 7f3f34ba5640 1 -- 172.21.15.150:0/3221900268 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f2c031830 con 0x7f3f3006b190 2024-03-17T23:49:23.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.070+0000 7f3f34ba5640 1 -- 172.21.15.150:0/3221900268 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c031c00 con 0x7f3f3006b190 2024-03-17T23:49:23.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.070+0000 7f3f37e0b640 1 -- 172.21.15.150:0/3221900268 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f3006b190 msgr2=0x7f3f300b2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:23.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.071+0000 7f3f37e0b640 1 --2- 172.21.15.150:0/3221900268 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f3006b190 0x7f3f300b2020 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c0097a0 tx=0x7f3f2c02b8d0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.071+0000 7f3f37e0b640 1 -- 172.21.15.150:0/3221900268 shutdown_connections 2024-03-17T23:49:23.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.071+0000 7f3f37e0b640 1 --2- 172.21.15.150:0/3221900268 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f3006b190 0x7f3f300b2020 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.071+0000 7f3f37e0b640 1 -- 172.21.15.150:0/3221900268 >> 172.21.15.150:0/3221900268 conn(0x7f3f300b6170 msgr2=0x7f3f30099a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:23.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.071+0000 7f3f37e0b640 1 -- 172.21.15.150:0/3221900268 shutdown_connections 2024-03-17T23:49:23.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.071+0000 7f3f37e0b640 1 -- 172.21.15.150:0/3221900268 wait complete. 2024-03-17T23:49:23.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.072+0000 7f3f37e0b640 1 Processor -- start 2024-03-17T23:49:23.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.072+0000 7f3f37e0b640 1 -- start start 2024-03-17T23:49:23.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.072+0000 7f3f37e0b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f3006b190 0x7f3f30155690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:23.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.072+0000 7f3f37e0b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f300b3c30 0x7f3f301f65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:23.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.072+0000 7f3f37e0b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f300b5730 0x7f3f3015f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:23.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.072+0000 7f3f37e0b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3009a380 con 0x7f3f300b5730 2024-03-17T23:49:23.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.072+0000 7f3f37e0b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3f300b5ef0 con 0x7f3f3006b190 2024-03-17T23:49:23.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.073+0000 7f3f37e0b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3f3015e260 con 0x7f3f300b3c30 2024-03-17T23:49:23.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.073+0000 7f3f363a8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f300b5730 0x7f3f3015f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:23.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.073+0000 7f3f353a6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f300b3c30 0x7f3f301f65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:23.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.073+0000 7f3f363a8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f300b5730 0x7f3f3015f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45680/0 (socket says 172.21.15.150:45680) 2024-03-17T23:49:23.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.073+0000 7f3f363a8640 1 -- 172.21.15.150:0/3754497362 learned_addr learned my addr 172.21.15.150:0/3754497362 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:23.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.073+0000 7f3f35ba7640 1 --2- 172.21.15.150:0/3754497362 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f3006b190 0x7f3f30155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:23.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.073+0000 7f3f363a8640 1 -- 172.21.15.150:0/3754497362 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f300b3c30 msgr2=0x7f3f301f65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:23.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.073+0000 7f3f363a8640 1 --2- 172.21.15.150:0/3754497362 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f300b3c30 0x7f3f301f65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.073+0000 7f3f363a8640 1 -- 172.21.15.150:0/3754497362 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f3006b190 msgr2=0x7f3f30155690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:23.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.073+0000 7f3f363a8640 1 --2- 172.21.15.150:0/3754497362 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f3006b190 0x7f3f30155690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.073+0000 7f3f363a8640 1 -- 172.21.15.150:0/3754497362 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f301f4af0 con 0x7f3f300b5730 2024-03-17T23:49:23.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.073+0000 7f3f363a8640 1 --2- 172.21.15.150:0/3754497362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f300b5730 0x7f3f3015f640 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f3f2800cbc0 tx=0x7f3f2800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:23.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.073+0000 7f3f267fc640 1 -- 172.21.15.150:0/3754497362 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f28017070 con 0x7f3f300b5730 2024-03-17T23:49:23.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.073+0000 7f3f267fc640 1 -- 172.21.15.150:0/3754497362 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f2800ad40 con 0x7f3f300b5730 2024-03-17T23:49:23.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.073+0000 7f3f37e0b640 1 -- 172.21.15.150:0/3754497362 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f301f5b10 con 0x7f3f300b5730 2024-03-17T23:49:23.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.073+0000 7f3f267fc640 1 -- 172.21.15.150:0/3754497362 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f28012600 con 0x7f3f300b5730 2024-03-17T23:49:23.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.073+0000 7f3f37e0b640 1 -- 172.21.15.150:0/3754497362 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f301f60d0 con 0x7f3f300b5730 2024-03-17T23:49:23.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.074+0000 7f3f37e0b640 1 -- 172.21.15.150:0/3754497362 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f300681a0 con 0x7f3f300b5730 2024-03-17T23:49:23.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.075+0000 7f3f267fc640 1 -- 172.21.15.150:0/3754497362 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3f2801a070 con 0x7f3f300b5730 2024-03-17T23:49:23.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.075+0000 7f3f267fc640 1 --2- 172.21.15.150:0/3754497362 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3f1406bbd0 0x7f3f1406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:23.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.075+0000 7f3f267fc640 1 -- 172.21.15.150:0/3754497362 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f3f2808a8e0 con 0x7f3f300b5730 2024-03-17T23:49:23.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.075+0000 7f3f35ba7640 1 --2- 172.21.15.150:0/3754497362 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3f1406bbd0 0x7f3f1406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:23.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.075+0000 7f3f35ba7640 1 --2- 172.21.15.150:0/3754497362 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3f1406bbd0 0x7f3f1406e090 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c036000 tx=0x7f3f2c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:23.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.079+0000 7f3f267fc640 1 -- 172.21.15.150:0/3754497362 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3f28058bf0 con 0x7f3f300b5730 2024-03-17T23:49:23.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.251+0000 7f3f37e0b640 1 -- 172.21.15.150:0/3754497362 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3f30073520 con 0x7f3f300b5730 2024-03-17T23:49:23.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.251+0000 7f3f267fc640 1 -- 172.21.15.150:0/3754497362 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) ==== 93+0+10 (secure 0 0 0) 0x7f3f2805cc90 con 0x7f3f300b5730 2024-03-17T23:49:23.252 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:49:23.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.253+0000 7f3f37e0b640 1 -- 172.21.15.150:0/3754497362 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3f1406bbd0 msgr2=0x7f3f1406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:23.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.253+0000 7f3f37e0b640 1 --2- 172.21.15.150:0/3754497362 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3f1406bbd0 0x7f3f1406e090 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c036000 tx=0x7f3f2c008030 comp rx=0 tx=0).stop 2024-03-17T23:49:23.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.253+0000 7f3f37e0b640 1 -- 172.21.15.150:0/3754497362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f300b5730 msgr2=0x7f3f3015f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:23.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.253+0000 7f3f37e0b640 1 --2- 172.21.15.150:0/3754497362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f300b5730 0x7f3f3015f640 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f3f2800cbc0 tx=0x7f3f2800a430 comp rx=0 tx=0).stop 2024-03-17T23:49:23.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.253+0000 7f3f37e0b640 1 -- 172.21.15.150:0/3754497362 shutdown_connections 2024-03-17T23:49:23.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.253+0000 7f3f37e0b640 1 --2- 172.21.15.150:0/3754497362 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f300b5730 0x7f3f3015f640 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.253+0000 7f3f37e0b640 1 --2- 172.21.15.150:0/3754497362 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3f300b3c30 0x7f3f301f65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.253+0000 7f3f37e0b640 1 --2- 172.21.15.150:0/3754497362 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3f1406bbd0 0x7f3f1406e090 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.253+0000 7f3f37e0b640 1 --2- 172.21.15.150:0/3754497362 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3f3006b190 0x7f3f30155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.253+0000 7f3f37e0b640 1 -- 172.21.15.150:0/3754497362 >> 172.21.15.150:0/3754497362 conn(0x7f3f300b6170 msgr2=0x7f3f300b1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:23.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.253+0000 7f3f37e0b640 1 -- 172.21.15.150:0/3754497362 shutdown_connections 2024-03-17T23:49:23.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.253+0000 7f3f37e0b640 1 -- 172.21.15.150:0/3754497362 wait complete. 2024-03-17T23:49:23.267 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:49:23.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.370+0000 7f5432540640 1 Processor -- start 2024-03-17T23:49:23.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.370+0000 7f5432540640 1 -- start start 2024-03-17T23:49:23.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.371+0000 7f5432540640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f542c15e460 0x7f542c06c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:23.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.371+0000 7f5432540640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f542c06c8c0 0x7f542c167ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:23.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.371+0000 7f5432540640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f542c15e990 con 0x7f542c099e20 2024-03-17T23:49:23.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.371+0000 7f5432540640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f542c15ec70 con 0x7f542c06c8c0 2024-03-17T23:49:23.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.371+0000 7f5432540640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f542c15ef50 con 0x7f542c15e460 2024-03-17T23:49:23.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.371+0000 7f542bfff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f542c099e20 0x7f542c09a200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:51804/0 (socket says 172.21.15.150:51804) 2024-03-17T23:49:23.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.371+0000 7f542bfff640 1 -- 172.21.15.150:0/458549374 learned_addr learned my addr 172.21.15.150:0/458549374 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:23.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.371+0000 7f542b7fe640 1 --2- 172.21.15.150:0/458549374 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f542c15e460 0x7f542c06c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:23.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.371+0000 7f5430add640 1 --2- 172.21.15.150:0/458549374 >> v2:172.21.15.157:3300/0 conn(0x7f542c06c8c0 0x7f542c167ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:23.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.372+0000 7f542affd640 1 -- 172.21.15.150:0/458549374 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 421924422 0 0) 0x7f542c15e990 con 0x7f542c099e20 2024-03-17T23:49:23.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.372+0000 7f542affd640 1 -- 172.21.15.150:0/458549374 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5414003520 con 0x7f542c099e20 2024-03-17T23:49:23.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.372+0000 7f542b7fe640 1 -- 172.21.15.150:0/458549374 >> v2:172.21.15.157:3300/0 conn(0x7f542c06c8c0 msgr2=0x7f542c167ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:23.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.372+0000 7f542b7fe640 1 --2- 172.21.15.150:0/458549374 >> v2:172.21.15.157:3300/0 conn(0x7f542c06c8c0 0x7f542c167ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.372+0000 7f542b7fe640 1 -- 172.21.15.150:0/458549374 >> v1:172.21.15.150:6789/0 conn(0x7f542c099e20 legacy=0x7f542c09a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:23.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.372+0000 7f542b7fe640 1 -- 172.21.15.150:0/458549374 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f542c15f230 con 0x7f542c15e460 2024-03-17T23:49:23.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.372+0000 7f542b7fe640 1 --2- 172.21.15.150:0/458549374 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f542c15e460 0x7f542c06c380 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f54200097a0 tx=0x7f542002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c0509afed6ba455 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:23.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.373+0000 7f542affd640 1 -- 172.21.15.150:0/458549374 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5420035020 con 0x7f542c15e460 2024-03-17T23:49:23.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.373+0000 7f542affd640 1 -- 172.21.15.150:0/458549374 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5420031830 con 0x7f542c15e460 2024-03-17T23:49:23.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.373+0000 7f542affd640 1 -- 172.21.15.150:0/458549374 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5420031be0 con 0x7f542c15e460 2024-03-17T23:49:23.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.373+0000 7f5432540640 1 -- 172.21.15.150:0/458549374 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f542c15e460 msgr2=0x7f542c06c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:23.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.373+0000 7f5432540640 1 --2- 172.21.15.150:0/458549374 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f542c15e460 0x7f542c06c380 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f54200097a0 tx=0x7f542002b8d0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.373+0000 7f5432540640 1 -- 172.21.15.150:0/458549374 shutdown_connections 2024-03-17T23:49:23.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.373+0000 7f5432540640 1 --2- 172.21.15.150:0/458549374 >> v2:172.21.15.157:3300/0 conn(0x7f542c06c8c0 0x7f542c167ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.373+0000 7f5432540640 1 --2- 172.21.15.150:0/458549374 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f542c15e460 0x7f542c06c380 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.373+0000 7f5432540640 1 -- 172.21.15.150:0/458549374 >> 172.21.15.150:0/458549374 conn(0x7f542c0b3630 msgr2=0x7f542c0b3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:23.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.373+0000 7f5432540640 1 -- 172.21.15.150:0/458549374 shutdown_connections 2024-03-17T23:49:23.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.373+0000 7f5432540640 1 -- 172.21.15.150:0/458549374 wait complete. 2024-03-17T23:49:23.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.374+0000 7f5432540640 1 Processor -- start 2024-03-17T23:49:23.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.375+0000 7f5432540640 1 -- start start 2024-03-17T23:49:23.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.375+0000 7f5432540640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f542c099e20 0x7f542c0b60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:23.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.375+0000 7f5432540640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f542c15e460 0x7f542c0b6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:23.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.375+0000 7f5432540640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f542c156340 0x7f542c0b6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:23.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.375+0000 7f5432540640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f542c0d1210 con 0x7f542c099e20 2024-03-17T23:49:23.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.375+0000 7f5432540640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f542c06d570 con 0x7f542c156340 2024-03-17T23:49:23.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.375+0000 7f5432540640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f542c0d1390 con 0x7f542c15e460 2024-03-17T23:49:23.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.375+0000 7f542bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f542c099e20 0x7f542c0b60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:23.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.375+0000 7f542bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f542c099e20 0x7f542c0b60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45686/0 (socket says 172.21.15.150:45686) 2024-03-17T23:49:23.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.375+0000 7f542bfff640 1 -- 172.21.15.150:0/1943743902 learned_addr learned my addr 172.21.15.150:0/1943743902 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:23.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.375+0000 7f542b7fe640 1 --2- 172.21.15.150:0/1943743902 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f542c15e460 0x7f542c0b6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:23.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.375+0000 7f5430add640 1 --2- 172.21.15.150:0/1943743902 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f542c156340 0x7f542c0b6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:23.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.376+0000 7f542bfff640 1 -- 172.21.15.150:0/1943743902 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f542c15e460 msgr2=0x7f542c0b6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:23.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.376+0000 7f542bfff640 1 --2- 172.21.15.150:0/1943743902 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f542c15e460 0x7f542c0b6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.376+0000 7f542bfff640 1 -- 172.21.15.150:0/1943743902 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f542c156340 msgr2=0x7f542c0b6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:23.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.376+0000 7f542bfff640 1 --2- 172.21.15.150:0/1943743902 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f542c156340 0x7f542c0b6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.376+0000 7f542bfff640 1 -- 172.21.15.150:0/1943743902 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f542c0c6fe0 con 0x7f542c099e20 2024-03-17T23:49:23.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.376+0000 7f542bfff640 1 --2- 172.21.15.150:0/1943743902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f542c099e20 0x7f542c0b60c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f541c00c430 tx=0x7f541c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:23.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.376+0000 7f5428ff9640 1 -- 172.21.15.150:0/1943743902 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541c017070 con 0x7f542c099e20 2024-03-17T23:49:23.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.376+0000 7f5428ff9640 1 -- 172.21.15.150:0/1943743902 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f541c005ce0 con 0x7f542c099e20 2024-03-17T23:49:23.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.376+0000 7f5428ff9640 1 -- 172.21.15.150:0/1943743902 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541c0072f0 con 0x7f542c099e20 2024-03-17T23:49:23.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.376+0000 7f5432540640 1 -- 172.21.15.150:0/1943743902 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f542c0c6220 con 0x7f542c099e20 2024-03-17T23:49:23.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.376+0000 7f5432540640 1 -- 172.21.15.150:0/1943743902 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f542c0c67e0 con 0x7f542c099e20 2024-03-17T23:49:23.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.377+0000 7f5432540640 1 -- 172.21.15.150:0/1943743902 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f542c0681a0 con 0x7f542c099e20 2024-03-17T23:49:23.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.377+0000 7f5428ff9640 1 -- 172.21.15.150:0/1943743902 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f541c01a070 con 0x7f542c099e20 2024-03-17T23:49:23.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.378+0000 7f5428ff9640 1 --2- 172.21.15.150:0/1943743902 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f541806bbd0 0x7f541806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:23.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.378+0000 7f5428ff9640 1 -- 172.21.15.150:0/1943743902 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f541c089f80 con 0x7f542c099e20 2024-03-17T23:49:23.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.378+0000 7f542b7fe640 1 --2- 172.21.15.150:0/1943743902 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f541806bbd0 0x7f541806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:23.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.378+0000 7f542b7fe640 1 --2- 172.21.15.150:0/1943743902 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f541806bbd0 0x7f541806e090 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5420036000 tx=0x7f542003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:23.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.382+0000 7f5428ff9640 1 -- 172.21.15.150:0/1943743902 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f541c058320 con 0x7f542c099e20 2024-03-17T23:49:23.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.557+0000 7f5432540640 1 -- 172.21.15.150:0/1943743902 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f542c09b500 con 0x7f542c099e20 2024-03-17T23:49:23.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.558+0000 7f5428ff9640 1 -- 172.21.15.150:0/1943743902 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) ==== 104+0+11 (secure 0 0 0) 0x7f541c05c3c0 con 0x7f542c099e20 2024-03-17T23:49:23.559 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:49:23.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.560+0000 7f5432540640 1 -- 172.21.15.150:0/1943743902 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f541806bbd0 msgr2=0x7f541806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:23.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.560+0000 7f5432540640 1 --2- 172.21.15.150:0/1943743902 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f541806bbd0 0x7f541806e090 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5420036000 tx=0x7f542003c040 comp rx=0 tx=0).stop 2024-03-17T23:49:23.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.560+0000 7f5432540640 1 -- 172.21.15.150:0/1943743902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f542c099e20 msgr2=0x7f542c0b60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:23.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.560+0000 7f5432540640 1 --2- 172.21.15.150:0/1943743902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f542c099e20 0x7f542c0b60c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f541c00c430 tx=0x7f541c00c8f0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.560+0000 7f5432540640 1 -- 172.21.15.150:0/1943743902 shutdown_connections 2024-03-17T23:49:23.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.560+0000 7f5432540640 1 --2- 172.21.15.150:0/1943743902 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f541806bbd0 0x7f541806e090 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.560+0000 7f5432540640 1 --2- 172.21.15.150:0/1943743902 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f542c156340 0x7f542c0b6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.560+0000 7f5432540640 1 --2- 172.21.15.150:0/1943743902 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f542c15e460 0x7f542c0b6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.560+0000 7f5432540640 1 --2- 172.21.15.150:0/1943743902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f542c099e20 0x7f542c0b60c0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.560+0000 7f5432540640 1 -- 172.21.15.150:0/1943743902 >> 172.21.15.150:0/1943743902 conn(0x7f542c0b3630 msgr2=0x7f542c1600e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:23.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.560+0000 7f5432540640 1 -- 172.21.15.150:0/1943743902 shutdown_connections 2024-03-17T23:49:23.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.560+0000 7f5432540640 1 -- 172.21.15.150:0/1943743902 wait complete. 2024-03-17T23:49:23.572 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:49:23.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.674+0000 7fd02871d640 1 Processor -- start 2024-03-17T23:49:23.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.674+0000 7fd02871d640 1 -- start start 2024-03-17T23:49:23.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.675+0000 7fd02871d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd0200ce6a0 0x7fd020166a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:23.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.675+0000 7fd02871d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd020145170 con 0x7fd02006f400 2024-03-17T23:49:23.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.675+0000 7fd02871d640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd020145450 con 0x7fd020144bb0 2024-03-17T23:49:23.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.675+0000 7fd02871d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd020145730 con 0x7fd0200ce6a0 2024-03-17T23:49:23.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.675+0000 7fd025cb8640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd02006f400 0x7fd0200cdf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:51810/0 (socket says 172.21.15.150:51810) 2024-03-17T23:49:23.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.675+0000 7fd025cb8640 1 -- 172.21.15.150:0/2024892129 learned_addr learned my addr 172.21.15.150:0/2024892129 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:23.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.676+0000 7fd026cba640 1 --2- 172.21.15.150:0/2024892129 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd0200ce6a0 0x7fd020166a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:23.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.676+0000 7fd0254b7640 1 -- 172.21.15.150:0/2024892129 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3254995641 0 0) 0x7fd020145170 con 0x7fd02006f400 2024-03-17T23:49:23.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.676+0000 7fd0254b7640 1 -- 172.21.15.150:0/2024892129 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd00c003520 con 0x7fd02006f400 2024-03-17T23:49:23.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.676+0000 7fd0254b7640 1 -- 172.21.15.150:0/2024892129 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 152202275 0 0) 0x7fd020145450 con 0x7fd020144bb0 2024-03-17T23:49:23.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.676+0000 7fd0254b7640 1 -- 172.21.15.150:0/2024892129 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd020145170 con 0x7fd020144bb0 2024-03-17T23:49:23.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.676+0000 7fd026cba640 1 -- 172.21.15.150:0/2024892129 >> v1:172.21.15.157:6789/0 conn(0x7fd020144bb0 legacy=0x7fd02015ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:23.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.676+0000 7fd026cba640 1 -- 172.21.15.150:0/2024892129 >> v1:172.21.15.150:6789/0 conn(0x7fd02006f400 legacy=0x7fd0200cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:23.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.676+0000 7fd026cba640 1 -- 172.21.15.150:0/2024892129 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd02015d560 con 0x7fd0200ce6a0 2024-03-17T23:49:23.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.677+0000 7fd026cba640 1 --2- 172.21.15.150:0/2024892129 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd0200ce6a0 0x7fd020166a40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd018009dc0 tx=0x7fd01802be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81a26efd97c21e4b server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:23.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.677+0000 7fd0254b7640 1 -- 172.21.15.150:0/2024892129 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd018039070 con 0x7fd0200ce6a0 2024-03-17T23:49:23.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.677+0000 7fd0254b7640 1 -- 172.21.15.150:0/2024892129 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd01802cc40 con 0x7fd0200ce6a0 2024-03-17T23:49:23.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.677+0000 7fd0254b7640 1 -- 172.21.15.150:0/2024892129 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd018034440 con 0x7fd0200ce6a0 2024-03-17T23:49:23.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.677+0000 7fd02871d640 1 -- 172.21.15.150:0/2024892129 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd0200ce6a0 msgr2=0x7fd020166a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:23.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.678+0000 7fd02871d640 1 --2- 172.21.15.150:0/2024892129 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd0200ce6a0 0x7fd020166a40 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd018009dc0 tx=0x7fd01802be90 comp rx=0 tx=0).stop 2024-03-17T23:49:23.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.678+0000 7fd02871d640 1 -- 172.21.15.150:0/2024892129 shutdown_connections 2024-03-17T23:49:23.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.678+0000 7fd02871d640 1 --2- 172.21.15.150:0/2024892129 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd0200ce6a0 0x7fd020166a40 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.678+0000 7fd02871d640 1 -- 172.21.15.150:0/2024892129 >> 172.21.15.150:0/2024892129 conn(0x7fd02006b190 msgr2=0x7fd02006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:23.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.678+0000 7fd02871d640 1 -- 172.21.15.150:0/2024892129 shutdown_connections 2024-03-17T23:49:23.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.678+0000 7fd02871d640 1 -- 172.21.15.150:0/2024892129 wait complete. 2024-03-17T23:49:23.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.679+0000 7fd02871d640 1 Processor -- start 2024-03-17T23:49:23.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.679+0000 7fd02871d640 1 -- start start 2024-03-17T23:49:23.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.680+0000 7fd02871d640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd02006f400 0x7fd020138dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:23.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.680+0000 7fd02871d640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd0200ce6a0 0x7fd02013daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:23.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.680+0000 7fd02871d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd020144bb0 0x7fd02013dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:23.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.680+0000 7fd02871d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd020167100 con 0x7fd020144bb0 2024-03-17T23:49:23.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.680+0000 7fd02871d640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd020166f80 con 0x7fd0200ce6a0 2024-03-17T23:49:23.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.680+0000 7fd02871d640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd020167280 con 0x7fd02006f400 2024-03-17T23:49:23.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.680+0000 7fd026cba640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd020144bb0 0x7fd02013dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:23.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.680+0000 7fd026cba640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd020144bb0 0x7fd02013dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45692/0 (socket says 172.21.15.150:45692) 2024-03-17T23:49:23.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.680+0000 7fd026cba640 1 -- 172.21.15.150:0/497299749 learned_addr learned my addr 172.21.15.150:0/497299749 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:23.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.680+0000 7fd0264b9640 1 --2- 172.21.15.150:0/497299749 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd02006f400 0x7fd020138dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:23.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.680+0000 7fd025cb8640 1 --2- 172.21.15.150:0/497299749 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd0200ce6a0 0x7fd02013daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:23.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.680+0000 7fd026cba640 1 -- 172.21.15.150:0/497299749 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd02006f400 msgr2=0x7fd020138dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:23.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.680+0000 7fd026cba640 1 --2- 172.21.15.150:0/497299749 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd02006f400 0x7fd020138dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.680+0000 7fd026cba640 1 -- 172.21.15.150:0/497299749 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd0200ce6a0 msgr2=0x7fd02013daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:23.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.680+0000 7fd026cba640 1 --2- 172.21.15.150:0/497299749 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd0200ce6a0 0x7fd02013daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.680+0000 7fd026cba640 1 -- 172.21.15.150:0/497299749 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd020145b10 con 0x7fd020144bb0 2024-03-17T23:49:23.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.680+0000 7fd026cba640 1 --2- 172.21.15.150:0/497299749 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd020144bb0 0x7fd02013dfe0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fd018009ee0 tx=0x7fd018004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:23.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.682+0000 7fd016ffd640 1 -- 172.21.15.150:0/497299749 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd018039050 con 0x7fd020144bb0 2024-03-17T23:49:23.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.682+0000 7fd016ffd640 1 -- 172.21.15.150:0/497299749 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd018045070 con 0x7fd020144bb0 2024-03-17T23:49:23.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.682+0000 7fd016ffd640 1 -- 172.21.15.150:0/497299749 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd018009450 con 0x7fd020144bb0 2024-03-17T23:49:23.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.682+0000 7fd02871d640 1 -- 172.21.15.150:0/497299749 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd020145df0 con 0x7fd020144bb0 2024-03-17T23:49:23.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.682+0000 7fd02871d640 1 -- 172.21.15.150:0/497299749 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0201463b0 con 0x7fd020144bb0 2024-03-17T23:49:23.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.683+0000 7fd02871d640 1 -- 172.21.15.150:0/497299749 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0200681a0 con 0x7fd020144bb0 2024-03-17T23:49:23.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.683+0000 7fd016ffd640 1 -- 172.21.15.150:0/497299749 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd018040690 con 0x7fd020144bb0 2024-03-17T23:49:23.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.684+0000 7fd016ffd640 1 --2- 172.21.15.150:0/497299749 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd00406bbd0 0x7fd00406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:23.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.684+0000 7fd016ffd640 1 -- 172.21.15.150:0/497299749 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7fd0180ad5e0 con 0x7fd020144bb0 2024-03-17T23:49:23.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.684+0000 7fd0264b9640 1 --2- 172.21.15.150:0/497299749 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd00406bbd0 0x7fd00406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:23.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.684+0000 7fd0264b9640 1 --2- 172.21.15.150:0/497299749 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd00406bbd0 0x7fd00406e090 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd01c007fd0 tx=0x7fd01c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:23.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.688+0000 7fd016ffd640 1 -- 172.21.15.150:0/497299749 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd01807b860 con 0x7fd020144bb0 2024-03-17T23:49:23.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.861+0000 7fd02871d640 1 -- 172.21.15.150:0/497299749 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd02015bdf0 con 0x7fd020144bb0 2024-03-17T23:49:23.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.861+0000 7fd016ffd640 1 -- 172.21.15.150:0/497299749 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) ==== 100+0+11 (secure 0 0 0) 0x7fd01807f900 con 0x7fd020144bb0 2024-03-17T23:49:23.862 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:49:23.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.863+0000 7fd02871d640 1 -- 172.21.15.150:0/497299749 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd00406bbd0 msgr2=0x7fd00406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:23.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.863+0000 7fd02871d640 1 --2- 172.21.15.150:0/497299749 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd00406bbd0 0x7fd00406e090 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd01c007fd0 tx=0x7fd01c009040 comp rx=0 tx=0).stop 2024-03-17T23:49:23.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.863+0000 7fd02871d640 1 -- 172.21.15.150:0/497299749 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd020144bb0 msgr2=0x7fd02013dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:23.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.863+0000 7fd02871d640 1 --2- 172.21.15.150:0/497299749 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd020144bb0 0x7fd02013dfe0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fd018009ee0 tx=0x7fd018004a40 comp rx=0 tx=0).stop 2024-03-17T23:49:23.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.863+0000 7fd02871d640 1 -- 172.21.15.150:0/497299749 shutdown_connections 2024-03-17T23:49:23.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.863+0000 7fd02871d640 1 --2- 172.21.15.150:0/497299749 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd020144bb0 0x7fd02013dfe0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.863+0000 7fd02871d640 1 --2- 172.21.15.150:0/497299749 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd00406bbd0 0x7fd00406e090 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.863+0000 7fd02871d640 1 --2- 172.21.15.150:0/497299749 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd0200ce6a0 0x7fd02013daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.863+0000 7fd02871d640 1 --2- 172.21.15.150:0/497299749 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd02006f400 0x7fd020138dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.863+0000 7fd02871d640 1 -- 172.21.15.150:0/497299749 >> 172.21.15.150:0/497299749 conn(0x7fd02006b190 msgr2=0x7fd020071cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:23.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.863+0000 7fd02871d640 1 -- 172.21.15.150:0/497299749 shutdown_connections 2024-03-17T23:49:23.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.863+0000 7fd02871d640 1 -- 172.21.15.150:0/497299749 wait complete. 2024-03-17T23:49:23.875 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-17T23:49:23.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.975+0000 7f5a0186c640 1 Processor -- start 2024-03-17T23:49:23.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.975+0000 7f5a0186c640 1 -- start start 2024-03-17T23:49:23.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.976+0000 7f5a0186c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f59fc161530 0x7f59fc0d3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:23.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.976+0000 7f5a0186c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59fc0ce6a0 con 0x7f59fc161910 2024-03-17T23:49:23.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.976+0000 7f5a0186c640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59fc06d470 con 0x7f59fc06cda0 2024-03-17T23:49:23.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.976+0000 7f5a0186c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f59fc06db30 con 0x7f59fc161530 2024-03-17T23:49:23.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.976+0000 7f59faffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f59fc161530 0x7f59fc0d3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:23.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.976+0000 7f59fa7fc640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f59fc161910 0x7f59fc06c690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:51816/0 (socket says 172.21.15.150:51816) 2024-03-17T23:49:23.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.976+0000 7f59faffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f59fc161530 0x7f59fc0d3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45694/0 (socket says 172.21.15.150:45694) 2024-03-17T23:49:23.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.976+0000 7f59fa7fc640 1 -- 172.21.15.150:0/1347650593 learned_addr learned my addr 172.21.15.150:0/1347650593 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:23.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.976+0000 7f59faffd640 1 -- 172.21.15.150:0/1347650593 >> v1:172.21.15.157:6789/0 conn(0x7f59fc06cda0 legacy=0x7f59fc0d1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:23.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.976+0000 7f59faffd640 1 -- 172.21.15.150:0/1347650593 >> v1:172.21.15.150:6789/0 conn(0x7f59fc161910 legacy=0x7f59fc06c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:23.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.976+0000 7f59f9ffb640 1 -- 172.21.15.150:0/1347650593 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4102650440 0 0) 0x7f59fc06d470 con 0x7f59fc06cda0 2024-03-17T23:49:23.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.976+0000 7f59faffd640 1 -- 172.21.15.150:0/1347650593 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59fc06d750 con 0x7f59fc161530 2024-03-17T23:49:23.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.977+0000 7f59faffd640 1 --2- 172.21.15.150:0/1347650593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59fc161530 0x7f59fc0d3690 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f59ec0097a0 tx=0x7f59ec02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b16d853a54fff6 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:23.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.977+0000 7f59f9ffb640 1 -- 172.21.15.150:0/1347650593 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ec037070 con 0x7f59fc161530 2024-03-17T23:49:23.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.977+0000 7f59f9ffb640 1 -- 172.21.15.150:0/1347650593 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59ec030710 con 0x7f59fc161530 2024-03-17T23:49:23.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.977+0000 7f59f9ffb640 1 -- 172.21.15.150:0/1347650593 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ec030ae0 con 0x7f59fc161530 2024-03-17T23:49:23.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.978+0000 7f5a0186c640 1 -- 172.21.15.150:0/1347650593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59fc161530 msgr2=0x7f59fc0d3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:23.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.978+0000 7f5a0186c640 1 --2- 172.21.15.150:0/1347650593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59fc161530 0x7f59fc0d3690 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f59ec0097a0 tx=0x7f59ec02b790 comp rx=0 tx=0).stop 2024-03-17T23:49:23.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.978+0000 7f5a0186c640 1 -- 172.21.15.150:0/1347650593 shutdown_connections 2024-03-17T23:49:23.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.978+0000 7f5a0186c640 1 --2- 172.21.15.150:0/1347650593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59fc161530 0x7f59fc0d3690 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.978+0000 7f5a0186c640 1 -- 172.21.15.150:0/1347650593 >> 172.21.15.150:0/1347650593 conn(0x7f59fc0d0d50 msgr2=0x7f59fc0d3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:23.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.978+0000 7f5a0186c640 1 -- 172.21.15.150:0/1347650593 shutdown_connections 2024-03-17T23:49:23.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.978+0000 7f5a0186c640 1 -- 172.21.15.150:0/1347650593 wait complete. 2024-03-17T23:49:23.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.979+0000 7f5a0186c640 1 Processor -- start 2024-03-17T23:49:23.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.979+0000 7f5a0186c640 1 -- start start 2024-03-17T23:49:23.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.980+0000 7f5a0186c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59fc06cda0 0x7f59fc155fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:23.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.980+0000 7f5a0186c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f59fc161530 0x7f59fc0cd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:23.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.980+0000 7f5a0186c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59fc161910 0x7f59fc1f6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:23.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.980+0000 7f5a0186c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc0b3210 con 0x7f59fc161910 2024-03-17T23:49:23.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.980+0000 7f5a0186c640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc0b3090 con 0x7f59fc06cda0 2024-03-17T23:49:23.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.980+0000 7f5a0186c640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f59fc0b3390 con 0x7f59fc161530 2024-03-17T23:49:23.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.980+0000 7f59fb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59fc161910 0x7f59fc1f6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:23.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.980+0000 7f59faffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59fc06cda0 0x7f59fc155fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:23.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.980+0000 7f59faffd640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59fc06cda0 0x7f59fc155fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:58566/0 (socket says 172.21.15.150:58566) 2024-03-17T23:49:23.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.980+0000 7f59faffd640 1 -- 172.21.15.150:0/565382250 learned_addr learned my addr 172.21.15.150:0/565382250 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:23.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.980+0000 7f59fa7fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f59fc161530 0x7f59fc0cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:23.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.980+0000 7f59fb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59fc161910 0x7f59fc1f6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45708/0 (socket says 172.21.15.150:45708) 2024-03-17T23:49:23.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.980+0000 7f59faffd640 1 -- 172.21.15.150:0/565382250 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f59fc161530 msgr2=0x7f59fc0cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:23.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.980+0000 7f59faffd640 1 --2- 172.21.15.150:0/565382250 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f59fc161530 0x7f59fc0cd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.980+0000 7f59faffd640 1 -- 172.21.15.150:0/565382250 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59fc161910 msgr2=0x7f59fc1f6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:23.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.980+0000 7f59faffd640 1 --2- 172.21.15.150:0/565382250 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59fc161910 0x7f59fc1f6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:23.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.980+0000 7f59faffd640 1 -- 172.21.15.150:0/565382250 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59fc0cb150 con 0x7f59fc06cda0 2024-03-17T23:49:23.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.980+0000 7f59faffd640 1 --2- 172.21.15.150:0/565382250 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59fc06cda0 0x7f59fc155fc0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f59ec0097a0 tx=0x7f59ec0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:23.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.981+0000 7f59e3fff640 1 -- 172.21.15.150:0/565382250 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ec037040 con 0x7f59fc06cda0 2024-03-17T23:49:23.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.981+0000 7f59e3fff640 1 -- 172.21.15.150:0/565382250 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59ec004030 con 0x7f59fc06cda0 2024-03-17T23:49:23.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.981+0000 7f5a0186c640 1 -- 172.21.15.150:0/565382250 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59fc0cb430 con 0x7f59fc06cda0 2024-03-17T23:49:23.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.981+0000 7f59e3fff640 1 -- 172.21.15.150:0/565382250 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ec004420 con 0x7f59fc06cda0 2024-03-17T23:49:23.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.981+0000 7f5a0186c640 1 -- 172.21.15.150:0/565382250 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59fc0cb9f0 con 0x7f59fc06cda0 2024-03-17T23:49:23.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.981+0000 7f5a0186c640 1 -- 172.21.15.150:0/565382250 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59fc0681a0 con 0x7f59fc06cda0 2024-03-17T23:49:23.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.982+0000 7f59e3fff640 1 -- 172.21.15.150:0/565382250 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f59ec032940 con 0x7f59fc06cda0 2024-03-17T23:49:23.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.982+0000 7f59e3fff640 1 --2- 172.21.15.150:0/565382250 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f59e806bbd0 0x7f59e806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:23.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.982+0000 7f59e3fff640 1 -- 172.21.15.150:0/565382250 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f59ec0ac940 con 0x7f59fc06cda0 2024-03-17T23:49:23.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.982+0000 7f59fa7fc640 1 --2- 172.21.15.150:0/565382250 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f59e806bbd0 0x7f59e806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:23.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.983+0000 7f59fa7fc640 1 --2- 172.21.15.150:0/565382250 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f59e806bbd0 0x7f59e806e090 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f59fc0b5050 tx=0x7f59f0007e20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:23.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:23.987+0000 7f59e3fff640 1 -- 172.21.15.150:0/565382250 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f59ec042020 con 0x7f59fc06cda0 2024-03-17T23:49:24.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.170+0000 7f5a0186c640 1 -- 172.21.15.150:0/565382250 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f59fc0d4a30 con 0x7f59fc06cda0 2024-03-17T23:49:24.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.170+0000 7f59e3fff640 1 -- 172.21.15.150:0/565382250 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v58) ==== 80+0+0 (secure 0 0 0) 0x7f59ec07ac50 con 0x7f59fc06cda0 2024-03-17T23:49:24.171 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-03-17T23:49:24.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.172+0000 7f5a0186c640 1 -- 172.21.15.150:0/565382250 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f59e806bbd0 msgr2=0x7f59e806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:24.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.172+0000 7f5a0186c640 1 --2- 172.21.15.150:0/565382250 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f59e806bbd0 0x7f59e806e090 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f59fc0b5050 tx=0x7f59f0007e20 comp rx=0 tx=0).stop 2024-03-17T23:49:24.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.172+0000 7f5a0186c640 1 -- 172.21.15.150:0/565382250 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59fc06cda0 msgr2=0x7f59fc155fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:24.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.172+0000 7f5a0186c640 1 --2- 172.21.15.150:0/565382250 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59fc06cda0 0x7f59fc155fc0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f59ec0097a0 tx=0x7f59ec0046c0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.172+0000 7f5a0186c640 1 -- 172.21.15.150:0/565382250 shutdown_connections 2024-03-17T23:49:24.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.172+0000 7f5a0186c640 1 --2- 172.21.15.150:0/565382250 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59fc161910 0x7f59fc1f6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.172+0000 7f5a0186c640 1 --2- 172.21.15.150:0/565382250 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f59fc161530 0x7f59fc0cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.172+0000 7f5a0186c640 1 --2- 172.21.15.150:0/565382250 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f59e806bbd0 0x7f59e806e090 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.172+0000 7f5a0186c640 1 --2- 172.21.15.150:0/565382250 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f59fc06cda0 0x7f59fc155fc0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.172+0000 7f5a0186c640 1 -- 172.21.15.150:0/565382250 >> 172.21.15.150:0/565382250 conn(0x7f59fc0d0d50 msgr2=0x7f59fc160750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:24.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.172+0000 7f5a0186c640 1 -- 172.21.15.150:0/565382250 shutdown_connections 2024-03-17T23:49:24.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.172+0000 7f5a0186c640 1 -- 172.21.15.150:0/565382250 wait complete. 2024-03-17T23:49:24.184 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:49:24.184 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:49:24.199 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:49:24.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.339+0000 7f0404d22640 1 Processor -- start 2024-03-17T23:49:24.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.339+0000 7f0404d22640 1 -- start start 2024-03-17T23:49:24.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.340+0000 7f0404d22640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f040006b190 0x7f04000b2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:24.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.340+0000 7f0404d22640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f04000b3c30 0x7f04000b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:24.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.340+0000 7f0404d22640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04000d1230 con 0x7f04000b5730 2024-03-17T23:49:24.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.340+0000 7f0404d22640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f04000b2560 con 0x7f04000b3c30 2024-03-17T23:49:24.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.340+0000 7f0404d22640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f04000b5c10 con 0x7f040006b190 2024-03-17T23:49:24.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.341+0000 7f03fdd9b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f04000b5730 0x7f040006b570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:51822/0 (socket says 172.21.15.150:51822) 2024-03-17T23:49:24.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.341+0000 7f03fdd9b640 1 -- 172.21.15.150:0/1644071202 learned_addr learned my addr 172.21.15.150:0/1644071202 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:24.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.341+0000 7f03fe59c640 1 --2- 172.21.15.150:0/1644071202 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f040006b190 0x7f04000b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:24.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.341+0000 7f03fed9d640 1 --2- 172.21.15.150:0/1644071202 >> v2:172.21.15.157:3300/0 conn(0x7f04000b3c30 0x7f04000b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:24.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.341+0000 7f03fe59c640 1 -- 172.21.15.150:0/1644071202 >> v2:172.21.15.157:3300/0 conn(0x7f04000b3c30 msgr2=0x7f04000b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:24.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.341+0000 7f03fe59c640 1 --2- 172.21.15.150:0/1644071202 >> v2:172.21.15.157:3300/0 conn(0x7f04000b3c30 0x7f04000b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.341+0000 7f03fe59c640 1 -- 172.21.15.150:0/1644071202 >> v1:172.21.15.150:6789/0 conn(0x7f04000b5730 legacy=0x7f040006b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:24.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.341+0000 7f03fd59a640 1 -- 172.21.15.150:0/1644071202 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2718274189 0 0) 0x7f04000d1230 con 0x7f04000b5730 2024-03-17T23:49:24.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.341+0000 7f03fe59c640 1 -- 172.21.15.150:0/1644071202 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04000d14d0 con 0x7f040006b190 2024-03-17T23:49:24.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.342+0000 7f03fe59c640 1 --2- 172.21.15.150:0/1644071202 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f040006b190 0x7f04000b2020 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f03f40097a0 tx=0x7f03f402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29d1d32175b1da61 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:24.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.342+0000 7f03fd59a640 1 -- 172.21.15.150:0/1644071202 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f40315a0 con 0x7f040006b190 2024-03-17T23:49:24.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.342+0000 7f03fd59a640 1 -- 172.21.15.150:0/1644071202 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03f4037070 con 0x7f040006b190 2024-03-17T23:49:24.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.342+0000 7f03fd59a640 1 -- 172.21.15.150:0/1644071202 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f4031960 con 0x7f040006b190 2024-03-17T23:49:24.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.342+0000 7f0404d22640 1 -- 172.21.15.150:0/1644071202 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f040006b190 msgr2=0x7f04000b2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:24.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.342+0000 7f0404d22640 1 --2- 172.21.15.150:0/1644071202 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f040006b190 0x7f04000b2020 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f03f40097a0 tx=0x7f03f402b840 comp rx=0 tx=0).stop 2024-03-17T23:49:24.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.342+0000 7f0404d22640 1 -- 172.21.15.150:0/1644071202 shutdown_connections 2024-03-17T23:49:24.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.342+0000 7f0404d22640 1 --2- 172.21.15.150:0/1644071202 >> v2:172.21.15.157:3300/0 conn(0x7f04000b3c30 0x7f04000b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.342+0000 7f0404d22640 1 --2- 172.21.15.150:0/1644071202 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f040006b190 0x7f04000b2020 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.342+0000 7f0404d22640 1 -- 172.21.15.150:0/1644071202 >> 172.21.15.150:0/1644071202 conn(0x7f04000b6170 msgr2=0x7f0400099a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:24.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.342+0000 7f0404d22640 1 -- 172.21.15.150:0/1644071202 shutdown_connections 2024-03-17T23:49:24.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.342+0000 7f0404d22640 1 -- 172.21.15.150:0/1644071202 wait complete. 2024-03-17T23:49:24.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.344+0000 7f0404d22640 1 Processor -- start 2024-03-17T23:49:24.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.344+0000 7f0404d22640 1 -- start start 2024-03-17T23:49:24.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.344+0000 7f0404d22640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f040006b190 0x7f040015ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:24.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.344+0000 7f0404d22640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04000b3c30 0x7f0400158950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:24.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.344+0000 7f0404d22640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04000b5730 0x7f040006fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:24.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.344+0000 7f0404d22640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f040009a380 con 0x7f04000b3c30 2024-03-17T23:49:24.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.344+0000 7f0404d22640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f04000b5ef0 con 0x7f04000b5730 2024-03-17T23:49:24.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.344+0000 7f0404d22640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f040015e260 con 0x7f040006b190 2024-03-17T23:49:24.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.345+0000 7f03fdd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04000b3c30 0x7f0400158950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:24.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.345+0000 7f03fdd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04000b3c30 0x7f0400158950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45714/0 (socket says 172.21.15.150:45714) 2024-03-17T23:49:24.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.345+0000 7f03fdd9b640 1 -- 172.21.15.150:0/2167988470 learned_addr learned my addr 172.21.15.150:0/2167988470 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:24.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.345+0000 7f03fe59c640 1 --2- 172.21.15.150:0/2167988470 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f040006b190 0x7f040015ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:24.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.345+0000 7f03fed9d640 1 --2- 172.21.15.150:0/2167988470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04000b5730 0x7f040006fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:24.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.345+0000 7f03fdd9b640 1 -- 172.21.15.150:0/2167988470 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f040006b190 msgr2=0x7f040015ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:24.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.345+0000 7f03fdd9b640 1 --2- 172.21.15.150:0/2167988470 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f040006b190 0x7f040015ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.345+0000 7f03fdd9b640 1 -- 172.21.15.150:0/2167988470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04000b5730 msgr2=0x7f040006fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:24.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.345+0000 7f03fdd9b640 1 --2- 172.21.15.150:0/2167988470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04000b5730 0x7f040006fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.345+0000 7f03fdd9b640 1 -- 172.21.15.150:0/2167988470 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04001f6cd0 con 0x7f04000b3c30 2024-03-17T23:49:24.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.345+0000 7f03fdd9b640 1 --2- 172.21.15.150:0/2167988470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04000b3c30 0x7f0400158950 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f03f000c430 tx=0x7f03f000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:24.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.345+0000 7f03eeffd640 1 -- 172.21.15.150:0/2167988470 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f00074e0 con 0x7f04000b3c30 2024-03-17T23:49:24.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.345+0000 7f03eeffd640 1 -- 172.21.15.150:0/2167988470 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03f00047e0 con 0x7f04000b3c30 2024-03-17T23:49:24.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.345+0000 7f0404d22640 1 -- 172.21.15.150:0/2167988470 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04001f6fb0 con 0x7f04000b3c30 2024-03-17T23:49:24.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.345+0000 7f03eeffd640 1 -- 172.21.15.150:0/2167988470 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03f0004bd0 con 0x7f04000b3c30 2024-03-17T23:49:24.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.345+0000 7f0404d22640 1 -- 172.21.15.150:0/2167988470 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04001f7570 con 0x7f04000b3c30 2024-03-17T23:49:24.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.346+0000 7f0404d22640 1 -- 172.21.15.150:0/2167988470 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04000681a0 con 0x7f04000b3c30 2024-03-17T23:49:24.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.347+0000 7f03eeffd640 1 -- 172.21.15.150:0/2167988470 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f03f0011070 con 0x7f04000b3c30 2024-03-17T23:49:24.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.347+0000 7f03eeffd640 1 --2- 172.21.15.150:0/2167988470 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f03e006bbd0 0x7f03e006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:24.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.347+0000 7f03eeffd640 1 -- 172.21.15.150:0/2167988470 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f03f008a710 con 0x7f04000b3c30 2024-03-17T23:49:24.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.347+0000 7f03fe59c640 1 --2- 172.21.15.150:0/2167988470 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f03e006bbd0 0x7f03e006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:24.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.348+0000 7f03fe59c640 1 --2- 172.21.15.150:0/2167988470 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f03e006bbd0 0x7f03e006e090 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f03f4036000 tx=0x7f03f4008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:24.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.351+0000 7f03eeffd640 1 -- 172.21.15.150:0/2167988470 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f03f0058a30 con 0x7f04000b3c30 2024-03-17T23:49:24.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.527+0000 7f0404d22640 1 -- 172.21.15.150:0/2167988470 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f04001f7950 con 0x7f04000b3c30 2024-03-17T23:49:24.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.528+0000 7f03eeffd640 1 -- 172.21.15.150:0/2167988470 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) ==== 74+0+16974 (secure 0 0 0) 0x7f03f005cad0 con 0x7f04000b3c30 2024-03-17T23:49:24.529 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:24.529 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":58,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:49:12.646307+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:49:09.223630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-17T23:49:09.956422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:49:24.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.531+0000 7f0404d22640 1 -- 172.21.15.150:0/2167988470 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f03e006bbd0 msgr2=0x7f03e006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:24.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.531+0000 7f0404d22640 1 --2- 172.21.15.150:0/2167988470 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f03e006bbd0 0x7f03e006e090 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f03f4036000 tx=0x7f03f4008030 comp rx=0 tx=0).stop 2024-03-17T23:49:24.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.531+0000 7f0404d22640 1 -- 172.21.15.150:0/2167988470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04000b3c30 msgr2=0x7f0400158950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:24.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.531+0000 7f0404d22640 1 --2- 172.21.15.150:0/2167988470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04000b3c30 0x7f0400158950 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f03f000c430 tx=0x7f03f000c8f0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.531+0000 7f0404d22640 1 -- 172.21.15.150:0/2167988470 shutdown_connections 2024-03-17T23:49:24.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.531+0000 7f0404d22640 1 --2- 172.21.15.150:0/2167988470 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f03e006bbd0 0x7f03e006e090 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.531+0000 7f0404d22640 1 --2- 172.21.15.150:0/2167988470 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04000b5730 0x7f040006fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.531+0000 7f0404d22640 1 --2- 172.21.15.150:0/2167988470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04000b3c30 0x7f0400158950 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.531+0000 7f0404d22640 1 --2- 172.21.15.150:0/2167988470 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f040006b190 0x7f040015ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.531+0000 7f0404d22640 1 -- 172.21.15.150:0/2167988470 >> 172.21.15.150:0/2167988470 conn(0x7f04000b6170 msgr2=0x7f040015d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:24.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.531+0000 7f0404d22640 1 -- 172.21.15.150:0/2167988470 shutdown_connections 2024-03-17T23:49:24.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.531+0000 7f0404d22640 1 -- 172.21.15.150:0/2167988470 wait complete. 2024-03-17T23:49:24.542 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:49:09.223630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:49:09.956422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-17T23:49:24.543 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:49:24.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.646+0000 7f87c2b90640 1 Processor -- start 2024-03-17T23:49:24.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.646+0000 7f87c2b90640 1 -- start start 2024-03-17T23:49:24.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.647+0000 7f87c2b90640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc155320 0x7f87bc15bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:24.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.647+0000 7f87c2b90640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87bc0cf3b0 con 0x7f87bc06dd30 2024-03-17T23:49:24.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.647+0000 7f87c2b90640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87bc0cf690 con 0x7f87bc0cd420 2024-03-17T23:49:24.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.647+0000 7f87c2b90640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f87bc15c550 con 0x7f87bc155320 2024-03-17T23:49:24.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.647+0000 7f87bbfff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f87bc06dd30 0x7f87bc0ccd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:51824/0 (socket says 172.21.15.150:51824) 2024-03-17T23:49:24.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.647+0000 7f87bbfff640 1 -- 172.21.15.150:0/555158939 learned_addr learned my addr 172.21.15.150:0/555158939 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:24.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.647+0000 7f87c092c640 1 --2- 172.21.15.150:0/555158939 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc155320 0x7f87bc15bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:24.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.647+0000 7f87bb7fe640 1 -- 172.21.15.150:0/555158939 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3078104240 0 0) 0x7f87bc0cf3b0 con 0x7f87bc06dd30 2024-03-17T23:49:24.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.647+0000 7f87bb7fe640 1 -- 172.21.15.150:0/555158939 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87ac003520 con 0x7f87bc06dd30 2024-03-17T23:49:24.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.647+0000 7f87bb7fe640 1 -- 172.21.15.150:0/555158939 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1849758106 0 0) 0x7f87bc0cf690 con 0x7f87bc0cd420 2024-03-17T23:49:24.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.647+0000 7f87bb7fe640 1 -- 172.21.15.150:0/555158939 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87bc0cf3b0 con 0x7f87bc0cd420 2024-03-17T23:49:24.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.647+0000 7f87c092c640 1 -- 172.21.15.150:0/555158939 >> v1:172.21.15.157:6789/0 conn(0x7f87bc0cd420 legacy=0x7f87bc15ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:24.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.647+0000 7f87c092c640 1 -- 172.21.15.150:0/555158939 >> v1:172.21.15.150:6789/0 conn(0x7f87bc06dd30 legacy=0x7f87bc0ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:24.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.647+0000 7f87c092c640 1 -- 172.21.15.150:0/555158939 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87bc15c7a0 con 0x7f87bc155320 2024-03-17T23:49:24.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.648+0000 7f87c092c640 1 --2- 172.21.15.150:0/555158939 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc155320 0x7f87bc15bfd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f87b00097d0 tx=0x7f87b002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fa9c40a4664cf23 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:24.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.648+0000 7f87bb7fe640 1 -- 172.21.15.150:0/555158939 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b0035020 con 0x7f87bc155320 2024-03-17T23:49:24.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.648+0000 7f87bb7fe640 1 -- 172.21.15.150:0/555158939 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87b0031890 con 0x7f87bc155320 2024-03-17T23:49:24.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.648+0000 7f87bb7fe640 1 -- 172.21.15.150:0/555158939 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b0031c40 con 0x7f87bc155320 2024-03-17T23:49:24.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.649+0000 7f87c2b90640 1 -- 172.21.15.150:0/555158939 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc155320 msgr2=0x7f87bc15bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:24.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.649+0000 7f87c2b90640 1 --2- 172.21.15.150:0/555158939 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc155320 0x7f87bc15bfd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f87b00097d0 tx=0x7f87b002b930 comp rx=0 tx=0).stop 2024-03-17T23:49:24.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.649+0000 7f87c2b90640 1 -- 172.21.15.150:0/555158939 shutdown_connections 2024-03-17T23:49:24.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.649+0000 7f87c2b90640 1 --2- 172.21.15.150:0/555158939 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc155320 0x7f87bc15bfd0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.649+0000 7f87c2b90640 1 -- 172.21.15.150:0/555158939 >> 172.21.15.150:0/555158939 conn(0x7f87bc06b190 msgr2=0x7f87bc06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:24.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.649+0000 7f87c2b90640 1 -- 172.21.15.150:0/555158939 shutdown_connections 2024-03-17T23:49:24.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.649+0000 7f87c2b90640 1 -- 172.21.15.150:0/555158939 wait complete. 2024-03-17T23:49:24.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.650+0000 7f87c2b90640 1 Processor -- start 2024-03-17T23:49:24.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.650+0000 7f87c2b90640 1 -- start start 2024-03-17T23:49:24.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.651+0000 7f87c2b90640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc06dd30 0x7f87bc1454c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:24.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.651+0000 7f87c2b90640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87bc0cd420 0x7f87bc0c93f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:24.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.651+0000 7f87c2b90640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87bc150700 0x7f87bc152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:24.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.651+0000 7f87c2b90640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc155700 con 0x7f87bc150700 2024-03-17T23:49:24.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.651+0000 7f87c2b90640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f87bc06e190 con 0x7f87bc0cd420 2024-03-17T23:49:24.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.651+0000 7f87c2b90640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f87bc155880 con 0x7f87bc06dd30 2024-03-17T23:49:24.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.651+0000 7f87c112d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87bc150700 0x7f87bc152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:24.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.651+0000 7f87c112d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87bc150700 0x7f87bc152220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45724/0 (socket says 172.21.15.150:45724) 2024-03-17T23:49:24.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.651+0000 7f87c112d640 1 -- 172.21.15.150:0/3041898948 learned_addr learned my addr 172.21.15.150:0/3041898948 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:24.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.651+0000 7f87c092c640 1 --2- 172.21.15.150:0/3041898948 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc06dd30 0x7f87bc1454c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:24.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.651+0000 7f87bbfff640 1 --2- 172.21.15.150:0/3041898948 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87bc0cd420 0x7f87bc0c93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:24.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.651+0000 7f87c112d640 1 -- 172.21.15.150:0/3041898948 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc06dd30 msgr2=0x7f87bc1454c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:24.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.651+0000 7f87c112d640 1 --2- 172.21.15.150:0/3041898948 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc06dd30 0x7f87bc1454c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.651+0000 7f87c112d640 1 -- 172.21.15.150:0/3041898948 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87bc0cd420 msgr2=0x7f87bc0c93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:24.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.651+0000 7f87c112d640 1 --2- 172.21.15.150:0/3041898948 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87bc0cd420 0x7f87bc0c93f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.651+0000 7f87c112d640 1 -- 172.21.15.150:0/3041898948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87bc152ef0 con 0x7f87bc150700 2024-03-17T23:49:24.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.652+0000 7f87c112d640 1 --2- 172.21.15.150:0/3041898948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87bc150700 0x7f87bc152220 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f87b400c7a0 tx=0x7f87b400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:24.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.652+0000 7f87b97fa640 1 -- 172.21.15.150:0/3041898948 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4003b70 con 0x7f87bc150700 2024-03-17T23:49:24.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.652+0000 7f87b97fa640 1 -- 172.21.15.150:0/3041898948 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87b4003d30 con 0x7f87bc150700 2024-03-17T23:49:24.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.652+0000 7f87b97fa640 1 -- 172.21.15.150:0/3041898948 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b400aa90 con 0x7f87bc150700 2024-03-17T23:49:24.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.652+0000 7f87c2b90640 1 -- 172.21.15.150:0/3041898948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87bc153190 con 0x7f87bc150700 2024-03-17T23:49:24.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.652+0000 7f87c2b90640 1 -- 172.21.15.150:0/3041898948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87bc153750 con 0x7f87bc150700 2024-03-17T23:49:24.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.653+0000 7f87c2b90640 1 -- 172.21.15.150:0/3041898948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87bc06f5d0 con 0x7f87bc150700 2024-03-17T23:49:24.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.653+0000 7f87b97fa640 1 -- 172.21.15.150:0/3041898948 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f87b400ac30 con 0x7f87bc150700 2024-03-17T23:49:24.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.653+0000 7f87b97fa640 1 --2- 172.21.15.150:0/3041898948 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f87a006bc00 0x7f87a006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:24.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.653+0000 7f87b97fa640 1 -- 172.21.15.150:0/3041898948 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f87b408a990 con 0x7f87bc150700 2024-03-17T23:49:24.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.654+0000 7f87c092c640 1 --2- 172.21.15.150:0/3041898948 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f87a006bc00 0x7f87a006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:24.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.654+0000 7f87c092c640 1 --2- 172.21.15.150:0/3041898948 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f87a006bc00 0x7f87a006e0c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f87b0036000 tx=0x7f87b0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:24.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.658+0000 7f87b97fa640 1 -- 172.21.15.150:0/3041898948 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f87bc06f5d0 con 0x7f87bc150700 2024-03-17T23:49:24.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.831+0000 7f87c2b90640 1 -- 172.21.15.150:0/3041898948 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f87bc0681a0 con 0x7f87bc150700 2024-03-17T23:49:24.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.831+0000 7f87b97fa640 1 -- 172.21.15.150:0/3041898948 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) ==== 93+0+10 (secure 0 0 0) 0x7f87b4058ca0 con 0x7f87bc150700 2024-03-17T23:49:24.832 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:49:24.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.833+0000 7f87c2b90640 1 -- 172.21.15.150:0/3041898948 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f87a006bc00 msgr2=0x7f87a006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:24.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.833+0000 7f87c2b90640 1 --2- 172.21.15.150:0/3041898948 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f87a006bc00 0x7f87a006e0c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f87b0036000 tx=0x7f87b0008030 comp rx=0 tx=0).stop 2024-03-17T23:49:24.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.833+0000 7f87c2b90640 1 -- 172.21.15.150:0/3041898948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87bc150700 msgr2=0x7f87bc152220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:24.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.833+0000 7f87c2b90640 1 --2- 172.21.15.150:0/3041898948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87bc150700 0x7f87bc152220 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f87b400c7a0 tx=0x7f87b400cc60 comp rx=0 tx=0).stop 2024-03-17T23:49:24.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.833+0000 7f87c2b90640 1 -- 172.21.15.150:0/3041898948 shutdown_connections 2024-03-17T23:49:24.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.833+0000 7f87c2b90640 1 --2- 172.21.15.150:0/3041898948 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87bc150700 0x7f87bc152220 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.834+0000 7f87c2b90640 1 --2- 172.21.15.150:0/3041898948 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f87a006bc00 0x7f87a006e0c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.834+0000 7f87c2b90640 1 --2- 172.21.15.150:0/3041898948 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87bc0cd420 0x7f87bc0c93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.834+0000 7f87c2b90640 1 --2- 172.21.15.150:0/3041898948 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87bc06dd30 0x7f87bc1454c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.834+0000 7f87c2b90640 1 -- 172.21.15.150:0/3041898948 >> 172.21.15.150:0/3041898948 conn(0x7f87bc06b190 msgr2=0x7f87bc15abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:24.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.834+0000 7f87c2b90640 1 -- 172.21.15.150:0/3041898948 shutdown_connections 2024-03-17T23:49:24.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.834+0000 7f87c2b90640 1 -- 172.21.15.150:0/3041898948 wait complete. 2024-03-17T23:49:24.845 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:49:24.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.949+0000 7f8dd92e8640 1 Processor -- start 2024-03-17T23:49:24.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.949+0000 7f8dd92e8640 1 -- start start 2024-03-17T23:49:24.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.950+0000 7f8dd92e8640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f8dd415de30 0x7f8dd41619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:24.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.950+0000 7f8dd92e8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8dd40cf120 0x7f8dd40d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:24.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.950+0000 7f8dd92e8640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dd40b9cf0 con 0x7f8dd40b9860 2024-03-17T23:49:24.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.950+0000 7f8dd92e8640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f8dd4162940 con 0x7f8dd415de30 2024-03-17T23:49:24.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.950+0000 7f8dd92e8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8dd41620a0 con 0x7f8dd40cf120 2024-03-17T23:49:24.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.950+0000 7f8dd27fc640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f8dd40b9860 0x7f8dd40cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:51834/0 (socket says 172.21.15.150:51834) 2024-03-17T23:49:24.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.950+0000 7f8dd27fc640 1 -- 172.21.15.150:0/3272088910 learned_addr learned my addr 172.21.15.150:0/3272088910 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:24.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.950+0000 7f8dd37fe640 1 --2- 172.21.15.150:0/3272088910 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8dd40cf120 0x7f8dd40d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:24.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.950+0000 7f8dd2ffd640 1 --2- 172.21.15.150:0/3272088910 >> v2:172.21.15.157:3300/0 conn(0x7f8dd415de30 0x7f8dd41619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:24.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.950+0000 7f8dd1ffb640 1 -- 172.21.15.150:0/3272088910 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3942701413 0 0) 0x7f8dd40b9cf0 con 0x7f8dd40b9860 2024-03-17T23:49:24.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.950+0000 7f8dd1ffb640 1 -- 172.21.15.150:0/3272088910 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dbc003520 con 0x7f8dd40b9860 2024-03-17T23:49:24.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.950+0000 7f8dd2ffd640 1 -- 172.21.15.150:0/3272088910 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8dd40cf120 msgr2=0x7f8dd40d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:24.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.950+0000 7f8dd2ffd640 1 --2- 172.21.15.150:0/3272088910 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8dd40cf120 0x7f8dd40d2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.950+0000 7f8dd2ffd640 1 -- 172.21.15.150:0/3272088910 >> v1:172.21.15.150:6789/0 conn(0x7f8dd40b9860 legacy=0x7f8dd40cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:24.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.950+0000 7f8dd2ffd640 1 -- 172.21.15.150:0/3272088910 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dd4162380 con 0x7f8dd415de30 2024-03-17T23:49:24.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.950+0000 7f8dd37fe640 1 --2- 172.21.15.150:0/3272088910 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8dd40cf120 0x7f8dd40d2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:24.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.951+0000 7f8dd2ffd640 1 --2- 172.21.15.150:0/3272088910 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8dd415de30 0x7f8dd41619a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8dc40097a0 tx=0x7f8dc402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7ad0a3f844734a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:24.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.951+0000 7f8dd1ffb640 1 -- 172.21.15.150:0/3272088910 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc40315a0 con 0x7f8dd415de30 2024-03-17T23:49:24.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.951+0000 7f8dd1ffb640 1 -- 172.21.15.150:0/3272088910 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dc4037070 con 0x7f8dd415de30 2024-03-17T23:49:24.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.951+0000 7f8dd1ffb640 1 -- 172.21.15.150:0/3272088910 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc4031940 con 0x7f8dd415de30 2024-03-17T23:49:24.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.952+0000 7f8dd92e8640 1 -- 172.21.15.150:0/3272088910 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8dd415de30 msgr2=0x7f8dd41619a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:24.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.952+0000 7f8dd92e8640 1 --2- 172.21.15.150:0/3272088910 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8dd415de30 0x7f8dd41619a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8dc40097a0 tx=0x7f8dc402b840 comp rx=0 tx=0).stop 2024-03-17T23:49:24.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.952+0000 7f8dd92e8640 1 -- 172.21.15.150:0/3272088910 shutdown_connections 2024-03-17T23:49:24.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.952+0000 7f8dd92e8640 1 --2- 172.21.15.150:0/3272088910 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8dd40cf120 0x7f8dd40d2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.952+0000 7f8dd92e8640 1 --2- 172.21.15.150:0/3272088910 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8dd415de30 0x7f8dd41619a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.952+0000 7f8dd92e8640 1 -- 172.21.15.150:0/3272088910 >> 172.21.15.150:0/3272088910 conn(0x7f8dd40730b0 msgr2=0x7f8dd40d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:24.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.952+0000 7f8dd92e8640 1 -- 172.21.15.150:0/3272088910 shutdown_connections 2024-03-17T23:49:24.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.952+0000 7f8dd92e8640 1 -- 172.21.15.150:0/3272088910 wait complete. 2024-03-17T23:49:24.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.953+0000 7f8dd92e8640 1 Processor -- start 2024-03-17T23:49:24.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.953+0000 7f8dd92e8640 1 -- start start 2024-03-17T23:49:24.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.954+0000 7f8dd92e8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dd40b9860 0x7f8dd4157a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:24.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.954+0000 7f8dd92e8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8dd40cf120 0x7f8dd4160ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:24.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.954+0000 7f8dd92e8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8dd415de30 0x7f8dd41f6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:24.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.954+0000 7f8dd92e8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd415e210 con 0x7f8dd40b9860 2024-03-17T23:49:24.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.954+0000 7f8dd92e8640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd40d2170 con 0x7f8dd40cf120 2024-03-17T23:49:24.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.954+0000 7f8dd92e8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8dd40d2bd0 con 0x7f8dd415de30 2024-03-17T23:49:24.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.954+0000 7f8dd2ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dd40b9860 0x7f8dd4157a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:24.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.954+0000 7f8dd2ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dd40b9860 0x7f8dd4157a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45732/0 (socket says 172.21.15.150:45732) 2024-03-17T23:49:24.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.954+0000 7f8dd2ffd640 1 -- 172.21.15.150:0/3918146445 learned_addr learned my addr 172.21.15.150:0/3918146445 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:24.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.954+0000 7f8dd27fc640 1 --2- 172.21.15.150:0/3918146445 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8dd40cf120 0x7f8dd4160ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:24.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.954+0000 7f8dd37fe640 1 --2- 172.21.15.150:0/3918146445 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8dd415de30 0x7f8dd41f6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:24.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.954+0000 7f8dd2ffd640 1 -- 172.21.15.150:0/3918146445 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8dd415de30 msgr2=0x7f8dd41f6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:24.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.954+0000 7f8dd2ffd640 1 --2- 172.21.15.150:0/3918146445 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8dd415de30 0x7f8dd41f6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.954+0000 7f8dd2ffd640 1 -- 172.21.15.150:0/3918146445 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8dd40cf120 msgr2=0x7f8dd4160ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:24.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.954+0000 7f8dd2ffd640 1 --2- 172.21.15.150:0/3918146445 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8dd40cf120 0x7f8dd4160ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:24.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.954+0000 7f8dd2ffd640 1 -- 172.21.15.150:0/3918146445 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dd415a0f0 con 0x7f8dd40b9860 2024-03-17T23:49:24.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.954+0000 7f8dd2ffd640 1 --2- 172.21.15.150:0/3918146445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dd40b9860 0x7f8dd4157a30 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4034040 tx=0x7f8dc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:24.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.954+0000 7f8dbb7fe640 1 -- 172.21.15.150:0/3918146445 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc4032630 con 0x7f8dd40b9860 2024-03-17T23:49:24.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.954+0000 7f8dbb7fe640 1 -- 172.21.15.150:0/3918146445 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dc40327f0 con 0x7f8dd40b9860 2024-03-17T23:49:24.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.954+0000 7f8dd92e8640 1 -- 172.21.15.150:0/3918146445 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dd415a3d0 con 0x7f8dd40b9860 2024-03-17T23:49:24.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.954+0000 7f8dbb7fe640 1 -- 172.21.15.150:0/3918146445 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc4032be0 con 0x7f8dd40b9860 2024-03-17T23:49:24.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.955+0000 7f8dd92e8640 1 -- 172.21.15.150:0/3918146445 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dd4159de0 con 0x7f8dd40b9860 2024-03-17T23:49:24.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.955+0000 7f8dd92e8640 1 -- 172.21.15.150:0/3918146445 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dd40681a0 con 0x7f8dd40b9860 2024-03-17T23:49:24.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.956+0000 7f8dbb7fe640 1 -- 172.21.15.150:0/3918146445 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8dc403b3f0 con 0x7f8dd40b9860 2024-03-17T23:49:24.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.956+0000 7f8dbb7fe640 1 --2- 172.21.15.150:0/3918146445 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8dc006bc00 0x7f8dc006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:24.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.956+0000 7f8dbb7fe640 1 -- 172.21.15.150:0/3918146445 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f8dc402f030 con 0x7f8dd40b9860 2024-03-17T23:49:24.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.956+0000 7f8dd27fc640 1 --2- 172.21.15.150:0/3918146445 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8dc006bc00 0x7f8dc006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:24.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.957+0000 7f8dd27fc640 1 --2- 172.21.15.150:0/3918146445 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8dc006bc00 0x7f8dc006e0c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8dd415e510 tx=0x7f8dc8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:24.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:24.961+0000 7f8dbb7fe640 1 -- 172.21.15.150:0/3918146445 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8dc4040070 con 0x7f8dd40b9860 2024-03-17T23:49:25.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.132+0000 7f8dd92e8640 1 -- 172.21.15.150:0/3918146445 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8dd40d1910 con 0x7f8dd40b9860 2024-03-17T23:49:25.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.132+0000 7f8dbb7fe640 1 -- 172.21.15.150:0/3918146445 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) ==== 104+0+11 (secure 0 0 0) 0x7f8dc407b120 con 0x7f8dd40b9860 2024-03-17T23:49:25.135 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:49:25.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.134+0000 7f8dd92e8640 1 -- 172.21.15.150:0/3918146445 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8dc006bc00 msgr2=0x7f8dc006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:25.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.134+0000 7f8dd92e8640 1 --2- 172.21.15.150:0/3918146445 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8dc006bc00 0x7f8dc006e0c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f8dd415e510 tx=0x7f8dc8009040 comp rx=0 tx=0).stop 2024-03-17T23:49:25.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.134+0000 7f8dd92e8640 1 -- 172.21.15.150:0/3918146445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dd40b9860 msgr2=0x7f8dd4157a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:25.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.134+0000 7f8dd92e8640 1 --2- 172.21.15.150:0/3918146445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dd40b9860 0x7f8dd4157a30 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4034040 tx=0x7f8dc40046c0 comp rx=0 tx=0).stop 2024-03-17T23:49:25.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.134+0000 7f8dd92e8640 1 -- 172.21.15.150:0/3918146445 shutdown_connections 2024-03-17T23:49:25.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.134+0000 7f8dd92e8640 1 --2- 172.21.15.150:0/3918146445 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8dd415de30 0x7f8dd41f6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:25.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.134+0000 7f8dd92e8640 1 --2- 172.21.15.150:0/3918146445 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8dc006bc00 0x7f8dc006e0c0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:25.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.134+0000 7f8dd92e8640 1 --2- 172.21.15.150:0/3918146445 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8dd40cf120 0x7f8dd4160ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:25.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.134+0000 7f8dd92e8640 1 --2- 172.21.15.150:0/3918146445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8dd40b9860 0x7f8dd4157a30 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:25.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.134+0000 7f8dd92e8640 1 -- 172.21.15.150:0/3918146445 >> 172.21.15.150:0/3918146445 conn(0x7f8dd40730b0 msgr2=0x7f8dd40b7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:25.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.134+0000 7f8dd92e8640 1 -- 172.21.15.150:0/3918146445 shutdown_connections 2024-03-17T23:49:25.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.134+0000 7f8dd92e8640 1 -- 172.21.15.150:0/3918146445 wait complete. 2024-03-17T23:49:25.146 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:49:25.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.247+0000 7f7a8fa63640 1 Processor -- start 2024-03-17T23:49:25.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.248+0000 7f7a8fa63640 1 -- start start 2024-03-17T23:49:25.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.249+0000 7f7a8fa63640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7a88155320 0x7f7a8815bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:25.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.249+0000 7f7a8fa63640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a880cd420 0x7f7a8815ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:25.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.249+0000 7f7a8fa63640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a880cf3b0 con 0x7f7a8806dd30 2024-03-17T23:49:25.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.249+0000 7f7a8fa63640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f7a8815c550 con 0x7f7a88155320 2024-03-17T23:49:25.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.249+0000 7f7a8fa63640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7a880cf690 con 0x7f7a880cd420 2024-03-17T23:49:25.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.249+0000 7f7a8e000640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a880cd420 0x7f7a8815ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:25.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.249+0000 7f7a8d7ff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7a88155320 0x7f7a8815bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:25.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.249+0000 7f7a8e000640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a880cd420 0x7f7a8815ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:45498/0 (socket says 172.21.15.150:45498) 2024-03-17T23:49:25.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.249+0000 7f7a8e000640 1 -- 172.21.15.150:0/210943950 learned_addr learned my addr 172.21.15.150:0/210943950 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:25.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.249+0000 7f7a8cffe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f7a8806dd30 0x7f7a880ccd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:35396/0 (socket says 172.21.15.150:35396) 2024-03-17T23:49:25.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.249+0000 7f7a8d7ff640 1 -- 172.21.15.150:0/210943950 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a880cd420 msgr2=0x7f7a8815ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:25.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.249+0000 7f7a8d7ff640 1 --2- 172.21.15.150:0/210943950 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a880cd420 0x7f7a8815ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:25.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.249+0000 7f7a8d7ff640 1 -- 172.21.15.150:0/210943950 >> v1:172.21.15.157:6789/0 conn(0x7f7a8806dd30 legacy=0x7f7a880ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:25.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.249+0000 7f7a8d7ff640 1 -- 172.21.15.150:0/210943950 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a8815c7a0 con 0x7f7a88155320 2024-03-17T23:49:25.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.249+0000 7f7a8e000640 1 --2- 172.21.15.150:0/210943950 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a880cd420 0x7f7a8815ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:25.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.250+0000 7f7a8d7ff640 1 --2- 172.21.15.150:0/210943950 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a88155320 0x7f7a8815bfd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7a840097a0 tx=0x7f7a8402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6842c5ac4eae03f server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:25.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.250+0000 7f7a7ffff640 1 -- 172.21.15.150:0/210943950 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a84035020 con 0x7f7a88155320 2024-03-17T23:49:25.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.250+0000 7f7a7ffff640 1 -- 172.21.15.150:0/210943950 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a84031830 con 0x7f7a88155320 2024-03-17T23:49:25.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.250+0000 7f7a7ffff640 1 -- 172.21.15.150:0/210943950 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a84031c00 con 0x7f7a88155320 2024-03-17T23:49:25.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.251+0000 7f7a8fa63640 1 -- 172.21.15.150:0/210943950 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a88155320 msgr2=0x7f7a8815bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:25.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.251+0000 7f7a8fa63640 1 --2- 172.21.15.150:0/210943950 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a88155320 0x7f7a8815bfd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7a840097a0 tx=0x7f7a8402b8d0 comp rx=0 tx=0).stop 2024-03-17T23:49:25.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.251+0000 7f7a8fa63640 1 -- 172.21.15.150:0/210943950 shutdown_connections 2024-03-17T23:49:25.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.251+0000 7f7a8fa63640 1 --2- 172.21.15.150:0/210943950 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a880cd420 0x7f7a8815ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:25.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.251+0000 7f7a8fa63640 1 --2- 172.21.15.150:0/210943950 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a88155320 0x7f7a8815bfd0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:25.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.251+0000 7f7a8fa63640 1 -- 172.21.15.150:0/210943950 >> 172.21.15.150:0/210943950 conn(0x7f7a8806b190 msgr2=0x7f7a8806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:25.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.251+0000 7f7a8fa63640 1 -- 172.21.15.150:0/210943950 shutdown_connections 2024-03-17T23:49:25.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.251+0000 7f7a8fa63640 1 -- 172.21.15.150:0/210943950 wait complete. 2024-03-17T23:49:25.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.252+0000 7f7a8fa63640 1 Processor -- start 2024-03-17T23:49:25.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.252+0000 7f7a8fa63640 1 -- start start 2024-03-17T23:49:25.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.253+0000 7f7a8fa63640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a8806dd30 0x7f7a8815f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:25.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.253+0000 7f7a8fa63640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a880cd420 0x7f7a8814a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:25.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.253+0000 7f7a8fa63640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a88155320 0x7f7a88147d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:25.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.253+0000 7f7a8fa63640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8806e9b0 con 0x7f7a8806dd30 2024-03-17T23:49:25.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.253+0000 7f7a8fa63640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8806e830 con 0x7f7a88155320 2024-03-17T23:49:25.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.253+0000 7f7a8fa63640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7a8806eb30 con 0x7f7a880cd420 2024-03-17T23:49:25.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.253+0000 7f7a8d7ff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a8806dd30 0x7f7a8815f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:25.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.253+0000 7f7a8cffe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a880cd420 0x7f7a8814a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:25.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.253+0000 7f7a8cffe640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a880cd420 0x7f7a8814a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:45502/0 (socket says 172.21.15.150:45502) 2024-03-17T23:49:25.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.253+0000 7f7a8cffe640 1 -- 172.21.15.150:0/3326962550 learned_addr learned my addr 172.21.15.150:0/3326962550 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:25.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.253+0000 7f7a8e000640 1 --2- 172.21.15.150:0/3326962550 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a88155320 0x7f7a88147d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:25.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.253+0000 7f7a8cffe640 1 -- 172.21.15.150:0/3326962550 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a88155320 msgr2=0x7f7a88147d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:25.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.253+0000 7f7a8cffe640 1 --2- 172.21.15.150:0/3326962550 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a88155320 0x7f7a88147d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:25.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.253+0000 7f7a8cffe640 1 -- 172.21.15.150:0/3326962550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a8806dd30 msgr2=0x7f7a8815f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:25.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.253+0000 7f7a8cffe640 1 --2- 172.21.15.150:0/3326962550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a8806dd30 0x7f7a8815f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:25.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.253+0000 7f7a8cffe640 1 -- 172.21.15.150:0/3326962550 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a88156980 con 0x7f7a880cd420 2024-03-17T23:49:25.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.253+0000 7f7a8cffe640 1 --2- 172.21.15.150:0/3326962550 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a880cd420 0x7f7a8814a560 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7a7800c400 tx=0x7f7a7800c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:25.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.254+0000 7f7a7dffb640 1 -- 172.21.15.150:0/3326962550 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a780074e0 con 0x7f7a880cd420 2024-03-17T23:49:25.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.254+0000 7f7a7dffb640 1 -- 172.21.15.150:0/3326962550 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a78007680 con 0x7f7a880cd420 2024-03-17T23:49:25.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.254+0000 7f7a8fa63640 1 -- 172.21.15.150:0/3326962550 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a88149c20 con 0x7f7a880cd420 2024-03-17T23:49:25.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.254+0000 7f7a7dffb640 1 -- 172.21.15.150:0/3326962550 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a78004a00 con 0x7f7a880cd420 2024-03-17T23:49:25.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.254+0000 7f7a8fa63640 1 -- 172.21.15.150:0/3326962550 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a88156c20 con 0x7f7a880cd420 2024-03-17T23:49:25.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.255+0000 7f7a8fa63640 1 -- 172.21.15.150:0/3326962550 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a8814a140 con 0x7f7a880cd420 2024-03-17T23:49:25.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.255+0000 7f7a7dffb640 1 -- 172.21.15.150:0/3326962550 <== mon.2 v2:172.21.15.157:3301/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7a7801a020 con 0x7f7a880cd420 2024-03-17T23:49:25.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.255+0000 7f7a7dffb640 1 --2- 172.21.15.150:0/3326962550 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7a6c06bbd0 0x7f7a6c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:25.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.255+0000 7f7a7dffb640 1 -- 172.21.15.150:0/3326962550 <== mon.2 v2:172.21.15.157:3301/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f7a7808a2e0 con 0x7f7a880cd420 2024-03-17T23:49:25.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.255+0000 7f7a8d7ff640 1 --2- 172.21.15.150:0/3326962550 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7a6c06bbd0 0x7f7a6c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:25.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.256+0000 7f7a8d7ff640 1 --2- 172.21.15.150:0/3326962550 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7a6c06bbd0 0x7f7a6c06e090 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7a84036000 tx=0x7f7a8403f040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:25.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.260+0000 7f7a7dffb640 1 -- 172.21.15.150:0/3326962550 <== mon.2 v2:172.21.15.157:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7a8814a140 con 0x7f7a880cd420 2024-03-17T23:49:25.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.431+0000 7f7a8fa63640 1 -- 172.21.15.150:0/3326962550 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7a880681a0 con 0x7f7a880cd420 2024-03-17T23:49:25.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.432+0000 7f7a7dffb640 1 -- 172.21.15.150:0/3326962550 <== mon.2 v2:172.21.15.157:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) ==== 100+0+11 (secure 0 0 0) 0x7f7a780585f0 con 0x7f7a880cd420 2024-03-17T23:49:25.432 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:49:25.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.434+0000 7f7a8fa63640 1 -- 172.21.15.150:0/3326962550 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7a6c06bbd0 msgr2=0x7f7a6c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:25.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.434+0000 7f7a8fa63640 1 --2- 172.21.15.150:0/3326962550 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7a6c06bbd0 0x7f7a6c06e090 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7a84036000 tx=0x7f7a8403f040 comp rx=0 tx=0).stop 2024-03-17T23:49:25.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.434+0000 7f7a8fa63640 1 -- 172.21.15.150:0/3326962550 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a880cd420 msgr2=0x7f7a8814a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:25.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.434+0000 7f7a8fa63640 1 --2- 172.21.15.150:0/3326962550 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a880cd420 0x7f7a8814a560 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7a7800c400 tx=0x7f7a7800c8c0 comp rx=0 tx=0).stop 2024-03-17T23:49:25.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.434+0000 7f7a8fa63640 1 -- 172.21.15.150:0/3326962550 shutdown_connections 2024-03-17T23:49:25.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.434+0000 7f7a8fa63640 1 --2- 172.21.15.150:0/3326962550 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7a6c06bbd0 0x7f7a6c06e090 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:25.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.434+0000 7f7a8fa63640 1 --2- 172.21.15.150:0/3326962550 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a88155320 0x7f7a88147d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:25.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.434+0000 7f7a8fa63640 1 --2- 172.21.15.150:0/3326962550 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7a880cd420 0x7f7a8814a560 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:25.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.434+0000 7f7a8fa63640 1 --2- 172.21.15.150:0/3326962550 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a8806dd30 0x7f7a8815f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:25.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.434+0000 7f7a8fa63640 1 -- 172.21.15.150:0/3326962550 >> 172.21.15.150:0/3326962550 conn(0x7f7a8806b190 msgr2=0x7f7a8815a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:25.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.434+0000 7f7a8fa63640 1 -- 172.21.15.150:0/3326962550 shutdown_connections 2024-03-17T23:49:25.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.434+0000 7f7a8fa63640 1 -- 172.21.15.150:0/3326962550 wait complete. 2024-03-17T23:49:25.445 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-17T23:49:25.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.548+0000 7f526ef48640 1 Processor -- start 2024-03-17T23:49:25.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.548+0000 7f526ef48640 1 -- start start 2024-03-17T23:49:25.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.548+0000 7f526ef48640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f52680b97f0 0x7f5268162160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:25.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.548+0000 7f526ef48640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f526815ec90 con 0x7f526815df40 2024-03-17T23:49:25.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.548+0000 7f526ef48640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f526815ef70 con 0x7f52681626a0 2024-03-17T23:49:25.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.548+0000 7f526ef48640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f52680b6d40 con 0x7f52680b97f0 2024-03-17T23:49:25.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.548+0000 7f526cce4640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f526815df40 0x7f526815e320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:42308/0 (socket says 172.21.15.150:42308) 2024-03-17T23:49:25.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.548+0000 7f526cce4640 1 -- 172.21.15.150:0/3629347341 learned_addr learned my addr 172.21.15.150:0/3629347341 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:25.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.549+0000 7f5267fff640 1 --2- 172.21.15.150:0/3629347341 >> v2:172.21.15.157:3300/0 conn(0x7f52680b97f0 0x7f5268162160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:25.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.549+0000 7f52677fe640 1 -- 172.21.15.150:0/3629347341 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2928167375 0 0) 0x7f526815ec90 con 0x7f526815df40 2024-03-17T23:49:25.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.549+0000 7f5267fff640 1 -- 172.21.15.150:0/3629347341 >> v1:172.21.15.157:6789/0 conn(0x7f52681626a0 legacy=0x7f52680d2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:25.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.549+0000 7f5267fff640 1 -- 172.21.15.150:0/3629347341 >> v1:172.21.15.150:6789/0 conn(0x7f526815df40 legacy=0x7f526815e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:25.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.549+0000 7f5267fff640 1 -- 172.21.15.150:0/3629347341 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f526815f250 con 0x7f52680b97f0 2024-03-17T23:49:25.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.550+0000 7f5267fff640 1 --2- 172.21.15.150:0/3629347341 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52680b97f0 0x7f5268162160 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f52540097a0 tx=0x7f525402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5382d3b9962a7c61 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:25.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.550+0000 7f52677fe640 1 -- 172.21.15.150:0/3629347341 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5254034070 con 0x7f52680b97f0 2024-03-17T23:49:25.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.550+0000 7f52677fe640 1 -- 172.21.15.150:0/3629347341 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5254030610 con 0x7f52680b97f0 2024-03-17T23:49:25.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.550+0000 7f52677fe640 1 -- 172.21.15.150:0/3629347341 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52540309e0 con 0x7f52680b97f0 2024-03-17T23:49:25.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.550+0000 7f526ef48640 1 -- 172.21.15.150:0/3629347341 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52680b97f0 msgr2=0x7f5268162160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:25.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.550+0000 7f526ef48640 1 --2- 172.21.15.150:0/3629347341 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52680b97f0 0x7f5268162160 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f52540097a0 tx=0x7f525402b960 comp rx=0 tx=0).stop 2024-03-17T23:49:25.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.550+0000 7f526ef48640 1 -- 172.21.15.150:0/3629347341 shutdown_connections 2024-03-17T23:49:25.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.550+0000 7f526ef48640 1 --2- 172.21.15.150:0/3629347341 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52680b97f0 0x7f5268162160 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:25.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.550+0000 7f526ef48640 1 -- 172.21.15.150:0/3629347341 >> 172.21.15.150:0/3629347341 conn(0x7f5268072ef0 msgr2=0x7f52680d3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:25.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.550+0000 7f526ef48640 1 -- 172.21.15.150:0/3629347341 shutdown_connections 2024-03-17T23:49:25.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.550+0000 7f526ef48640 1 -- 172.21.15.150:0/3629347341 wait complete. 2024-03-17T23:49:25.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.551+0000 7f526ef48640 1 Processor -- start 2024-03-17T23:49:25.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.551+0000 7f526ef48640 1 -- start start 2024-03-17T23:49:25.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.552+0000 7f526ef48640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52680b97f0 0x7f52680cda00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:25.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.552+0000 7f526ef48640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f526815df40 0x7f52680b5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:25.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.552+0000 7f526ef48640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52681626a0 0x7f5268153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:25.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.552+0000 7f526ef48640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f52680b9ca0 con 0x7f526815df40 2024-03-17T23:49:25.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.552+0000 7f526ef48640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f526815ea30 con 0x7f52680b97f0 2024-03-17T23:49:25.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.552+0000 7f526ef48640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f52680d2810 con 0x7f52681626a0 2024-03-17T23:49:25.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.552+0000 7f5267fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f526815df40 0x7f52680b5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:25.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.552+0000 7f5267fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f526815df40 0x7f52680b5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59034/0 (socket says 172.21.15.150:59034) 2024-03-17T23:49:25.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.552+0000 7f5267fff640 1 -- 172.21.15.150:0/1427550216 learned_addr learned my addr 172.21.15.150:0/1427550216 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:25.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.552+0000 7f526cce4640 1 --2- 172.21.15.150:0/1427550216 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52680b97f0 0x7f52680cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:25.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.553+0000 7f526d4e5640 1 --2- 172.21.15.150:0/1427550216 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52681626a0 0x7f5268153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:25.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.553+0000 7f5267fff640 1 -- 172.21.15.150:0/1427550216 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52681626a0 msgr2=0x7f5268153100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:25.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.553+0000 7f5267fff640 1 --2- 172.21.15.150:0/1427550216 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52681626a0 0x7f5268153100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:25.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.553+0000 7f5267fff640 1 -- 172.21.15.150:0/1427550216 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52680b97f0 msgr2=0x7f52680cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:25.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.553+0000 7f5267fff640 1 --2- 172.21.15.150:0/1427550216 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52680b97f0 0x7f52680cda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:25.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.553+0000 7f5267fff640 1 -- 172.21.15.150:0/1427550216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52681589f0 con 0x7f526815df40 2024-03-17T23:49:25.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.553+0000 7f526cce4640 1 --2- 172.21.15.150:0/1427550216 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52680b97f0 0x7f52680cda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:25.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.553+0000 7f5267fff640 1 --2- 172.21.15.150:0/1427550216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f526815df40 0x7f52680b5e70 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f5254000c00 tx=0x7f52540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:25.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.553+0000 7f52657fa640 1 -- 172.21.15.150:0/1427550216 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5254034040 con 0x7f526815df40 2024-03-17T23:49:25.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.553+0000 7f52657fa640 1 -- 172.21.15.150:0/1427550216 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5254004030 con 0x7f526815df40 2024-03-17T23:49:25.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.553+0000 7f526ef48640 1 -- 172.21.15.150:0/1427550216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5268158cd0 con 0x7f526815df40 2024-03-17T23:49:25.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.553+0000 7f52657fa640 1 -- 172.21.15.150:0/1427550216 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5254004420 con 0x7f526815df40 2024-03-17T23:49:25.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.553+0000 7f526ef48640 1 -- 172.21.15.150:0/1427550216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5268159290 con 0x7f526815df40 2024-03-17T23:49:25.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.554+0000 7f526ef48640 1 -- 172.21.15.150:0/1427550216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52680681a0 con 0x7f526815df40 2024-03-17T23:49:25.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.555+0000 7f52657fa640 1 -- 172.21.15.150:0/1427550216 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5254037070 con 0x7f526815df40 2024-03-17T23:49:25.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.555+0000 7f52657fa640 1 --2- 172.21.15.150:0/1427550216 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f524c06bbd0 0x7f524c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:25.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.555+0000 7f52657fa640 1 -- 172.21.15.150:0/1427550216 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f52540ac7d0 con 0x7f526815df40 2024-03-17T23:49:25.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.555+0000 7f526cce4640 1 --2- 172.21.15.150:0/1427550216 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f524c06bbd0 0x7f524c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:25.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.555+0000 7f526cce4640 1 --2- 172.21.15.150:0/1427550216 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f524c06bbd0 0x7f524c06e090 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f525c007f70 tx=0x7f525c007f00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:25.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.559+0000 7f52657fa640 1 -- 172.21.15.150:0/1427550216 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f525407ac00 con 0x7f526815df40 2024-03-17T23:49:25.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.746+0000 7f526ef48640 1 -- 172.21.15.150:0/1427550216 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f52681610d0 con 0x7f526815df40 2024-03-17T23:49:25.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.746+0000 7f52657fa640 1 -- 172.21.15.150:0/1427550216 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v58) ==== 80+0+0 (secure 0 0 0) 0x7f525407eca0 con 0x7f526815df40 2024-03-17T23:49:25.747 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-03-17T23:49:25.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.748+0000 7f526ef48640 1 -- 172.21.15.150:0/1427550216 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f524c06bbd0 msgr2=0x7f524c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:25.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.748+0000 7f526ef48640 1 --2- 172.21.15.150:0/1427550216 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f524c06bbd0 0x7f524c06e090 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f525c007f70 tx=0x7f525c007f00 comp rx=0 tx=0).stop 2024-03-17T23:49:25.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.748+0000 7f526ef48640 1 -- 172.21.15.150:0/1427550216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f526815df40 msgr2=0x7f52680b5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:25.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.748+0000 7f526ef48640 1 --2- 172.21.15.150:0/1427550216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f526815df40 0x7f52680b5e70 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f5254000c00 tx=0x7f52540046c0 comp rx=0 tx=0).stop 2024-03-17T23:49:25.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.749+0000 7f526ef48640 1 -- 172.21.15.150:0/1427550216 shutdown_connections 2024-03-17T23:49:25.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.749+0000 7f526ef48640 1 --2- 172.21.15.150:0/1427550216 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f52681626a0 0x7f5268153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:25.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.749+0000 7f526ef48640 1 --2- 172.21.15.150:0/1427550216 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f526815df40 0x7f52680b5e70 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:25.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.749+0000 7f526ef48640 1 --2- 172.21.15.150:0/1427550216 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f524c06bbd0 0x7f524c06e090 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:25.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.749+0000 7f526ef48640 1 --2- 172.21.15.150:0/1427550216 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52680b97f0 0x7f52680cda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:25.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.749+0000 7f526ef48640 1 -- 172.21.15.150:0/1427550216 >> 172.21.15.150:0/1427550216 conn(0x7f5268072ef0 msgr2=0x7f52681608e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:25.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.749+0000 7f526ef48640 1 -- 172.21.15.150:0/1427550216 shutdown_connections 2024-03-17T23:49:25.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:25.749+0000 7f526ef48640 1 -- 172.21.15.150:0/1427550216 wait complete. 2024-03-17T23:49:25.760 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi150.front.sepia.ceph.com... 2024-03-17T23:49:25.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:25.760 DEBUG:teuthology.orchestra.run.smithi150:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-17T23:49:25.785 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:ceph-fuse[60301]: fuse finished with error 0 and tester_r 0 2024-03-17T23:49:25.788 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:49:27.783 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-17T23:49:27.787 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-17T23:49:31.889 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi150.front.sepia.ceph.com 2024-03-17T23:49:31.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:31.890 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-17T23:49:31.909 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-17T23:49:32.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.058+0000 7fcbefad1640 1 Processor -- start 2024-03-17T23:49:32.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.058+0000 7fcbefad1640 1 -- start start 2024-03-17T23:49:32.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.059+0000 7fcbefad1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcbe8155320 0x7fcbe815bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:32.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.059+0000 7fcbefad1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbe80cf3b0 con 0x7fcbe806dd30 2024-03-17T23:49:32.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.059+0000 7fcbefad1640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbe80cf690 con 0x7fcbe80cd420 2024-03-17T23:49:32.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.059+0000 7fcbefad1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fcbe815c550 con 0x7fcbe8155320 2024-03-17T23:49:32.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.059+0000 7fcbed06c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fcbe806dd30 0x7fcbe80ccd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:42324/0 (socket says 172.21.15.150:42324) 2024-03-17T23:49:32.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.059+0000 7fcbed06c640 1 -- 172.21.15.150:0/3697070575 learned_addr learned my addr 172.21.15.150:0/3697070575 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:32.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.059+0000 7fcbed86d640 1 --2- 172.21.15.150:0/3697070575 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcbe8155320 0x7fcbe815bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:32.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.059+0000 7fcbec86b640 1 -- 172.21.15.150:0/3697070575 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1048995134 0 0) 0x7fcbe80cf3b0 con 0x7fcbe806dd30 2024-03-17T23:49:32.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.059+0000 7fcbec86b640 1 -- 172.21.15.150:0/3697070575 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbd4005730 con 0x7fcbe806dd30 2024-03-17T23:49:32.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.059+0000 7fcbec86b640 1 -- 172.21.15.150:0/3697070575 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1927340125 0 0) 0x7fcbe80cf690 con 0x7fcbe80cd420 2024-03-17T23:49:32.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.059+0000 7fcbec86b640 1 -- 172.21.15.150:0/3697070575 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbe80cf3b0 con 0x7fcbe80cd420 2024-03-17T23:49:32.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.059+0000 7fcbed86d640 1 -- 172.21.15.150:0/3697070575 >> v1:172.21.15.157:6789/0 conn(0x7fcbe80cd420 legacy=0x7fcbe815ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:32.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.059+0000 7fcbed86d640 1 -- 172.21.15.150:0/3697070575 >> v1:172.21.15.150:6789/0 conn(0x7fcbe806dd30 legacy=0x7fcbe80ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:32.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.059+0000 7fcbed86d640 1 -- 172.21.15.150:0/3697070575 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe815c7a0 con 0x7fcbe8155320 2024-03-17T23:49:32.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.060+0000 7fcbed86d640 1 --2- 172.21.15.150:0/3697070575 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcbe8155320 0x7fcbe815bfd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc0097d0 tx=0x7fcbdc02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a912adc76e560cc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:32.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.060+0000 7fcbec86b640 1 -- 172.21.15.150:0/3697070575 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc035020 con 0x7fcbe8155320 2024-03-17T23:49:32.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.060+0000 7fcbec86b640 1 -- 172.21.15.150:0/3697070575 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbdc031890 con 0x7fcbe8155320 2024-03-17T23:49:32.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.060+0000 7fcbec86b640 1 -- 172.21.15.150:0/3697070575 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc031c40 con 0x7fcbe8155320 2024-03-17T23:49:32.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.061+0000 7fcbefad1640 1 -- 172.21.15.150:0/3697070575 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcbe8155320 msgr2=0x7fcbe815bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:32.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.061+0000 7fcbefad1640 1 --2- 172.21.15.150:0/3697070575 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcbe8155320 0x7fcbe815bfd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc0097d0 tx=0x7fcbdc02b930 comp rx=0 tx=0).stop 2024-03-17T23:49:32.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.061+0000 7fcbefad1640 1 -- 172.21.15.150:0/3697070575 shutdown_connections 2024-03-17T23:49:32.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.061+0000 7fcbefad1640 1 --2- 172.21.15.150:0/3697070575 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcbe8155320 0x7fcbe815bfd0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:32.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.061+0000 7fcbefad1640 1 -- 172.21.15.150:0/3697070575 >> 172.21.15.150:0/3697070575 conn(0x7fcbe806b190 msgr2=0x7fcbe806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:32.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.061+0000 7fcbefad1640 1 -- 172.21.15.150:0/3697070575 shutdown_connections 2024-03-17T23:49:32.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.061+0000 7fcbefad1640 1 -- 172.21.15.150:0/3697070575 wait complete. 2024-03-17T23:49:32.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.062+0000 7fcbefad1640 1 Processor -- start 2024-03-17T23:49:32.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.062+0000 7fcbefad1640 1 -- start start 2024-03-17T23:49:32.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.063+0000 7fcbefad1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcbe806dd30 0x7fcbe813d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:32.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.063+0000 7fcbefad1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcbe80cd420 0x7fcbe8149640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:32.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.063+0000 7fcbefad1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbe8155320 0x7fcbe8158d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:32.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.063+0000 7fcbefad1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe8155700 con 0x7fcbe8155320 2024-03-17T23:49:32.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.063+0000 7fcbefad1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe806e190 con 0x7fcbe806dd30 2024-03-17T23:49:32.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.063+0000 7fcbefad1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fcbe8155880 con 0x7fcbe80cd420 2024-03-17T23:49:32.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.063+0000 7fcbee06e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbe8155320 0x7fcbe8158d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:32.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.063+0000 7fcbee06e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbe8155320 0x7fcbe8158d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59040/0 (socket says 172.21.15.150:59040) 2024-03-17T23:49:32.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.063+0000 7fcbee06e640 1 -- 172.21.15.150:0/1956285891 learned_addr learned my addr 172.21.15.150:0/1956285891 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:32.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.063+0000 7fcbed86d640 1 --2- 172.21.15.150:0/1956285891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcbe806dd30 0x7fcbe813d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:32.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.063+0000 7fcbed06c640 1 --2- 172.21.15.150:0/1956285891 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcbe80cd420 0x7fcbe8149640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:32.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.063+0000 7fcbee06e640 1 -- 172.21.15.150:0/1956285891 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcbe80cd420 msgr2=0x7fcbe8149640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:32.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.063+0000 7fcbee06e640 1 --2- 172.21.15.150:0/1956285891 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcbe80cd420 0x7fcbe8149640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:32.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.063+0000 7fcbee06e640 1 -- 172.21.15.150:0/1956285891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcbe806dd30 msgr2=0x7fcbe813d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:32.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.063+0000 7fcbee06e640 1 --2- 172.21.15.150:0/1956285891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcbe806dd30 0x7fcbe813d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:32.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.063+0000 7fcbee06e640 1 -- 172.21.15.150:0/1956285891 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe814a020 con 0x7fcbe8155320 2024-03-17T23:49:32.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.063+0000 7fcbee06e640 1 --2- 172.21.15.150:0/1956285891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbe8155320 0x7fcbe8158d90 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fcbe000c790 tx=0x7fcbe000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:32.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.063+0000 7fcbda7fc640 1 -- 172.21.15.150:0/1956285891 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe0003b70 con 0x7fcbe8155320 2024-03-17T23:49:32.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.063+0000 7fcbda7fc640 1 -- 172.21.15.150:0/1956285891 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbe0003d10 con 0x7fcbe8155320 2024-03-17T23:49:32.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.063+0000 7fcbefad1640 1 -- 172.21.15.150:0/1956285891 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbe814a300 con 0x7fcbe8155320 2024-03-17T23:49:32.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.063+0000 7fcbda7fc640 1 -- 172.21.15.150:0/1956285891 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe000aa50 con 0x7fcbe8155320 2024-03-17T23:49:32.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.064+0000 7fcbefad1640 1 -- 172.21.15.150:0/1956285891 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbe814a8c0 con 0x7fcbe8155320 2024-03-17T23:49:32.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.064+0000 7fcbefad1640 1 -- 172.21.15.150:0/1956285891 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbe815c9c0 con 0x7fcbe8155320 2024-03-17T23:49:32.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.065+0000 7fcbda7fc640 1 -- 172.21.15.150:0/1956285891 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fcbe000abf0 con 0x7fcbe8155320 2024-03-17T23:49:32.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.065+0000 7fcbda7fc640 1 --2- 172.21.15.150:0/1956285891 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcbcc06bbd0 0x7fcbcc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:32.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.065+0000 7fcbda7fc640 1 -- 172.21.15.150:0/1956285891 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7fcbe008a9c0 con 0x7fcbe8155320 2024-03-17T23:49:32.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.065+0000 7fcbed86d640 1 --2- 172.21.15.150:0/1956285891 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcbcc06bbd0 0x7fcbcc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:32.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.066+0000 7fcbed86d640 1 --2- 172.21.15.150:0/1956285891 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcbcc06bbd0 0x7fcbcc06e090 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc036000 tx=0x7fcbdc03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:32.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.070+0000 7fcbda7fc640 1 -- 172.21.15.150:0/1956285891 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fcbe815c9c0 con 0x7fcbe8155320 2024-03-17T23:49:32.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.278+0000 7fcbefad1640 1 -- 172.21.15.150:0/1956285891 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fcbe80681a0 con 0x7fcbe8155320 2024-03-17T23:49:32.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.279+0000 7fcbda7fc640 1 -- 172.21.15.150:0/1956285891 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v22) ==== 53+0+73 (secure 0 0 0) 0x7fcbe008ce20 con 0x7fcbe8155320 2024-03-17T23:49:32.280 INFO:teuthology.orchestra.run.smithi150.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-17T23:49:32.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.281+0000 7fcbefad1640 1 -- 172.21.15.150:0/1956285891 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcbcc06bbd0 msgr2=0x7fcbcc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:32.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.281+0000 7fcbefad1640 1 --2- 172.21.15.150:0/1956285891 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcbcc06bbd0 0x7fcbcc06e090 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc036000 tx=0x7fcbdc03c040 comp rx=0 tx=0).stop 2024-03-17T23:49:32.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.281+0000 7fcbefad1640 1 -- 172.21.15.150:0/1956285891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbe8155320 msgr2=0x7fcbe8158d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:32.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.281+0000 7fcbefad1640 1 --2- 172.21.15.150:0/1956285891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbe8155320 0x7fcbe8158d90 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fcbe000c790 tx=0x7fcbe000cc50 comp rx=0 tx=0).stop 2024-03-17T23:49:32.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.281+0000 7fcbefad1640 1 -- 172.21.15.150:0/1956285891 shutdown_connections 2024-03-17T23:49:32.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.281+0000 7fcbefad1640 1 --2- 172.21.15.150:0/1956285891 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcbe8155320 0x7fcbe8158d90 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:32.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.281+0000 7fcbefad1640 1 --2- 172.21.15.150:0/1956285891 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fcbe80cd420 0x7fcbe8149640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:32.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.281+0000 7fcbefad1640 1 --2- 172.21.15.150:0/1956285891 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fcbcc06bbd0 0x7fcbcc06e090 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:32.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.281+0000 7fcbefad1640 1 --2- 172.21.15.150:0/1956285891 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcbe806dd30 0x7fcbe813d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:32.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.281+0000 7fcbefad1640 1 -- 172.21.15.150:0/1956285891 >> 172.21.15.150:0/1956285891 conn(0x7fcbe806b190 msgr2=0x7fcbe80cbd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:32.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.281+0000 7fcbefad1640 1 -- 172.21.15.150:0/1956285891 shutdown_connections 2024-03-17T23:49:32.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32.281+0000 7fcbefad1640 1 -- 172.21.15.150:0/1956285891 wait complete. 2024-03-17T23:49:32.293 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-17T23:49:32.293 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-17T23:49:32.293 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi150.front.sepia.ceph.com 2024-03-17T23:49:32.293 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-17T23:49:32.293 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-17T23:49:32.293 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-17T23:49:32.293 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-17T23:49:32.293 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-17T23:49:32.293 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-17T23:49:32.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:32.294 DEBUG:teuthology.orchestra.run.smithi150:> ip addr 2024-03-17T23:49:32.309 INFO:teuthology.orchestra.run.smithi150.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-17T23:49:32.309 INFO:teuthology.orchestra.run.smithi150.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-17T23:49:32.309 INFO:teuthology.orchestra.run.smithi150.stdout: inet 127.0.0.1/8 scope host lo 2024-03-17T23:49:32.309 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:49:32.309 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 ::1/128 scope host 2024-03-17T23:49:32.310 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:49:32.310 INFO:teuthology.orchestra.run.smithi150.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:49:32.310 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bc brd ff:ff:ff:ff:ff:ff 2024-03-17T23:49:32.310 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f0 2024-03-17T23:49:32.310 INFO:teuthology.orchestra.run.smithi150.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:49:32.310 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bd brd ff:ff:ff:ff:ff:ff 2024-03-17T23:49:32.310 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f1 2024-03-17T23:49:32.310 INFO:teuthology.orchestra.run.smithi150.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:49:32.310 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:34 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:49:32.310 INFO:teuthology.orchestra.run.smithi150.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-17T23:49:32.310 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:35 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:49:32.310 INFO:teuthology.orchestra.run.smithi150.stdout: inet 172.21.15.150/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-17T23:49:32.310 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft 42282sec preferred_lft 42282sec 2024-03-17T23:49:32.311 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::ec4:7aff:fe88:6e35/64 scope link 2024-03-17T23:49:32.311 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:49:32.311 INFO:teuthology.orchestra.run.smithi150.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-17T23:49:32.311 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether f6:2d:aa:e3:d6:a3 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:49:32.311 INFO:teuthology.orchestra.run.smithi150.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-17T23:49:32.311 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:49:32.311 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::bc2f:65ff:fe6d:abc3/64 scope link 2024-03-17T23:49:32.311 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:49:32.311 INFO:teuthology.orchestra.run.smithi150.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-17T23:49:32.311 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether f6:2d:aa:e3:d6:a3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:49:32.311 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::f42d:aaff:fee3:d6a3/64 scope link 2024-03-17T23:49:32.311 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:49:32.311 INFO:teuthology.orchestra.run.smithi150.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-17T23:49:32.311 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether fa:95:93:4e:f1:dd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:49:32.312 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::f895:93ff:fe4e:f1dd/64 scope link 2024-03-17T23:49:32.312 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:49:32.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:32.312 DEBUG:teuthology.orchestra.run.smithi150:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-17T23:49:32.375 INFO:teuthology.orchestra.run.smithi150.stdout:1 2024-03-17T23:49:32.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:32.376 DEBUG:teuthology.orchestra.run.smithi150:> route 2024-03-17T23:49:32.432 INFO:teuthology.orchestra.run.smithi150.stdout:Kernel IP routing table 2024-03-17T23:49:32.432 INFO:teuthology.orchestra.run.smithi150.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-17T23:49:32.432 INFO:teuthology.orchestra.run.smithi150.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-17T23:49:32.433 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-17T23:49:32.433 INFO:teuthology.orchestra.run.smithi150.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-17T23:49:32.433 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:49:32.433 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-03-17T23:49:32.433 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-17T23:49:32.433 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-17T23:49:32.433 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-17T23:49:32.433 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-03-17T23:49:32.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:49:32.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:49:32.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:32.568 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-03-17T23:49:32.583 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-17T23:49:32.583 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-17T23:49:32.583 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-17T23:49:32.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:32.583 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set brx.0 up 2024-03-17T23:49:32.650 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-17T23:49:32.650 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-17T23:49:32.650 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-17T23:49:32.705 INFO:teuthology.orchestra.run.smithi150.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-17T23:49:32.705 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-17T23:49:32.705 DEBUG:teuthology.orchestra.run.smithi150:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-17T23:49:32.760 DEBUG:teuthology.orchestra.run.smithi150:> sudo modprobe fuse 2024-03-17T23:49:32.828 DEBUG:teuthology.orchestra.run.smithi150:> cat /proc/self/mounts | awk '{print $2}' 2024-03-17T23:49:32.884 INFO:teuthology.orchestra.run.smithi150.stdout:/proc 2024-03-17T23:49:32.884 INFO:teuthology.orchestra.run.smithi150.stdout:/sys 2024-03-17T23:49:32.884 INFO:teuthology.orchestra.run.smithi150.stdout:/dev 2024-03-17T23:49:32.884 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/security 2024-03-17T23:49:32.884 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/shm 2024-03-17T23:49:32.884 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/pts 2024-03-17T23:49:32.884 INFO:teuthology.orchestra.run.smithi150.stdout:/run 2024-03-17T23:49:32.884 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/cgroup 2024-03-17T23:49:32.885 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/pstore 2024-03-17T23:49:32.885 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/bpf 2024-03-17T23:49:32.885 INFO:teuthology.orchestra.run.smithi150.stdout:/ 2024-03-17T23:49:32.885 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/selinux 2024-03-17T23:49:32.885 INFO:teuthology.orchestra.run.smithi150.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:49:32.885 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/hugepages 2024-03-17T23:49:32.885 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/mqueue 2024-03-17T23:49:32.885 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/debug 2024-03-17T23:49:32.885 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/tracing 2024-03-17T23:49:32.885 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-17T23:49:32.885 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/fuse/connections 2024-03-17T23:49:32.885 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/config 2024-03-17T23:49:32.885 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-sysctl.service 2024-03-17T23:49:32.885 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-17T23:49:32.886 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/nfs/rpc_pipefs 2024-03-17T23:49:32.886 INFO:teuthology.orchestra.run.smithi150.stdout:/run/user/1000 2024-03-17T23:49:32.886 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph 2024-03-17T23:49:32.886 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-0 2024-03-17T23:49:32.886 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-1 2024-03-17T23:49:32.886 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-2 2024-03-17T23:49:32.886 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-3 2024-03-17T23:49:32.886 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns 2024-03-17T23:49:32.886 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:49:32.886 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:49:32.886 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:49:32.886 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:49:32.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:32.887 DEBUG:teuthology.orchestra.run.smithi150:> ls /sys/fs/fuse/connections 2024-03-17T23:49:32.939 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-17T23:49:32.939 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-17T23:49:32.982 DEBUG:teuthology.orchestra.run.smithi150:> sudo modprobe fuse 2024-03-17T23:49:33.008 DEBUG:teuthology.orchestra.run.smithi150:> cat /proc/self/mounts | awk '{print $2}' 2024-03-17T23:49:33.064 INFO:teuthology.orchestra.run.smithi150.stdout:/proc 2024-03-17T23:49:33.064 INFO:teuthology.orchestra.run.smithi150.stdout:/sys 2024-03-17T23:49:33.064 INFO:teuthology.orchestra.run.smithi150.stdout:/dev 2024-03-17T23:49:33.064 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/security 2024-03-17T23:49:33.064 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/shm 2024-03-17T23:49:33.064 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/pts 2024-03-17T23:49:33.064 INFO:teuthology.orchestra.run.smithi150.stdout:/run 2024-03-17T23:49:33.064 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/cgroup 2024-03-17T23:49:33.064 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/pstore 2024-03-17T23:49:33.065 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/bpf 2024-03-17T23:49:33.065 INFO:teuthology.orchestra.run.smithi150.stdout:/ 2024-03-17T23:49:33.065 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/selinux 2024-03-17T23:49:33.065 INFO:teuthology.orchestra.run.smithi150.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:49:33.065 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/hugepages 2024-03-17T23:49:33.065 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/mqueue 2024-03-17T23:49:33.065 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/debug 2024-03-17T23:49:33.065 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/tracing 2024-03-17T23:49:33.065 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-17T23:49:33.065 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/fuse/connections 2024-03-17T23:49:33.065 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/config 2024-03-17T23:49:33.065 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-sysctl.service 2024-03-17T23:49:33.065 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-17T23:49:33.065 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/nfs/rpc_pipefs 2024-03-17T23:49:33.066 INFO:teuthology.orchestra.run.smithi150.stdout:/run/user/1000 2024-03-17T23:49:33.066 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph 2024-03-17T23:49:33.066 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-0 2024-03-17T23:49:33.066 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-1 2024-03-17T23:49:33.066 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-2 2024-03-17T23:49:33.066 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-3 2024-03-17T23:49:33.066 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns 2024-03-17T23:49:33.066 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:49:33.066 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:49:33.066 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:49:33.066 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:49:33.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:33.067 DEBUG:teuthology.orchestra.run.smithi150:> ls /sys/fs/fuse/connections 2024-03-17T23:49:33.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:2024-03-17T23:49:33.066+0000 7faa7ba4a480 -1 init, newargv = 0x555d05eed040 newargc=15 2024-03-17T23:49:33.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:ceph-fuse[61283]: starting ceph client 2024-03-17T23:49:33.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:ceph-fuse[61283]: starting fuse 2024-03-17T23:49:33.082 INFO:teuthology.orchestra.run.smithi150.stdout:43 2024-03-17T23:49:33.082 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-17T23:49:33.083 DEBUG:teuthology.orchestra.run.smithi150:> sudo stdin-killer -- python3 -c ' 2024-03-17T23:49:33.083 DEBUG:teuthology.orchestra.run.smithi150:> import glob 2024-03-17T23:49:33.083 DEBUG:teuthology.orchestra.run.smithi150:> import re 2024-03-17T23:49:33.083 DEBUG:teuthology.orchestra.run.smithi150:> import os 2024-03-17T23:49:33.083 DEBUG:teuthology.orchestra.run.smithi150:> import subprocess 2024-03-17T23:49:33.083 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:49:33.083 DEBUG:teuthology.orchestra.run.smithi150:> def _find_admin_socket(client_name): 2024-03-17T23:49:33.083 DEBUG:teuthology.orchestra.run.smithi150:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-17T23:49:33.083 DEBUG:teuthology.orchestra.run.smithi150:> files = glob.glob(asok_path) 2024-03-17T23:49:33.083 DEBUG:teuthology.orchestra.run.smithi150:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-17T23:49:33.083 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:49:33.083 DEBUG:teuthology.orchestra.run.smithi150:> # Given a non-glob path, it better be there 2024-03-17T23:49:33.083 DEBUG:teuthology.orchestra.run.smithi150:> if "*" not in asok_path: 2024-03-17T23:49:33.084 DEBUG:teuthology.orchestra.run.smithi150:> assert(len(files) == 1) 2024-03-17T23:49:33.084 DEBUG:teuthology.orchestra.run.smithi150:> return files[0] 2024-03-17T23:49:33.084 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:49:33.084 DEBUG:teuthology.orchestra.run.smithi150:> for f in files: 2024-03-17T23:49:33.084 DEBUG:teuthology.orchestra.run.smithi150:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-17T23:49:33.084 DEBUG:teuthology.orchestra.run.smithi150:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-17T23:49:33.084 DEBUG:teuthology.orchestra.run.smithi150:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-17T23:49:33.084 DEBUG:teuthology.orchestra.run.smithi150:> contents = proc_f.read() 2024-03-17T23:49:33.084 DEBUG:teuthology.orchestra.run.smithi150:> if mountpoint in contents: 2024-03-17T23:49:33.084 DEBUG:teuthology.orchestra.run.smithi150:> return f 2024-03-17T23:49:33.084 DEBUG:teuthology.orchestra.run.smithi150:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-17T23:49:33.084 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:49:33.084 DEBUG:teuthology.orchestra.run.smithi150:> print(_find_admin_socket("client.0")) 2024-03-17T23:49:33.084 DEBUG:teuthology.orchestra.run.smithi150:> ' 2024-03-17T23:49:33.202 INFO:teuthology.orchestra.run.smithi150.stdout:/var/run/ceph/ceph-client.0.61283.asok 2024-03-17T23:49:33.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:49:33.213 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.61283.asok 2024-03-17T23:49:33.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:33.214 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.61283.asok status 2024-03-17T23:49:33.350 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-17T23:49:33.350 INFO:teuthology.orchestra.run.smithi150.stdout: "metadata": { 2024-03-17T23:49:33.350 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-17T23:49:33.350 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-17T23:49:33.350 INFO:teuthology.orchestra.run.smithi150.stdout: "entity_id": "0", 2024-03-17T23:49:33.350 INFO:teuthology.orchestra.run.smithi150.stdout: "hostname": "smithi150", 2024-03-17T23:49:33.350 INFO:teuthology.orchestra.run.smithi150.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-17T23:49:33.350 INFO:teuthology.orchestra.run.smithi150.stdout: "pid": "61283", 2024-03-17T23:49:33.350 INFO:teuthology.orchestra.run.smithi150.stdout: "root": "/" 2024-03-17T23:49:33.350 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:33.351 INFO:teuthology.orchestra.run.smithi150.stdout: "dentry_count": 0, 2024-03-17T23:49:33.351 INFO:teuthology.orchestra.run.smithi150.stdout: "dentry_pinned_count": 0, 2024-03-17T23:49:33.351 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 6459, 2024-03-17T23:49:33.351 INFO:teuthology.orchestra.run.smithi150.stdout: "inst": { 2024-03-17T23:49:33.351 INFO:teuthology.orchestra.run.smithi150.stdout: "name": { 2024-03-17T23:49:33.351 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "client", 2024-03-17T23:49:33.351 INFO:teuthology.orchestra.run.smithi150.stdout: "num": 6459 2024-03-17T23:49:33.351 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:33.351 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": { 2024-03-17T23:49:33.351 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v1", 2024-03-17T23:49:33.351 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "192.168.0.1:0", 2024-03-17T23:49:33.351 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1314074314 2024-03-17T23:49:33.351 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:49:33.351 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:33.352 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": { 2024-03-17T23:49:33.352 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v1", 2024-03-17T23:49:33.352 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "192.168.0.1:0", 2024-03-17T23:49:33.352 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1314074314 2024-03-17T23:49:33.352 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:33.352 INFO:teuthology.orchestra.run.smithi150.stdout: "inst_str": "client.6459 192.168.0.1:0/1314074314", 2024-03-17T23:49:33.352 INFO:teuthology.orchestra.run.smithi150.stdout: "addr_str": "192.168.0.1:0/1314074314", 2024-03-17T23:49:33.352 INFO:teuthology.orchestra.run.smithi150.stdout: "inode_count": 1, 2024-03-17T23:49:33.352 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_epoch": 22, 2024-03-17T23:49:33.352 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_epoch": 58, 2024-03-17T23:49:33.352 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_epoch_barrier": 0, 2024-03-17T23:49:33.352 INFO:teuthology.orchestra.run.smithi150.stdout: "blocklisted": false, 2024-03-17T23:49:33.352 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "cephfs" 2024-03-17T23:49:33.352 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-17T23:49:33.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:33.358 DEBUG:teuthology.orchestra.run.smithi150:> stat --file-system '--printf=%T 2024-03-17T23:49:33.358 DEBUG:teuthology.orchestra.run.smithi150:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-17T23:49:33.373 INFO:teuthology.orchestra.run.smithi150.stdout:fuseblk 2024-03-17T23:49:33.373 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-17T23:49:33.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:33.374 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-17T23:49:33.440 DEBUG:teuthology.orchestra.run.smithi150:> stdin-killer -- python3 -c ' 2024-03-17T23:49:33.441 DEBUG:teuthology.orchestra.run.smithi150:> import os 2024-03-17T23:49:33.441 DEBUG:teuthology.orchestra.run.smithi150:> import errno 2024-03-17T23:49:33.441 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:49:33.441 DEBUG:teuthology.orchestra.run.smithi150:> fd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR) 2024-03-17T23:49:33.441 DEBUG:teuthology.orchestra.run.smithi150:> try: 2024-03-17T23:49:33.441 DEBUG:teuthology.orchestra.run.smithi150:> if False: 2024-03-17T23:49:33.441 DEBUG:teuthology.orchestra.run.smithi150:> ret = os.read(fd, 1024) 2024-03-17T23:49:33.441 DEBUG:teuthology.orchestra.run.smithi150:> else: 2024-03-17T23:49:33.441 DEBUG:teuthology.orchestra.run.smithi150:> os.write(fd, b'"'"'content'"'"') 2024-03-17T23:49:33.441 DEBUG:teuthology.orchestra.run.smithi150:> except OSError as e: 2024-03-17T23:49:33.441 DEBUG:teuthology.orchestra.run.smithi150:> if e.errno != errno.EPERM: 2024-03-17T23:49:33.441 DEBUG:teuthology.orchestra.run.smithi150:> raise 2024-03-17T23:49:33.441 DEBUG:teuthology.orchestra.run.smithi150:> else: 2024-03-17T23:49:33.442 DEBUG:teuthology.orchestra.run.smithi150:> raise RuntimeError("client does not check permission of data pool") 2024-03-17T23:49:33.442 DEBUG:teuthology.orchestra.run.smithi150:> ' 2024-03-17T23:49:33.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:49:33.557 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow w pool=cephfs_data' 2024-03-17T23:49:33.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.703+0000 7f389c8f1640 1 Processor -- start 2024-03-17T23:49:33.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.704+0000 7f389c8f1640 1 -- start start 2024-03-17T23:49:33.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.704+0000 7f389c8f1640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3898145660 0x7f3898071c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:33.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.704+0000 7f389c8f1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38980724b0 con 0x7f38980cef10 2024-03-17T23:49:33.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.704+0000 7f389c8f1640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3898072790 con 0x7f389806bae0 2024-03-17T23:49:33.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.704+0000 7f389c8f1640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3898072a70 con 0x7f3898145660 2024-03-17T23:49:33.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.705+0000 7f3895d9b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f38980cef10 0x7f389806b3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:42336/0 (socket says 172.21.15.150:42336) 2024-03-17T23:49:33.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.705+0000 7f3895d9b640 1 -- 172.21.15.150:0/969153863 learned_addr learned my addr 172.21.15.150:0/969153863 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:33.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.705+0000 7f389659c640 1 --2- 172.21.15.150:0/969153863 >> v2:172.21.15.157:3300/0 conn(0x7f3898145660 0x7f3898071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:33.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.705+0000 7f389559a640 1 -- 172.21.15.150:0/969153863 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2985348521 0 0) 0x7f38980724b0 con 0x7f38980cef10 2024-03-17T23:49:33.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.705+0000 7f389559a640 1 -- 172.21.15.150:0/969153863 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3880003520 con 0x7f38980cef10 2024-03-17T23:49:33.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.705+0000 7f389559a640 1 -- 172.21.15.150:0/969153863 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2775689088 0 0) 0x7f3898072790 con 0x7f389806bae0 2024-03-17T23:49:33.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.705+0000 7f389559a640 1 -- 172.21.15.150:0/969153863 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38980724b0 con 0x7f389806bae0 2024-03-17T23:49:33.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.705+0000 7f389659c640 1 -- 172.21.15.150:0/969153863 >> v1:172.21.15.157:6789/0 conn(0x7f389806bae0 legacy=0x7f38981657f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:33.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.705+0000 7f389659c640 1 -- 172.21.15.150:0/969153863 >> v1:172.21.15.150:6789/0 conn(0x7f38980cef10 legacy=0x7f389806b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:33.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.705+0000 7f389659c640 1 -- 172.21.15.150:0/969153863 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38980b38c0 con 0x7f3898145660 2024-03-17T23:49:33.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.706+0000 7f389659c640 1 --2- 172.21.15.150:0/969153863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3898145660 0x7f3898071c60 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f388c0097a0 tx=0x7f388c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c73e33eaec8f9fcb server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:33.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.706+0000 7f389559a640 1 -- 172.21.15.150:0/969153863 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f388c035020 con 0x7f3898145660 2024-03-17T23:49:33.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.706+0000 7f389559a640 1 -- 172.21.15.150:0/969153863 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f388c031830 con 0x7f3898145660 2024-03-17T23:49:33.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.706+0000 7f389559a640 1 -- 172.21.15.150:0/969153863 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f388c031be0 con 0x7f3898145660 2024-03-17T23:49:33.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.706+0000 7f389c8f1640 1 -- 172.21.15.150:0/969153863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3898145660 msgr2=0x7f3898071c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:33.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.706+0000 7f389c8f1640 1 --2- 172.21.15.150:0/969153863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3898145660 0x7f3898071c60 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f388c0097a0 tx=0x7f388c02b8d0 comp rx=0 tx=0).stop 2024-03-17T23:49:33.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.707+0000 7f389c8f1640 1 -- 172.21.15.150:0/969153863 shutdown_connections 2024-03-17T23:49:33.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.707+0000 7f389c8f1640 1 --2- 172.21.15.150:0/969153863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3898145660 0x7f3898071c60 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:33.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.707+0000 7f389c8f1640 1 -- 172.21.15.150:0/969153863 >> 172.21.15.150:0/969153863 conn(0x7f38980b18c0 msgr2=0x7f38980b1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:33.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.707+0000 7f389c8f1640 1 -- 172.21.15.150:0/969153863 shutdown_connections 2024-03-17T23:49:33.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.707+0000 7f389c8f1640 1 -- 172.21.15.150:0/969153863 wait complete. 2024-03-17T23:49:33.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.708+0000 7f389c8f1640 1 Processor -- start 2024-03-17T23:49:33.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.708+0000 7f389c8f1640 1 -- start start 2024-03-17T23:49:33.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.709+0000 7f389c8f1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f389806bae0 0x7f3898158c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:33.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.709+0000 7f389c8f1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38980cef10 0x7f3898157a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:33.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.709+0000 7f389c8f1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3898145660 0x7f38981f70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:33.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.709+0000 7f389c8f1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f389815c1d0 con 0x7f3898145660 2024-03-17T23:49:33.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.709+0000 7f389c8f1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f38980b4df0 con 0x7f389806bae0 2024-03-17T23:49:33.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.709+0000 7f389c8f1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3898166f10 con 0x7f38980cef10 2024-03-17T23:49:33.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.709+0000 7f3896d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3898145660 0x7f38981f70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:33.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.709+0000 7f3896d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3898145660 0x7f38981f70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59056/0 (socket says 172.21.15.150:59056) 2024-03-17T23:49:33.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.709+0000 7f3896d9d640 1 -- 172.21.15.150:0/184268931 learned_addr learned my addr 172.21.15.150:0/184268931 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:33.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.709+0000 7f389659c640 1 --2- 172.21.15.150:0/184268931 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f389806bae0 0x7f3898158c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:33.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.709+0000 7f3895d9b640 1 --2- 172.21.15.150:0/184268931 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38980cef10 0x7f3898157a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:33.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.709+0000 7f3896d9d640 1 -- 172.21.15.150:0/184268931 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38980cef10 msgr2=0x7f3898157a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:33.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.709+0000 7f3896d9d640 1 --2- 172.21.15.150:0/184268931 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38980cef10 0x7f3898157a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:33.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.709+0000 7f3896d9d640 1 -- 172.21.15.150:0/184268931 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f389806bae0 msgr2=0x7f3898158c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:33.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.709+0000 7f3896d9d640 1 --2- 172.21.15.150:0/184268931 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f389806bae0 0x7f3898158c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:33.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.709+0000 7f3896d9d640 1 -- 172.21.15.150:0/184268931 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3898155b80 con 0x7f3898145660 2024-03-17T23:49:33.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.709+0000 7f3896d9d640 1 --2- 172.21.15.150:0/184268931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3898145660 0x7f38981f70a0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f389000c7a0 tx=0x7f389000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:33.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.709+0000 7f3886ffd640 1 -- 172.21.15.150:0/184268931 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3890003b70 con 0x7f3898145660 2024-03-17T23:49:33.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.709+0000 7f3886ffd640 1 -- 172.21.15.150:0/184268931 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3890003d10 con 0x7f3898145660 2024-03-17T23:49:33.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.709+0000 7f389c8f1640 1 -- 172.21.15.150:0/184268931 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3898155e60 con 0x7f3898145660 2024-03-17T23:49:33.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.709+0000 7f3886ffd640 1 -- 172.21.15.150:0/184268931 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389000aa90 con 0x7f3898145660 2024-03-17T23:49:33.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.709+0000 7f389c8f1640 1 -- 172.21.15.150:0/184268931 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38980c4950 con 0x7f3898145660 2024-03-17T23:49:33.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.710+0000 7f389c8f1640 1 -- 172.21.15.150:0/184268931 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38980681a0 con 0x7f3898145660 2024-03-17T23:49:33.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.711+0000 7f3886ffd640 1 -- 172.21.15.150:0/184268931 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f389000ac30 con 0x7f3898145660 2024-03-17T23:49:33.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.711+0000 7f3886ffd640 1 --2- 172.21.15.150:0/184268931 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f387806bbd0 0x7f387806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:33.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.711+0000 7f3886ffd640 1 -- 172.21.15.150:0/184268931 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f389008aa80 con 0x7f3898145660 2024-03-17T23:49:33.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.711+0000 7f389659c640 1 --2- 172.21.15.150:0/184268931 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f387806bbd0 0x7f387806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:33.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.712+0000 7f389659c640 1 --2- 172.21.15.150:0/184268931 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f387806bbd0 0x7f387806e090 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f388c036000 tx=0x7f388c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:33.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.716+0000 7f3886ffd640 1 -- 172.21.15.150:0/184268931 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3890058d90 con 0x7f3898145660 2024-03-17T23:49:33.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.952+0000 7f389c8f1640 1 -- 172.21.15.150:0/184268931 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow w pool=cephfs_data"]} v 0) -- 0x7f38980af7f0 con 0x7f3898145660 2024-03-17T23:49:33.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.953+0000 7f3886ffd640 1 -- 172.21.15.150:0/184268931 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow w pool=cephfs_data"]}]=0 updated caps for client.0 v19) ==== 183+0+140 (secure 0 0 0) 0x7f389005ce30 con 0x7f3898145660 2024-03-17T23:49:33.955 INFO:teuthology.orchestra.run.smithi150.stdout:[client.0] 2024-03-17T23:49:33.955 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQC3gPdlvlPJNRAAkUfetPJK4hZQG5rNiSDozw== 2024-03-17T23:49:33.955 INFO:teuthology.orchestra.run.smithi150.stdout: caps mds = "allow" 2024-03-17T23:49:33.955 INFO:teuthology.orchestra.run.smithi150.stdout: caps mon = "allow r" 2024-03-17T23:49:33.956 INFO:teuthology.orchestra.run.smithi150.stdout: caps osd = "allow w pool=cephfs_data" 2024-03-17T23:49:33.956 INFO:teuthology.orchestra.run.smithi150.stderr:updated caps for client.0 2024-03-17T23:49:33.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.956+0000 7f389c8f1640 1 -- 172.21.15.150:0/184268931 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f387806bbd0 msgr2=0x7f387806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:33.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.956+0000 7f389c8f1640 1 --2- 172.21.15.150:0/184268931 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f387806bbd0 0x7f387806e090 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f388c036000 tx=0x7f388c03c040 comp rx=0 tx=0).stop 2024-03-17T23:49:33.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.956+0000 7f389c8f1640 1 -- 172.21.15.150:0/184268931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3898145660 msgr2=0x7f38981f70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:33.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.956+0000 7f389c8f1640 1 --2- 172.21.15.150:0/184268931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3898145660 0x7f38981f70a0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f389000c7a0 tx=0x7f389000cc60 comp rx=0 tx=0).stop 2024-03-17T23:49:33.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.956+0000 7f389c8f1640 1 -- 172.21.15.150:0/184268931 shutdown_connections 2024-03-17T23:49:33.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.956+0000 7f389c8f1640 1 --2- 172.21.15.150:0/184268931 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3898145660 0x7f38981f70a0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:33.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.956+0000 7f389c8f1640 1 --2- 172.21.15.150:0/184268931 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f38980cef10 0x7f3898157a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:33.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.956+0000 7f389c8f1640 1 --2- 172.21.15.150:0/184268931 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f387806bbd0 0x7f387806e090 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:33.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.956+0000 7f389c8f1640 1 --2- 172.21.15.150:0/184268931 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f389806bae0 0x7f3898158c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:33.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.956+0000 7f389c8f1640 1 -- 172.21.15.150:0/184268931 >> 172.21.15.150:0/184268931 conn(0x7f38980b18c0 msgr2=0x7f389815c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:33.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.957+0000 7f389c8f1640 1 -- 172.21.15.150:0/184268931 shutdown_connections 2024-03-17T23:49:33.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:33.957+0000 7f389c8f1640 1 -- 172.21.15.150:0/184268931 wait complete. 2024-03-17T23:49:33.969 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:49:33.969 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:49:33.984 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:49:34.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.123+0000 7f22ba4c6640 1 Processor -- start 2024-03-17T23:49:34.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.123+0000 7f22ba4c6640 1 -- start start 2024-03-17T23:49:34.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.124+0000 7f22ba4c6640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f22b40b31c0 0x7f22b40b4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:34.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.124+0000 7f22ba4c6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22b40cc220 con 0x7f22b406ccb0 2024-03-17T23:49:34.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.124+0000 7f22ba4c6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22b40cc500 con 0x7f22b409a7d0 2024-03-17T23:49:34.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.124+0000 7f22ba4c6640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f22b409ac00 con 0x7f22b40b31c0 2024-03-17T23:49:34.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.124+0000 7f22b37fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f22b406ccb0 0x7f22b409a0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:42344/0 (socket says 172.21.15.150:42344) 2024-03-17T23:49:34.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.124+0000 7f22b37fe640 1 -- 172.21.15.150:0/3460823051 learned_addr learned my addr 172.21.15.150:0/3460823051 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:34.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.124+0000 7f22b3fff640 1 --2- 172.21.15.150:0/3460823051 >> v2:172.21.15.157:3300/0 conn(0x7f22b40b31c0 0x7f22b40b4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:34.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.125+0000 7f22b2ffd640 1 -- 172.21.15.150:0/3460823051 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2466439440 0 0) 0x7f22b40cc220 con 0x7f22b406ccb0 2024-03-17T23:49:34.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.125+0000 7f22b2ffd640 1 -- 172.21.15.150:0/3460823051 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f229c003520 con 0x7f22b406ccb0 2024-03-17T23:49:34.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.125+0000 7f22b2ffd640 1 -- 172.21.15.150:0/3460823051 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 632660733 0 0) 0x7f22b40cc500 con 0x7f22b409a7d0 2024-03-17T23:49:34.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.125+0000 7f22b2ffd640 1 -- 172.21.15.150:0/3460823051 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22b40cc220 con 0x7f22b409a7d0 2024-03-17T23:49:34.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.125+0000 7f22b3fff640 1 -- 172.21.15.150:0/3460823051 >> v1:172.21.15.157:6789/0 conn(0x7f22b409a7d0 legacy=0x7f22b415d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:34.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.125+0000 7f22b3fff640 1 -- 172.21.15.150:0/3460823051 >> v1:172.21.15.150:6789/0 conn(0x7f22b406ccb0 legacy=0x7f22b409a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:34.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.125+0000 7f22b3fff640 1 -- 172.21.15.150:0/3460823051 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22b40cc7e0 con 0x7f22b40b31c0 2024-03-17T23:49:34.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.126+0000 7f22b3fff640 1 --2- 172.21.15.150:0/3460823051 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22b40b31c0 0x7f22b40b4980 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f22a40097a0 tx=0x7f22a402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa14b862c45a6f9f server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:34.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.126+0000 7f22b2ffd640 1 -- 172.21.15.150:0/3460823051 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a4034070 con 0x7f22b40b31c0 2024-03-17T23:49:34.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.126+0000 7f22b2ffd640 1 -- 172.21.15.150:0/3460823051 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22a4030610 con 0x7f22b40b31c0 2024-03-17T23:49:34.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.126+0000 7f22b2ffd640 1 -- 172.21.15.150:0/3460823051 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a40309c0 con 0x7f22b40b31c0 2024-03-17T23:49:34.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.126+0000 7f22ba4c6640 1 -- 172.21.15.150:0/3460823051 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22b40b31c0 msgr2=0x7f22b40b4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:34.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.126+0000 7f22ba4c6640 1 --2- 172.21.15.150:0/3460823051 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22b40b31c0 0x7f22b40b4980 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f22a40097a0 tx=0x7f22a402b960 comp rx=0 tx=0).stop 2024-03-17T23:49:34.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.126+0000 7f22ba4c6640 1 -- 172.21.15.150:0/3460823051 shutdown_connections 2024-03-17T23:49:34.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.126+0000 7f22ba4c6640 1 --2- 172.21.15.150:0/3460823051 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22b40b31c0 0x7f22b40b4980 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.126+0000 7f22ba4c6640 1 -- 172.21.15.150:0/3460823051 >> 172.21.15.150:0/3460823051 conn(0x7f22b40b3620 msgr2=0x7f22b40b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:34.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.126+0000 7f22ba4c6640 1 -- 172.21.15.150:0/3460823051 shutdown_connections 2024-03-17T23:49:34.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.126+0000 7f22ba4c6640 1 -- 172.21.15.150:0/3460823051 wait complete. 2024-03-17T23:49:34.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.127+0000 7f22ba4c6640 1 Processor -- start 2024-03-17T23:49:34.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.128+0000 7f22ba4c6640 1 -- start start 2024-03-17T23:49:34.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.128+0000 7f22ba4c6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f22b406ccb0 0x7f22b4149360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:34.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.128+0000 7f22ba4c6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22b409a7d0 0x7f22b41f7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:34.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.128+0000 7f22ba4c6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22b40b31c0 0x7f22b415e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:34.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.128+0000 7f22ba4c6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f22b406d140 con 0x7f22b40b31c0 2024-03-17T23:49:34.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.128+0000 7f22ba4c6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f22b40ccda0 con 0x7f22b409a7d0 2024-03-17T23:49:34.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.128+0000 7f22ba4c6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f22b406d2c0 con 0x7f22b406ccb0 2024-03-17T23:49:34.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.128+0000 7f22b8a63640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22b40b31c0 0x7f22b415e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:34.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.128+0000 7f22b8a63640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22b40b31c0 0x7f22b415e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59062/0 (socket says 172.21.15.150:59062) 2024-03-17T23:49:34.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.128+0000 7f22b8a63640 1 -- 172.21.15.150:0/904450876 learned_addr learned my addr 172.21.15.150:0/904450876 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:34.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.128+0000 7f22b37fe640 1 --2- 172.21.15.150:0/904450876 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22b409a7d0 0x7f22b41f7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:34.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.128+0000 7f22b3fff640 1 --2- 172.21.15.150:0/904450876 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f22b406ccb0 0x7f22b4149360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:34.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.129+0000 7f22b8a63640 1 -- 172.21.15.150:0/904450876 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f22b406ccb0 msgr2=0x7f22b4149360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:34.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.129+0000 7f22b8a63640 1 --2- 172.21.15.150:0/904450876 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f22b406ccb0 0x7f22b4149360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.129+0000 7f22b8a63640 1 -- 172.21.15.150:0/904450876 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22b409a7d0 msgr2=0x7f22b41f7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:34.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.129+0000 7f22b8a63640 1 --2- 172.21.15.150:0/904450876 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22b409a7d0 0x7f22b41f7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.129+0000 7f22b8a63640 1 -- 172.21.15.150:0/904450876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22b415d500 con 0x7f22b40b31c0 2024-03-17T23:49:34.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.129+0000 7f22b8a63640 1 --2- 172.21.15.150:0/904450876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22b40b31c0 0x7f22b415e5a0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f22ac00c7c0 tx=0x7f22ac00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:34.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.129+0000 7f22b0ff9640 1 -- 172.21.15.150:0/904450876 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22ac003b70 con 0x7f22b40b31c0 2024-03-17T23:49:34.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.129+0000 7f22ba4c6640 1 -- 172.21.15.150:0/904450876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22b415d7e0 con 0x7f22b40b31c0 2024-03-17T23:49:34.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.129+0000 7f22b0ff9640 1 -- 172.21.15.150:0/904450876 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22ac003d10 con 0x7f22b40b31c0 2024-03-17T23:49:34.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.129+0000 7f22b0ff9640 1 -- 172.21.15.150:0/904450876 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22ac00aa50 con 0x7f22b40b31c0 2024-03-17T23:49:34.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.129+0000 7f22ba4c6640 1 -- 172.21.15.150:0/904450876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22b415dda0 con 0x7f22b40b31c0 2024-03-17T23:49:34.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.130+0000 7f22ba4c6640 1 -- 172.21.15.150:0/904450876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22b40688b0 con 0x7f22b40b31c0 2024-03-17T23:49:34.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.130+0000 7f22b0ff9640 1 -- 172.21.15.150:0/904450876 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f22ac00abf0 con 0x7f22b40b31c0 2024-03-17T23:49:34.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.131+0000 7f22b0ff9640 1 --2- 172.21.15.150:0/904450876 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f22a006bc00 0x7f22a006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:34.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.131+0000 7f22b0ff9640 1 -- 172.21.15.150:0/904450876 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f22ac08aa20 con 0x7f22b40b31c0 2024-03-17T23:49:34.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.131+0000 7f22b3fff640 1 --2- 172.21.15.150:0/904450876 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f22a006bc00 0x7f22a006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:34.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.131+0000 7f22b3fff640 1 --2- 172.21.15.150:0/904450876 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f22a006bc00 0x7f22a006e0c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f22a4036000 tx=0x7f22a403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:34.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.135+0000 7f22b0ff9640 1 -- 172.21.15.150:0/904450876 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f22ac058d40 con 0x7f22b40b31c0 2024-03-17T23:49:34.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.310+0000 7f22ba4c6640 1 -- 172.21.15.150:0/904450876 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f22b4070af0 con 0x7f22b40b31c0 2024-03-17T23:49:34.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.311+0000 7f22b0ff9640 1 -- 172.21.15.150:0/904450876 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) ==== 74+0+16974 (secure 0 0 0) 0x7f22ac05cde0 con 0x7f22b40b31c0 2024-03-17T23:49:34.313 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:34.313 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":58,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:49:12.646307+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:49:09.223630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-17T23:49:09.956422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:49:34.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.313+0000 7f22ba4c6640 1 -- 172.21.15.150:0/904450876 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f22a006bc00 msgr2=0x7f22a006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:34.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.313+0000 7f22ba4c6640 1 --2- 172.21.15.150:0/904450876 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f22a006bc00 0x7f22a006e0c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f22a4036000 tx=0x7f22a403b040 comp rx=0 tx=0).stop 2024-03-17T23:49:34.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.313+0000 7f22ba4c6640 1 -- 172.21.15.150:0/904450876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22b40b31c0 msgr2=0x7f22b415e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:34.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.313+0000 7f22ba4c6640 1 --2- 172.21.15.150:0/904450876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22b40b31c0 0x7f22b415e5a0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f22ac00c7c0 tx=0x7f22ac00cc80 comp rx=0 tx=0).stop 2024-03-17T23:49:34.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.313+0000 7f22ba4c6640 1 -- 172.21.15.150:0/904450876 shutdown_connections 2024-03-17T23:49:34.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.313+0000 7f22ba4c6640 1 --2- 172.21.15.150:0/904450876 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f22b40b31c0 0x7f22b415e5a0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.313+0000 7f22ba4c6640 1 --2- 172.21.15.150:0/904450876 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f22a006bc00 0x7f22a006e0c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.313+0000 7f22ba4c6640 1 --2- 172.21.15.150:0/904450876 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22b409a7d0 0x7f22b41f7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.313+0000 7f22ba4c6640 1 --2- 172.21.15.150:0/904450876 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f22b406ccb0 0x7f22b4149360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.313+0000 7f22ba4c6640 1 -- 172.21.15.150:0/904450876 >> 172.21.15.150:0/904450876 conn(0x7f22b40b3620 msgr2=0x7f22b4073b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:34.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.314+0000 7f22ba4c6640 1 -- 172.21.15.150:0/904450876 shutdown_connections 2024-03-17T23:49:34.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.314+0000 7f22ba4c6640 1 -- 172.21.15.150:0/904450876 wait complete. 2024-03-17T23:49:34.326 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:49:09.223630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:49:09.956422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-17T23:49:34.327 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:49:34.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.429+0000 7f1a4f31b640 1 Processor -- start 2024-03-17T23:49:34.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.429+0000 7f1a4f31b640 1 -- start start 2024-03-17T23:49:34.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.430+0000 7f1a4f31b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a4806ecd0 0x7f1a4815bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:34.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.430+0000 7f1a4f31b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a480ce790 con 0x7f1a481514f0 2024-03-17T23:49:34.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.430+0000 7f1a4f31b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a4815b050 con 0x7f1a4815ab90 2024-03-17T23:49:34.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.430+0000 7f1a4f31b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1a4815b330 con 0x7f1a4806ecd0 2024-03-17T23:49:34.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.430+0000 7f1a4d0b7640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f1a481514f0 0x7f1a4806e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:42356/0 (socket says 172.21.15.150:42356) 2024-03-17T23:49:34.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.430+0000 7f1a4d0b7640 1 -- 172.21.15.150:0/3923999750 learned_addr learned my addr 172.21.15.150:0/3923999750 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:34.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.430+0000 7f1a4d8b8640 1 --2- 172.21.15.150:0/3923999750 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a4806ecd0 0x7f1a4815bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:34.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.430+0000 7f1a3ffff640 1 -- 172.21.15.150:0/3923999750 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1455629445 0 0) 0x7f1a44001110 con 0x7f1a481514f0 2024-03-17T23:49:34.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.430+0000 7f1a3ffff640 1 -- 172.21.15.150:0/3923999750 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a34003440 con 0x7f1a481514f0 2024-03-17T23:49:34.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.430+0000 7f1a3ffff640 1 -- 172.21.15.150:0/3923999750 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1844937722 0 0) 0x7f1a4815b050 con 0x7f1a4815ab90 2024-03-17T23:49:34.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.430+0000 7f1a3ffff640 1 -- 172.21.15.150:0/3923999750 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a44001110 con 0x7f1a4815ab90 2024-03-17T23:49:34.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.430+0000 7f1a4d8b8640 1 -- 172.21.15.150:0/3923999750 >> v1:172.21.15.157:6789/0 conn(0x7f1a4815ab90 legacy=0x7f1a480cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:34.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.430+0000 7f1a4d8b8640 1 -- 172.21.15.150:0/3923999750 >> v1:172.21.15.150:6789/0 conn(0x7f1a481514f0 legacy=0x7f1a4806e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:34.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.430+0000 7f1a4d8b8640 1 -- 172.21.15.150:0/3923999750 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a4815b610 con 0x7f1a4806ecd0 2024-03-17T23:49:34.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.431+0000 7f1a4d8b8640 1 --2- 172.21.15.150:0/3923999750 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a4806ecd0 0x7f1a4815bd10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1a40009dc0 tx=0x7f1a4002bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9110e00191bcd34 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:34.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.431+0000 7f1a3ffff640 1 -- 172.21.15.150:0/3923999750 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a40039070 con 0x7f1a4806ecd0 2024-03-17T23:49:34.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.431+0000 7f1a3ffff640 1 -- 172.21.15.150:0/3923999750 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a4002ccd0 con 0x7f1a4806ecd0 2024-03-17T23:49:34.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.431+0000 7f1a3ffff640 1 -- 172.21.15.150:0/3923999750 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a40034660 con 0x7f1a4806ecd0 2024-03-17T23:49:34.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.432+0000 7f1a4f31b640 1 -- 172.21.15.150:0/3923999750 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a4806ecd0 msgr2=0x7f1a4815bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:34.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.432+0000 7f1a4f31b640 1 --2- 172.21.15.150:0/3923999750 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a4806ecd0 0x7f1a4815bd10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1a40009dc0 tx=0x7f1a4002bf20 comp rx=0 tx=0).stop 2024-03-17T23:49:34.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.432+0000 7f1a4f31b640 1 -- 172.21.15.150:0/3923999750 shutdown_connections 2024-03-17T23:49:34.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.432+0000 7f1a4f31b640 1 --2- 172.21.15.150:0/3923999750 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a4806ecd0 0x7f1a4815bd10 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.432+0000 7f1a4f31b640 1 -- 172.21.15.150:0/3923999750 >> 172.21.15.150:0/3923999750 conn(0x7f1a4806b190 msgr2=0x7f1a4806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:34.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.432+0000 7f1a4f31b640 1 -- 172.21.15.150:0/3923999750 shutdown_connections 2024-03-17T23:49:34.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.432+0000 7f1a4f31b640 1 -- 172.21.15.150:0/3923999750 wait complete. 2024-03-17T23:49:34.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.433+0000 7f1a4f31b640 1 Processor -- start 2024-03-17T23:49:34.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.433+0000 7f1a4f31b640 1 -- start start 2024-03-17T23:49:34.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.433+0000 7f1a4f31b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a4806ecd0 0x7f1a480b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:34.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.433+0000 7f1a4f31b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a481514f0 0x7f1a4815c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:34.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.433+0000 7f1a4f31b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a4815ab90 0x7f1a48147e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:34.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.433+0000 7f1a4f31b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1a481518d0 con 0x7f1a4806ecd0 2024-03-17T23:49:34.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.434+0000 7f1a4f31b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1a480af510 con 0x7f1a4815ab90 2024-03-17T23:49:34.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.434+0000 7f1a4f31b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1a48151a50 con 0x7f1a481514f0 2024-03-17T23:49:34.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.434+0000 7f1a4d0b7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a4806ecd0 0x7f1a480b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:34.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.434+0000 7f1a4d0b7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a4806ecd0 0x7f1a480b3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59074/0 (socket says 172.21.15.150:59074) 2024-03-17T23:49:34.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.434+0000 7f1a4d0b7640 1 -- 172.21.15.150:0/1750716635 learned_addr learned my addr 172.21.15.150:0/1750716635 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:34.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.434+0000 7f1a4d8b8640 1 --2- 172.21.15.150:0/1750716635 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a4815ab90 0x7f1a48147e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:34.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.434+0000 7f1a4c8b6640 1 --2- 172.21.15.150:0/1750716635 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a481514f0 0x7f1a4815c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:34.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.434+0000 7f1a4d0b7640 1 -- 172.21.15.150:0/1750716635 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a481514f0 msgr2=0x7f1a4815c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:34.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.434+0000 7f1a4d0b7640 1 --2- 172.21.15.150:0/1750716635 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a481514f0 0x7f1a4815c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.434+0000 7f1a4d0b7640 1 -- 172.21.15.150:0/1750716635 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a4815ab90 msgr2=0x7f1a48147e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:34.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.434+0000 7f1a4d0b7640 1 --2- 172.21.15.150:0/1750716635 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a4815ab90 0x7f1a48147e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.434+0000 7f1a4d0b7640 1 -- 172.21.15.150:0/1750716635 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a481effa0 con 0x7f1a4806ecd0 2024-03-17T23:49:34.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.434+0000 7f1a4d0b7640 1 --2- 172.21.15.150:0/1750716635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a4806ecd0 0x7f1a480b3ee0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f1a4400c680 tx=0x7f1a4400cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:34.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.434+0000 7f1a3dffb640 1 -- 172.21.15.150:0/1750716635 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a44017070 con 0x7f1a4806ecd0 2024-03-17T23:49:34.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.434+0000 7f1a3dffb640 1 -- 172.21.15.150:0/1750716635 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a44005ab0 con 0x7f1a4806ecd0 2024-03-17T23:49:34.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.434+0000 7f1a4f31b640 1 -- 172.21.15.150:0/1750716635 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a481f0280 con 0x7f1a4806ecd0 2024-03-17T23:49:34.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.434+0000 7f1a3dffb640 1 -- 172.21.15.150:0/1750716635 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a44007070 con 0x7f1a4806ecd0 2024-03-17T23:49:34.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.434+0000 7f1a4f31b640 1 -- 172.21.15.150:0/1750716635 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a48154870 con 0x7f1a4806ecd0 2024-03-17T23:49:34.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.436+0000 7f1a4f31b640 1 -- 172.21.15.150:0/1750716635 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a481f0670 con 0x7f1a4806ecd0 2024-03-17T23:49:34.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.436+0000 7f1a3dffb640 1 -- 172.21.15.150:0/1750716635 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1a4401a070 con 0x7f1a4806ecd0 2024-03-17T23:49:34.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.436+0000 7f1a3dffb640 1 --2- 172.21.15.150:0/1750716635 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1a2c06bbd0 0x7f1a2c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:34.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.436+0000 7f1a3dffb640 1 -- 172.21.15.150:0/1750716635 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f1a4408a150 con 0x7f1a4806ecd0 2024-03-17T23:49:34.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.436+0000 7f1a4c8b6640 1 --2- 172.21.15.150:0/1750716635 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1a2c06bbd0 0x7f1a2c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:34.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.437+0000 7f1a4c8b6640 1 --2- 172.21.15.150:0/1750716635 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1a2c06bbd0 0x7f1a2c06e090 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1a38007fd0 tx=0x7f1a38009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:34.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.441+0000 7f1a3dffb640 1 -- 172.21.15.150:0/1750716635 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1a480ce790 con 0x7f1a4806ecd0 2024-03-17T23:49:34.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.613+0000 7f1a4f31b640 1 -- 172.21.15.150:0/1750716635 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1a480681a0 con 0x7f1a4806ecd0 2024-03-17T23:49:34.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.613+0000 7f1a3dffb640 1 -- 172.21.15.150:0/1750716635 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) ==== 93+0+10 (secure 0 0 0) 0x7f1a44058460 con 0x7f1a4806ecd0 2024-03-17T23:49:34.615 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:49:34.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.615+0000 7f1a4f31b640 1 -- 172.21.15.150:0/1750716635 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1a2c06bbd0 msgr2=0x7f1a2c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:34.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.615+0000 7f1a4f31b640 1 --2- 172.21.15.150:0/1750716635 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1a2c06bbd0 0x7f1a2c06e090 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1a38007fd0 tx=0x7f1a38009040 comp rx=0 tx=0).stop 2024-03-17T23:49:34.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.615+0000 7f1a4f31b640 1 -- 172.21.15.150:0/1750716635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a4806ecd0 msgr2=0x7f1a480b3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:34.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.615+0000 7f1a4f31b640 1 --2- 172.21.15.150:0/1750716635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a4806ecd0 0x7f1a480b3ee0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f1a4400c680 tx=0x7f1a4400cb40 comp rx=0 tx=0).stop 2024-03-17T23:49:34.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.615+0000 7f1a4f31b640 1 -- 172.21.15.150:0/1750716635 shutdown_connections 2024-03-17T23:49:34.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.615+0000 7f1a4f31b640 1 --2- 172.21.15.150:0/1750716635 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1a2c06bbd0 0x7f1a2c06e090 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.615+0000 7f1a4f31b640 1 --2- 172.21.15.150:0/1750716635 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a4815ab90 0x7f1a48147e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.615+0000 7f1a4f31b640 1 --2- 172.21.15.150:0/1750716635 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a481514f0 0x7f1a4815c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.615+0000 7f1a4f31b640 1 --2- 172.21.15.150:0/1750716635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a4806ecd0 0x7f1a480b3ee0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.615+0000 7f1a4f31b640 1 -- 172.21.15.150:0/1750716635 >> 172.21.15.150:0/1750716635 conn(0x7f1a4806b190 msgr2=0x7f1a480aebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:34.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.615+0000 7f1a4f31b640 1 -- 172.21.15.150:0/1750716635 shutdown_connections 2024-03-17T23:49:34.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.616+0000 7f1a4f31b640 1 -- 172.21.15.150:0/1750716635 wait complete. 2024-03-17T23:49:34.628 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:49:34.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.732+0000 7f47a8b05640 1 Processor -- start 2024-03-17T23:49:34.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.732+0000 7f47a8b05640 1 -- start start 2024-03-17T23:49:34.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.732+0000 7f47a8b05640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47a409a470 0x7f47a409a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:34.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.732+0000 7f47a8b05640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f47a409ad90 0x7f47a415f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:34.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.732+0000 7f47a8b05640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47a40cfad0 con 0x7f47a415fcd0 2024-03-17T23:49:34.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.732+0000 7f47a8b05640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f47a40cfdb0 con 0x7f47a409ad90 2024-03-17T23:49:34.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.732+0000 7f47a8b05640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f47a40d0090 con 0x7f47a409a470 2024-03-17T23:49:34.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.733+0000 7f479bfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f47a409ad90 0x7f47a415f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:34.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.733+0000 7f479bfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f47a409ad90 0x7f47a415f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59088/0 (socket says 172.21.15.150:59088) 2024-03-17T23:49:34.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.733+0000 7f479bfff640 1 -- 172.21.15.150:0/340234373 learned_addr learned my addr 172.21.15.150:0/340234373 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:34.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.733+0000 7f47a259c640 1 --2- 172.21.15.150:0/340234373 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47a409a470 0x7f47a409a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:34.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.733+0000 7f479bfff640 1 -- 172.21.15.150:0/340234373 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47a409a470 msgr2=0x7f47a409a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:34.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.733+0000 7f479bfff640 1 --2- 172.21.15.150:0/340234373 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47a409a470 0x7f47a409a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.733+0000 7f479bfff640 1 -- 172.21.15.150:0/340234373 >> v1:172.21.15.157:6789/0 conn(0x7f47a415fcd0 legacy=0x7f47a415c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:34.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.733+0000 7f479bfff640 1 -- 172.21.15.150:0/340234373 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47a40d0370 con 0x7f47a409ad90 2024-03-17T23:49:34.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.733+0000 7f47a1d9b640 1 -- 172.21.15.150:0/340234373 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 835098189 0 0) 0x7f47a40cfad0 con 0x7f47a415fcd0 2024-03-17T23:49:34.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.734+0000 7f479bfff640 1 --2- 172.21.15.150:0/340234373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a409ad90 0x7f47a415f790 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f478c009870 tx=0x7f478c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc4b70d88890366a server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:34.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.734+0000 7f47a1d9b640 1 -- 172.21.15.150:0/340234373 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c034070 con 0x7f47a409ad90 2024-03-17T23:49:34.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.734+0000 7f47a1d9b640 1 -- 172.21.15.150:0/340234373 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f478c030840 con 0x7f47a409ad90 2024-03-17T23:49:34.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.734+0000 7f47a1d9b640 1 -- 172.21.15.150:0/340234373 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c030c10 con 0x7f47a409ad90 2024-03-17T23:49:34.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.734+0000 7f47a8b05640 1 -- 172.21.15.150:0/340234373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a409ad90 msgr2=0x7f47a415f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:34.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.734+0000 7f47a8b05640 1 --2- 172.21.15.150:0/340234373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a409ad90 0x7f47a415f790 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f478c009870 tx=0x7f478c02b980 comp rx=0 tx=0).stop 2024-03-17T23:49:34.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.734+0000 7f47a8b05640 1 -- 172.21.15.150:0/340234373 shutdown_connections 2024-03-17T23:49:34.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.734+0000 7f47a8b05640 1 --2- 172.21.15.150:0/340234373 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a409ad90 0x7f47a415f790 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.734+0000 7f47a8b05640 1 --2- 172.21.15.150:0/340234373 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47a409a470 0x7f47a409a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.734+0000 7f47a8b05640 1 -- 172.21.15.150:0/340234373 >> 172.21.15.150:0/340234373 conn(0x7f47a40b3630 msgr2=0x7f47a40b3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:34.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.734+0000 7f47a8b05640 1 -- 172.21.15.150:0/340234373 shutdown_connections 2024-03-17T23:49:34.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.735+0000 7f47a8b05640 1 -- 172.21.15.150:0/340234373 wait complete. 2024-03-17T23:49:34.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.736+0000 7f47a8b05640 1 Processor -- start 2024-03-17T23:49:34.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.736+0000 7f47a8b05640 1 -- start start 2024-03-17T23:49:34.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.736+0000 7f47a8b05640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a409a470 0x7f47a414bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:34.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.736+0000 7f47a8b05640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47a409ad90 0x7f47a40c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:34.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.737+0000 7f47a8b05640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47a415fcd0 0x7f47a40c7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:34.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.737+0000 7f47a8b05640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f47a40cf890 con 0x7f47a409a470 2024-03-17T23:49:34.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.737+0000 7f47a8b05640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f47a40cf710 con 0x7f47a415fcd0 2024-03-17T23:49:34.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.737+0000 7f47a8b05640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f47a40cd5f0 con 0x7f47a409ad90 2024-03-17T23:49:34.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.737+0000 7f47a259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a409a470 0x7f47a414bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:34.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.737+0000 7f47a259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a409a470 0x7f47a414bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59094/0 (socket says 172.21.15.150:59094) 2024-03-17T23:49:34.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.737+0000 7f47a259c640 1 -- 172.21.15.150:0/1499824658 learned_addr learned my addr 172.21.15.150:0/1499824658 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:34.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.737+0000 7f479bfff640 1 --2- 172.21.15.150:0/1499824658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47a409ad90 0x7f47a40c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:34.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.737+0000 7f47a2d9d640 1 --2- 172.21.15.150:0/1499824658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47a415fcd0 0x7f47a40c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:34.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.737+0000 7f47a259c640 1 -- 172.21.15.150:0/1499824658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47a409ad90 msgr2=0x7f47a40c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:34.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.737+0000 7f47a259c640 1 --2- 172.21.15.150:0/1499824658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47a409ad90 0x7f47a40c6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.737+0000 7f47a259c640 1 -- 172.21.15.150:0/1499824658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47a415fcd0 msgr2=0x7f47a40c7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:34.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.737+0000 7f47a259c640 1 --2- 172.21.15.150:0/1499824658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47a415fcd0 0x7f47a40c7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.737+0000 7f47a259c640 1 -- 172.21.15.150:0/1499824658 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47a40af640 con 0x7f47a409a470 2024-03-17T23:49:34.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.737+0000 7f47a259c640 1 --2- 172.21.15.150:0/1499824658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a409a470 0x7f47a414bef0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f479400e9b0 tx=0x7f479400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:34.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.737+0000 7f479affd640 1 -- 172.21.15.150:0/1499824658 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4794017070 con 0x7f47a409a470 2024-03-17T23:49:34.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.737+0000 7f479affd640 1 -- 172.21.15.150:0/1499824658 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f479400c9a0 con 0x7f47a409a470 2024-03-17T23:49:34.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.737+0000 7f479affd640 1 -- 172.21.15.150:0/1499824658 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f479400cd90 con 0x7f47a409a470 2024-03-17T23:49:34.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.737+0000 7f47a8b05640 1 -- 172.21.15.150:0/1499824658 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47a40af920 con 0x7f47a409a470 2024-03-17T23:49:34.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.737+0000 7f47a8b05640 1 -- 172.21.15.150:0/1499824658 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47a40afee0 con 0x7f47a409a470 2024-03-17T23:49:34.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.738+0000 7f47a8b05640 1 -- 172.21.15.150:0/1499824658 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47a40681a0 con 0x7f47a409a470 2024-03-17T23:49:34.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.739+0000 7f479affd640 1 -- 172.21.15.150:0/1499824658 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f479401a070 con 0x7f47a409a470 2024-03-17T23:49:34.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.739+0000 7f479affd640 1 --2- 172.21.15.150:0/1499824658 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f478406bbd0 0x7f478406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:34.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.739+0000 7f479affd640 1 -- 172.21.15.150:0/1499824658 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f479408aee0 con 0x7f47a409a470 2024-03-17T23:49:34.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.739+0000 7f479bfff640 1 --2- 172.21.15.150:0/1499824658 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f478406bbd0 0x7f478406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:34.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.740+0000 7f479bfff640 1 --2- 172.21.15.150:0/1499824658 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f478406bbd0 0x7f478406e090 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f478c036000 tx=0x7f478c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:34.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.744+0000 7f479affd640 1 -- 172.21.15.150:0/1499824658 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f47940591f0 con 0x7f47a409a470 2024-03-17T23:49:34.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.917+0000 7f47a8b05640 1 -- 172.21.15.150:0/1499824658 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f47a40d1120 con 0x7f47a409a470 2024-03-17T23:49:34.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.917+0000 7f479affd640 1 -- 172.21.15.150:0/1499824658 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) ==== 104+0+11 (secure 0 0 0) 0x7f479405f070 con 0x7f47a409a470 2024-03-17T23:49:34.919 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:49:34.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.919+0000 7f47a8b05640 1 -- 172.21.15.150:0/1499824658 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f478406bbd0 msgr2=0x7f478406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:34.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.919+0000 7f47a8b05640 1 --2- 172.21.15.150:0/1499824658 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f478406bbd0 0x7f478406e090 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f478c036000 tx=0x7f478c008030 comp rx=0 tx=0).stop 2024-03-17T23:49:34.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.919+0000 7f47a8b05640 1 -- 172.21.15.150:0/1499824658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a409a470 msgr2=0x7f47a414bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:34.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.919+0000 7f47a8b05640 1 --2- 172.21.15.150:0/1499824658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a409a470 0x7f47a414bef0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f479400e9b0 tx=0x7f479400ee70 comp rx=0 tx=0).stop 2024-03-17T23:49:34.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.919+0000 7f47a8b05640 1 -- 172.21.15.150:0/1499824658 shutdown_connections 2024-03-17T23:49:34.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.919+0000 7f47a8b05640 1 --2- 172.21.15.150:0/1499824658 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f478406bbd0 0x7f478406e090 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.919+0000 7f47a8b05640 1 --2- 172.21.15.150:0/1499824658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f47a415fcd0 0x7f47a40c7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.919+0000 7f47a8b05640 1 --2- 172.21.15.150:0/1499824658 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f47a409ad90 0x7f47a40c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.919+0000 7f47a8b05640 1 --2- 172.21.15.150:0/1499824658 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a409a470 0x7f47a414bef0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:34.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.920+0000 7f47a8b05640 1 -- 172.21.15.150:0/1499824658 >> 172.21.15.150:0/1499824658 conn(0x7f47a40b3630 msgr2=0x7f47a415f520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:34.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.920+0000 7f47a8b05640 1 -- 172.21.15.150:0/1499824658 shutdown_connections 2024-03-17T23:49:34.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:34.920+0000 7f47a8b05640 1 -- 172.21.15.150:0/1499824658 wait complete. 2024-03-17T23:49:34.933 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:49:35.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.035+0000 7f7894c0b640 1 Processor -- start 2024-03-17T23:49:35.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.035+0000 7f7894c0b640 1 -- start start 2024-03-17T23:49:35.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.036+0000 7f7894c0b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7890144bb0 0x7f789015ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:35.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.036+0000 7f7894c0b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f78900ce6a0 0x7f7890165a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:35.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.036+0000 7f7894c0b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7890145170 con 0x7f789015d390 2024-03-17T23:49:35.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.036+0000 7f7894c0b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f7890145450 con 0x7f78900ce6a0 2024-03-17T23:49:35.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.036+0000 7f7894c0b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7890145730 con 0x7f7890144bb0 2024-03-17T23:49:35.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.036+0000 7f788ed9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f78900ce6a0 0x7f7890165a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:35.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.036+0000 7f788dd9b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f789015d390 0x7f78900cdf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:42372/0 (socket says 172.21.15.150:42372) 2024-03-17T23:49:35.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.036+0000 7f788dd9b640 1 -- 172.21.15.150:0/807705597 learned_addr learned my addr 172.21.15.150:0/807705597 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:35.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.036+0000 7f788e59c640 1 --2- 172.21.15.150:0/807705597 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7890144bb0 0x7f789015ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:35.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.037+0000 7f788ed9d640 1 -- 172.21.15.150:0/807705597 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7890144bb0 msgr2=0x7f789015ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:35.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.037+0000 7f788ed9d640 1 --2- 172.21.15.150:0/807705597 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7890144bb0 0x7f789015ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.037+0000 7f788ed9d640 1 -- 172.21.15.150:0/807705597 >> v1:172.21.15.150:6789/0 conn(0x7f789015d390 legacy=0x7f78900cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:35.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.037+0000 7f788d59a640 1 -- 172.21.15.150:0/807705597 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2684867747 0 0) 0x7f7890145170 con 0x7f789015d390 2024-03-17T23:49:35.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.037+0000 7f788ed9d640 1 -- 172.21.15.150:0/807705597 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f789006bd60 con 0x7f78900ce6a0 2024-03-17T23:49:35.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.038+0000 7f788ed9d640 1 --2- 172.21.15.150:0/807705597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78900ce6a0 0x7f7890165a30 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f7888009d90 tx=0x7f788802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afd3443e6f00771c server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:35.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.038+0000 7f788d59a640 1 -- 172.21.15.150:0/807705597 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7888039070 con 0x7f78900ce6a0 2024-03-17T23:49:35.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.038+0000 7f788d59a640 1 -- 172.21.15.150:0/807705597 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f788802cbc0 con 0x7f78900ce6a0 2024-03-17T23:49:35.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.038+0000 7f788d59a640 1 -- 172.21.15.150:0/807705597 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7888034440 con 0x7f78900ce6a0 2024-03-17T23:49:35.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.038+0000 7f7894c0b640 1 -- 172.21.15.150:0/807705597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78900ce6a0 msgr2=0x7f7890165a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:35.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.038+0000 7f7894c0b640 1 --2- 172.21.15.150:0/807705597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78900ce6a0 0x7f7890165a30 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f7888009d90 tx=0x7f788802be10 comp rx=0 tx=0).stop 2024-03-17T23:49:35.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.038+0000 7f7894c0b640 1 -- 172.21.15.150:0/807705597 shutdown_connections 2024-03-17T23:49:35.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.038+0000 7f7894c0b640 1 --2- 172.21.15.150:0/807705597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78900ce6a0 0x7f7890165a30 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.038+0000 7f7894c0b640 1 --2- 172.21.15.150:0/807705597 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7890144bb0 0x7f789015ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.038+0000 7f7894c0b640 1 -- 172.21.15.150:0/807705597 >> 172.21.15.150:0/807705597 conn(0x7f789006b190 msgr2=0x7f789006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:35.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.038+0000 7f7894c0b640 1 -- 172.21.15.150:0/807705597 shutdown_connections 2024-03-17T23:49:35.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.038+0000 7f7894c0b640 1 -- 172.21.15.150:0/807705597 wait complete. 2024-03-17T23:49:35.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.040+0000 7f7894c0b640 1 Processor -- start 2024-03-17T23:49:35.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.040+0000 7f7894c0b640 1 -- start start 2024-03-17T23:49:35.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.040+0000 7f7894c0b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78900ce6a0 0x7f7890159c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:35.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.040+0000 7f7894c0b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7890144bb0 0x7f789015b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:35.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.040+0000 7f7894c0b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f789015d390 0x7f7890136200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:35.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.040+0000 7f7894c0b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7890165f70 con 0x7f789015d390 2024-03-17T23:49:35.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.040+0000 7f7894c0b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f789006f650 con 0x7f78900ce6a0 2024-03-17T23:49:35.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.040+0000 7f7894c0b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f78901660f0 con 0x7f7890144bb0 2024-03-17T23:49:35.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.040+0000 7f788ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f789015d390 0x7f7890136200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:35.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.040+0000 7f788ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f789015d390 0x7f7890136200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59116/0 (socket says 172.21.15.150:59116) 2024-03-17T23:49:35.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.041+0000 7f788ed9d640 1 -- 172.21.15.150:0/1053345897 learned_addr learned my addr 172.21.15.150:0/1053345897 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:35.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.041+0000 7f788dd9b640 1 --2- 172.21.15.150:0/1053345897 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7890144bb0 0x7f789015b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:35.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.041+0000 7f788e59c640 1 --2- 172.21.15.150:0/1053345897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78900ce6a0 0x7f7890159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:35.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.041+0000 7f788ed9d640 1 -- 172.21.15.150:0/1053345897 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7890144bb0 msgr2=0x7f789015b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:35.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.041+0000 7f788ed9d640 1 --2- 172.21.15.150:0/1053345897 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7890144bb0 0x7f789015b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.041+0000 7f788ed9d640 1 -- 172.21.15.150:0/1053345897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78900ce6a0 msgr2=0x7f7890159c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:35.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.041+0000 7f788ed9d640 1 --2- 172.21.15.150:0/1053345897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78900ce6a0 0x7f7890159c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.041+0000 7f788ed9d640 1 -- 172.21.15.150:0/1053345897 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f789013bd50 con 0x7f789015d390 2024-03-17T23:49:35.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.041+0000 7f788dd9b640 1 --2- 172.21.15.150:0/1053345897 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7890144bb0 0x7f789015b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:35.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.041+0000 7f788e59c640 1 --2- 172.21.15.150:0/1053345897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78900ce6a0 0x7f7890159c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:35.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.041+0000 7f788ed9d640 1 --2- 172.21.15.150:0/1053345897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f789015d390 0x7f7890136200 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f7888009d90 tx=0x7f7888004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:35.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.041+0000 7f787effd640 1 -- 172.21.15.150:0/1053345897 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7888039050 con 0x7f789015d390 2024-03-17T23:49:35.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.041+0000 7f787effd640 1 -- 172.21.15.150:0/1053345897 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7888045070 con 0x7f789015d390 2024-03-17T23:49:35.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.041+0000 7f7894c0b640 1 -- 172.21.15.150:0/1053345897 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f789013bff0 con 0x7f789015d390 2024-03-17T23:49:35.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.041+0000 7f787effd640 1 -- 172.21.15.150:0/1053345897 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7888009450 con 0x7f789015d390 2024-03-17T23:49:35.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.041+0000 7f7894c0b640 1 -- 172.21.15.150:0/1053345897 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f789013c5b0 con 0x7f789015d390 2024-03-17T23:49:35.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.042+0000 7f7894c0b640 1 -- 172.21.15.150:0/1053345897 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78900681a0 con 0x7f789015d390 2024-03-17T23:49:35.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.043+0000 7f787effd640 1 -- 172.21.15.150:0/1053345897 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f788803e850 con 0x7f789015d390 2024-03-17T23:49:35.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.043+0000 7f787effd640 1 --2- 172.21.15.150:0/1053345897 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f787006bbd0 0x7f787006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:35.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.043+0000 7f787effd640 1 -- 172.21.15.150:0/1053345897 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f78880ad820 con 0x7f789015d390 2024-03-17T23:49:35.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.043+0000 7f788e59c640 1 --2- 172.21.15.150:0/1053345897 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f787006bbd0 0x7f787006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:35.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.043+0000 7f788e59c640 1 --2- 172.21.15.150:0/1053345897 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f787006bbd0 0x7f787006e090 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7884009f30 tx=0x7f7884009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:35.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.047+0000 7f787effd640 1 -- 172.21.15.150:0/1053345897 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f788807bc50 con 0x7f789015d390 2024-03-17T23:49:35.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.218+0000 7f7894c0b640 1 -- 172.21.15.150:0/1053345897 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7890138260 con 0x7f789015d390 2024-03-17T23:49:35.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.218+0000 7f787effd640 1 -- 172.21.15.150:0/1053345897 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) ==== 100+0+11 (secure 0 0 0) 0x7f788807fcf0 con 0x7f789015d390 2024-03-17T23:49:35.220 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:49:35.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.220+0000 7f7894c0b640 1 -- 172.21.15.150:0/1053345897 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f787006bbd0 msgr2=0x7f787006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:35.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.220+0000 7f7894c0b640 1 --2- 172.21.15.150:0/1053345897 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f787006bbd0 0x7f787006e090 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f7884009f30 tx=0x7f7884009190 comp rx=0 tx=0).stop 2024-03-17T23:49:35.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.220+0000 7f7894c0b640 1 -- 172.21.15.150:0/1053345897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f789015d390 msgr2=0x7f7890136200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:35.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.220+0000 7f7894c0b640 1 --2- 172.21.15.150:0/1053345897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f789015d390 0x7f7890136200 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f7888009d90 tx=0x7f7888004a40 comp rx=0 tx=0).stop 2024-03-17T23:49:35.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.220+0000 7f7894c0b640 1 -- 172.21.15.150:0/1053345897 shutdown_connections 2024-03-17T23:49:35.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.220+0000 7f7894c0b640 1 --2- 172.21.15.150:0/1053345897 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f789015d390 0x7f7890136200 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.220+0000 7f7894c0b640 1 --2- 172.21.15.150:0/1053345897 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7890144bb0 0x7f789015b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.220+0000 7f7894c0b640 1 --2- 172.21.15.150:0/1053345897 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f787006bbd0 0x7f787006e090 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.220+0000 7f7894c0b640 1 --2- 172.21.15.150:0/1053345897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78900ce6a0 0x7f7890159c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.220+0000 7f7894c0b640 1 -- 172.21.15.150:0/1053345897 >> 172.21.15.150:0/1053345897 conn(0x7f789006b190 msgr2=0x7f7890071cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:35.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.220+0000 7f7894c0b640 1 -- 172.21.15.150:0/1053345897 shutdown_connections 2024-03-17T23:49:35.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.220+0000 7f7894c0b640 1 -- 172.21.15.150:0/1053345897 wait complete. 2024-03-17T23:49:35.233 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-17T23:49:35.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.333+0000 7fdabf868640 1 Processor -- start 2024-03-17T23:49:35.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.333+0000 7fdabf868640 1 -- start start 2024-03-17T23:49:35.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.334+0000 7fdabf868640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdab80ccf90 0x7fdab80cf100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:35.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.334+0000 7fdabf868640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdab815bce0 0x7fdab80cd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:35.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.334+0000 7fdabf868640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdab80cd8b0 con 0x7fdab815e830 2024-03-17T23:49:35.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.334+0000 7fdabf868640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fdab806e420 con 0x7fdab80ccf90 2024-03-17T23:49:35.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.334+0000 7fdabf868640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdab80cf8a0 con 0x7fdab815bce0 2024-03-17T23:49:35.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.334+0000 7fdabde05640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fdab815e830 0x7fdab815ecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:42376/0 (socket says 172.21.15.150:42376) 2024-03-17T23:49:35.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.334+0000 7fdabde05640 1 -- 172.21.15.150:0/4195246238 learned_addr learned my addr 172.21.15.150:0/4195246238 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:35.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.334+0000 7fdabd604640 1 --2- 172.21.15.150:0/4195246238 >> v2:172.21.15.150:3300/0 conn(0x7fdab80ccf90 0x7fdab80cf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:35.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.334+0000 7fdabce03640 1 --2- 172.21.15.150:0/4195246238 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdab815bce0 0x7fdab80cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:35.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.334+0000 7fdaaffff640 1 -- 172.21.15.150:0/4195246238 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2987609131 0 0) 0x7fdab80cd8b0 con 0x7fdab815e830 2024-03-17T23:49:35.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.334+0000 7fdaaffff640 1 -- 172.21.15.150:0/4195246238 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdaa4003520 con 0x7fdab815e830 2024-03-17T23:49:35.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.334+0000 7fdabd604640 1 -- 172.21.15.150:0/4195246238 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdab815bce0 msgr2=0x7fdab80cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:35.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.334+0000 7fdabd604640 1 --2- 172.21.15.150:0/4195246238 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdab815bce0 0x7fdab80cd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.334+0000 7fdabd604640 1 -- 172.21.15.150:0/4195246238 >> v1:172.21.15.150:6789/0 conn(0x7fdab815e830 legacy=0x7fdab815ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:35.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.334+0000 7fdabd604640 1 -- 172.21.15.150:0/4195246238 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdab806de60 con 0x7fdab80ccf90 2024-03-17T23:49:35.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.334+0000 7fdabce03640 1 --2- 172.21.15.150:0/4195246238 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdab815bce0 0x7fdab80cd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:35.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.335+0000 7fdabd604640 1 --2- 172.21.15.150:0/4195246238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdab80ccf90 0x7fdab80cf100 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fdab40097a0 tx=0x7fdab402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f9bc27197885618 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:35.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.335+0000 7fdaaffff640 1 -- 172.21.15.150:0/4195246238 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab40315a0 con 0x7fdab80ccf90 2024-03-17T23:49:35.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.335+0000 7fdaaffff640 1 -- 172.21.15.150:0/4195246238 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdab4037070 con 0x7fdab80ccf90 2024-03-17T23:49:35.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.335+0000 7fdaaffff640 1 -- 172.21.15.150:0/4195246238 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab4031960 con 0x7fdab80ccf90 2024-03-17T23:49:35.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.336+0000 7fdabf868640 1 -- 172.21.15.150:0/4195246238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdab80ccf90 msgr2=0x7fdab80cf100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:35.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.336+0000 7fdabf868640 1 --2- 172.21.15.150:0/4195246238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdab80ccf90 0x7fdab80cf100 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fdab40097a0 tx=0x7fdab402b840 comp rx=0 tx=0).stop 2024-03-17T23:49:35.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.336+0000 7fdabf868640 1 -- 172.21.15.150:0/4195246238 shutdown_connections 2024-03-17T23:49:35.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.336+0000 7fdabf868640 1 --2- 172.21.15.150:0/4195246238 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdab815bce0 0x7fdab80cd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.336+0000 7fdabf868640 1 --2- 172.21.15.150:0/4195246238 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdab80ccf90 0x7fdab80cf100 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.336+0000 7fdabf868640 1 -- 172.21.15.150:0/4195246238 >> 172.21.15.150:0/4195246238 conn(0x7fdab806b190 msgr2=0x7fdab806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:35.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.336+0000 7fdabf868640 1 -- 172.21.15.150:0/4195246238 shutdown_connections 2024-03-17T23:49:35.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.336+0000 7fdabf868640 1 -- 172.21.15.150:0/4195246238 wait complete. 2024-03-17T23:49:35.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.337+0000 7fdabf868640 1 Processor -- start 2024-03-17T23:49:35.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.337+0000 7fdabf868640 1 -- start start 2024-03-17T23:49:35.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.338+0000 7fdabf868640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdab80ccf90 0x7fdab80c5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:35.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.338+0000 7fdabf868640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdab815bce0 0x7fdab8150960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:35.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.338+0000 7fdabf868640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdab815e830 0x7fdab806e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:35.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.338+0000 7fdabf868640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdab815c5a0 con 0x7fdab80ccf90 2024-03-17T23:49:35.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.338+0000 7fdabf868640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fdab80cf640 con 0x7fdab815bce0 2024-03-17T23:49:35.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.338+0000 7fdabf868640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fdab815c160 con 0x7fdab815e830 2024-03-17T23:49:35.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.338+0000 7fdabd604640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdab80ccf90 0x7fdab80c5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:35.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.338+0000 7fdabd604640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdab80ccf90 0x7fdab80c5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59132/0 (socket says 172.21.15.150:59132) 2024-03-17T23:49:35.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.338+0000 7fdabd604640 1 -- 172.21.15.150:0/37390311 learned_addr learned my addr 172.21.15.150:0/37390311 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:35.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.338+0000 7fdabce03640 1 --2- 172.21.15.150:0/37390311 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdab815bce0 0x7fdab8150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:35.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.338+0000 7fdabde05640 1 --2- 172.21.15.150:0/37390311 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdab815e830 0x7fdab806e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:35.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.338+0000 7fdabd604640 1 -- 172.21.15.150:0/37390311 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdab815e830 msgr2=0x7fdab806e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:35.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.338+0000 7fdabd604640 1 --2- 172.21.15.150:0/37390311 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdab815e830 0x7fdab806e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.338+0000 7fdabd604640 1 -- 172.21.15.150:0/37390311 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdab815bce0 msgr2=0x7fdab8150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:35.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.338+0000 7fdabd604640 1 --2- 172.21.15.150:0/37390311 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdab815bce0 0x7fdab8150960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.338+0000 7fdabd604640 1 -- 172.21.15.150:0/37390311 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdab80c63c0 con 0x7fdab80ccf90 2024-03-17T23:49:35.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.338+0000 7fdabd604640 1 --2- 172.21.15.150:0/37390311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdab80ccf90 0x7fdab80c5e80 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fdab40098c0 tx=0x7fdab40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:35.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.339+0000 7fdaadffb640 1 -- 172.21.15.150:0/37390311 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab4032630 con 0x7fdab80ccf90 2024-03-17T23:49:35.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.339+0000 7fdaadffb640 1 -- 172.21.15.150:0/37390311 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdab40327f0 con 0x7fdab80ccf90 2024-03-17T23:49:35.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.339+0000 7fdabf868640 1 -- 172.21.15.150:0/37390311 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdab8152350 con 0x7fdab80ccf90 2024-03-17T23:49:35.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.339+0000 7fdaadffb640 1 -- 172.21.15.150:0/37390311 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab4032be0 con 0x7fdab80ccf90 2024-03-17T23:49:35.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.339+0000 7fdabf868640 1 -- 172.21.15.150:0/37390311 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdab8151590 con 0x7fdab80ccf90 2024-03-17T23:49:35.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.340+0000 7fdabf868640 1 -- 172.21.15.150:0/37390311 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdab80681a0 con 0x7fdab80ccf90 2024-03-17T23:49:35.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.340+0000 7fdaadffb640 1 -- 172.21.15.150:0/37390311 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fdab404e430 con 0x7fdab80ccf90 2024-03-17T23:49:35.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.341+0000 7fdaadffb640 1 --2- 172.21.15.150:0/37390311 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fda9c06bbd0 0x7fda9c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:35.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.341+0000 7fdaadffb640 1 -- 172.21.15.150:0/37390311 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7fdab402f030 con 0x7fdab80ccf90 2024-03-17T23:49:35.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.341+0000 7fdabce03640 1 --2- 172.21.15.150:0/37390311 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fda9c06bbd0 0x7fda9c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:35.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.341+0000 7fdabce03640 1 --2- 172.21.15.150:0/37390311 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fda9c06bbd0 0x7fda9c06e090 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fdab81f5080 tx=0x7fdaa8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:35.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.345+0000 7fdaadffb640 1 -- 172.21.15.150:0/37390311 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fdab80c63c0 con 0x7fdab80ccf90 2024-03-17T23:49:35.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.528+0000 7fdabf868640 1 -- 172.21.15.150:0/37390311 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fdab80b4010 con 0x7fdab80ccf90 2024-03-17T23:49:35.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.529+0000 7fdaadffb640 1 -- 172.21.15.150:0/37390311 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v58) ==== 80+0+0 (secure 0 0 0) 0x7fdab4040070 con 0x7fdab80ccf90 2024-03-17T23:49:35.530 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-03-17T23:49:35.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.531+0000 7fdabf868640 1 -- 172.21.15.150:0/37390311 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fda9c06bbd0 msgr2=0x7fda9c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:35.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.531+0000 7fdabf868640 1 --2- 172.21.15.150:0/37390311 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fda9c06bbd0 0x7fda9c06e090 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fdab81f5080 tx=0x7fdaa8009190 comp rx=0 tx=0).stop 2024-03-17T23:49:35.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.531+0000 7fdabf868640 1 -- 172.21.15.150:0/37390311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdab80ccf90 msgr2=0x7fdab80c5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:35.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.531+0000 7fdabf868640 1 --2- 172.21.15.150:0/37390311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdab80ccf90 0x7fdab80c5e80 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fdab40098c0 tx=0x7fdab40046c0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.531+0000 7fdabf868640 1 -- 172.21.15.150:0/37390311 shutdown_connections 2024-03-17T23:49:35.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.531+0000 7fdabf868640 1 --2- 172.21.15.150:0/37390311 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fdab815e830 0x7fdab806e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.531+0000 7fdabf868640 1 --2- 172.21.15.150:0/37390311 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fda9c06bbd0 0x7fda9c06e090 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.531+0000 7fdabf868640 1 --2- 172.21.15.150:0/37390311 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdab815bce0 0x7fdab8150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.531+0000 7fdabf868640 1 --2- 172.21.15.150:0/37390311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdab80ccf90 0x7fdab80c5e80 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.531+0000 7fdabf868640 1 -- 172.21.15.150:0/37390311 >> 172.21.15.150:0/37390311 conn(0x7fdab806b190 msgr2=0x7fdab815c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:35.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.531+0000 7fdabf868640 1 -- 172.21.15.150:0/37390311 shutdown_connections 2024-03-17T23:49:35.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.531+0000 7fdabf868640 1 -- 172.21.15.150:0/37390311 wait complete. 2024-03-17T23:49:35.543 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:49:35.544 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:49:35.559 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:49:35.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.703+0000 7f04b3263640 1 Processor -- start 2024-03-17T23:49:35.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.703+0000 7f04b3263640 1 -- start start 2024-03-17T23:49:35.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.704+0000 7f04b3263640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f04ac15ac90 0x7f04ac06f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:35.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.704+0000 7f04b3263640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f04ac15b070 0x7f04ac0d0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:35.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.704+0000 7f04b3263640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04ac0cb4d0 con 0x7f04ac0d0bb0 2024-03-17T23:49:35.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.704+0000 7f04b3263640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f04ac0cf3c0 con 0x7f04ac15ac90 2024-03-17T23:49:35.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.704+0000 7f04b3263640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f04ac0cf6a0 con 0x7f04ac15b070 2024-03-17T23:49:35.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.704+0000 7f04b0fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f04ac15ac90 0x7f04ac06f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:35.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.704+0000 7f04b0fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f04ac15ac90 0x7f04ac06f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60876/0 (socket says 172.21.15.150:60876) 2024-03-17T23:49:35.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.704+0000 7f04b0fff640 1 -- 172.21.15.150:0/2955195294 learned_addr learned my addr 172.21.15.150:0/2955195294 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:35.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.704+0000 7f04abfff640 1 --2- 172.21.15.150:0/2955195294 >> v2:172.21.15.157:3300/0 conn(0x7f04ac15b070 0x7f04ac0d0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:35.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.704+0000 7f04b0fff640 1 -- 172.21.15.150:0/2955195294 >> v2:172.21.15.157:3300/0 conn(0x7f04ac15b070 msgr2=0x7f04ac0d0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:35.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.704+0000 7f04b0fff640 1 --2- 172.21.15.150:0/2955195294 >> v2:172.21.15.157:3300/0 conn(0x7f04ac15b070 0x7f04ac0d0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.704+0000 7f04b0fff640 1 -- 172.21.15.150:0/2955195294 >> v1:172.21.15.157:6789/0 conn(0x7f04ac0d0bb0 legacy=0x7f04ac0cbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:35.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.704+0000 7f04b0fff640 1 -- 172.21.15.150:0/2955195294 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ac0cb770 con 0x7f04ac15ac90 2024-03-17T23:49:35.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.704+0000 7f04ab7fe640 1 -- 172.21.15.150:0/2955195294 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3054764762 0 0) 0x7f04ac0cb4d0 con 0x7f04ac0d0bb0 2024-03-17T23:49:35.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.704+0000 7f04abfff640 1 --2- 172.21.15.150:0/2955195294 >> v2:172.21.15.157:3300/0 conn(0x7f04ac15b070 0x7f04ac0d0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:35.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.705+0000 7f04b0fff640 1 --2- 172.21.15.150:0/2955195294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04ac15ac90 0x7f04ac06f1e0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0ce550 tx=0x7f04a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e80381345c24b527 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:35.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.705+0000 7f04ab7fe640 1 -- 172.21.15.150:0/2955195294 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a4035020 con 0x7f04ac15ac90 2024-03-17T23:49:35.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.705+0000 7f04ab7fe640 1 -- 172.21.15.150:0/2955195294 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04a4031870 con 0x7f04ac15ac90 2024-03-17T23:49:35.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.705+0000 7f04ab7fe640 1 -- 172.21.15.150:0/2955195294 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a4031c40 con 0x7f04ac15ac90 2024-03-17T23:49:35.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.706+0000 7f04b3263640 1 -- 172.21.15.150:0/2955195294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04ac15ac90 msgr2=0x7f04ac06f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:35.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.706+0000 7f04b3263640 1 --2- 172.21.15.150:0/2955195294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04ac15ac90 0x7f04ac06f1e0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0ce550 tx=0x7f04a402b910 comp rx=0 tx=0).stop 2024-03-17T23:49:35.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.706+0000 7f04b3263640 1 -- 172.21.15.150:0/2955195294 shutdown_connections 2024-03-17T23:49:35.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.706+0000 7f04b3263640 1 --2- 172.21.15.150:0/2955195294 >> v2:172.21.15.157:3300/0 conn(0x7f04ac15b070 0x7f04ac0d0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.706+0000 7f04b3263640 1 --2- 172.21.15.150:0/2955195294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04ac15ac90 0x7f04ac06f1e0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.706+0000 7f04b3263640 1 -- 172.21.15.150:0/2955195294 >> 172.21.15.150:0/2955195294 conn(0x7f04ac099a70 msgr2=0x7f04ac099e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:35.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.706+0000 7f04b3263640 1 -- 172.21.15.150:0/2955195294 shutdown_connections 2024-03-17T23:49:35.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.706+0000 7f04b3263640 1 -- 172.21.15.150:0/2955195294 wait complete. 2024-03-17T23:49:35.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.707+0000 7f04b3263640 1 Processor -- start 2024-03-17T23:49:35.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.707+0000 7f04b3263640 1 -- start start 2024-03-17T23:49:35.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.708+0000 7f04b3263640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04ac0d0bb0 0x7f04ac149b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:35.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.708+0000 7f04b3263640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04ac15ac90 0x7f04ac167740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:35.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.708+0000 7f04b3263640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04ac15b070 0x7f04ac0aec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:35.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.708+0000 7f04b3263640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f04ac06f8a0 con 0x7f04ac15ac90 2024-03-17T23:49:35.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.708+0000 7f04b3263640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f04ac06f720 con 0x7f04ac15b070 2024-03-17T23:49:35.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.708+0000 7f04b3263640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f04ac0cc680 con 0x7f04ac0d0bb0 2024-03-17T23:49:35.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.708+0000 7f04b0fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04ac0d0bb0 0x7f04ac149b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:35.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.708+0000 7f04abfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04ac15ac90 0x7f04ac167740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:35.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.708+0000 7f04b0fff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04ac0d0bb0 0x7f04ac149b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:36476/0 (socket says 172.21.15.150:36476) 2024-03-17T23:49:35.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.708+0000 7f04b0fff640 1 -- 172.21.15.150:0/11381614 learned_addr learned my addr 172.21.15.150:0/11381614 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:35.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.708+0000 7f04b1800640 1 --2- 172.21.15.150:0/11381614 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04ac15b070 0x7f04ac0aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:35.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.708+0000 7f04abfff640 1 -- 172.21.15.150:0/11381614 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04ac0d0bb0 msgr2=0x7f04ac149b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:35.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.708+0000 7f04abfff640 1 --2- 172.21.15.150:0/11381614 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04ac0d0bb0 0x7f04ac149b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.708+0000 7f04abfff640 1 -- 172.21.15.150:0/11381614 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04ac15b070 msgr2=0x7f04ac0aec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:35.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.708+0000 7f04abfff640 1 --2- 172.21.15.150:0/11381614 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04ac15b070 0x7f04ac0aec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.708+0000 7f04abfff640 1 -- 172.21.15.150:0/11381614 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ac167eb0 con 0x7f04ac15ac90 2024-03-17T23:49:35.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.708+0000 7f04b0fff640 1 --2- 172.21.15.150:0/11381614 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04ac0d0bb0 0x7f04ac149b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:35.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.708+0000 7f04abfff640 1 --2- 172.21.15.150:0/11381614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04ac15ac90 0x7f04ac167740 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f04a000e870 tx=0x7f04a000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:35.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.709+0000 7f04a97fa640 1 -- 172.21.15.150:0/11381614 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a0017070 con 0x7f04ac15ac90 2024-03-17T23:49:35.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.709+0000 7f04a97fa640 1 -- 172.21.15.150:0/11381614 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04a000c940 con 0x7f04ac15ac90 2024-03-17T23:49:35.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.709+0000 7f04a97fa640 1 -- 172.21.15.150:0/11381614 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a000cd10 con 0x7f04ac15ac90 2024-03-17T23:49:35.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.709+0000 7f04b3263640 1 -- 172.21.15.150:0/11381614 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04ac168190 con 0x7f04ac15ac90 2024-03-17T23:49:35.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.709+0000 7f04b3263640 1 -- 172.21.15.150:0/11381614 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04ac168750 con 0x7f04ac15ac90 2024-03-17T23:49:35.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.710+0000 7f04b3263640 1 -- 172.21.15.150:0/11381614 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04ac151580 con 0x7f04ac15ac90 2024-03-17T23:49:35.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.710+0000 7f04a97fa640 1 -- 172.21.15.150:0/11381614 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f04a001a070 con 0x7f04ac15ac90 2024-03-17T23:49:35.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.710+0000 7f04a97fa640 1 --2- 172.21.15.150:0/11381614 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f049806bbd0 0x7f049806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:35.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.711+0000 7f04a97fa640 1 -- 172.21.15.150:0/11381614 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f04a008ac50 con 0x7f04ac15ac90 2024-03-17T23:49:35.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.711+0000 7f04b0fff640 1 --2- 172.21.15.150:0/11381614 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f049806bbd0 0x7f049806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:35.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.711+0000 7f04b0fff640 1 --2- 172.21.15.150:0/11381614 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f049806bbd0 0x7f049806e090 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f04a4009630 tx=0x7f04a403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:35.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.715+0000 7f04a97fa640 1 -- 172.21.15.150:0/11381614 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f04a0090020 con 0x7f04ac15ac90 2024-03-17T23:49:35.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.889+0000 7f04b3263640 1 -- 172.21.15.150:0/11381614 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f04ac0688b0 con 0x7f04ac15ac90 2024-03-17T23:49:35.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.890+0000 7f04a97fa640 1 -- 172.21.15.150:0/11381614 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) ==== 74+0+16974 (secure 0 0 0) 0x7f04a0058ff0 con 0x7f04ac15ac90 2024-03-17T23:49:35.891 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:35.892 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":58,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:49:12.646307+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:49:09.223630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-17T23:49:09.956422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:49:35.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.892+0000 7f04b3263640 1 -- 172.21.15.150:0/11381614 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f049806bbd0 msgr2=0x7f049806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:35.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.892+0000 7f04b3263640 1 --2- 172.21.15.150:0/11381614 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f049806bbd0 0x7f049806e090 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f04a4009630 tx=0x7f04a403c040 comp rx=0 tx=0).stop 2024-03-17T23:49:35.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.892+0000 7f04b3263640 1 -- 172.21.15.150:0/11381614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04ac15ac90 msgr2=0x7f04ac167740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:35.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.892+0000 7f04b3263640 1 --2- 172.21.15.150:0/11381614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04ac15ac90 0x7f04ac167740 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f04a000e870 tx=0x7f04a000ed30 comp rx=0 tx=0).stop 2024-03-17T23:49:35.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.892+0000 7f04b3263640 1 -- 172.21.15.150:0/11381614 shutdown_connections 2024-03-17T23:49:35.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.892+0000 7f04b3263640 1 --2- 172.21.15.150:0/11381614 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f049806bbd0 0x7f049806e090 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.892+0000 7f04b3263640 1 --2- 172.21.15.150:0/11381614 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f04ac15b070 0x7f04ac0aec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.892+0000 7f04b3263640 1 --2- 172.21.15.150:0/11381614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04ac15ac90 0x7f04ac167740 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.892+0000 7f04b3263640 1 --2- 172.21.15.150:0/11381614 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f04ac0d0bb0 0x7f04ac149b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:35.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.892+0000 7f04b3263640 1 -- 172.21.15.150:0/11381614 >> 172.21.15.150:0/11381614 conn(0x7f04ac099a70 msgr2=0x7f04ac0ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:35.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.892+0000 7f04b3263640 1 -- 172.21.15.150:0/11381614 shutdown_connections 2024-03-17T23:49:35.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:35.892+0000 7f04b3263640 1 -- 172.21.15.150:0/11381614 wait complete. 2024-03-17T23:49:35.905 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:49:09.223630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:49:09.956422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-17T23:49:35.905 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:49:36.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.008+0000 7fd4dcc4b640 1 Processor -- start 2024-03-17T23:49:36.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.008+0000 7fd4dcc4b640 1 -- start start 2024-03-17T23:49:36.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.009+0000 7fd4dcc4b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd4d815ab90 0x7fd4d815bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:36.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.009+0000 7fd4dcc4b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd4d806ecd0 0x7fd4d806f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:36.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.009+0000 7fd4dcc4b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4d80ce790 con 0x7fd4d81514f0 2024-03-17T23:49:36.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.009+0000 7fd4dcc4b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fd4d806fba0 con 0x7fd4d806ecd0 2024-03-17T23:49:36.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.009+0000 7fd4dcc4b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd4d815b050 con 0x7fd4d815ab90 2024-03-17T23:49:36.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.009+0000 7fd4d6d9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd4d806ecd0 0x7fd4d806f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:36.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.009+0000 7fd4cffff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd4d815ab90 0x7fd4d815bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:36.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.009+0000 7fd4d6d9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd4d806ecd0 0x7fd4d806f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60900/0 (socket says 172.21.15.150:60900) 2024-03-17T23:49:36.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.009+0000 7fd4d6d9d640 1 -- 172.21.15.150:0/303039626 learned_addr learned my addr 172.21.15.150:0/303039626 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:36.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.009+0000 7fd4d659c640 1 --1- 172.21.15.150:0/303039626 >> v1:172.21.15.157:6789/0 conn(0x7fd4d81514f0 0x7fd4d806e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:44490/0 (socket says 172.21.15.150:44490) 2024-03-17T23:49:36.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.009+0000 7fd4cffff640 1 -- 172.21.15.150:0/303039626 >> v2:172.21.15.150:3300/0 conn(0x7fd4d806ecd0 msgr2=0x7fd4d806f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:36.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.009+0000 7fd4cffff640 1 --2- 172.21.15.150:0/303039626 >> v2:172.21.15.150:3300/0 conn(0x7fd4d806ecd0 0x7fd4d806f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.009+0000 7fd4cffff640 1 -- 172.21.15.150:0/303039626 >> v1:172.21.15.157:6789/0 conn(0x7fd4d81514f0 legacy=0x7fd4d806e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:36.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.009+0000 7fd4cffff640 1 -- 172.21.15.150:0/303039626 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4d815b330 con 0x7fd4d815ab90 2024-03-17T23:49:36.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.009+0000 7fd4d5d9b640 1 -- 172.21.15.150:0/303039626 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3870984409 0 0) 0x7fd4c8001110 con 0x7fd4d81514f0 2024-03-17T23:49:36.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.010+0000 7fd4cffff640 1 --2- 172.21.15.150:0/303039626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4d815ab90 0x7fd4d815bd10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd4d806f2d0 tx=0x7fd4c002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85543d1e25a258fa server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:36.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.010+0000 7fd4d5d9b640 1 -- 172.21.15.150:0/303039626 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c0035020 con 0x7fd4d815ab90 2024-03-17T23:49:36.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.010+0000 7fd4d5d9b640 1 -- 172.21.15.150:0/303039626 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4c0031830 con 0x7fd4d815ab90 2024-03-17T23:49:36.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.010+0000 7fd4d5d9b640 1 -- 172.21.15.150:0/303039626 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c0031c00 con 0x7fd4d815ab90 2024-03-17T23:49:36.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.010+0000 7fd4dcc4b640 1 -- 172.21.15.150:0/303039626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4d815ab90 msgr2=0x7fd4d815bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:36.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.011+0000 7fd4dcc4b640 1 --2- 172.21.15.150:0/303039626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4d815ab90 0x7fd4d815bd10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd4d806f2d0 tx=0x7fd4c002b8d0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.011+0000 7fd4dcc4b640 1 -- 172.21.15.150:0/303039626 shutdown_connections 2024-03-17T23:49:36.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.011+0000 7fd4dcc4b640 1 --2- 172.21.15.150:0/303039626 >> v2:172.21.15.150:3300/0 conn(0x7fd4d806ecd0 0x7fd4d806f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.011+0000 7fd4dcc4b640 1 --2- 172.21.15.150:0/303039626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4d815ab90 0x7fd4d815bd10 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.011+0000 7fd4dcc4b640 1 -- 172.21.15.150:0/303039626 >> 172.21.15.150:0/303039626 conn(0x7fd4d806b190 msgr2=0x7fd4d806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:36.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.011+0000 7fd4dcc4b640 1 -- 172.21.15.150:0/303039626 shutdown_connections 2024-03-17T23:49:36.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.011+0000 7fd4dcc4b640 1 -- 172.21.15.150:0/303039626 wait complete. 2024-03-17T23:49:36.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.012+0000 7fd4dcc4b640 1 Processor -- start 2024-03-17T23:49:36.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.012+0000 7fd4dcc4b640 1 -- start start 2024-03-17T23:49:36.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.013+0000 7fd4dcc4b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4d806ecd0 0x7fd4d8157e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:36.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.013+0000 7fd4dcc4b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4d81514f0 0x7fd4d8158e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:36.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.013+0000 7fd4dcc4b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4d815ab90 0x7fd4d814fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:36.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.013+0000 7fd4dcc4b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d80af510 con 0x7fd4d81514f0 2024-03-17T23:49:36.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.013+0000 7fd4dcc4b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd4d815b8f0 con 0x7fd4d806ecd0 2024-03-17T23:49:36.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.013+0000 7fd4dcc4b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd4d81518d0 con 0x7fd4d815ab90 2024-03-17T23:49:36.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.013+0000 7fd4cffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4d81514f0 0x7fd4d8158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:36.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.013+0000 7fd4cffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4d81514f0 0x7fd4d8158e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60906/0 (socket says 172.21.15.150:60906) 2024-03-17T23:49:36.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.013+0000 7fd4cffff640 1 -- 172.21.15.150:0/947350538 learned_addr learned my addr 172.21.15.150:0/947350538 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:36.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.013+0000 7fd4d659c640 1 --2- 172.21.15.150:0/947350538 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4d806ecd0 0x7fd4d8157e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:36.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.013+0000 7fd4d6d9d640 1 --2- 172.21.15.150:0/947350538 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4d815ab90 0x7fd4d814fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:36.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.013+0000 7fd4cffff640 1 -- 172.21.15.150:0/947350538 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4d815ab90 msgr2=0x7fd4d814fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:36.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.013+0000 7fd4cffff640 1 --2- 172.21.15.150:0/947350538 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4d815ab90 0x7fd4d814fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.013+0000 7fd4cffff640 1 -- 172.21.15.150:0/947350538 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4d806ecd0 msgr2=0x7fd4d8157e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:36.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.013+0000 7fd4cffff640 1 --2- 172.21.15.150:0/947350538 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4d806ecd0 0x7fd4d8157e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.013+0000 7fd4cffff640 1 -- 172.21.15.150:0/947350538 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4d8159340 con 0x7fd4d81514f0 2024-03-17T23:49:36.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.013+0000 7fd4cffff640 1 --2- 172.21.15.150:0/947350538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4d81514f0 0x7fd4d8158e00 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fd4c0009890 tx=0x7fd4c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:36.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.013+0000 7fd4ceffd640 1 -- 172.21.15.150:0/947350538 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c0041070 con 0x7fd4d81514f0 2024-03-17T23:49:36.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.013+0000 7fd4ceffd640 1 -- 172.21.15.150:0/947350538 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4c003b920 con 0x7fd4d81514f0 2024-03-17T23:49:36.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.013+0000 7fd4dcc4b640 1 -- 172.21.15.150:0/947350538 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4d8155340 con 0x7fd4d81514f0 2024-03-17T23:49:36.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.013+0000 7fd4ceffd640 1 -- 172.21.15.150:0/947350538 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4c0035040 con 0x7fd4d81514f0 2024-03-17T23:49:36.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.014+0000 7fd4dcc4b640 1 -- 172.21.15.150:0/947350538 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4d8155900 con 0x7fd4d81514f0 2024-03-17T23:49:36.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.014+0000 7fd4dcc4b640 1 -- 172.21.15.150:0/947350538 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4d80681a0 con 0x7fd4d81514f0 2024-03-17T23:49:36.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.015+0000 7fd4ceffd640 1 -- 172.21.15.150:0/947350538 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd4c0038020 con 0x7fd4d81514f0 2024-03-17T23:49:36.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.015+0000 7fd4ceffd640 1 --2- 172.21.15.150:0/947350538 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd4b806bc00 0x7fd4b806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:36.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.015+0000 7fd4ceffd640 1 -- 172.21.15.150:0/947350538 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7fd4c00ac350 con 0x7fd4d81514f0 2024-03-17T23:49:36.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.015+0000 7fd4d659c640 1 --2- 172.21.15.150:0/947350538 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd4b806bc00 0x7fd4b806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:36.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.016+0000 7fd4d659c640 1 --2- 172.21.15.150:0/947350538 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd4b806bc00 0x7fd4b806e0c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd4c80013b0 tx=0x7fd4c8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:36.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.020+0000 7fd4ceffd640 1 -- 172.21.15.150:0/947350538 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd4c0031800 con 0x7fd4d81514f0 2024-03-17T23:49:36.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.193+0000 7fd4dcc4b640 1 -- 172.21.15.150:0/947350538 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd4d806bf90 con 0x7fd4d81514f0 2024-03-17T23:49:36.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.194+0000 7fd4ceffd640 1 -- 172.21.15.150:0/947350538 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) ==== 93+0+10 (secure 0 0 0) 0x7fd4c007a6f0 con 0x7fd4d81514f0 2024-03-17T23:49:36.196 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:49:36.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.197+0000 7fd4dcc4b640 1 -- 172.21.15.150:0/947350538 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd4b806bc00 msgr2=0x7fd4b806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:36.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.197+0000 7fd4dcc4b640 1 --2- 172.21.15.150:0/947350538 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd4b806bc00 0x7fd4b806e0c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd4c80013b0 tx=0x7fd4c8009040 comp rx=0 tx=0).stop 2024-03-17T23:49:36.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.197+0000 7fd4dcc4b640 1 -- 172.21.15.150:0/947350538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4d81514f0 msgr2=0x7fd4d8158e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:36.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.197+0000 7fd4dcc4b640 1 --2- 172.21.15.150:0/947350538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4d81514f0 0x7fd4d8158e00 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fd4c0009890 tx=0x7fd4c00046c0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.197+0000 7fd4dcc4b640 1 -- 172.21.15.150:0/947350538 shutdown_connections 2024-03-17T23:49:36.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.197+0000 7fd4dcc4b640 1 --2- 172.21.15.150:0/947350538 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd4d815ab90 0x7fd4d814fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.197+0000 7fd4dcc4b640 1 --2- 172.21.15.150:0/947350538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd4d81514f0 0x7fd4d8158e00 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.197+0000 7fd4dcc4b640 1 --2- 172.21.15.150:0/947350538 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd4b806bc00 0x7fd4b806e0c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.197+0000 7fd4dcc4b640 1 --2- 172.21.15.150:0/947350538 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd4d806ecd0 0x7fd4d8157e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.197+0000 7fd4dcc4b640 1 -- 172.21.15.150:0/947350538 >> 172.21.15.150:0/947350538 conn(0x7fd4d806b190 msgr2=0x7fd4d8150600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:36.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.197+0000 7fd4dcc4b640 1 -- 172.21.15.150:0/947350538 shutdown_connections 2024-03-17T23:49:36.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.197+0000 7fd4dcc4b640 1 -- 172.21.15.150:0/947350538 wait complete. 2024-03-17T23:49:36.209 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:49:36.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.311+0000 7fbbc0cff640 1 Processor -- start 2024-03-17T23:49:36.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.311+0000 7fbbc0cff640 1 -- start start 2024-03-17T23:49:36.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.312+0000 7fbbc0cff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fbbbc0ce6a0 0x7fbbbc166a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:36.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.312+0000 7fbbc0cff640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbbc145170 con 0x7fbbbc06f400 2024-03-17T23:49:36.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.312+0000 7fbbc0cff640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbbc145450 con 0x7fbbbc144bb0 2024-03-17T23:49:36.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.312+0000 7fbbc0cff640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fbbbc145730 con 0x7fbbbc0ce6a0 2024-03-17T23:49:36.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.312+0000 7fbbb9d9b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fbbbc06f400 0x7fbbbc0cdf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41766/0 (socket says 172.21.15.150:41766) 2024-03-17T23:49:36.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.312+0000 7fbbb9d9b640 1 -- 172.21.15.150:0/2244720693 learned_addr learned my addr 172.21.15.150:0/2244720693 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:36.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.312+0000 7fbbbad9d640 1 --2- 172.21.15.150:0/2244720693 >> v2:172.21.15.157:3300/0 conn(0x7fbbbc0ce6a0 0x7fbbbc166a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:36.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.312+0000 7fbbb959a640 1 -- 172.21.15.150:0/2244720693 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2808298709 0 0) 0x7fbbbc145170 con 0x7fbbbc06f400 2024-03-17T23:49:36.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.312+0000 7fbbb959a640 1 -- 172.21.15.150:0/2244720693 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbba4003520 con 0x7fbbbc06f400 2024-03-17T23:49:36.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.312+0000 7fbbb959a640 1 -- 172.21.15.150:0/2244720693 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2061289951 0 0) 0x7fbbbc145450 con 0x7fbbbc144bb0 2024-03-17T23:49:36.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.312+0000 7fbbb959a640 1 -- 172.21.15.150:0/2244720693 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbbc145170 con 0x7fbbbc144bb0 2024-03-17T23:49:36.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.312+0000 7fbbbad9d640 1 -- 172.21.15.150:0/2244720693 >> v1:172.21.15.157:6789/0 conn(0x7fbbbc144bb0 legacy=0x7fbbbc15ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:36.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.312+0000 7fbbbad9d640 1 -- 172.21.15.150:0/2244720693 >> v1:172.21.15.150:6789/0 conn(0x7fbbbc06f400 legacy=0x7fbbbc0cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:36.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.312+0000 7fbbbad9d640 1 -- 172.21.15.150:0/2244720693 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbbc15d560 con 0x7fbbbc0ce6a0 2024-03-17T23:49:36.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.313+0000 7fbbbad9d640 1 --2- 172.21.15.150:0/2244720693 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbbbc0ce6a0 0x7fbbbc166a40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4009d90 tx=0x7fbbb402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f8a26be8fc3b4b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:36.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.313+0000 7fbbb959a640 1 -- 172.21.15.150:0/2244720693 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4039070 con 0x7fbbbc0ce6a0 2024-03-17T23:49:36.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.313+0000 7fbbb959a640 1 -- 172.21.15.150:0/2244720693 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbb402cc60 con 0x7fbbbc0ce6a0 2024-03-17T23:49:36.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.313+0000 7fbbb959a640 1 -- 172.21.15.150:0/2244720693 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4034660 con 0x7fbbbc0ce6a0 2024-03-17T23:49:36.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.314+0000 7fbbc0cff640 1 -- 172.21.15.150:0/2244720693 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbbbc0ce6a0 msgr2=0x7fbbbc166a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:36.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.314+0000 7fbbc0cff640 1 --2- 172.21.15.150:0/2244720693 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbbbc0ce6a0 0x7fbbbc166a40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4009d90 tx=0x7fbbb402beb0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.314+0000 7fbbc0cff640 1 -- 172.21.15.150:0/2244720693 shutdown_connections 2024-03-17T23:49:36.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.314+0000 7fbbc0cff640 1 --2- 172.21.15.150:0/2244720693 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbbbc0ce6a0 0x7fbbbc166a40 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.314+0000 7fbbc0cff640 1 -- 172.21.15.150:0/2244720693 >> 172.21.15.150:0/2244720693 conn(0x7fbbbc06b190 msgr2=0x7fbbbc06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:36.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.314+0000 7fbbc0cff640 1 -- 172.21.15.150:0/2244720693 shutdown_connections 2024-03-17T23:49:36.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.314+0000 7fbbc0cff640 1 -- 172.21.15.150:0/2244720693 wait complete. 2024-03-17T23:49:36.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.315+0000 7fbbc0cff640 1 Processor -- start 2024-03-17T23:49:36.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.315+0000 7fbbc0cff640 1 -- start start 2024-03-17T23:49:36.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.316+0000 7fbbc0cff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbbc06f400 0x7fbbbc157f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:36.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.316+0000 7fbbc0cff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbbbc0ce6a0 0x7fbbbc0aea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:36.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.316+0000 7fbbc0cff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbbbc144bb0 0x7fbbbc157530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:36.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.316+0000 7fbbc0cff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc167100 con 0x7fbbbc06f400 2024-03-17T23:49:36.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.316+0000 7fbbc0cff640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc166f80 con 0x7fbbbc144bb0 2024-03-17T23:49:36.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.316+0000 7fbbc0cff640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbbbc167280 con 0x7fbbbc0ce6a0 2024-03-17T23:49:36.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.316+0000 7fbbba59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbbc06f400 0x7fbbbc157f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:36.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.316+0000 7fbbba59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbbc06f400 0x7fbbbc157f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60908/0 (socket says 172.21.15.150:60908) 2024-03-17T23:49:36.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.316+0000 7fbbba59c640 1 -- 172.21.15.150:0/296446018 learned_addr learned my addr 172.21.15.150:0/296446018 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:36.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.316+0000 7fbbb9d9b640 1 --2- 172.21.15.150:0/296446018 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbbbc0ce6a0 0x7fbbbc0aea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:36.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.316+0000 7fbbbad9d640 1 --2- 172.21.15.150:0/296446018 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbbbc144bb0 0x7fbbbc157530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:36.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.316+0000 7fbbba59c640 1 -- 172.21.15.150:0/296446018 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbbbc0ce6a0 msgr2=0x7fbbbc0aea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:36.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.316+0000 7fbbba59c640 1 --2- 172.21.15.150:0/296446018 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbbbc0ce6a0 0x7fbbbc0aea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.316+0000 7fbbba59c640 1 -- 172.21.15.150:0/296446018 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbbbc144bb0 msgr2=0x7fbbbc157530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:36.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.316+0000 7fbbba59c640 1 --2- 172.21.15.150:0/296446018 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbbbc144bb0 0x7fbbbc157530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.316+0000 7fbbba59c640 1 -- 172.21.15.150:0/296446018 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbbc0aef60 con 0x7fbbbc06f400 2024-03-17T23:49:36.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.317+0000 7fbbba59c640 1 --2- 172.21.15.150:0/296446018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbbc06f400 0x7fbbbc157f30 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fbbac00c430 tx=0x7fbbac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:36.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.317+0000 7fbbaaffd640 1 -- 172.21.15.150:0/296446018 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbac017070 con 0x7fbbbc06f400 2024-03-17T23:49:36.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.317+0000 7fbbaaffd640 1 -- 172.21.15.150:0/296446018 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbac0052f0 con 0x7fbbbc06f400 2024-03-17T23:49:36.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.317+0000 7fbbaaffd640 1 -- 172.21.15.150:0/296446018 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbac003830 con 0x7fbbbc06f400 2024-03-17T23:49:36.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.317+0000 7fbbc0cff640 1 -- 172.21.15.150:0/296446018 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbbc156490 con 0x7fbbbc06f400 2024-03-17T23:49:36.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.317+0000 7fbbc0cff640 1 -- 172.21.15.150:0/296446018 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbbc156a50 con 0x7fbbbc06f400 2024-03-17T23:49:36.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.318+0000 7fbbc0cff640 1 -- 172.21.15.150:0/296446018 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbbc0b5060 con 0x7fbbbc06f400 2024-03-17T23:49:36.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.318+0000 7fbbaaffd640 1 -- 172.21.15.150:0/296446018 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbbac01a070 con 0x7fbbbc06f400 2024-03-17T23:49:36.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.318+0000 7fbbaaffd640 1 --2- 172.21.15.150:0/296446018 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbb9c06bbd0 0x7fbb9c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:36.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.319+0000 7fbbaaffd640 1 -- 172.21.15.150:0/296446018 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7fbbac08a060 con 0x7fbbbc06f400 2024-03-17T23:49:36.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.319+0000 7fbbb9d9b640 1 --2- 172.21.15.150:0/296446018 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbb9c06bbd0 0x7fbb9c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:36.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.319+0000 7fbbb9d9b640 1 --2- 172.21.15.150:0/296446018 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbb9c06bbd0 0x7fbb9c06e090 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc13acb0 tx=0x7fbbb0007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:36.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.323+0000 7fbbaaffd640 1 -- 172.21.15.150:0/296446018 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbbac058260 con 0x7fbbbc06f400 2024-03-17T23:49:36.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.495+0000 7fbbc0cff640 1 -- 172.21.15.150:0/296446018 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbbbc0681a0 con 0x7fbbbc06f400 2024-03-17T23:49:36.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.495+0000 7fbbaaffd640 1 -- 172.21.15.150:0/296446018 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) ==== 104+0+11 (secure 0 0 0) 0x7fbbac05c300 con 0x7fbbbc06f400 2024-03-17T23:49:36.497 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:49:36.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.497+0000 7fbbc0cff640 1 -- 172.21.15.150:0/296446018 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbb9c06bbd0 msgr2=0x7fbb9c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:36.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.497+0000 7fbbc0cff640 1 --2- 172.21.15.150:0/296446018 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbb9c06bbd0 0x7fbb9c06e090 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc13acb0 tx=0x7fbbb0007d30 comp rx=0 tx=0).stop 2024-03-17T23:49:36.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.497+0000 7fbbc0cff640 1 -- 172.21.15.150:0/296446018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbbc06f400 msgr2=0x7fbbbc157f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:36.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.497+0000 7fbbc0cff640 1 --2- 172.21.15.150:0/296446018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbbc06f400 0x7fbbbc157f30 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fbbac00c430 tx=0x7fbbac00c8f0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.497+0000 7fbbc0cff640 1 -- 172.21.15.150:0/296446018 shutdown_connections 2024-03-17T23:49:36.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.497+0000 7fbbc0cff640 1 --2- 172.21.15.150:0/296446018 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbb9c06bbd0 0x7fbb9c06e090 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.497+0000 7fbbc0cff640 1 --2- 172.21.15.150:0/296446018 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbbbc144bb0 0x7fbbbc157530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.497+0000 7fbbc0cff640 1 --2- 172.21.15.150:0/296446018 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbbbc0ce6a0 0x7fbbbc0aea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.497+0000 7fbbc0cff640 1 --2- 172.21.15.150:0/296446018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbbbc06f400 0x7fbbbc157f30 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.497+0000 7fbbc0cff640 1 -- 172.21.15.150:0/296446018 >> 172.21.15.150:0/296446018 conn(0x7fbbbc06b190 msgr2=0x7fbbbc139d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:36.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.497+0000 7fbbc0cff640 1 -- 172.21.15.150:0/296446018 shutdown_connections 2024-03-17T23:49:36.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.497+0000 7fbbc0cff640 1 -- 172.21.15.150:0/296446018 wait complete. 2024-03-17T23:49:36.510 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:49:36.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.612+0000 7fbff113b640 1 Processor -- start 2024-03-17T23:49:36.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.612+0000 7fbff113b640 1 -- start start 2024-03-17T23:49:36.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.612+0000 7fbff113b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbfec099e20 0x7fbfec09a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:36.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.612+0000 7fbff113b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfec09a740 con 0x7fbfec0d1210 2024-03-17T23:49:36.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.612+0000 7fbff113b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfec06ca90 con 0x7fbfec15e460 2024-03-17T23:49:36.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.612+0000 7fbff113b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbfec15e990 con 0x7fbfec099e20 2024-03-17T23:49:36.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.613+0000 7fbfeb59e640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fbfec0d1210 0x7fbfec167ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41772/0 (socket says 172.21.15.150:41772) 2024-03-17T23:49:36.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.613+0000 7fbfeb59e640 1 -- 172.21.15.150:0/2977643641 learned_addr learned my addr 172.21.15.150:0/2977643641 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:36.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.613+0000 7fbfead9d640 1 --2- 172.21.15.150:0/2977643641 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbfec099e20 0x7fbfec09a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:36.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.613+0000 7fbfea59c640 1 -- 172.21.15.150:0/2977643641 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2962839908 0 0) 0x7fbfec09a740 con 0x7fbfec0d1210 2024-03-17T23:49:36.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.613+0000 7fbfea59c640 1 -- 172.21.15.150:0/2977643641 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfd8003520 con 0x7fbfec0d1210 2024-03-17T23:49:36.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.613+0000 7fbfea59c640 1 -- 172.21.15.150:0/2977643641 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 998385700 0 0) 0x7fbfec06ca90 con 0x7fbfec15e460 2024-03-17T23:49:36.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.613+0000 7fbfea59c640 1 -- 172.21.15.150:0/2977643641 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfec09a740 con 0x7fbfec15e460 2024-03-17T23:49:36.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.613+0000 7fbfead9d640 1 -- 172.21.15.150:0/2977643641 >> v1:172.21.15.157:6789/0 conn(0x7fbfec15e460 legacy=0x7fbfec06c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:36.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.613+0000 7fbfead9d640 1 -- 172.21.15.150:0/2977643641 >> v1:172.21.15.150:6789/0 conn(0x7fbfec0d1210 legacy=0x7fbfec167ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:36.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.613+0000 7fbfead9d640 1 -- 172.21.15.150:0/2977643641 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfec15ec70 con 0x7fbfec099e20 2024-03-17T23:49:36.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.614+0000 7fbfead9d640 1 --2- 172.21.15.150:0/2977643641 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbfec099e20 0x7fbfec09a200 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc0097d0 tx=0x7fbfdc02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcdff65a73d0f8b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:36.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.614+0000 7fbfea59c640 1 -- 172.21.15.150:0/2977643641 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc035020 con 0x7fbfec099e20 2024-03-17T23:49:36.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.614+0000 7fbfea59c640 1 -- 172.21.15.150:0/2977643641 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfdc031890 con 0x7fbfec099e20 2024-03-17T23:49:36.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.614+0000 7fbfea59c640 1 -- 172.21.15.150:0/2977643641 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc031c40 con 0x7fbfec099e20 2024-03-17T23:49:36.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.614+0000 7fbff113b640 1 -- 172.21.15.150:0/2977643641 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbfec099e20 msgr2=0x7fbfec09a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:36.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.614+0000 7fbff113b640 1 --2- 172.21.15.150:0/2977643641 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbfec099e20 0x7fbfec09a200 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc0097d0 tx=0x7fbfdc02b930 comp rx=0 tx=0).stop 2024-03-17T23:49:36.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.614+0000 7fbff113b640 1 -- 172.21.15.150:0/2977643641 shutdown_connections 2024-03-17T23:49:36.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.614+0000 7fbff113b640 1 --2- 172.21.15.150:0/2977643641 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbfec099e20 0x7fbfec09a200 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.614+0000 7fbff113b640 1 -- 172.21.15.150:0/2977643641 >> 172.21.15.150:0/2977643641 conn(0x7fbfec0b3630 msgr2=0x7fbfec0b3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:36.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.614+0000 7fbff113b640 1 -- 172.21.15.150:0/2977643641 shutdown_connections 2024-03-17T23:49:36.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.614+0000 7fbff113b640 1 -- 172.21.15.150:0/2977643641 wait complete. 2024-03-17T23:49:36.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.616+0000 7fbff113b640 1 Processor -- start 2024-03-17T23:49:36.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.616+0000 7fbff113b640 1 -- start start 2024-03-17T23:49:36.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.617+0000 7fbff113b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfec099e20 0x7fbfec154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:36.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.617+0000 7fbff113b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbfec0d1210 0x7fbfec0cd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:36.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.617+0000 7fbff113b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfec15e460 0x7fbfec0cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:36.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.617+0000 7fbff113b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbfec0cfcd0 con 0x7fbfec099e20 2024-03-17T23:49:36.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.617+0000 7fbff113b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbfec15f230 con 0x7fbfec15e460 2024-03-17T23:49:36.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.617+0000 7fbff113b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbfec06d570 con 0x7fbfec0d1210 2024-03-17T23:49:36.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.617+0000 7fbfead9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfec099e20 0x7fbfec154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:36.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.617+0000 7fbfead9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfec099e20 0x7fbfec154100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60924/0 (socket says 172.21.15.150:60924) 2024-03-17T23:49:36.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.617+0000 7fbfead9d640 1 -- 172.21.15.150:0/919420026 learned_addr learned my addr 172.21.15.150:0/919420026 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:36.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.617+0000 7fbfeb59e640 1 --2- 172.21.15.150:0/919420026 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfec15e460 0x7fbfec0cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:36.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.617+0000 7fbfe3fff640 1 --2- 172.21.15.150:0/919420026 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbfec0d1210 0x7fbfec0cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:36.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.617+0000 7fbfead9d640 1 -- 172.21.15.150:0/919420026 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbfec0d1210 msgr2=0x7fbfec0cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:36.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.617+0000 7fbfead9d640 1 --2- 172.21.15.150:0/919420026 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbfec0d1210 0x7fbfec0cd190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.617+0000 7fbfead9d640 1 -- 172.21.15.150:0/919420026 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfec15e460 msgr2=0x7fbfec0cd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:36.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.617+0000 7fbfead9d640 1 --2- 172.21.15.150:0/919420026 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfec15e460 0x7fbfec0cd6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.617+0000 7fbfead9d640 1 -- 172.21.15.150:0/919420026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfec0caa70 con 0x7fbfec099e20 2024-03-17T23:49:36.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.617+0000 7fbfeb59e640 1 --2- 172.21.15.150:0/919420026 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfec15e460 0x7fbfec0cd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:36.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.617+0000 7fbfead9d640 1 --2- 172.21.15.150:0/919420026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfec099e20 0x7fbfec154100 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc000c00 tx=0x7fbfdc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:36.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.617+0000 7fbfe37fe640 1 -- 172.21.15.150:0/919420026 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc040070 con 0x7fbfec099e20 2024-03-17T23:49:36.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.617+0000 7fbfe37fe640 1 -- 172.21.15.150:0/919420026 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfdc030d20 con 0x7fbfec099e20 2024-03-17T23:49:36.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.617+0000 7fbff113b640 1 -- 172.21.15.150:0/919420026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfec0cad50 con 0x7fbfec099e20 2024-03-17T23:49:36.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.617+0000 7fbfe37fe640 1 -- 172.21.15.150:0/919420026 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc035040 con 0x7fbfec099e20 2024-03-17T23:49:36.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.617+0000 7fbff113b640 1 -- 172.21.15.150:0/919420026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfec0cb310 con 0x7fbfec099e20 2024-03-17T23:49:36.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.619+0000 7fbff113b640 1 -- 172.21.15.150:0/919420026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfec0681a0 con 0x7fbfec099e20 2024-03-17T23:49:36.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.619+0000 7fbfe37fe640 1 -- 172.21.15.150:0/919420026 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbfdc038020 con 0x7fbfec099e20 2024-03-17T23:49:36.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.619+0000 7fbfe37fe640 1 --2- 172.21.15.150:0/919420026 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbfcc06bc00 0x7fbfcc06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:36.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.619+0000 7fbfe37fe640 1 -- 172.21.15.150:0/919420026 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7fbfdc0ac800 con 0x7fbfec099e20 2024-03-17T23:49:36.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.619+0000 7fbfe3fff640 1 --2- 172.21.15.150:0/919420026 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbfcc06bc00 0x7fbfcc06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:36.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.620+0000 7fbfe3fff640 1 --2- 172.21.15.150:0/919420026 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbfcc06bc00 0x7fbfcc06e0c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4007fd0 tx=0x7fbfd4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:36.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.624+0000 7fbfe37fe640 1 -- 172.21.15.150:0/919420026 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbfdc02d040 con 0x7fbfec099e20 2024-03-17T23:49:36.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.799+0000 7fbff113b640 1 -- 172.21.15.150:0/919420026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbfec09b500 con 0x7fbfec099e20 2024-03-17T23:49:36.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.799+0000 7fbfe37fe640 1 -- 172.21.15.150:0/919420026 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) ==== 100+0+11 (secure 0 0 0) 0x7fbfdc07ab10 con 0x7fbfec099e20 2024-03-17T23:49:36.801 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:49:36.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.801+0000 7fbff113b640 1 -- 172.21.15.150:0/919420026 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbfcc06bc00 msgr2=0x7fbfcc06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:36.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.801+0000 7fbff113b640 1 --2- 172.21.15.150:0/919420026 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbfcc06bc00 0x7fbfcc06e0c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4007fd0 tx=0x7fbfd4009040 comp rx=0 tx=0).stop 2024-03-17T23:49:36.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.801+0000 7fbff113b640 1 -- 172.21.15.150:0/919420026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfec099e20 msgr2=0x7fbfec154100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:36.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.801+0000 7fbff113b640 1 --2- 172.21.15.150:0/919420026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfec099e20 0x7fbfec154100 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc000c00 tx=0x7fbfdc0046c0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.801+0000 7fbff113b640 1 -- 172.21.15.150:0/919420026 shutdown_connections 2024-03-17T23:49:36.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.801+0000 7fbff113b640 1 --2- 172.21.15.150:0/919420026 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbfcc06bc00 0x7fbfcc06e0c0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.801+0000 7fbff113b640 1 --2- 172.21.15.150:0/919420026 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbfec15e460 0x7fbfec0cd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.801+0000 7fbff113b640 1 --2- 172.21.15.150:0/919420026 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbfec0d1210 0x7fbfec0cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.801+0000 7fbff113b640 1 --2- 172.21.15.150:0/919420026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbfec099e20 0x7fbfec154100 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.801+0000 7fbff113b640 1 -- 172.21.15.150:0/919420026 >> 172.21.15.150:0/919420026 conn(0x7fbfec0b3630 msgr2=0x7fbfec160e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:36.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.801+0000 7fbff113b640 1 -- 172.21.15.150:0/919420026 shutdown_connections 2024-03-17T23:49:36.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.801+0000 7fbff113b640 1 -- 172.21.15.150:0/919420026 wait complete. 2024-03-17T23:49:36.814 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-17T23:49:36.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.914+0000 7fd6612c0640 1 Processor -- start 2024-03-17T23:49:36.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.914+0000 7fd6612c0640 1 -- start start 2024-03-17T23:49:36.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.915+0000 7fd6612c0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd65c142520 0x7fd65c0cc510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:36.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.915+0000 7fd6612c0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd65c0cca50 0x7fd65c0cdf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:36.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.915+0000 7fd6612c0640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd65c0ce440 0x7fd65c06e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:36.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.915+0000 7fd6612c0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fd65c142900 con 0x7fd65c142520 2024-03-17T23:49:36.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.915+0000 7fd6612c0640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd65c06bf60 con 0x7fd65c0ce440 2024-03-17T23:49:36.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.915+0000 7fd6612c0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd65c0cd900 con 0x7fd65c0cca50 2024-03-17T23:49:36.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.915+0000 7fd65affd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd65c142520 0x7fd65c0cc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:36.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.915+0000 7fd65a7fc640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd65c0cca50 0x7fd65c0cdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:36.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.915+0000 7fd65b7fe640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd65c0ce440 0x7fd65c06e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:36.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.915+0000 7fd65affd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd65c142520 0x7fd65c0cc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60934/0 (socket says 172.21.15.150:60934) 2024-03-17T23:49:36.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.915+0000 7fd65affd640 1 -- 172.21.15.150:0/16803782 learned_addr learned my addr 172.21.15.150:0/16803782 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:36.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.915+0000 7fd65affd640 1 -- 172.21.15.150:0/16803782 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd65c0cca50 msgr2=0x7fd65c0cdf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:36.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.915+0000 7fd65affd640 1 --2- 172.21.15.150:0/16803782 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd65c0cca50 0x7fd65c0cdf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.915+0000 7fd65affd640 1 -- 172.21.15.150:0/16803782 >> v2:172.21.15.157:3300/0 conn(0x7fd65c0ce440 msgr2=0x7fd65c06e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:36.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.915+0000 7fd65affd640 1 --2- 172.21.15.150:0/16803782 >> v2:172.21.15.157:3300/0 conn(0x7fd65c0ce440 0x7fd65c06e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.915+0000 7fd65affd640 1 -- 172.21.15.150:0/16803782 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd65c0cd060 con 0x7fd65c142520 2024-03-17T23:49:36.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.916+0000 7fd65affd640 1 --2- 172.21.15.150:0/16803782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd65c142520 0x7fd65c0cc510 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd65c0cfa30 tx=0x7fd64c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d1a2fa10ff331c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:36.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.916+0000 7fd659ffb640 1 -- 172.21.15.150:0/16803782 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64c02c9e0 con 0x7fd65c142520 2024-03-17T23:49:36.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.916+0000 7fd659ffb640 1 -- 172.21.15.150:0/16803782 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd64c02cb80 con 0x7fd65c142520 2024-03-17T23:49:36.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.916+0000 7fd659ffb640 1 -- 172.21.15.150:0/16803782 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64c0343f0 con 0x7fd65c142520 2024-03-17T23:49:36.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.917+0000 7fd6612c0640 1 -- 172.21.15.150:0/16803782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd65c142520 msgr2=0x7fd65c0cc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:36.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.917+0000 7fd6612c0640 1 --2- 172.21.15.150:0/16803782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd65c142520 0x7fd65c0cc510 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd65c0cfa30 tx=0x7fd64c02be40 comp rx=0 tx=0).stop 2024-03-17T23:49:36.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.917+0000 7fd6612c0640 1 -- 172.21.15.150:0/16803782 shutdown_connections 2024-03-17T23:49:36.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.917+0000 7fd6612c0640 1 --2- 172.21.15.150:0/16803782 >> v2:172.21.15.157:3300/0 conn(0x7fd65c0ce440 0x7fd65c06e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.917+0000 7fd6612c0640 1 --2- 172.21.15.150:0/16803782 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd65c0cca50 0x7fd65c0cdf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.917+0000 7fd6612c0640 1 --2- 172.21.15.150:0/16803782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd65c142520 0x7fd65c0cc510 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.917+0000 7fd6612c0640 1 -- 172.21.15.150:0/16803782 >> 172.21.15.150:0/16803782 conn(0x7fd65c06b190 msgr2=0x7fd65c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:36.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.917+0000 7fd6612c0640 1 -- 172.21.15.150:0/16803782 shutdown_connections 2024-03-17T23:49:36.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.917+0000 7fd6612c0640 1 -- 172.21.15.150:0/16803782 wait complete. 2024-03-17T23:49:36.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.918+0000 7fd6612c0640 1 Processor -- start 2024-03-17T23:49:36.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.918+0000 7fd6612c0640 1 -- start start 2024-03-17T23:49:36.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.919+0000 7fd6612c0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd65c0cca50 0x7fd65c0ad9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:36.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.919+0000 7fd6612c0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd65c0ce440 0x7fd65c0adf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:36.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.919+0000 7fd6612c0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd65c142520 0x7fd65c0b3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:36.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.919+0000 7fd6612c0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd65c06de60 con 0x7fd65c0cca50 2024-03-17T23:49:36.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.919+0000 7fd6612c0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd65c0cd620 con 0x7fd65c0ce440 2024-03-17T23:49:36.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.919+0000 7fd6612c0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd65c06e8c0 con 0x7fd65c142520 2024-03-17T23:49:36.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.919+0000 7fd65affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd65c0cca50 0x7fd65c0ad9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:36.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.919+0000 7fd65affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd65c0cca50 0x7fd65c0ad9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60948/0 (socket says 172.21.15.150:60948) 2024-03-17T23:49:36.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.919+0000 7fd65affd640 1 -- 172.21.15.150:0/1146177714 learned_addr learned my addr 172.21.15.150:0/1146177714 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:36.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.919+0000 7fd65a7fc640 1 --2- 172.21.15.150:0/1146177714 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd65c0ce440 0x7fd65c0adf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:36.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.919+0000 7fd65b7fe640 1 --2- 172.21.15.150:0/1146177714 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd65c142520 0x7fd65c0b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:36.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.919+0000 7fd65affd640 1 -- 172.21.15.150:0/1146177714 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd65c142520 msgr2=0x7fd65c0b3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:36.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.919+0000 7fd65affd640 1 --2- 172.21.15.150:0/1146177714 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd65c142520 0x7fd65c0b3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.919+0000 7fd65affd640 1 -- 172.21.15.150:0/1146177714 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd65c0ce440 msgr2=0x7fd65c0adf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:36.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.919+0000 7fd65affd640 1 --2- 172.21.15.150:0/1146177714 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd65c0ce440 0x7fd65c0adf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:36.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.919+0000 7fd65affd640 1 -- 172.21.15.150:0/1146177714 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd65c0ae5f0 con 0x7fd65c0cca50 2024-03-17T23:49:36.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.919+0000 7fd65affd640 1 --2- 172.21.15.150:0/1146177714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd65c0cca50 0x7fd65c0ad9c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd64c000f80 tx=0x7fd64c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:36.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.919+0000 7fd6437fe640 1 -- 172.21.15.150:0/1146177714 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64c043020 con 0x7fd65c0cca50 2024-03-17T23:49:36.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.919+0000 7fd6437fe640 1 -- 172.21.15.150:0/1146177714 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd64c03e450 con 0x7fd65c0cca50 2024-03-17T23:49:36.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.919+0000 7fd6437fe640 1 -- 172.21.15.150:0/1146177714 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64c009510 con 0x7fd65c0cca50 2024-03-17T23:49:36.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.920+0000 7fd6612c0640 1 -- 172.21.15.150:0/1146177714 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd65c0ae8d0 con 0x7fd65c0cca50 2024-03-17T23:49:36.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.920+0000 7fd6612c0640 1 -- 172.21.15.150:0/1146177714 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd65c13b400 con 0x7fd65c0cca50 2024-03-17T23:49:36.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.920+0000 7fd6612c0640 1 -- 172.21.15.150:0/1146177714 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd65c0b4e40 con 0x7fd65c0cca50 2024-03-17T23:49:36.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.921+0000 7fd6437fe640 1 -- 172.21.15.150:0/1146177714 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd64c046020 con 0x7fd65c0cca50 2024-03-17T23:49:36.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.921+0000 7fd6437fe640 1 --2- 172.21.15.150:0/1146177714 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd64806bbd0 0x7fd64806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:36.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.921+0000 7fd6437fe640 1 -- 172.21.15.150:0/1146177714 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7fd64c0ad200 con 0x7fd65c0cca50 2024-03-17T23:49:36.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.921+0000 7fd65a7fc640 1 --2- 172.21.15.150:0/1146177714 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd64806bbd0 0x7fd64806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:36.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.922+0000 7fd65a7fc640 1 --2- 172.21.15.150:0/1146177714 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd64806bbd0 0x7fd64806e090 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd65c13f650 tx=0x7fd650009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:36.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:36.927+0000 7fd6437fe640 1 -- 172.21.15.150:0/1146177714 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd65c0b4e40 con 0x7fd65c0cca50 2024-03-17T23:49:37.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:37.112+0000 7fd6612c0640 1 -- 172.21.15.150:0/1146177714 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd65c0681a0 con 0x7fd65c0cca50 2024-03-17T23:49:37.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:37.113+0000 7fd6437fe640 1 -- 172.21.15.150:0/1146177714 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v58) ==== 80+0+0 (secure 0 0 0) 0x7fd64c04d030 con 0x7fd65c0cca50 2024-03-17T23:49:37.113 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-03-17T23:49:37.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:37.114+0000 7fd6612c0640 1 -- 172.21.15.150:0/1146177714 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd64806bbd0 msgr2=0x7fd64806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:37.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:37.114+0000 7fd6612c0640 1 --2- 172.21.15.150:0/1146177714 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd64806bbd0 0x7fd64806e090 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd65c13f650 tx=0x7fd650009190 comp rx=0 tx=0).stop 2024-03-17T23:49:37.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:37.115+0000 7fd6612c0640 1 -- 172.21.15.150:0/1146177714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd65c0cca50 msgr2=0x7fd65c0ad9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:37.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:37.115+0000 7fd6612c0640 1 --2- 172.21.15.150:0/1146177714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd65c0cca50 0x7fd65c0ad9c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd64c000f80 tx=0x7fd64c004a40 comp rx=0 tx=0).stop 2024-03-17T23:49:37.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:37.115+0000 7fd6612c0640 1 -- 172.21.15.150:0/1146177714 shutdown_connections 2024-03-17T23:49:37.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:37.115+0000 7fd6612c0640 1 --2- 172.21.15.150:0/1146177714 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd65c142520 0x7fd65c0b3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:37.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:37.115+0000 7fd6612c0640 1 --2- 172.21.15.150:0/1146177714 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd64806bbd0 0x7fd64806e090 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:37.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:37.115+0000 7fd6612c0640 1 --2- 172.21.15.150:0/1146177714 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd65c0ce440 0x7fd65c0adf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:37.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:37.115+0000 7fd6612c0640 1 --2- 172.21.15.150:0/1146177714 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd65c0cca50 0x7fd65c0ad9c0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:37.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:37.115+0000 7fd6612c0640 1 -- 172.21.15.150:0/1146177714 >> 172.21.15.150:0/1146177714 conn(0x7fd65c06b190 msgr2=0x7fd65c0afbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:37.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:37.115+0000 7fd6612c0640 1 -- 172.21.15.150:0/1146177714 shutdown_connections 2024-03-17T23:49:37.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:37.115+0000 7fd6612c0640 1 -- 172.21.15.150:0/1146177714 wait complete. 2024-03-17T23:49:37.126 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi150.front.sepia.ceph.com... 2024-03-17T23:49:37.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:37.127 DEBUG:teuthology.orchestra.run.smithi150:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-17T23:49:37.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:ceph-fuse[61283]: fuse finished with error 0 and tester_r 0 2024-03-17T23:49:37.154 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:49:43.256 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi150.front.sepia.ceph.com 2024-03-17T23:49:43.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:43.257 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-17T23:49:43.276 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-17T23:49:43.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.426+0000 7fd8f0e85640 1 Processor -- start 2024-03-17T23:49:43.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.426+0000 7fd8f0e85640 1 -- start start 2024-03-17T23:49:43.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.427+0000 7fd8f0e85640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd8ec0cf120 0x7fd8ec1619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:43.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.427+0000 7fd8f0e85640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fd8ec0b9860 0x7fd8ec0d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:43.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.427+0000 7fd8f0e85640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8ec0b9cb0 con 0x7fd8ec15de30 2024-03-17T23:49:43.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.427+0000 7fd8f0e85640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fd8ec162940 con 0x7fd8ec0b9860 2024-03-17T23:49:43.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.427+0000 7fd8f0e85640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd8ec1620a0 con 0x7fd8ec0cf120 2024-03-17T23:49:43.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.427+0000 7fd8ea59c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd8ec15de30 0x7fd8ec0cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41774/0 (socket says 172.21.15.150:41774) 2024-03-17T23:49:43.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.427+0000 7fd8ea59c640 1 -- 172.21.15.150:0/4077799791 learned_addr learned my addr 172.21.15.150:0/4077799791 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:43.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.428+0000 7fd8e9d9b640 1 --2- 172.21.15.150:0/4077799791 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd8ec0cf120 0x7fd8ec1619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:43.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.428+0000 7fd8ead9d640 1 --2- 172.21.15.150:0/4077799791 >> v2:172.21.15.157:3300/0 conn(0x7fd8ec0b9860 0x7fd8ec0d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:43.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.428+0000 7fd8e959a640 1 -- 172.21.15.150:0/4077799791 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1975103796 0 0) 0x7fd8ec0b9cb0 con 0x7fd8ec15de30 2024-03-17T23:49:43.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.428+0000 7fd8e959a640 1 -- 172.21.15.150:0/4077799791 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8d4003520 con 0x7fd8ec15de30 2024-03-17T23:49:43.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.428+0000 7fd8e9d9b640 1 -- 172.21.15.150:0/4077799791 >> v2:172.21.15.157:3300/0 conn(0x7fd8ec0b9860 msgr2=0x7fd8ec0d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:43.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.428+0000 7fd8e9d9b640 1 --2- 172.21.15.150:0/4077799791 >> v2:172.21.15.157:3300/0 conn(0x7fd8ec0b9860 0x7fd8ec0d2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:43.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.428+0000 7fd8e9d9b640 1 -- 172.21.15.150:0/4077799791 >> v1:172.21.15.150:6789/0 conn(0x7fd8ec15de30 legacy=0x7fd8ec0cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:43.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.428+0000 7fd8e9d9b640 1 -- 172.21.15.150:0/4077799791 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8ec162380 con 0x7fd8ec0cf120 2024-03-17T23:49:43.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.428+0000 7fd8ead9d640 1 --2- 172.21.15.150:0/4077799791 >> v2:172.21.15.157:3300/0 conn(0x7fd8ec0b9860 0x7fd8ec0d2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:43.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.429+0000 7fd8e9d9b640 1 --2- 172.21.15.150:0/4077799791 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd8ec0cf120 0x7fd8ec1619a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00097a0 tx=0x7fd8e002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be9f512d209a9060 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:43.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.429+0000 7fd8e959a640 1 -- 172.21.15.150:0/4077799791 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e00315a0 con 0x7fd8ec0cf120 2024-03-17T23:49:43.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.429+0000 7fd8e959a640 1 -- 172.21.15.150:0/4077799791 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8e0037070 con 0x7fd8ec0cf120 2024-03-17T23:49:43.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.429+0000 7fd8e959a640 1 -- 172.21.15.150:0/4077799791 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0031960 con 0x7fd8ec0cf120 2024-03-17T23:49:43.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.429+0000 7fd8f0e85640 1 -- 172.21.15.150:0/4077799791 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd8ec0cf120 msgr2=0x7fd8ec1619a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:43.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.429+0000 7fd8f0e85640 1 --2- 172.21.15.150:0/4077799791 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd8ec0cf120 0x7fd8ec1619a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00097a0 tx=0x7fd8e002b840 comp rx=0 tx=0).stop 2024-03-17T23:49:43.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.430+0000 7fd8f0e85640 1 -- 172.21.15.150:0/4077799791 shutdown_connections 2024-03-17T23:49:43.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.430+0000 7fd8f0e85640 1 --2- 172.21.15.150:0/4077799791 >> v2:172.21.15.157:3300/0 conn(0x7fd8ec0b9860 0x7fd8ec0d2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:43.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.430+0000 7fd8f0e85640 1 --2- 172.21.15.150:0/4077799791 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd8ec0cf120 0x7fd8ec1619a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:43.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.430+0000 7fd8f0e85640 1 -- 172.21.15.150:0/4077799791 >> 172.21.15.150:0/4077799791 conn(0x7fd8ec0730b0 msgr2=0x7fd8ec0d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:43.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.430+0000 7fd8f0e85640 1 -- 172.21.15.150:0/4077799791 shutdown_connections 2024-03-17T23:49:43.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.430+0000 7fd8f0e85640 1 -- 172.21.15.150:0/4077799791 wait complete. 2024-03-17T23:49:43.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.431+0000 7fd8f0e85640 1 Processor -- start 2024-03-17T23:49:43.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.431+0000 7fd8f0e85640 1 -- start start 2024-03-17T23:49:43.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.432+0000 7fd8f0e85640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd8ec0b9860 0x7fd8ec1591c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:43.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.432+0000 7fd8f0e85640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8ec0cf120 0x7fd8ec0cb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:43.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.432+0000 7fd8f0e85640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8ec15de30 0x7fd8ec1f3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:43.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.432+0000 7fd8f0e85640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ec15e210 con 0x7fd8ec0cf120 2024-03-17T23:49:43.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.432+0000 7fd8f0e85640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ec0d2170 con 0x7fd8ec15de30 2024-03-17T23:49:43.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.432+0000 7fd8f0e85640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fd8ec0d2bd0 con 0x7fd8ec0b9860 2024-03-17T23:49:43.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.432+0000 7fd8e9d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8ec0cf120 0x7fd8ec0cb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:43.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.432+0000 7fd8e9d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8ec0cf120 0x7fd8ec0cb860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60964/0 (socket says 172.21.15.150:60964) 2024-03-17T23:49:43.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.432+0000 7fd8e9d9b640 1 -- 172.21.15.150:0/3806821325 learned_addr learned my addr 172.21.15.150:0/3806821325 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:43.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.432+0000 7fd8ead9d640 1 --2- 172.21.15.150:0/3806821325 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8ec15de30 0x7fd8ec1f3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:43.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.432+0000 7fd8ea59c640 1 --2- 172.21.15.150:0/3806821325 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd8ec0b9860 0x7fd8ec1591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:43.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.432+0000 7fd8e9d9b640 1 -- 172.21.15.150:0/3806821325 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd8ec0b9860 msgr2=0x7fd8ec1591c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:43.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.432+0000 7fd8e9d9b640 1 --2- 172.21.15.150:0/3806821325 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd8ec0b9860 0x7fd8ec1591c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:43.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.432+0000 7fd8e9d9b640 1 -- 172.21.15.150:0/3806821325 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8ec15de30 msgr2=0x7fd8ec1f3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:43.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.432+0000 7fd8e9d9b640 1 --2- 172.21.15.150:0/3806821325 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8ec15de30 0x7fd8ec1f3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:43.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.432+0000 7fd8e9d9b640 1 -- 172.21.15.150:0/3806821325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8ec1f4350 con 0x7fd8ec0cf120 2024-03-17T23:49:43.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.432+0000 7fd8e9d9b640 1 --2- 172.21.15.150:0/3806821325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8ec0cf120 0x7fd8ec0cb860 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00098c0 tx=0x7fd8e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:43.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.432+0000 7fd8daffd640 1 -- 172.21.15.150:0/3806821325 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0032630 con 0x7fd8ec0cf120 2024-03-17T23:49:43.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.433+0000 7fd8daffd640 1 -- 172.21.15.150:0/3806821325 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8e00327f0 con 0x7fd8ec0cf120 2024-03-17T23:49:43.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.433+0000 7fd8f0e85640 1 -- 172.21.15.150:0/3806821325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8ec1f4630 con 0x7fd8ec0cf120 2024-03-17T23:49:43.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.433+0000 7fd8daffd640 1 -- 172.21.15.150:0/3806821325 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0032be0 con 0x7fd8ec0cf120 2024-03-17T23:49:43.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.433+0000 7fd8f0e85640 1 -- 172.21.15.150:0/3806821325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8ec159700 con 0x7fd8ec0cf120 2024-03-17T23:49:43.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.433+0000 7fd8f0e85640 1 -- 172.21.15.150:0/3806821325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8ec0681a0 con 0x7fd8ec0cf120 2024-03-17T23:49:43.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.434+0000 7fd8daffd640 1 -- 172.21.15.150:0/3806821325 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fd8e004e430 con 0x7fd8ec0cf120 2024-03-17T23:49:43.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.434+0000 7fd8daffd640 1 --2- 172.21.15.150:0/3806821325 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd8cc06bbd0 0x7fd8cc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:43.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.434+0000 7fd8daffd640 1 -- 172.21.15.150:0/3806821325 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7fd8e002f030 con 0x7fd8ec0cf120 2024-03-17T23:49:43.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.434+0000 7fd8ea59c640 1 --2- 172.21.15.150:0/3806821325 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd8cc06bbd0 0x7fd8cc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:43.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.435+0000 7fd8ea59c640 1 --2- 172.21.15.150:0/3806821325 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd8cc06bbd0 0x7fd8cc06e090 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc005c00 tx=0x7fd8dc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:43.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.439+0000 7fd8daffd640 1 -- 172.21.15.150:0/3806821325 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fd8e0040070 con 0x7fd8ec0cf120 2024-03-17T23:49:43.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.647+0000 7fd8f0e85640 1 -- 172.21.15.150:0/3806821325 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd8ec0d1910 con 0x7fd8ec0cf120 2024-03-17T23:49:43.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.647+0000 7fd8daffd640 1 -- 172.21.15.150:0/3806821325 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v22) ==== 53+0+73 (secure 0 0 0) 0x7fd8e007b210 con 0x7fd8ec0cf120 2024-03-17T23:49:43.648 INFO:teuthology.orchestra.run.smithi150.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-17T23:49:43.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.649+0000 7fd8f0e85640 1 -- 172.21.15.150:0/3806821325 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd8cc06bbd0 msgr2=0x7fd8cc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:43.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.649+0000 7fd8f0e85640 1 --2- 172.21.15.150:0/3806821325 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd8cc06bbd0 0x7fd8cc06e090 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc005c00 tx=0x7fd8dc009040 comp rx=0 tx=0).stop 2024-03-17T23:49:43.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.649+0000 7fd8f0e85640 1 -- 172.21.15.150:0/3806821325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8ec0cf120 msgr2=0x7fd8ec0cb860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:43.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.649+0000 7fd8f0e85640 1 --2- 172.21.15.150:0/3806821325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8ec0cf120 0x7fd8ec0cb860 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fd8e00098c0 tx=0x7fd8e00046c0 comp rx=0 tx=0).stop 2024-03-17T23:49:43.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.649+0000 7fd8f0e85640 1 -- 172.21.15.150:0/3806821325 shutdown_connections 2024-03-17T23:49:43.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.649+0000 7fd8f0e85640 1 --2- 172.21.15.150:0/3806821325 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fd8cc06bbd0 0x7fd8cc06e090 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:43.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.649+0000 7fd8f0e85640 1 --2- 172.21.15.150:0/3806821325 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd8ec15de30 0x7fd8ec1f3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:43.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.649+0000 7fd8f0e85640 1 --2- 172.21.15.150:0/3806821325 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8ec0cf120 0x7fd8ec0cb860 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:43.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.649+0000 7fd8f0e85640 1 --2- 172.21.15.150:0/3806821325 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fd8ec0b9860 0x7fd8ec1591c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:43.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.649+0000 7fd8f0e85640 1 -- 172.21.15.150:0/3806821325 >> 172.21.15.150:0/3806821325 conn(0x7fd8ec0730b0 msgr2=0x7fd8ec0b75e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:43.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.649+0000 7fd8f0e85640 1 -- 172.21.15.150:0/3806821325 shutdown_connections 2024-03-17T23:49:43.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43.649+0000 7fd8f0e85640 1 -- 172.21.15.150:0/3806821325 wait complete. 2024-03-17T23:49:43.662 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-17T23:49:43.662 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-17T23:49:43.662 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi150.front.sepia.ceph.com 2024-03-17T23:49:43.662 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-17T23:49:43.662 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-17T23:49:43.662 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-17T23:49:43.662 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-17T23:49:43.662 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-17T23:49:43.662 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-17T23:49:43.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:43.663 DEBUG:teuthology.orchestra.run.smithi150:> ip addr 2024-03-17T23:49:43.678 INFO:teuthology.orchestra.run.smithi150.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-17T23:49:43.678 INFO:teuthology.orchestra.run.smithi150.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-17T23:49:43.678 INFO:teuthology.orchestra.run.smithi150.stdout: inet 127.0.0.1/8 scope host lo 2024-03-17T23:49:43.678 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:49:43.679 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 ::1/128 scope host 2024-03-17T23:49:43.679 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:49:43.679 INFO:teuthology.orchestra.run.smithi150.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:49:43.679 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bc brd ff:ff:ff:ff:ff:ff 2024-03-17T23:49:43.679 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f0 2024-03-17T23:49:43.679 INFO:teuthology.orchestra.run.smithi150.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:49:43.679 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bd brd ff:ff:ff:ff:ff:ff 2024-03-17T23:49:43.679 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f1 2024-03-17T23:49:43.679 INFO:teuthology.orchestra.run.smithi150.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-17T23:49:43.679 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:34 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:49:43.679 INFO:teuthology.orchestra.run.smithi150.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-17T23:49:43.679 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:35 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:49:43.679 INFO:teuthology.orchestra.run.smithi150.stdout: inet 172.21.15.150/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-17T23:49:43.679 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft 42271sec preferred_lft 42271sec 2024-03-17T23:49:43.679 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::ec4:7aff:fe88:6e35/64 scope link 2024-03-17T23:49:43.680 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:49:43.680 INFO:teuthology.orchestra.run.smithi150.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-17T23:49:43.680 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether f6:2d:aa:e3:d6:a3 brd ff:ff:ff:ff:ff:ff 2024-03-17T23:49:43.680 INFO:teuthology.orchestra.run.smithi150.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-17T23:49:43.680 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:49:43.680 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::bc2f:65ff:fe6d:abc3/64 scope link 2024-03-17T23:49:43.680 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:49:43.680 INFO:teuthology.orchestra.run.smithi150.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-17T23:49:43.680 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether f6:2d:aa:e3:d6:a3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:49:43.680 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::f42d:aaff:fee3:d6a3/64 scope link 2024-03-17T23:49:43.680 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:49:43.680 INFO:teuthology.orchestra.run.smithi150.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-17T23:49:43.680 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether fa:95:93:4e:f1:dd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:49:43.680 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::f895:93ff:fe4e:f1dd/64 scope link 2024-03-17T23:49:43.680 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-03-17T23:49:43.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:43.681 DEBUG:teuthology.orchestra.run.smithi150:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-17T23:49:43.744 INFO:teuthology.orchestra.run.smithi150.stdout:1 2024-03-17T23:49:43.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:43.745 DEBUG:teuthology.orchestra.run.smithi150:> route 2024-03-17T23:49:43.801 INFO:teuthology.orchestra.run.smithi150.stdout:Kernel IP routing table 2024-03-17T23:49:43.801 INFO:teuthology.orchestra.run.smithi150.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-17T23:49:43.801 INFO:teuthology.orchestra.run.smithi150.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-17T23:49:43.801 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-17T23:49:43.801 INFO:teuthology.orchestra.run.smithi150.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-17T23:49:43.802 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-17T23:49:43.802 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-03-17T23:49:43.802 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-17T23:49:43.802 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-17T23:49:43.802 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-17T23:49:43.802 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-03-17T23:49:43.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-17T23:49:43.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:49:43.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:43.936 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-03-17T23:49:43.951 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-17T23:49:43.951 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-17T23:49:43.951 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-17T23:49:43.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:43.951 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set brx.0 up 2024-03-17T23:49:44.018 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-17T23:49:44.019 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-17T23:49:44.019 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-17T23:49:44.074 INFO:teuthology.orchestra.run.smithi150.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-17T23:49:44.075 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-17T23:49:44.075 DEBUG:teuthology.orchestra.run.smithi150:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-17T23:49:44.130 DEBUG:teuthology.orchestra.run.smithi150:> sudo modprobe fuse 2024-03-17T23:49:44.198 DEBUG:teuthology.orchestra.run.smithi150:> cat /proc/self/mounts | awk '{print $2}' 2024-03-17T23:49:44.254 INFO:teuthology.orchestra.run.smithi150.stdout:/proc 2024-03-17T23:49:44.254 INFO:teuthology.orchestra.run.smithi150.stdout:/sys 2024-03-17T23:49:44.254 INFO:teuthology.orchestra.run.smithi150.stdout:/dev 2024-03-17T23:49:44.254 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/security 2024-03-17T23:49:44.254 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/shm 2024-03-17T23:49:44.254 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/pts 2024-03-17T23:49:44.255 INFO:teuthology.orchestra.run.smithi150.stdout:/run 2024-03-17T23:49:44.255 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/cgroup 2024-03-17T23:49:44.255 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/pstore 2024-03-17T23:49:44.255 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/bpf 2024-03-17T23:49:44.255 INFO:teuthology.orchestra.run.smithi150.stdout:/ 2024-03-17T23:49:44.255 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/selinux 2024-03-17T23:49:44.255 INFO:teuthology.orchestra.run.smithi150.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:49:44.255 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/hugepages 2024-03-17T23:49:44.255 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/mqueue 2024-03-17T23:49:44.255 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/debug 2024-03-17T23:49:44.255 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/tracing 2024-03-17T23:49:44.255 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-17T23:49:44.255 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/fuse/connections 2024-03-17T23:49:44.256 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/config 2024-03-17T23:49:44.256 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-sysctl.service 2024-03-17T23:49:44.256 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-17T23:49:44.256 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/nfs/rpc_pipefs 2024-03-17T23:49:44.256 INFO:teuthology.orchestra.run.smithi150.stdout:/run/user/1000 2024-03-17T23:49:44.256 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph 2024-03-17T23:49:44.256 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-0 2024-03-17T23:49:44.256 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-1 2024-03-17T23:49:44.256 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-2 2024-03-17T23:49:44.256 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-3 2024-03-17T23:49:44.256 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns 2024-03-17T23:49:44.256 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:49:44.256 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:49:44.257 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:49:44.257 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:49:44.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:44.257 DEBUG:teuthology.orchestra.run.smithi150:> ls /sys/fs/fuse/connections 2024-03-17T23:49:44.309 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-17T23:49:44.309 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-17T23:49:44.351 DEBUG:teuthology.orchestra.run.smithi150:> sudo modprobe fuse 2024-03-17T23:49:44.378 DEBUG:teuthology.orchestra.run.smithi150:> cat /proc/self/mounts | awk '{print $2}' 2024-03-17T23:49:44.434 INFO:teuthology.orchestra.run.smithi150.stdout:/proc 2024-03-17T23:49:44.435 INFO:teuthology.orchestra.run.smithi150.stdout:/sys 2024-03-17T23:49:44.435 INFO:teuthology.orchestra.run.smithi150.stdout:/dev 2024-03-17T23:49:44.435 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/security 2024-03-17T23:49:44.435 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/shm 2024-03-17T23:49:44.435 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/pts 2024-03-17T23:49:44.435 INFO:teuthology.orchestra.run.smithi150.stdout:/run 2024-03-17T23:49:44.435 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/cgroup 2024-03-17T23:49:44.435 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/pstore 2024-03-17T23:49:44.435 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/bpf 2024-03-17T23:49:44.435 INFO:teuthology.orchestra.run.smithi150.stdout:/ 2024-03-17T23:49:44.435 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/selinux 2024-03-17T23:49:44.435 INFO:teuthology.orchestra.run.smithi150.stdout:/proc/sys/fs/binfmt_misc 2024-03-17T23:49:44.435 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/hugepages 2024-03-17T23:49:44.435 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/mqueue 2024-03-17T23:49:44.436 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/debug 2024-03-17T23:49:44.436 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/tracing 2024-03-17T23:49:44.436 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-17T23:49:44.436 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/fuse/connections 2024-03-17T23:49:44.436 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/config 2024-03-17T23:49:44.436 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-sysctl.service 2024-03-17T23:49:44.436 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-17T23:49:44.436 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/nfs/rpc_pipefs 2024-03-17T23:49:44.436 INFO:teuthology.orchestra.run.smithi150.stdout:/run/user/1000 2024-03-17T23:49:44.436 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph 2024-03-17T23:49:44.436 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-0 2024-03-17T23:49:44.436 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-1 2024-03-17T23:49:44.436 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-2 2024-03-17T23:49:44.436 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-3 2024-03-17T23:49:44.437 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns 2024-03-17T23:49:44.437 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:49:44.437 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:49:44.437 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:49:44.437 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:49:44.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:44.437 DEBUG:teuthology.orchestra.run.smithi150:> ls /sys/fs/fuse/connections 2024-03-17T23:49:44.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:2024-03-17T23:49:44.438+0000 7f78bc838480 -1 init, newargv = 0x55593c0b0ee0 newargc=15 2024-03-17T23:49:44.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:ceph-fuse[62247]: starting ceph client 2024-03-17T23:49:44.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:ceph-fuse[62247]: starting fuse 2024-03-17T23:49:44.453 INFO:teuthology.orchestra.run.smithi150.stdout:43 2024-03-17T23:49:44.453 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-17T23:49:44.453 DEBUG:teuthology.orchestra.run.smithi150:> sudo stdin-killer -- python3 -c ' 2024-03-17T23:49:44.453 DEBUG:teuthology.orchestra.run.smithi150:> import glob 2024-03-17T23:49:44.453 DEBUG:teuthology.orchestra.run.smithi150:> import re 2024-03-17T23:49:44.453 DEBUG:teuthology.orchestra.run.smithi150:> import os 2024-03-17T23:49:44.453 DEBUG:teuthology.orchestra.run.smithi150:> import subprocess 2024-03-17T23:49:44.453 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:49:44.453 DEBUG:teuthology.orchestra.run.smithi150:> def _find_admin_socket(client_name): 2024-03-17T23:49:44.454 DEBUG:teuthology.orchestra.run.smithi150:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-17T23:49:44.454 DEBUG:teuthology.orchestra.run.smithi150:> files = glob.glob(asok_path) 2024-03-17T23:49:44.454 DEBUG:teuthology.orchestra.run.smithi150:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-17T23:49:44.454 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:49:44.454 DEBUG:teuthology.orchestra.run.smithi150:> # Given a non-glob path, it better be there 2024-03-17T23:49:44.454 DEBUG:teuthology.orchestra.run.smithi150:> if "*" not in asok_path: 2024-03-17T23:49:44.454 DEBUG:teuthology.orchestra.run.smithi150:> assert(len(files) == 1) 2024-03-17T23:49:44.454 DEBUG:teuthology.orchestra.run.smithi150:> return files[0] 2024-03-17T23:49:44.454 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:49:44.454 DEBUG:teuthology.orchestra.run.smithi150:> for f in files: 2024-03-17T23:49:44.454 DEBUG:teuthology.orchestra.run.smithi150:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-17T23:49:44.454 DEBUG:teuthology.orchestra.run.smithi150:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-17T23:49:44.454 DEBUG:teuthology.orchestra.run.smithi150:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-17T23:49:44.454 DEBUG:teuthology.orchestra.run.smithi150:> contents = proc_f.read() 2024-03-17T23:49:44.455 DEBUG:teuthology.orchestra.run.smithi150:> if mountpoint in contents: 2024-03-17T23:49:44.455 DEBUG:teuthology.orchestra.run.smithi150:> return f 2024-03-17T23:49:44.455 DEBUG:teuthology.orchestra.run.smithi150:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-17T23:49:44.455 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:49:44.455 DEBUG:teuthology.orchestra.run.smithi150:> print(_find_admin_socket("client.0")) 2024-03-17T23:49:44.455 DEBUG:teuthology.orchestra.run.smithi150:> ' 2024-03-17T23:49:44.574 INFO:teuthology.orchestra.run.smithi150.stdout:/var/run/ceph/ceph-client.0.62247.asok 2024-03-17T23:49:44.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-17T23:49:44.584 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.62247.asok 2024-03-17T23:49:44.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:44.585 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.62247.asok status 2024-03-17T23:49:44.722 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-17T23:49:44.722 INFO:teuthology.orchestra.run.smithi150.stdout: "metadata": { 2024-03-17T23:49:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-17T23:49:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-17T23:49:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: "entity_id": "0", 2024-03-17T23:49:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: "hostname": "smithi150", 2024-03-17T23:49:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-17T23:49:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: "pid": "62247", 2024-03-17T23:49:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: "root": "/" 2024-03-17T23:49:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: "dentry_count": 0, 2024-03-17T23:49:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: "dentry_pinned_count": 0, 2024-03-17T23:49:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 5804, 2024-03-17T23:49:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: "inst": { 2024-03-17T23:49:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: "name": { 2024-03-17T23:49:44.723 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "client", 2024-03-17T23:49:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: "num": 5804 2024-03-17T23:49:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": { 2024-03-17T23:49:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v1", 2024-03-17T23:49:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:0", 2024-03-17T23:49:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1423561143 2024-03-17T23:49:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-03-17T23:49:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": { 2024-03-17T23:49:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v1", 2024-03-17T23:49:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "172.21.15.150:0", 2024-03-17T23:49:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 1423561143 2024-03-17T23:49:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-03-17T23:49:44.724 INFO:teuthology.orchestra.run.smithi150.stdout: "inst_str": "client.5804 172.21.15.150:0/1423561143", 2024-03-17T23:49:44.725 INFO:teuthology.orchestra.run.smithi150.stdout: "addr_str": "172.21.15.150:0/1423561143", 2024-03-17T23:49:44.725 INFO:teuthology.orchestra.run.smithi150.stdout: "inode_count": 1, 2024-03-17T23:49:44.725 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_epoch": 22, 2024-03-17T23:49:44.725 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_epoch": 58, 2024-03-17T23:49:44.725 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_epoch_barrier": 0, 2024-03-17T23:49:44.725 INFO:teuthology.orchestra.run.smithi150.stdout: "blocklisted": false, 2024-03-17T23:49:44.725 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "cephfs" 2024-03-17T23:49:44.725 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-17T23:49:44.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:44.731 DEBUG:teuthology.orchestra.run.smithi150:> stat --file-system '--printf=%T 2024-03-17T23:49:44.731 DEBUG:teuthology.orchestra.run.smithi150:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-17T23:49:44.747 INFO:teuthology.orchestra.run.smithi150.stdout:fuseblk 2024-03-17T23:49:44.747 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-17T23:49:44.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:44.748 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-17T23:49:44.816 DEBUG:teuthology.orchestra.run.smithi150:> stdin-killer -- python3 -c ' 2024-03-17T23:49:44.816 DEBUG:teuthology.orchestra.run.smithi150:> import os 2024-03-17T23:49:44.816 DEBUG:teuthology.orchestra.run.smithi150:> import errno 2024-03-17T23:49:44.816 DEBUG:teuthology.orchestra.run.smithi150:> 2024-03-17T23:49:44.816 DEBUG:teuthology.orchestra.run.smithi150:> fd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR) 2024-03-17T23:49:44.817 DEBUG:teuthology.orchestra.run.smithi150:> try: 2024-03-17T23:49:44.817 DEBUG:teuthology.orchestra.run.smithi150:> if True: 2024-03-17T23:49:44.817 DEBUG:teuthology.orchestra.run.smithi150:> ret = os.read(fd, 1024) 2024-03-17T23:49:44.817 DEBUG:teuthology.orchestra.run.smithi150:> else: 2024-03-17T23:49:44.817 DEBUG:teuthology.orchestra.run.smithi150:> os.write(fd, b'"'"'content'"'"') 2024-03-17T23:49:44.817 DEBUG:teuthology.orchestra.run.smithi150:> except OSError as e: 2024-03-17T23:49:44.817 DEBUG:teuthology.orchestra.run.smithi150:> if e.errno != errno.EPERM: 2024-03-17T23:49:44.817 DEBUG:teuthology.orchestra.run.smithi150:> raise 2024-03-17T23:49:44.817 DEBUG:teuthology.orchestra.run.smithi150:> else: 2024-03-17T23:49:44.817 DEBUG:teuthology.orchestra.run.smithi150:> raise RuntimeError("client does not check permission of data pool") 2024-03-17T23:49:44.817 DEBUG:teuthology.orchestra.run.smithi150:> ' 2024-03-17T23:49:44.923 INFO:teuthology.orchestra.run.smithi150.stderr:Traceback (most recent call last): 2024-03-17T23:49:44.923 INFO:teuthology.orchestra.run.smithi150.stderr: File "", line 15, in 2024-03-17T23:49:44.923 INFO:teuthology.orchestra.run.smithi150.stderr:RuntimeError: client does not check permission of data pool 2024-03-17T23:49:44.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:44 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-17T23:49:44.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-17T23:49:44.932 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2024-03-17T23:49:45.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.078+0000 7f8d0f4ac640 1 Processor -- start 2024-03-17T23:49:45.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.079+0000 7f8d0f4ac640 1 -- start start 2024-03-17T23:49:45.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.079+0000 7f8d0f4ac640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8d080d0ce0 0x7f8d080b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:45.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.079+0000 7f8d0f4ac640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d08161d30 con 0x7f8d08161520 2024-03-17T23:49:45.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.079+0000 7f8d0f4ac640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d08162010 con 0x7f8d080b2bf0 2024-03-17T23:49:45.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.079+0000 7f8d0f4ac640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f8d080b9bf0 con 0x7f8d080d0ce0 2024-03-17T23:49:45.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.080+0000 7f8d0ca47640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8d080d0ce0 0x7f8d080b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:45.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.080+0000 7f8d0d248640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f8d08161520 0x7f8d080d05d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41784/0 (socket says 172.21.15.150:41784) 2024-03-17T23:49:45.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.080+0000 7f8d0d248640 1 -- 172.21.15.150:0/2014577909 learned_addr learned my addr 172.21.15.150:0/2014577909 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:45.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.080+0000 7f8d0ca47640 1 -- 172.21.15.150:0/2014577909 >> v1:172.21.15.157:6789/0 conn(0x7f8d080b2bf0 legacy=0x7f8d080d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:45.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.080+0000 7f8cfffff640 1 -- 172.21.15.150:0/2014577909 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1528475843 0 0) 0x7f8d08162010 con 0x7f8d080b2bf0 2024-03-17T23:49:45.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.080+0000 7f8d0ca47640 1 -- 172.21.15.150:0/2014577909 >> v1:172.21.15.150:6789/0 conn(0x7f8d08161520 legacy=0x7f8d080d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:45.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.080+0000 7f8d0ca47640 1 -- 172.21.15.150:0/2014577909 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d081622f0 con 0x7f8d080d0ce0 2024-03-17T23:49:45.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.081+0000 7f8d0ca47640 1 --2- 172.21.15.150:0/2014577909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d080d0ce0 0x7f8d080b96b0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f8cf80097a0 tx=0x7f8cf802b680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f01665badcb5af server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:45.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.081+0000 7f8cfffff640 1 -- 172.21.15.150:0/2014577909 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf8037070 con 0x7f8d080d0ce0 2024-03-17T23:49:45.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.081+0000 7f8cfffff640 1 -- 172.21.15.150:0/2014577909 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cf8030710 con 0x7f8d080d0ce0 2024-03-17T23:49:45.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.081+0000 7f8cfffff640 1 -- 172.21.15.150:0/2014577909 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf8030ae0 con 0x7f8d080d0ce0 2024-03-17T23:49:45.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.081+0000 7f8d0f4ac640 1 -- 172.21.15.150:0/2014577909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d080d0ce0 msgr2=0x7f8d080b96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:45.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.081+0000 7f8d0f4ac640 1 --2- 172.21.15.150:0/2014577909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d080d0ce0 0x7f8d080b96b0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f8cf80097a0 tx=0x7f8cf802b680 comp rx=0 tx=0).stop 2024-03-17T23:49:45.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.081+0000 7f8d0f4ac640 1 -- 172.21.15.150:0/2014577909 shutdown_connections 2024-03-17T23:49:45.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.081+0000 7f8d0f4ac640 1 --2- 172.21.15.150:0/2014577909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d080d0ce0 0x7f8d080b96b0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:45.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.081+0000 7f8d0f4ac640 1 -- 172.21.15.150:0/2014577909 >> 172.21.15.150:0/2014577909 conn(0x7f8d0806ca00 msgr2=0x7f8d0806ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:45.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.081+0000 7f8d0f4ac640 1 -- 172.21.15.150:0/2014577909 shutdown_connections 2024-03-17T23:49:45.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.081+0000 7f8d0f4ac640 1 -- 172.21.15.150:0/2014577909 wait complete. 2024-03-17T23:49:45.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.083+0000 7f8d0f4ac640 1 Processor -- start 2024-03-17T23:49:45.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.083+0000 7f8d0f4ac640 1 -- start start 2024-03-17T23:49:45.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.083+0000 7f8d0f4ac640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d080b2bf0 0x7f8d081518b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:45.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.083+0000 7f8d0f4ac640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d080d0ce0 0x7f8d080c9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:45.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.083+0000 7f8d0f4ac640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d08161520 0x7f8d080c9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:45.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.083+0000 7f8d0f4ac640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8d08161900 con 0x7f8d08161520 2024-03-17T23:49:45.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.083+0000 7f8d0f4ac640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f8d08162940 con 0x7f8d080d0ce0 2024-03-17T23:49:45.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.083+0000 7f8d0f4ac640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f8d080d26b0 con 0x7f8d080b2bf0 2024-03-17T23:49:45.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.083+0000 7f8d0da49640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d08161520 0x7f8d080c9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:45.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.084+0000 7f8d0da49640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d08161520 0x7f8d080c9ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60990/0 (socket says 172.21.15.150:60990) 2024-03-17T23:49:45.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.084+0000 7f8d0da49640 1 -- 172.21.15.150:0/989749679 learned_addr learned my addr 172.21.15.150:0/989749679 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:45.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.084+0000 7f8d0d248640 1 --2- 172.21.15.150:0/989749679 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d080b2bf0 0x7f8d081518b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:45.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.084+0000 7f8d0ca47640 1 --2- 172.21.15.150:0/989749679 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d080d0ce0 0x7f8d080c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:45.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.084+0000 7f8d0da49640 1 -- 172.21.15.150:0/989749679 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d080b2bf0 msgr2=0x7f8d081518b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:45.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.084+0000 7f8d0da49640 1 --2- 172.21.15.150:0/989749679 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d080b2bf0 0x7f8d081518b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:45.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.084+0000 7f8d0da49640 1 -- 172.21.15.150:0/989749679 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d080d0ce0 msgr2=0x7f8d080c9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:45.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.084+0000 7f8d0da49640 1 --2- 172.21.15.150:0/989749679 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d080d0ce0 0x7f8d080c9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:45.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.084+0000 7f8d0da49640 1 -- 172.21.15.150:0/989749679 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d080caf90 con 0x7f8d08161520 2024-03-17T23:49:45.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.084+0000 7f8d0da49640 1 --2- 172.21.15.150:0/989749679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d08161520 0x7f8d080c9ac0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f8d0000c760 tx=0x7f8d0000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:45.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.084+0000 7f8cfdffb640 1 -- 172.21.15.150:0/989749679 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d00003da0 con 0x7f8d08161520 2024-03-17T23:49:45.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.084+0000 7f8cfdffb640 1 -- 172.21.15.150:0/989749679 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d00005070 con 0x7f8d08161520 2024-03-17T23:49:45.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.084+0000 7f8d0f4ac640 1 -- 172.21.15.150:0/989749679 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d0815d610 con 0x7f8d08161520 2024-03-17T23:49:45.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.084+0000 7f8cfdffb640 1 -- 172.21.15.150:0/989749679 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d00005460 con 0x7f8d08161520 2024-03-17T23:49:45.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.084+0000 7f8d0f4ac640 1 -- 172.21.15.150:0/989749679 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d080cb510 con 0x7f8d08161520 2024-03-17T23:49:45.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.085+0000 7f8d0f4ac640 1 -- 172.21.15.150:0/989749679 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d080681a0 con 0x7f8d08161520 2024-03-17T23:49:45.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.086+0000 7f8cfdffb640 1 -- 172.21.15.150:0/989749679 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8d00005600 con 0x7f8d08161520 2024-03-17T23:49:45.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.086+0000 7f8cfdffb640 1 --2- 172.21.15.150:0/989749679 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8cec06bbd0 0x7f8cec06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:45.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.086+0000 7f8cfdffb640 1 -- 172.21.15.150:0/989749679 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f8d0008a9d0 con 0x7f8d08161520 2024-03-17T23:49:45.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.086+0000 7f8d0d248640 1 --2- 172.21.15.150:0/989749679 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8cec06bbd0 0x7f8cec06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:45.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.087+0000 7f8d0d248640 1 --2- 172.21.15.150:0/989749679 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8cec06bbd0 0x7f8cec06e090 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8d04000f60 tx=0x7f8d04007e20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:45.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.090+0000 7f8cfdffb640 1 -- 172.21.15.150:0/989749679 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8d00058ce0 con 0x7f8d08161520 2024-03-17T23:49:45.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.327+0000 7f8d0f4ac640 1 -- 172.21.15.150:0/989749679 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) -- 0x7f8d080d19c0 con 0x7f8d08161520 2024-03-17T23:49:45.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.329+0000 7f8cfdffb640 1 -- 172.21.15.150:0/989749679 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v20) ==== 184+0+141 (secure 0 0 0) 0x7f8d0005cd80 con 0x7f8d08161520 2024-03-17T23:49:45.329 INFO:teuthology.orchestra.run.smithi150.stdout:[client.0] 2024-03-17T23:49:45.330 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQC3gPdlvlPJNRAAkUfetPJK4hZQG5rNiSDozw== 2024-03-17T23:49:45.330 INFO:teuthology.orchestra.run.smithi150.stdout: caps mds = "allow" 2024-03-17T23:49:45.330 INFO:teuthology.orchestra.run.smithi150.stdout: caps mon = "allow r" 2024-03-17T23:49:45.330 INFO:teuthology.orchestra.run.smithi150.stdout: caps osd = "allow rw pool=cephfs_data" 2024-03-17T23:49:45.330 INFO:teuthology.orchestra.run.smithi150.stderr:updated caps for client.0 2024-03-17T23:49:45.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.331+0000 7f8d0f4ac640 1 -- 172.21.15.150:0/989749679 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8cec06bbd0 msgr2=0x7f8cec06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:45.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.331+0000 7f8d0f4ac640 1 --2- 172.21.15.150:0/989749679 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8cec06bbd0 0x7f8cec06e090 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8d04000f60 tx=0x7f8d04007e20 comp rx=0 tx=0).stop 2024-03-17T23:49:45.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.331+0000 7f8d0f4ac640 1 -- 172.21.15.150:0/989749679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d08161520 msgr2=0x7f8d080c9ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:45.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.331+0000 7f8d0f4ac640 1 --2- 172.21.15.150:0/989749679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d08161520 0x7f8d080c9ac0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f8d0000c760 tx=0x7f8d0000cc20 comp rx=0 tx=0).stop 2024-03-17T23:49:45.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.331+0000 7f8d0f4ac640 1 -- 172.21.15.150:0/989749679 shutdown_connections 2024-03-17T23:49:45.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.331+0000 7f8d0f4ac640 1 --2- 172.21.15.150:0/989749679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8d08161520 0x7f8d080c9ac0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:45.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.331+0000 7f8d0f4ac640 1 --2- 172.21.15.150:0/989749679 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f8cec06bbd0 0x7f8cec06e090 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:45.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.331+0000 7f8d0f4ac640 1 --2- 172.21.15.150:0/989749679 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d080d0ce0 0x7f8d080c9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:45.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.331+0000 7f8d0f4ac640 1 --2- 172.21.15.150:0/989749679 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f8d080b2bf0 0x7f8d081518b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:45.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.331+0000 7f8d0f4ac640 1 -- 172.21.15.150:0/989749679 >> 172.21.15.150:0/989749679 conn(0x7f8d0806ca00 msgr2=0x7f8d080ce7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:45.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.331+0000 7f8d0f4ac640 1 -- 172.21.15.150:0/989749679 shutdown_connections 2024-03-17T23:49:45.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.331+0000 7f8d0f4ac640 1 -- 172.21.15.150:0/989749679 wait complete. 2024-03-17T23:49:45.343 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-17T23:49:45.343 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-17T23:49:45.384 DEBUG:teuthology.orchestra.run.smithi157:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-17T23:49:45.434 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-17T23:49:45.434 INFO:tasks.cephfs.mount:Terminating background process 2024-03-17T23:49:45.434 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:49:45.434 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:49:45.450 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:49:45.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.589+0000 7f3a69ff4640 1 Processor -- start 2024-03-17T23:49:45.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.589+0000 7f3a69ff4640 1 -- start start 2024-03-17T23:49:45.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.589+0000 7f3a69ff4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a640cf120 0x7f3a641619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:45.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.589+0000 7f3a69ff4640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3a640b9860 0x7f3a640d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:45.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.589+0000 7f3a69ff4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a640b9cb0 con 0x7f3a6415de30 2024-03-17T23:49:45.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.589+0000 7f3a69ff4640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3a64162940 con 0x7f3a640b9860 2024-03-17T23:49:45.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.589+0000 7f3a69ff4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3a641620a0 con 0x7f3a640cf120 2024-03-17T23:49:45.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.590+0000 7f3a5affd640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a640cf120 0x7f3a641619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:45.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.590+0000 7f3a63fff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3a640b9860 0x7f3a640d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:45.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.590+0000 7f3a637fe640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f3a6415de30 0x7f3a640cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:56882/0 (socket says 172.21.15.150:56882) 2024-03-17T23:49:45.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.590+0000 7f3a637fe640 1 -- 172.21.15.150:0/4236418552 learned_addr learned my addr 172.21.15.150:0/4236418552 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:45.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.590+0000 7f3a5affd640 1 -- 172.21.15.150:0/4236418552 >> v2:172.21.15.157:3300/0 conn(0x7f3a640b9860 msgr2=0x7f3a640d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:45.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.590+0000 7f3a5affd640 1 --2- 172.21.15.150:0/4236418552 >> v2:172.21.15.157:3300/0 conn(0x7f3a640b9860 0x7f3a640d2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:45.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.590+0000 7f3a5affd640 1 -- 172.21.15.150:0/4236418552 >> v1:172.21.15.157:6789/0 conn(0x7f3a6415de30 legacy=0x7f3a640cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:45.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.590+0000 7f3a62ffd640 1 -- 172.21.15.150:0/4236418552 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2125505215 0 0) 0x7f3a640b9cb0 con 0x7f3a6415de30 2024-03-17T23:49:45.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.590+0000 7f3a5affd640 1 -- 172.21.15.150:0/4236418552 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a64162380 con 0x7f3a640cf120 2024-03-17T23:49:45.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.590+0000 7f3a63fff640 1 --2- 172.21.15.150:0/4236418552 >> v2:172.21.15.157:3300/0 conn(0x7f3a640b9860 0x7f3a640d2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:45.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.591+0000 7f3a5affd640 1 --2- 172.21.15.150:0/4236418552 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a640cf120 0x7f3a641619a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c0097a0 tx=0x7f3a4c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb2cbd41ffcb4edb server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:45.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.591+0000 7f3a62ffd640 1 -- 172.21.15.150:0/4236418552 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c0315a0 con 0x7f3a640cf120 2024-03-17T23:49:45.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.591+0000 7f3a62ffd640 1 -- 172.21.15.150:0/4236418552 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a4c037070 con 0x7f3a640cf120 2024-03-17T23:49:45.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.591+0000 7f3a62ffd640 1 -- 172.21.15.150:0/4236418552 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c031960 con 0x7f3a640cf120 2024-03-17T23:49:45.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.591+0000 7f3a69ff4640 1 -- 172.21.15.150:0/4236418552 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a640cf120 msgr2=0x7f3a641619a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:45.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.591+0000 7f3a69ff4640 1 --2- 172.21.15.150:0/4236418552 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a640cf120 0x7f3a641619a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c0097a0 tx=0x7f3a4c02b840 comp rx=0 tx=0).stop 2024-03-17T23:49:45.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.591+0000 7f3a69ff4640 1 -- 172.21.15.150:0/4236418552 shutdown_connections 2024-03-17T23:49:45.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.591+0000 7f3a69ff4640 1 --2- 172.21.15.150:0/4236418552 >> v2:172.21.15.157:3300/0 conn(0x7f3a640b9860 0x7f3a640d2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:45.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.591+0000 7f3a69ff4640 1 --2- 172.21.15.150:0/4236418552 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a640cf120 0x7f3a641619a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:45.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.591+0000 7f3a69ff4640 1 -- 172.21.15.150:0/4236418552 >> 172.21.15.150:0/4236418552 conn(0x7f3a640730b0 msgr2=0x7f3a640d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:45.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.592+0000 7f3a69ff4640 1 -- 172.21.15.150:0/4236418552 shutdown_connections 2024-03-17T23:49:45.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.592+0000 7f3a69ff4640 1 -- 172.21.15.150:0/4236418552 wait complete. 2024-03-17T23:49:45.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.593+0000 7f3a69ff4640 1 Processor -- start 2024-03-17T23:49:45.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.593+0000 7f3a69ff4640 1 -- start start 2024-03-17T23:49:45.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.593+0000 7f3a69ff4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a640b9860 0x7f3a640cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:45.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.593+0000 7f3a69ff4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a640cf120 0x7f3a640acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:45.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.593+0000 7f3a69ff4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a6415de30 0x7f3a641f3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:45.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.594+0000 7f3a69ff4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3a6415e210 con 0x7f3a6415de30 2024-03-17T23:49:45.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.594+0000 7f3a69ff4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3a640d2170 con 0x7f3a640b9860 2024-03-17T23:49:45.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.594+0000 7f3a69ff4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3a640d2bd0 con 0x7f3a640cf120 2024-03-17T23:49:45.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.594+0000 7f3a63fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a6415de30 0x7f3a641f3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:45.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.594+0000 7f3a637fe640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a640b9860 0x7f3a640cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:45.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.594+0000 7f3a63fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a6415de30 0x7f3a641f3880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60384/0 (socket says 172.21.15.150:60384) 2024-03-17T23:49:45.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.594+0000 7f3a63fff640 1 -- 172.21.15.150:0/1867692857 learned_addr learned my addr 172.21.15.150:0/1867692857 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:45.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.594+0000 7f3a5affd640 1 --2- 172.21.15.150:0/1867692857 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a640cf120 0x7f3a640acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:45.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.594+0000 7f3a637fe640 1 -- 172.21.15.150:0/1867692857 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a640cf120 msgr2=0x7f3a640acdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:45.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.594+0000 7f3a637fe640 1 --2- 172.21.15.150:0/1867692857 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a640cf120 0x7f3a640acdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:45.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.594+0000 7f3a637fe640 1 -- 172.21.15.150:0/1867692857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a6415de30 msgr2=0x7f3a641f3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:45.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.594+0000 7f3a637fe640 1 --2- 172.21.15.150:0/1867692857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a6415de30 0x7f3a641f3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:45.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.594+0000 7f3a637fe640 1 -- 172.21.15.150:0/1867692857 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a640ccdf0 con 0x7f3a640b9860 2024-03-17T23:49:45.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.594+0000 7f3a637fe640 1 --2- 172.21.15.150:0/1867692857 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a640b9860 0x7f3a640cc8b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3a5400c410 tx=0x7f3a5400c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:45.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.594+0000 7f3a60ff9640 1 -- 172.21.15.150:0/1867692857 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a540074e0 con 0x7f3a640b9860 2024-03-17T23:49:45.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.594+0000 7f3a60ff9640 1 -- 172.21.15.150:0/1867692857 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a540047e0 con 0x7f3a640b9860 2024-03-17T23:49:45.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.594+0000 7f3a69ff4640 1 -- 172.21.15.150:0/1867692857 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a641f4340 con 0x7f3a640b9860 2024-03-17T23:49:45.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.594+0000 7f3a60ff9640 1 -- 172.21.15.150:0/1867692857 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a54004bd0 con 0x7f3a640b9860 2024-03-17T23:49:45.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.594+0000 7f3a69ff4640 1 -- 172.21.15.150:0/1867692857 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a640cc0b0 con 0x7f3a640b9860 2024-03-17T23:49:45.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.596+0000 7f3a69ff4640 1 -- 172.21.15.150:0/1867692857 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a640681a0 con 0x7f3a640b9860 2024-03-17T23:49:45.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.596+0000 7f3a60ff9640 1 -- 172.21.15.150:0/1867692857 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3a54011070 con 0x7f3a640b9860 2024-03-17T23:49:45.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.596+0000 7f3a60ff9640 1 --2- 172.21.15.150:0/1867692857 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3a4406bbd0 0x7f3a4406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:45.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.596+0000 7f3a60ff9640 1 -- 172.21.15.150:0/1867692857 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f3a5408a590 con 0x7f3a640b9860 2024-03-17T23:49:45.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.596+0000 7f3a5affd640 1 --2- 172.21.15.150:0/1867692857 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3a4406bbd0 0x7f3a4406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:45.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.597+0000 7f3a5affd640 1 --2- 172.21.15.150:0/1867692857 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3a4406bbd0 0x7f3a4406e090 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c036000 tx=0x7f3a4c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:45.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.601+0000 7f3a60ff9640 1 -- 172.21.15.150:0/1867692857 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3a540589f0 con 0x7f3a640b9860 2024-03-17T23:49:45.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.778+0000 7f3a69ff4640 1 -- 172.21.15.150:0/1867692857 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3a640d1910 con 0x7f3a640b9860 2024-03-17T23:49:45.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.779+0000 7f3a60ff9640 1 -- 172.21.15.150:0/1867692857 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) ==== 74+0+16974 (secure 0 0 0) 0x7f3a5405ca90 con 0x7f3a640b9860 2024-03-17T23:49:45.779 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:45.780 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":58,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:49:12.646307+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:49:09.223630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-17T23:49:09.956422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:49:45.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.781+0000 7f3a69ff4640 1 -- 172.21.15.150:0/1867692857 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3a4406bbd0 msgr2=0x7f3a4406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:45.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.781+0000 7f3a69ff4640 1 --2- 172.21.15.150:0/1867692857 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3a4406bbd0 0x7f3a4406e090 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c036000 tx=0x7f3a4c008030 comp rx=0 tx=0).stop 2024-03-17T23:49:45.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.781+0000 7f3a69ff4640 1 -- 172.21.15.150:0/1867692857 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a640b9860 msgr2=0x7f3a640cc8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:45.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.781+0000 7f3a69ff4640 1 --2- 172.21.15.150:0/1867692857 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a640b9860 0x7f3a640cc8b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3a5400c410 tx=0x7f3a5400c8d0 comp rx=0 tx=0).stop 2024-03-17T23:49:45.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.781+0000 7f3a69ff4640 1 -- 172.21.15.150:0/1867692857 shutdown_connections 2024-03-17T23:49:45.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.781+0000 7f3a69ff4640 1 --2- 172.21.15.150:0/1867692857 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a6415de30 0x7f3a641f3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:45.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.781+0000 7f3a69ff4640 1 --2- 172.21.15.150:0/1867692857 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3a640cf120 0x7f3a640acdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:45.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.781+0000 7f3a69ff4640 1 --2- 172.21.15.150:0/1867692857 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3a4406bbd0 0x7f3a4406e090 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:45.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.781+0000 7f3a69ff4640 1 --2- 172.21.15.150:0/1867692857 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a640b9860 0x7f3a640cc8b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:45.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.781+0000 7f3a69ff4640 1 -- 172.21.15.150:0/1867692857 >> 172.21.15.150:0/1867692857 conn(0x7f3a640730b0 msgr2=0x7f3a640b6e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:45.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.781+0000 7f3a69ff4640 1 -- 172.21.15.150:0/1867692857 shutdown_connections 2024-03-17T23:49:45.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.781+0000 7f3a69ff4640 1 -- 172.21.15.150:0/1867692857 wait complete. 2024-03-17T23:49:45.792 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:49:09.223630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:49:09.956422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-17T23:49:45.793 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:49:45.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.896+0000 7f06e0085640 1 Processor -- start 2024-03-17T23:49:45.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.896+0000 7f06e0085640 1 -- start start 2024-03-17T23:49:45.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.897+0000 7f06e0085640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f06d815bfd0 0x7f06d80ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:45.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.897+0000 7f06e0085640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06d80cd250 0x7f06d815ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:45.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.897+0000 7f06e0085640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06d80cf3b0 con 0x7f06d8155320 2024-03-17T23:49:45.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.897+0000 7f06e0085640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f06d815c550 con 0x7f06d815bfd0 2024-03-17T23:49:45.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.897+0000 7f06e0085640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f06d80cf690 con 0x7f06d80cd250 2024-03-17T23:49:45.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.897+0000 7f06dd620640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f06d815bfd0 0x7f06d80ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:45.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.897+0000 7f06dde21640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f06d8155320 0x7f06d806dd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41828/0 (socket says 172.21.15.150:41828) 2024-03-17T23:49:45.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.897+0000 7f06dde21640 1 -- 172.21.15.150:0/760291750 learned_addr learned my addr 172.21.15.150:0/760291750 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:45.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.897+0000 7f06de622640 1 --2- 172.21.15.150:0/760291750 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06d80cd250 0x7f06d815ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:45.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.897+0000 7f06dd620640 1 -- 172.21.15.150:0/760291750 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06d80cd250 msgr2=0x7f06d815ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:45.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.897+0000 7f06dd620640 1 --2- 172.21.15.150:0/760291750 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06d80cd250 0x7f06d815ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:45.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.897+0000 7f06dd620640 1 -- 172.21.15.150:0/760291750 >> v1:172.21.15.150:6789/0 conn(0x7f06d8155320 legacy=0x7f06d806dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:45.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.897+0000 7f06dce1f640 1 -- 172.21.15.150:0/760291750 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1445796814 0 0) 0x7f06d80cf3b0 con 0x7f06d8155320 2024-03-17T23:49:45.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.897+0000 7f06dd620640 1 -- 172.21.15.150:0/760291750 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d815c7a0 con 0x7f06d815bfd0 2024-03-17T23:49:45.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.898+0000 7f06dd620640 1 --2- 172.21.15.150:0/760291750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06d815bfd0 0x7f06d80ccd10 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f06c80097a0 tx=0x7f06c802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=654ee5ee2878f0f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:45.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.898+0000 7f06dce1f640 1 -- 172.21.15.150:0/760291750 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c8035020 con 0x7f06d815bfd0 2024-03-17T23:49:45.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.898+0000 7f06dce1f640 1 -- 172.21.15.150:0/760291750 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06c8031830 con 0x7f06d815bfd0 2024-03-17T23:49:45.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.898+0000 7f06dce1f640 1 -- 172.21.15.150:0/760291750 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c8031c00 con 0x7f06d815bfd0 2024-03-17T23:49:45.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.898+0000 7f06e0085640 1 -- 172.21.15.150:0/760291750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06d815bfd0 msgr2=0x7f06d80ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:45.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.898+0000 7f06e0085640 1 --2- 172.21.15.150:0/760291750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06d815bfd0 0x7f06d80ccd10 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f06c80097a0 tx=0x7f06c802b8d0 comp rx=0 tx=0).stop 2024-03-17T23:49:45.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.899+0000 7f06e0085640 1 -- 172.21.15.150:0/760291750 shutdown_connections 2024-03-17T23:49:45.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.899+0000 7f06e0085640 1 --2- 172.21.15.150:0/760291750 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06d80cd250 0x7f06d815ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:45.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.899+0000 7f06e0085640 1 --2- 172.21.15.150:0/760291750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06d815bfd0 0x7f06d80ccd10 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:45.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.899+0000 7f06e0085640 1 -- 172.21.15.150:0/760291750 >> 172.21.15.150:0/760291750 conn(0x7f06d806b190 msgr2=0x7f06d806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:45.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.899+0000 7f06e0085640 1 -- 172.21.15.150:0/760291750 shutdown_connections 2024-03-17T23:49:45.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.899+0000 7f06e0085640 1 -- 172.21.15.150:0/760291750 wait complete. 2024-03-17T23:49:45.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.900+0000 7f06e0085640 1 Processor -- start 2024-03-17T23:49:45.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.900+0000 7f06e0085640 1 -- start start 2024-03-17T23:49:45.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.900+0000 7f06e0085640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06d80cd250 0x7f06d814cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:45.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.900+0000 7f06e0085640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06d8155320 0x7f06d8159de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:45.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.901+0000 7f06e0085640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06d815bfd0 0x7f06d815a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:45.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.901+0000 7f06e0085640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f06d806e9b0 con 0x7f06d80cd250 2024-03-17T23:49:45.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.901+0000 7f06e0085640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f06d806e830 con 0x7f06d8155320 2024-03-17T23:49:45.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.901+0000 7f06e0085640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f06d806eb30 con 0x7f06d815bfd0 2024-03-17T23:49:45.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.901+0000 7f06dde21640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06d80cd250 0x7f06d814cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:45.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.901+0000 7f06dde21640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06d80cd250 0x7f06d814cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60404/0 (socket says 172.21.15.150:60404) 2024-03-17T23:49:45.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.901+0000 7f06dde21640 1 -- 172.21.15.150:0/4124301952 learned_addr learned my addr 172.21.15.150:0/4124301952 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:45.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.901+0000 7f06de622640 1 --2- 172.21.15.150:0/4124301952 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06d815bfd0 0x7f06d815a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:45.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.901+0000 7f06dd620640 1 --2- 172.21.15.150:0/4124301952 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06d8155320 0x7f06d8159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:45.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.901+0000 7f06dde21640 1 -- 172.21.15.150:0/4124301952 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06d815bfd0 msgr2=0x7f06d815a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:45.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.901+0000 7f06dde21640 1 --2- 172.21.15.150:0/4124301952 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06d815bfd0 0x7f06d815a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:45.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.901+0000 7f06dde21640 1 -- 172.21.15.150:0/4124301952 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06d8155320 msgr2=0x7f06d8159de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:45.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.901+0000 7f06dde21640 1 --2- 172.21.15.150:0/4124301952 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06d8155320 0x7f06d8159de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:45.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.901+0000 7f06dde21640 1 -- 172.21.15.150:0/4124301952 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d81f3c00 con 0x7f06d80cd250 2024-03-17T23:49:45.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.901+0000 7f06dde21640 1 --2- 172.21.15.150:0/4124301952 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06d80cd250 0x7f06d814cb50 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f06d400c4d0 tx=0x7f06d400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:45.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.901+0000 7f06ce7fc640 1 -- 172.21.15.150:0/4124301952 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d4018070 con 0x7f06d80cd250 2024-03-17T23:49:45.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.901+0000 7f06ce7fc640 1 -- 172.21.15.150:0/4124301952 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06d4007500 con 0x7f06d80cd250 2024-03-17T23:49:45.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.901+0000 7f06e0085640 1 -- 172.21.15.150:0/4124301952 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06d8150b20 con 0x7f06d80cd250 2024-03-17T23:49:45.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.901+0000 7f06ce7fc640 1 -- 172.21.15.150:0/4124301952 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d4004a20 con 0x7f06d80cd250 2024-03-17T23:49:45.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.901+0000 7f06e0085640 1 -- 172.21.15.150:0/4124301952 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06d81510e0 con 0x7f06d80cd250 2024-03-17T23:49:45.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.902+0000 7f06e0085640 1 -- 172.21.15.150:0/4124301952 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06d80681a0 con 0x7f06d80cd250 2024-03-17T23:49:45.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.903+0000 7f06ce7fc640 1 -- 172.21.15.150:0/4124301952 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f06d401b070 con 0x7f06d80cd250 2024-03-17T23:49:45.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.903+0000 7f06ce7fc640 1 --2- 172.21.15.150:0/4124301952 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f06bc06bbd0 0x7f06bc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:45.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.903+0000 7f06ce7fc640 1 -- 172.21.15.150:0/4124301952 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f06d4089e50 con 0x7f06d80cd250 2024-03-17T23:49:45.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.903+0000 7f06dd620640 1 --2- 172.21.15.150:0/4124301952 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f06bc06bbd0 0x7f06bc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:45.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.903+0000 7f06dd620640 1 --2- 172.21.15.150:0/4124301952 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f06bc06bbd0 0x7f06bc06e090 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f06c8036000 tx=0x7f06c8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:45.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:45.907+0000 7f06ce7fc640 1 -- 172.21.15.150:0/4124301952 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f06d4058160 con 0x7f06d80cd250 2024-03-17T23:49:46.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.077+0000 7f06e0085640 1 -- 172.21.15.150:0/4124301952 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f06d8074200 con 0x7f06d80cd250 2024-03-17T23:49:46.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.078+0000 7f06ce7fc640 1 -- 172.21.15.150:0/4124301952 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) ==== 93+0+10 (secure 0 0 0) 0x7f06d405c200 con 0x7f06d80cd250 2024-03-17T23:49:46.079 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:49:46.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.080+0000 7f06e0085640 1 -- 172.21.15.150:0/4124301952 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f06bc06bbd0 msgr2=0x7f06bc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:46.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.080+0000 7f06e0085640 1 --2- 172.21.15.150:0/4124301952 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f06bc06bbd0 0x7f06bc06e090 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f06c8036000 tx=0x7f06c8008030 comp rx=0 tx=0).stop 2024-03-17T23:49:46.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.080+0000 7f06e0085640 1 -- 172.21.15.150:0/4124301952 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06d80cd250 msgr2=0x7f06d814cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:46.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.080+0000 7f06e0085640 1 --2- 172.21.15.150:0/4124301952 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06d80cd250 0x7f06d814cb50 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f06d400c4d0 tx=0x7f06d400c990 comp rx=0 tx=0).stop 2024-03-17T23:49:46.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.080+0000 7f06e0085640 1 -- 172.21.15.150:0/4124301952 shutdown_connections 2024-03-17T23:49:46.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.080+0000 7f06e0085640 1 --2- 172.21.15.150:0/4124301952 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f06d815bfd0 0x7f06d815a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:46.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.080+0000 7f06e0085640 1 --2- 172.21.15.150:0/4124301952 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f06bc06bbd0 0x7f06bc06e090 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:46.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.080+0000 7f06e0085640 1 --2- 172.21.15.150:0/4124301952 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f06d8155320 0x7f06d8159de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:46.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.080+0000 7f06e0085640 1 --2- 172.21.15.150:0/4124301952 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06d80cd250 0x7f06d814cb50 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:46.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.080+0000 7f06e0085640 1 -- 172.21.15.150:0/4124301952 >> 172.21.15.150:0/4124301952 conn(0x7f06d806b190 msgr2=0x7f06d8153a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:46.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.080+0000 7f06e0085640 1 -- 172.21.15.150:0/4124301952 shutdown_connections 2024-03-17T23:49:46.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.080+0000 7f06e0085640 1 -- 172.21.15.150:0/4124301952 wait complete. 2024-03-17T23:49:46.091 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:49:46.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.195+0000 7f782e0ea640 1 Processor -- start 2024-03-17T23:49:46.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.195+0000 7f782e0ea640 1 -- start start 2024-03-17T23:49:46.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.196+0000 7f782e0ea640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f78280cd420 0x7f7828165d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:46.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.196+0000 7f782e0ea640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78280cf3b0 con 0x7f7828155320 2024-03-17T23:49:46.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.196+0000 7f782e0ea640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78280cf690 con 0x7f782815bfd0 2024-03-17T23:49:46.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.196+0000 7f782e0ea640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f782815c550 con 0x7f78280cd420 2024-03-17T23:49:46.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.196+0000 7f7827fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f78280cd420 0x7f7828165d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:46.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.196+0000 7f78277fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f7828155320 0x7f782806dd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41830/0 (socket says 172.21.15.150:41830) 2024-03-17T23:49:46.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.196+0000 7f78277fe640 1 -- 172.21.15.150:0/4260633532 learned_addr learned my addr 172.21.15.150:0/4260633532 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:46.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.196+0000 7f7827fff640 1 -- 172.21.15.150:0/4260633532 >> v1:172.21.15.157:6789/0 conn(0x7f782815bfd0 legacy=0x7f78280ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:46.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.196+0000 7f78267fc640 1 -- 172.21.15.150:0/4260633532 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3465496612 0 0) 0x7f78280cf690 con 0x7f782815bfd0 2024-03-17T23:49:46.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.196+0000 7f7827fff640 1 -- 172.21.15.150:0/4260633532 >> v1:172.21.15.150:6789/0 conn(0x7f7828155320 legacy=0x7f782806dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:46.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.196+0000 7f7827fff640 1 -- 172.21.15.150:0/4260633532 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f782815c7a0 con 0x7f78280cd420 2024-03-17T23:49:46.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.197+0000 7f7827fff640 1 --2- 172.21.15.150:0/4260633532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78280cd420 0x7f7828165d10 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f7820009d90 tx=0x7f782002bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fef00c9d87dc5750 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:46.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.197+0000 7f78267fc640 1 -- 172.21.15.150:0/4260633532 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7820039070 con 0x7f78280cd420 2024-03-17T23:49:46.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.197+0000 7f78267fc640 1 -- 172.21.15.150:0/4260633532 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f782002cab0 con 0x7f78280cd420 2024-03-17T23:49:46.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.197+0000 7f78267fc640 1 -- 172.21.15.150:0/4260633532 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7820034440 con 0x7f78280cd420 2024-03-17T23:49:46.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.198+0000 7f782e0ea640 1 -- 172.21.15.150:0/4260633532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78280cd420 msgr2=0x7f7828165d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:46.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.198+0000 7f782e0ea640 1 --2- 172.21.15.150:0/4260633532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78280cd420 0x7f7828165d10 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f7820009d90 tx=0x7f782002bd80 comp rx=0 tx=0).stop 2024-03-17T23:49:46.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.198+0000 7f782e0ea640 1 -- 172.21.15.150:0/4260633532 shutdown_connections 2024-03-17T23:49:46.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.198+0000 7f782e0ea640 1 --2- 172.21.15.150:0/4260633532 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f78280cd420 0x7f7828165d10 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:46.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.198+0000 7f782e0ea640 1 -- 172.21.15.150:0/4260633532 >> 172.21.15.150:0/4260633532 conn(0x7f782806b190 msgr2=0x7f782806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:46.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.198+0000 7f782e0ea640 1 -- 172.21.15.150:0/4260633532 shutdown_connections 2024-03-17T23:49:46.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.198+0000 7f782e0ea640 1 -- 172.21.15.150:0/4260633532 wait complete. 2024-03-17T23:49:46.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.199+0000 7f782e0ea640 1 Processor -- start 2024-03-17T23:49:46.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.199+0000 7f782e0ea640 1 -- start start 2024-03-17T23:49:46.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.200+0000 7f782e0ea640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78280cd420 0x7f782814a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:46.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.200+0000 7f782e0ea640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7828155320 0x7f7828141250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:46.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.200+0000 7f782e0ea640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f782815bfd0 0x7f78281f51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:46.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.200+0000 7f782e0ea640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7828155880 con 0x7f782815bfd0 2024-03-17T23:49:46.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.200+0000 7f782e0ea640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7828155700 con 0x7f7828155320 2024-03-17T23:49:46.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.200+0000 7f782e0ea640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f782815dcd0 con 0x7f78280cd420 2024-03-17T23:49:46.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.200+0000 7f7827fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f782815bfd0 0x7f78281f51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:46.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.200+0000 7f7827fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f782815bfd0 0x7f78281f51f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60422/0 (socket says 172.21.15.150:60422) 2024-03-17T23:49:46.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.200+0000 7f7827fff640 1 -- 172.21.15.150:0/3173518115 learned_addr learned my addr 172.21.15.150:0/3173518115 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:46.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.200+0000 7f78277fe640 1 --2- 172.21.15.150:0/3173518115 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78280cd420 0x7f782814a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:46.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.200+0000 7f7826ffd640 1 --2- 172.21.15.150:0/3173518115 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7828155320 0x7f7828141250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:46.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.200+0000 7f7827fff640 1 -- 172.21.15.150:0/3173518115 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78280cd420 msgr2=0x7f782814a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:46.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.200+0000 7f7827fff640 1 --2- 172.21.15.150:0/3173518115 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78280cd420 0x7f782814a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:46.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.200+0000 7f7827fff640 1 -- 172.21.15.150:0/3173518115 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7828155320 msgr2=0x7f7828141250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:46.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.200+0000 7f7827fff640 1 --2- 172.21.15.150:0/3173518115 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7828155320 0x7f7828141250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:46.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.200+0000 7f7827fff640 1 -- 172.21.15.150:0/3173518115 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f782814fb20 con 0x7f782815bfd0 2024-03-17T23:49:46.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.200+0000 7f7827fff640 1 --2- 172.21.15.150:0/3173518115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f782815bfd0 0x7f78281f51f0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f7820037040 tx=0x7f7820004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:46.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.200+0000 7f780ffff640 1 -- 172.21.15.150:0/3173518115 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7820044070 con 0x7f782815bfd0 2024-03-17T23:49:46.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.200+0000 7f780ffff640 1 -- 172.21.15.150:0/3173518115 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7820003bb0 con 0x7f782815bfd0 2024-03-17T23:49:46.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.200+0000 7f782e0ea640 1 -- 172.21.15.150:0/3173518115 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7828158280 con 0x7f782815bfd0 2024-03-17T23:49:46.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.200+0000 7f780ffff640 1 -- 172.21.15.150:0/3173518115 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7820039040 con 0x7f782815bfd0 2024-03-17T23:49:46.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.200+0000 7f782e0ea640 1 -- 172.21.15.150:0/3173518115 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7828158520 con 0x7f782815bfd0 2024-03-17T23:49:46.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.202+0000 7f782e0ea640 1 -- 172.21.15.150:0/3173518115 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78280681a0 con 0x7f782815bfd0 2024-03-17T23:49:46.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.202+0000 7f780ffff640 1 -- 172.21.15.150:0/3173518115 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f782002c930 con 0x7f782815bfd0 2024-03-17T23:49:46.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.202+0000 7f780ffff640 1 --2- 172.21.15.150:0/3173518115 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f781406bbd0 0x7f781406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:46.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.202+0000 7f780ffff640 1 -- 172.21.15.150:0/3173518115 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f78200ad480 con 0x7f782815bfd0 2024-03-17T23:49:46.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.202+0000 7f78277fe640 1 --2- 172.21.15.150:0/3173518115 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f781406bbd0 0x7f781406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:46.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.203+0000 7f78277fe640 1 --2- 172.21.15.150:0/3173518115 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f781406bbd0 0x7f781406e090 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7818000f60 tx=0x7f7818007e20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:46.207 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.207+0000 7f780ffff640 1 -- 172.21.15.150:0/3173518115 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f782004e020 con 0x7f782815bfd0 2024-03-17T23:49:46.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.381+0000 7f782e0ea640 1 -- 172.21.15.150:0/3173518115 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f78280ce960 con 0x7f782815bfd0 2024-03-17T23:49:46.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.382+0000 7f780ffff640 1 -- 172.21.15.150:0/3173518115 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) ==== 104+0+11 (secure 0 0 0) 0x7f782814fb20 con 0x7f782815bfd0 2024-03-17T23:49:46.383 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:49:46.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.384+0000 7f782e0ea640 1 -- 172.21.15.150:0/3173518115 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f781406bbd0 msgr2=0x7f781406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:46.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.384+0000 7f782e0ea640 1 --2- 172.21.15.150:0/3173518115 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f781406bbd0 0x7f781406e090 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7818000f60 tx=0x7f7818007e20 comp rx=0 tx=0).stop 2024-03-17T23:49:46.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.384+0000 7f782e0ea640 1 -- 172.21.15.150:0/3173518115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f782815bfd0 msgr2=0x7f78281f51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:46.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.384+0000 7f782e0ea640 1 --2- 172.21.15.150:0/3173518115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f782815bfd0 0x7f78281f51f0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f7820037040 tx=0x7f7820004a40 comp rx=0 tx=0).stop 2024-03-17T23:49:46.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.384+0000 7f782e0ea640 1 -- 172.21.15.150:0/3173518115 shutdown_connections 2024-03-17T23:49:46.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.384+0000 7f782e0ea640 1 --2- 172.21.15.150:0/3173518115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f782815bfd0 0x7f78281f51f0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:46.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.384+0000 7f782e0ea640 1 --2- 172.21.15.150:0/3173518115 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f781406bbd0 0x7f781406e090 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:46.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.384+0000 7f782e0ea640 1 --2- 172.21.15.150:0/3173518115 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7828155320 0x7f7828141250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:46.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.384+0000 7f782e0ea640 1 --2- 172.21.15.150:0/3173518115 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f78280cd420 0x7f782814a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:46.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.384+0000 7f782e0ea640 1 -- 172.21.15.150:0/3173518115 >> 172.21.15.150:0/3173518115 conn(0x7f782806b190 msgr2=0x7f7828071cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:46.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.384+0000 7f782e0ea640 1 -- 172.21.15.150:0/3173518115 shutdown_connections 2024-03-17T23:49:46.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.384+0000 7f782e0ea640 1 -- 172.21.15.150:0/3173518115 wait complete. 2024-03-17T23:49:46.395 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:49:46.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.499+0000 7f7ac35a9640 1 Processor -- start 2024-03-17T23:49:46.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.499+0000 7f7ac35a9640 1 -- start start 2024-03-17T23:49:46.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.499+0000 7f7ac35a9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7abc155320 0x7f7abc15bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:46.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.500+0000 7f7ac35a9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7abc0cd420 0x7f7abc15ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:46.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.500+0000 7f7ac35a9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7abc0cf3b0 con 0x7f7abc06dd30 2024-03-17T23:49:46.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.500+0000 7f7ac35a9640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f7abc15c550 con 0x7f7abc155320 2024-03-17T23:49:46.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.500+0000 7f7ac35a9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7abc0cf690 con 0x7f7abc0cd420 2024-03-17T23:49:46.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.500+0000 7f7ac1345640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7abc155320 0x7f7abc15bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:46.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.500+0000 7f7ac1b46640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7abc0cd420 0x7f7abc15ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:46.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.500+0000 7f7ac1345640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7abc155320 0x7f7abc15bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:44826/0 (socket says 172.21.15.150:44826) 2024-03-17T23:49:46.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.500+0000 7f7ac0b44640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f7abc06dd30 0x7f7abc0ccd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:56902/0 (socket says 172.21.15.150:56902) 2024-03-17T23:49:46.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.500+0000 7f7ac1345640 1 -- 172.21.15.150:0/2396596781 learned_addr learned my addr 172.21.15.150:0/2396596781 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:46.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.500+0000 7f7ac1345640 1 -- 172.21.15.150:0/2396596781 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7abc0cd420 msgr2=0x7f7abc15ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:46.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.500+0000 7f7ac1345640 1 --2- 172.21.15.150:0/2396596781 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7abc0cd420 0x7f7abc15ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:46.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.500+0000 7f7ac1345640 1 -- 172.21.15.150:0/2396596781 >> v1:172.21.15.157:6789/0 conn(0x7f7abc06dd30 legacy=0x7f7abc0ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:46.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.500+0000 7f7ac1345640 1 -- 172.21.15.150:0/2396596781 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7abc15c7a0 con 0x7f7abc155320 2024-03-17T23:49:46.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.500+0000 7f7ab3fff640 1 -- 172.21.15.150:0/2396596781 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1674971928 0 0) 0x7f7abc0cf3b0 con 0x7f7abc06dd30 2024-03-17T23:49:46.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.500+0000 7f7ac1b46640 1 --2- 172.21.15.150:0/2396596781 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7abc0cd420 0x7f7abc15ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:46.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.501+0000 7f7ac1345640 1 --2- 172.21.15.150:0/2396596781 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7abc155320 0x7f7abc15bfd0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8009e50 tx=0x7f7ab802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d289154f818b3fed server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:46.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.501+0000 7f7ab3fff640 1 -- 172.21.15.150:0/2396596781 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab802caf0 con 0x7f7abc155320 2024-03-17T23:49:46.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.501+0000 7f7ab3fff640 1 -- 172.21.15.150:0/2396596781 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ab802cc90 con 0x7f7abc155320 2024-03-17T23:49:46.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.501+0000 7f7ab3fff640 1 -- 172.21.15.150:0/2396596781 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab8034610 con 0x7f7abc155320 2024-03-17T23:49:46.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.501+0000 7f7ac35a9640 1 -- 172.21.15.150:0/2396596781 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7abc155320 msgr2=0x7f7abc15bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:46.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.501+0000 7f7ac35a9640 1 --2- 172.21.15.150:0/2396596781 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7abc155320 0x7f7abc15bfd0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8009e50 tx=0x7f7ab802bf50 comp rx=0 tx=0).stop 2024-03-17T23:49:46.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.502+0000 7f7ac35a9640 1 -- 172.21.15.150:0/2396596781 shutdown_connections 2024-03-17T23:49:46.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.502+0000 7f7ac35a9640 1 --2- 172.21.15.150:0/2396596781 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7abc0cd420 0x7f7abc15ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:46.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.502+0000 7f7ac35a9640 1 --2- 172.21.15.150:0/2396596781 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7abc155320 0x7f7abc15bfd0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:46.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.502+0000 7f7ac35a9640 1 -- 172.21.15.150:0/2396596781 >> 172.21.15.150:0/2396596781 conn(0x7f7abc06b190 msgr2=0x7f7abc06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:46.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.502+0000 7f7ac35a9640 1 -- 172.21.15.150:0/2396596781 shutdown_connections 2024-03-17T23:49:46.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.502+0000 7f7ac35a9640 1 -- 172.21.15.150:0/2396596781 wait complete. 2024-03-17T23:49:46.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.503+0000 7f7ac35a9640 1 Processor -- start 2024-03-17T23:49:46.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.503+0000 7f7ac35a9640 1 -- start start 2024-03-17T23:49:46.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.503+0000 7f7ac35a9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7abc06dd30 0x7f7abc15a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:46.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.503+0000 7f7ac35a9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7abc0cd420 0x7f7abc14bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:46.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.504+0000 7f7ac35a9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7abc155320 0x7f7abc157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:46.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.504+0000 7f7ac35a9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7abc155700 con 0x7f7abc06dd30 2024-03-17T23:49:46.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.504+0000 7f7ac35a9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f7abc06e190 con 0x7f7abc155320 2024-03-17T23:49:46.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.504+0000 7f7ac35a9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f7abc155880 con 0x7f7abc0cd420 2024-03-17T23:49:46.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.504+0000 7f7ac1345640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7abc06dd30 0x7f7abc15a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:46.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.504+0000 7f7ac1345640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7abc06dd30 0x7f7abc15a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60424/0 (socket says 172.21.15.150:60424) 2024-03-17T23:49:46.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.504+0000 7f7ac1345640 1 -- 172.21.15.150:0/1218537941 learned_addr learned my addr 172.21.15.150:0/1218537941 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:46.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.504+0000 7f7ac0b44640 1 --2- 172.21.15.150:0/1218537941 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7abc0cd420 0x7f7abc14bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:46.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.504+0000 7f7ac1b46640 1 --2- 172.21.15.150:0/1218537941 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7abc155320 0x7f7abc157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:46.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.504+0000 7f7ac1345640 1 -- 172.21.15.150:0/1218537941 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7abc0cd420 msgr2=0x7f7abc14bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:46.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.504+0000 7f7ac1345640 1 --2- 172.21.15.150:0/1218537941 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7abc0cd420 0x7f7abc14bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:46.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.504+0000 7f7ac1345640 1 -- 172.21.15.150:0/1218537941 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7abc155320 msgr2=0x7f7abc157c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:46.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.504+0000 7f7ac1345640 1 --2- 172.21.15.150:0/1218537941 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7abc155320 0x7f7abc157c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:46.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.504+0000 7f7ac1345640 1 -- 172.21.15.150:0/1218537941 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7abc06eb80 con 0x7f7abc06dd30 2024-03-17T23:49:46.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.504+0000 7f7ac1345640 1 --2- 172.21.15.150:0/1218537941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7abc06dd30 0x7f7abc15a640 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8036040 tx=0x7f7ab8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:46.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.504+0000 7f7ab1ffb640 1 -- 172.21.15.150:0/1218537941 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab8043020 con 0x7f7abc06dd30 2024-03-17T23:49:46.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.504+0000 7f7ab1ffb640 1 -- 172.21.15.150:0/1218537941 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ab803e450 con 0x7f7abc06dd30 2024-03-17T23:49:46.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.504+0000 7f7ab1ffb640 1 -- 172.21.15.150:0/1218537941 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab8009510 con 0x7f7abc06dd30 2024-03-17T23:49:46.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.504+0000 7f7ac35a9640 1 -- 172.21.15.150:0/1218537941 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7abc06ee60 con 0x7f7abc06dd30 2024-03-17T23:49:46.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.505+0000 7f7ac35a9640 1 -- 172.21.15.150:0/1218537941 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7abc1f2da0 con 0x7f7abc06dd30 2024-03-17T23:49:46.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.506+0000 7f7ac35a9640 1 -- 172.21.15.150:0/1218537941 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7abc1f2fa0 con 0x7f7abc06dd30 2024-03-17T23:49:46.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.506+0000 7f7ab1ffb640 1 -- 172.21.15.150:0/1218537941 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7ab8046020 con 0x7f7abc06dd30 2024-03-17T23:49:46.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.506+0000 7f7ab1ffb640 1 --2- 172.21.15.150:0/1218537941 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7aa006bc00 0x7f7aa006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:46.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.507+0000 7f7ab1ffb640 1 -- 172.21.15.150:0/1218537941 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f7ab80acef0 con 0x7f7abc06dd30 2024-03-17T23:49:46.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.507+0000 7f7ac0b44640 1 --2- 172.21.15.150:0/1218537941 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7aa006bc00 0x7f7aa006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:46.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.507+0000 7f7ac0b44640 1 --2- 172.21.15.150:0/1218537941 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7aa006bc00 0x7f7aa006e0c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7abc0728b0 tx=0x7f7aac007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:46.511 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.511+0000 7f7ab1ffb640 1 -- 172.21.15.150:0/1218537941 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7ab804d070 con 0x7f7abc06dd30 2024-03-17T23:49:46.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.684+0000 7f7ac35a9640 1 -- 172.21.15.150:0/1218537941 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7abc0681a0 con 0x7f7abc06dd30 2024-03-17T23:49:46.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.684+0000 7f7ab1ffb640 1 -- 172.21.15.150:0/1218537941 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) ==== 100+0+11 (secure 0 0 0) 0x7f7ab807b8e0 con 0x7f7abc06dd30 2024-03-17T23:49:46.685 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:49:46.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.686+0000 7f7ac35a9640 1 -- 172.21.15.150:0/1218537941 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7aa006bc00 msgr2=0x7f7aa006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:46.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.686+0000 7f7ac35a9640 1 --2- 172.21.15.150:0/1218537941 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7aa006bc00 0x7f7aa006e0c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7abc0728b0 tx=0x7f7aac007d50 comp rx=0 tx=0).stop 2024-03-17T23:49:46.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.686+0000 7f7ac35a9640 1 -- 172.21.15.150:0/1218537941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7abc06dd30 msgr2=0x7f7abc15a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:46.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.686+0000 7f7ac35a9640 1 --2- 172.21.15.150:0/1218537941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7abc06dd30 0x7f7abc15a640 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8036040 tx=0x7f7ab8004a40 comp rx=0 tx=0).stop 2024-03-17T23:49:46.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.686+0000 7f7ac35a9640 1 -- 172.21.15.150:0/1218537941 shutdown_connections 2024-03-17T23:49:46.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.686+0000 7f7ac35a9640 1 --2- 172.21.15.150:0/1218537941 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f7aa006bc00 0x7f7aa006e0c0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:46.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.686+0000 7f7ac35a9640 1 --2- 172.21.15.150:0/1218537941 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7abc155320 0x7f7abc157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:46.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.686+0000 7f7ac35a9640 1 --2- 172.21.15.150:0/1218537941 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7abc0cd420 0x7f7abc14bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:46.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.686+0000 7f7ac35a9640 1 --2- 172.21.15.150:0/1218537941 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7abc06dd30 0x7f7abc15a640 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:46.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.686+0000 7f7ac35a9640 1 -- 172.21.15.150:0/1218537941 >> 172.21.15.150:0/1218537941 conn(0x7f7abc06b190 msgr2=0x7f7abc0b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:46.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.686+0000 7f7ac35a9640 1 -- 172.21.15.150:0/1218537941 shutdown_connections 2024-03-17T23:49:46.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.686+0000 7f7ac35a9640 1 -- 172.21.15.150:0/1218537941 wait complete. 2024-03-17T23:49:46.698 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-17T23:49:46.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.801+0000 7f5e85553640 1 Processor -- start 2024-03-17T23:49:46.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.801+0000 7f5e85553640 1 -- start start 2024-03-17T23:49:46.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.802+0000 7f5e85553640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e8006ccb0 0x7f5e8009a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:46.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.802+0000 7f5e85553640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5e800d04c0 0x7f5e800d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:46.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.802+0000 7f5e85553640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e8006c750 con 0x7f5e8006d090 2024-03-17T23:49:46.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.802+0000 7f5e85553640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f5e800d0030 con 0x7f5e800d04c0 2024-03-17T23:49:46.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.802+0000 7f5e85553640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5e8015edf0 con 0x7f5e8006ccb0 2024-03-17T23:49:46.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.802+0000 7f5e7f7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5e800d04c0 0x7f5e800d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:46.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.802+0000 7f5e7e7fc640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f5e8006d090 0x7f5e8015e4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41840/0 (socket says 172.21.15.150:41840) 2024-03-17T23:49:46.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.802+0000 7f5e7e7fc640 1 -- 172.21.15.150:0/586845303 learned_addr learned my addr 172.21.15.150:0/586845303 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:46.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.802+0000 7f5e7effd640 1 --2- 172.21.15.150:0/586845303 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e8006ccb0 0x7f5e8009a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:46.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.802+0000 7f5e7f7fe640 1 -- 172.21.15.150:0/586845303 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e8006ccb0 msgr2=0x7f5e8009a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:46.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.802+0000 7f5e7f7fe640 1 --2- 172.21.15.150:0/586845303 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e8006ccb0 0x7f5e8009a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:46.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.802+0000 7f5e7f7fe640 1 -- 172.21.15.150:0/586845303 >> v1:172.21.15.150:6789/0 conn(0x7f5e8006d090 legacy=0x7f5e8015e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:46.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.802+0000 7f5e7f7fe640 1 -- 172.21.15.150:0/586845303 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e8015f0d0 con 0x7f5e800d04c0 2024-03-17T23:49:46.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.802+0000 7f5e7dffb640 1 -- 172.21.15.150:0/586845303 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4288795058 0 0) 0x7f5e8006c750 con 0x7f5e8006d090 2024-03-17T23:49:46.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.803+0000 7f5e7f7fe640 1 --2- 172.21.15.150:0/586845303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e800d04c0 0x7f5e800d0950 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f5e78009d90 tx=0x7f5e7802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d4e30ae7b85cad7 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:46.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.803+0000 7f5e7dffb640 1 -- 172.21.15.150:0/586845303 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e78039070 con 0x7f5e800d04c0 2024-03-17T23:49:46.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.803+0000 7f5e7dffb640 1 -- 172.21.15.150:0/586845303 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e7802cc50 con 0x7f5e800d04c0 2024-03-17T23:49:46.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.803+0000 7f5e7dffb640 1 -- 172.21.15.150:0/586845303 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e78034440 con 0x7f5e800d04c0 2024-03-17T23:49:46.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.804+0000 7f5e85553640 1 -- 172.21.15.150:0/586845303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e800d04c0 msgr2=0x7f5e800d0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:46.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.804+0000 7f5e85553640 1 --2- 172.21.15.150:0/586845303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e800d04c0 0x7f5e800d0950 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f5e78009d90 tx=0x7f5e7802bea0 comp rx=0 tx=0).stop 2024-03-17T23:49:46.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.804+0000 7f5e85553640 1 -- 172.21.15.150:0/586845303 shutdown_connections 2024-03-17T23:49:46.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.804+0000 7f5e85553640 1 --2- 172.21.15.150:0/586845303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e800d04c0 0x7f5e800d0950 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:46.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.804+0000 7f5e85553640 1 --2- 172.21.15.150:0/586845303 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e8006ccb0 0x7f5e8009a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:46.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.804+0000 7f5e85553640 1 -- 172.21.15.150:0/586845303 >> 172.21.15.150:0/586845303 conn(0x7f5e800b36b0 msgr2=0x7f5e800b3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:46.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.804+0000 7f5e85553640 1 -- 172.21.15.150:0/586845303 shutdown_connections 2024-03-17T23:49:46.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.804+0000 7f5e85553640 1 -- 172.21.15.150:0/586845303 wait complete. 2024-03-17T23:49:46.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.805+0000 7f5e85553640 1 Processor -- start 2024-03-17T23:49:46.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.805+0000 7f5e85553640 1 -- start start 2024-03-17T23:49:46.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.806+0000 7f5e85553640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e8006ccb0 0x7f5e800c7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:46.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.806+0000 7f5e85553640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e8006d090 0x7f5e800b6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:46.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.806+0000 7f5e85553640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e800d04c0 0x7f5e8014db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:46.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.806+0000 7f5e85553640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5e8006d4f0 con 0x7f5e8006ccb0 2024-03-17T23:49:46.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.806+0000 7f5e85553640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5e8015f690 con 0x7f5e8006d090 2024-03-17T23:49:46.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.806+0000 7f5e85553640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5e800d0e90 con 0x7f5e800d04c0 2024-03-17T23:49:46.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.806+0000 7f5e7effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e8006ccb0 0x7f5e800c7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:46.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.806+0000 7f5e7effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e8006ccb0 0x7f5e800c7ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60442/0 (socket says 172.21.15.150:60442) 2024-03-17T23:49:46.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.806+0000 7f5e7effd640 1 -- 172.21.15.150:0/3165241206 learned_addr learned my addr 172.21.15.150:0/3165241206 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:46.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.806+0000 7f5e7f7fe640 1 --2- 172.21.15.150:0/3165241206 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e800d04c0 0x7f5e8014db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:46.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.806+0000 7f5e7e7fc640 1 --2- 172.21.15.150:0/3165241206 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e8006d090 0x7f5e800b6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:46.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.806+0000 7f5e7effd640 1 -- 172.21.15.150:0/3165241206 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e800d04c0 msgr2=0x7f5e8014db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:46.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.806+0000 7f5e7effd640 1 --2- 172.21.15.150:0/3165241206 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e800d04c0 0x7f5e8014db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:46.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.806+0000 7f5e7effd640 1 -- 172.21.15.150:0/3165241206 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e8006d090 msgr2=0x7f5e800b6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:46.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.806+0000 7f5e7effd640 1 --2- 172.21.15.150:0/3165241206 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e8006d090 0x7f5e800b6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:46.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.806+0000 7f5e7effd640 1 -- 172.21.15.150:0/3165241206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e800afab0 con 0x7f5e8006ccb0 2024-03-17T23:49:46.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.807+0000 7f5e7effd640 1 --2- 172.21.15.150:0/3165241206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e8006ccb0 0x7f5e800c7ef0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f5e7000e9b0 tx=0x7f5e7000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:46.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.807+0000 7f5e677fe640 1 -- 172.21.15.150:0/3165241206 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e70017070 con 0x7f5e8006ccb0 2024-03-17T23:49:46.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.807+0000 7f5e677fe640 1 -- 172.21.15.150:0/3165241206 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e7000c9a0 con 0x7f5e8006ccb0 2024-03-17T23:49:46.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.807+0000 7f5e85553640 1 -- 172.21.15.150:0/3165241206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e800afd90 con 0x7f5e8006ccb0 2024-03-17T23:49:46.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.807+0000 7f5e677fe640 1 -- 172.21.15.150:0/3165241206 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e7000cd90 con 0x7f5e8006ccb0 2024-03-17T23:49:46.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.807+0000 7f5e85553640 1 -- 172.21.15.150:0/3165241206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e800b0350 con 0x7f5e8006ccb0 2024-03-17T23:49:46.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.808+0000 7f5e85553640 1 -- 172.21.15.150:0/3165241206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e800681a0 con 0x7f5e8006ccb0 2024-03-17T23:49:46.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.808+0000 7f5e677fe640 1 -- 172.21.15.150:0/3165241206 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5e7001a070 con 0x7f5e8006ccb0 2024-03-17T23:49:46.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.809+0000 7f5e677fe640 1 --2- 172.21.15.150:0/3165241206 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5e6c06bbd0 0x7f5e6c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:46.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.809+0000 7f5e7e7fc640 1 --2- 172.21.15.150:0/3165241206 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5e6c06bbd0 0x7f5e6c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:46.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.809+0000 7f5e677fe640 1 -- 172.21.15.150:0/3165241206 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f5e7008abc0 con 0x7f5e8006ccb0 2024-03-17T23:49:46.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.809+0000 7f5e7e7fc640 1 --2- 172.21.15.150:0/3165241206 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5e6c06bbd0 0x7f5e6c06e090 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5e801578a0 tx=0x7f5e74007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:46.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.813+0000 7f5e677fe640 1 -- 172.21.15.150:0/3165241206 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5e70058ed0 con 0x7f5e8006ccb0 2024-03-17T23:49:46.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.999+0000 7f5e85553640 1 -- 172.21.15.150:0/3165241206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5e800b7b40 con 0x7f5e8006ccb0 2024-03-17T23:49:47.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:46.999+0000 7f5e677fe640 1 -- 172.21.15.150:0/3165241206 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v58) ==== 80+0+0 (secure 0 0 0) 0x7f5e7005cf70 con 0x7f5e8006ccb0 2024-03-17T23:49:47.000 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-03-17T23:49:47.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.001+0000 7f5e85553640 1 -- 172.21.15.150:0/3165241206 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5e6c06bbd0 msgr2=0x7f5e6c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:47.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.001+0000 7f5e85553640 1 --2- 172.21.15.150:0/3165241206 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5e6c06bbd0 0x7f5e6c06e090 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f5e801578a0 tx=0x7f5e74007d50 comp rx=0 tx=0).stop 2024-03-17T23:49:47.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.001+0000 7f5e85553640 1 -- 172.21.15.150:0/3165241206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e8006ccb0 msgr2=0x7f5e800c7ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:47.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.001+0000 7f5e85553640 1 --2- 172.21.15.150:0/3165241206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e8006ccb0 0x7f5e800c7ef0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f5e7000e9b0 tx=0x7f5e7000ee70 comp rx=0 tx=0).stop 2024-03-17T23:49:47.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.001+0000 7f5e85553640 1 -- 172.21.15.150:0/3165241206 shutdown_connections 2024-03-17T23:49:47.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.001+0000 7f5e85553640 1 --2- 172.21.15.150:0/3165241206 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5e800d04c0 0x7f5e8014db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:47.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.001+0000 7f5e85553640 1 --2- 172.21.15.150:0/3165241206 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5e6c06bbd0 0x7f5e6c06e090 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:47.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.001+0000 7f5e85553640 1 --2- 172.21.15.150:0/3165241206 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5e8006d090 0x7f5e800b6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:47.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.001+0000 7f5e85553640 1 --2- 172.21.15.150:0/3165241206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5e8006ccb0 0x7f5e800c7ef0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:47.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.001+0000 7f5e85553640 1 -- 172.21.15.150:0/3165241206 >> 172.21.15.150:0/3165241206 conn(0x7f5e800b36b0 msgr2=0x7f5e800b1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:47.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.001+0000 7f5e85553640 1 -- 172.21.15.150:0/3165241206 shutdown_connections 2024-03-17T23:49:47.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.001+0000 7f5e85553640 1 -- 172.21.15.150:0/3165241206 wait complete. 2024-03-17T23:49:47.012 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi150.front.sepia.ceph.com... 2024-03-17T23:49:47.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:47.013 DEBUG:teuthology.orchestra.run.smithi150:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-17T23:49:47.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi150.stderr:ceph-fuse[62247]: fuse finished with error 0 and tester_r 0 2024-03-17T23:49:47.040 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi150.front.sepia.ceph.com 2024-03-17T23:49:47.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:47.041 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-17T23:49:47.206 INFO:tasks.cephfs.mount:Terminating background process 2024-03-17T23:49:47.206 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:49:47.206 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:49:47.222 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=626 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-17T23:49:47.222 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi157.front.sepia.ceph.com 2024-03-17T23:49:47.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:47.222 DEBUG:teuthology.orchestra.run.smithi157:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-17T23:49:47.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-17T23:49:47.277 INFO:teuthology.orchestra.run.smithi157.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-17T23:49:47.277 INFO:tasks.cephfs.mount:Terminating background process 2024-03-17T23:49:47.278 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:49:47.278 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:49:47.293 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19488 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-17T23:49:47.293 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi150.front.sepia.ceph.com 2024-03-17T23:49:47.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:47.294 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-17T23:49:47.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-17T23:49:47.348 INFO:teuthology.orchestra.run.smithi150.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-17T23:49:47.348 INFO:tasks.cephfs.mount:Terminating background process 2024-03-17T23:49:47.348 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:49:47.348 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:49:47.364 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=626 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-17T23:49:47.364 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi157.front.sepia.ceph.com 2024-03-17T23:49:47.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:47.364 DEBUG:teuthology.orchestra.run.smithi157:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-17T23:49:47.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-17T23:49:47.419 INFO:teuthology.orchestra.run.smithi157.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-17T23:49:47.419 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:49:47.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.519+0000 7f1b343b9640 1 Processor -- start 2024-03-17T23:49:47.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.519+0000 7f1b343b9640 1 -- start start 2024-03-17T23:49:47.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.520+0000 7f1b343b9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1b2c0cf120 0x7f1b2c1619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:47.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.520+0000 7f1b343b9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f1b2c0b9860 0x7f1b2c0d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:47.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.520+0000 7f1b343b9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b2c1620a0 con 0x7f1b2c15de30 2024-03-17T23:49:47.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.520+0000 7f1b343b9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f1b2c162940 con 0x7f1b2c0cf120 2024-03-17T23:49:47.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.520+0000 7f1b343b9640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f1b2c162380 con 0x7f1b2c0b9860 2024-03-17T23:49:47.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.520+0000 7f1b31954640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1b2c0cf120 0x7f1b2c1619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:47.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.520+0000 7f1b31954640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1b2c0cf120 0x7f1b2c1619a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60452/0 (socket says 172.21.15.150:60452) 2024-03-17T23:49:47.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.520+0000 7f1b31954640 1 -- 172.21.15.150:0/3855905848 learned_addr learned my addr 172.21.15.150:0/3855905848 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:47.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.520+0000 7f1b32956640 1 --2- 172.21.15.150:0/3855905848 >> v2:172.21.15.157:3300/0 conn(0x7f1b2c0b9860 0x7f1b2c0d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:47.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.520+0000 7f1b31954640 1 -- 172.21.15.150:0/3855905848 >> v2:172.21.15.157:3300/0 conn(0x7f1b2c0b9860 msgr2=0x7f1b2c0d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:47.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.520+0000 7f1b31954640 1 --2- 172.21.15.150:0/3855905848 >> v2:172.21.15.157:3300/0 conn(0x7f1b2c0b9860 0x7f1b2c0d2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:47.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.520+0000 7f1b31954640 1 -- 172.21.15.150:0/3855905848 >> v1:172.21.15.157:6789/0 conn(0x7f1b2c15de30 legacy=0x7f1b2c0cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:47.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.520+0000 7f1b31954640 1 -- 172.21.15.150:0/3855905848 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b2c162660 con 0x7f1b2c0cf120 2024-03-17T23:49:47.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.521+0000 7f1b31954640 1 --2- 172.21.15.150:0/3855905848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b2c0cf120 0x7f1b2c1619a0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c0d0400 tx=0x7f1b1c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77f0aea0d4f0a0d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:47.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.521+0000 7f1b31153640 1 -- 172.21.15.150:0/3855905848 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c035020 con 0x7f1b2c0cf120 2024-03-17T23:49:47.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.521+0000 7f1b31153640 1 -- 172.21.15.150:0/3855905848 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b1c031870 con 0x7f1b2c0cf120 2024-03-17T23:49:47.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.521+0000 7f1b31153640 1 -- 172.21.15.150:0/3855905848 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c031c40 con 0x7f1b2c0cf120 2024-03-17T23:49:47.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.521+0000 7f1b343b9640 1 -- 172.21.15.150:0/3855905848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b2c0cf120 msgr2=0x7f1b2c1619a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:47.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.521+0000 7f1b343b9640 1 --2- 172.21.15.150:0/3855905848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b2c0cf120 0x7f1b2c1619a0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c0d0400 tx=0x7f1b1c02b910 comp rx=0 tx=0).stop 2024-03-17T23:49:47.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.521+0000 7f1b343b9640 1 -- 172.21.15.150:0/3855905848 shutdown_connections 2024-03-17T23:49:47.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.521+0000 7f1b343b9640 1 --2- 172.21.15.150:0/3855905848 >> v2:172.21.15.157:3300/0 conn(0x7f1b2c0b9860 0x7f1b2c0d2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:47.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.521+0000 7f1b343b9640 1 --2- 172.21.15.150:0/3855905848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b2c0cf120 0x7f1b2c1619a0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:47.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.521+0000 7f1b343b9640 1 -- 172.21.15.150:0/3855905848 >> 172.21.15.150:0/3855905848 conn(0x7f1b2c0730b0 msgr2=0x7f1b2c0d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:47.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.522+0000 7f1b343b9640 1 -- 172.21.15.150:0/3855905848 shutdown_connections 2024-03-17T23:49:47.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.522+0000 7f1b343b9640 1 -- 172.21.15.150:0/3855905848 wait complete. 2024-03-17T23:49:47.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.523+0000 7f1b343b9640 1 Processor -- start 2024-03-17T23:49:47.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.523+0000 7f1b343b9640 1 -- start start 2024-03-17T23:49:47.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.523+0000 7f1b343b9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b2c0b9860 0x7f1b2c0cb820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:47.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.523+0000 7f1b343b9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b2c0cf120 0x7f1b2c1f6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:47.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.523+0000 7f1b343b9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b2c15de30 0x7f1b2c0b7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:47.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.523+0000 7f1b343b9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1b2c15e210 con 0x7f1b2c0b9860 2024-03-17T23:49:47.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.523+0000 7f1b343b9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1b2c0b9c90 con 0x7f1b2c15de30 2024-03-17T23:49:47.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.523+0000 7f1b343b9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1b2c0d2bd0 con 0x7f1b2c0cf120 2024-03-17T23:49:47.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.523+0000 7f1b32155640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b2c0b9860 0x7f1b2c0cb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:47.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.523+0000 7f1b32155640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b2c0b9860 0x7f1b2c0cb820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60468/0 (socket says 172.21.15.150:60468) 2024-03-17T23:49:47.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.523+0000 7f1b32155640 1 -- 172.21.15.150:0/2883030229 learned_addr learned my addr 172.21.15.150:0/2883030229 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:47.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.524+0000 7f1b32956640 1 --2- 172.21.15.150:0/2883030229 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b2c15de30 0x7f1b2c0b7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:47.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.524+0000 7f1b31954640 1 --2- 172.21.15.150:0/2883030229 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b2c0cf120 0x7f1b2c1f6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:47.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.524+0000 7f1b32155640 1 -- 172.21.15.150:0/2883030229 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b2c0cf120 msgr2=0x7f1b2c1f6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:47.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.524+0000 7f1b32155640 1 --2- 172.21.15.150:0/2883030229 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b2c0cf120 0x7f1b2c1f6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:47.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.524+0000 7f1b32155640 1 -- 172.21.15.150:0/2883030229 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b2c15de30 msgr2=0x7f1b2c0b7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:47.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.524+0000 7f1b32155640 1 --2- 172.21.15.150:0/2883030229 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b2c15de30 0x7f1b2c0b7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:47.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.524+0000 7f1b32155640 1 -- 172.21.15.150:0/2883030229 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b2c15e5e0 con 0x7f1b2c0b9860 2024-03-17T23:49:47.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.524+0000 7f1b32155640 1 --2- 172.21.15.150:0/2883030229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b2c0b9860 0x7f1b2c0cb820 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f1b2800c3c0 tx=0x7f1b2800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:47.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.524+0000 7f1b22ffd640 1 -- 172.21.15.150:0/2883030229 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b280052d0 con 0x7f1b2c0b9860 2024-03-17T23:49:47.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.524+0000 7f1b22ffd640 1 -- 172.21.15.150:0/2883030229 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b28005490 con 0x7f1b2c0b9860 2024-03-17T23:49:47.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.524+0000 7f1b22ffd640 1 -- 172.21.15.150:0/2883030229 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b28007260 con 0x7f1b2c0b9860 2024-03-17T23:49:47.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.524+0000 7f1b343b9640 1 -- 172.21.15.150:0/2883030229 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b2c15e8c0 con 0x7f1b2c0b9860 2024-03-17T23:49:47.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.524+0000 7f1b343b9640 1 -- 172.21.15.150:0/2883030229 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b2c0cbd60 con 0x7f1b2c0b9860 2024-03-17T23:49:47.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.525+0000 7f1b343b9640 1 -- 172.21.15.150:0/2883030229 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b2c0681a0 con 0x7f1b2c0b9860 2024-03-17T23:49:47.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.526+0000 7f1b22ffd640 1 -- 172.21.15.150:0/2883030229 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1b28007400 con 0x7f1b2c0b9860 2024-03-17T23:49:47.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.526+0000 7f1b22ffd640 1 --2- 172.21.15.150:0/2883030229 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1b1006bbd0 0x7f1b1006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:47.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.526+0000 7f1b22ffd640 1 -- 172.21.15.150:0/2883030229 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f1b28089fa0 con 0x7f1b2c0b9860 2024-03-17T23:49:47.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.526+0000 7f1b31954640 1 --2- 172.21.15.150:0/2883030229 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1b1006bbd0 0x7f1b1006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:47.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.526+0000 7f1b31954640 1 --2- 172.21.15.150:0/2883030229 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1b1006bbd0 0x7f1b1006e090 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c036000 tx=0x7f1b1c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:47.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.530+0000 7f1b22ffd640 1 -- 172.21.15.150:0/2883030229 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1b28058340 con 0x7f1b2c0b9860 2024-03-17T23:49:47.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.745+0000 7f1b343b9640 1 -- 172.21.15.150:0/2883030229 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b2c0d1910 con 0x7f1b2c0b9860 2024-03-17T23:49:47.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.746+0000 7f1b22ffd640 1 -- 172.21.15.150:0/2883030229 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+5187 (secure 0 0 0) 0x7f1b2805c3e0 con 0x7f1b2c0b9860 2024-03-17T23:49:47.747 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:47.747 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":22,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/2082413564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2082413564},{"type":"v1","addr":"172.21.15.150:6836","nonce":2082413564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":6043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/2350077845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":2350077845},{"type":"v1","addr":"172.21.15.150:6837","nonce":2350077845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":6052,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6834/137606728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":137606728},{"type":"v1","addr":"172.21.15.157:6834","nonce":137606728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-17T23:49:10.996124+0000","modified":"2024-03-17T23:49:12.994518+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6835/479821580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6833","nonce":479821580},{"type":"v1","addr":"172.21.15.157:6835","nonce":479821580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6204,"qdb_cluster":[6204]},"id":3}]} 2024-03-17T23:49:47.747 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 22 2024-03-17T23:49:47.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.748+0000 7f1b343b9640 1 -- 172.21.15.150:0/2883030229 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1b1006bbd0 msgr2=0x7f1b1006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:47.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.748+0000 7f1b343b9640 1 --2- 172.21.15.150:0/2883030229 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1b1006bbd0 0x7f1b1006e090 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c036000 tx=0x7f1b1c03c040 comp rx=0 tx=0).stop 2024-03-17T23:49:47.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.748+0000 7f1b343b9640 1 -- 172.21.15.150:0/2883030229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b2c0b9860 msgr2=0x7f1b2c0cb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:47.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.748+0000 7f1b343b9640 1 --2- 172.21.15.150:0/2883030229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b2c0b9860 0x7f1b2c0cb820 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f1b2800c3c0 tx=0x7f1b2800c880 comp rx=0 tx=0).stop 2024-03-17T23:49:47.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.748+0000 7f1b343b9640 1 -- 172.21.15.150:0/2883030229 shutdown_connections 2024-03-17T23:49:47.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.748+0000 7f1b343b9640 1 --2- 172.21.15.150:0/2883030229 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1b1006bbd0 0x7f1b1006e090 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:47.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.748+0000 7f1b343b9640 1 --2- 172.21.15.150:0/2883030229 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1b2c15de30 0x7f1b2c0b7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:47.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.748+0000 7f1b343b9640 1 --2- 172.21.15.150:0/2883030229 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1b2c0cf120 0x7f1b2c1f6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:47.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.748+0000 7f1b343b9640 1 --2- 172.21.15.150:0/2883030229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b2c0b9860 0x7f1b2c0cb820 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:47.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.748+0000 7f1b343b9640 1 -- 172.21.15.150:0/2883030229 >> 172.21.15.150:0/2883030229 conn(0x7f1b2c0730b0 msgr2=0x7f1b2c0b85f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:47.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.748+0000 7f1b343b9640 1 -- 172.21.15.150:0/2883030229 shutdown_connections 2024-03-17T23:49:47.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.748+0000 7f1b343b9640 1 -- 172.21.15.150:0/2883030229 wait complete. 2024-03-17T23:49:47.760 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:49:47.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.860+0000 7fb614987640 1 Processor -- start 2024-03-17T23:49:47.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.860+0000 7fb614987640 1 -- start start 2024-03-17T23:49:47.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.861+0000 7fb614987640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fb61006f400 0x7fb6100cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:47.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.861+0000 7fb614987640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb610145170 con 0x7fb6100ce4d0 2024-03-17T23:49:47.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.861+0000 7fb614987640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb610145450 con 0x7fb610144bb0 2024-03-17T23:49:47.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.861+0000 7fb614987640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fb6100ce940 con 0x7fb61006f400 2024-03-17T23:49:47.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.861+0000 7fb60ed9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb6100ce4d0 0x7fb610165a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41854/0 (socket says 172.21.15.150:41854) 2024-03-17T23:49:47.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.861+0000 7fb60ed9d640 1 -- 172.21.15.150:0/2711765480 learned_addr learned my addr 172.21.15.150:0/2711765480 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:47.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.861+0000 7fb60dd9b640 1 --2- 172.21.15.150:0/2711765480 >> v2:172.21.15.157:3300/0 conn(0x7fb61006f400 0x7fb6100cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:47.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.862+0000 7fb60d59a640 1 -- 172.21.15.150:0/2711765480 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2782674514 0 0) 0x7fb610145170 con 0x7fb6100ce4d0 2024-03-17T23:49:47.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.862+0000 7fb60d59a640 1 -- 172.21.15.150:0/2711765480 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5f8003520 con 0x7fb6100ce4d0 2024-03-17T23:49:47.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.862+0000 7fb60d59a640 1 -- 172.21.15.150:0/2711765480 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2139694660 0 0) 0x7fb610145450 con 0x7fb610144bb0 2024-03-17T23:49:47.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.862+0000 7fb60d59a640 1 -- 172.21.15.150:0/2711765480 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb610145170 con 0x7fb610144bb0 2024-03-17T23:49:47.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.862+0000 7fb60dd9b640 1 -- 172.21.15.150:0/2711765480 >> v1:172.21.15.157:6789/0 conn(0x7fb610144bb0 legacy=0x7fb61015ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:47.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.862+0000 7fb60dd9b640 1 -- 172.21.15.150:0/2711765480 >> v1:172.21.15.150:6789/0 conn(0x7fb6100ce4d0 legacy=0x7fb610165a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:47.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.862+0000 7fb60dd9b640 1 -- 172.21.15.150:0/2711765480 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb610145730 con 0x7fb61006f400 2024-03-17T23:49:47.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.862+0000 7fb60dd9b640 1 --2- 172.21.15.150:0/2711765480 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb61006f400 0x7fb6100cdf90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb6040097a0 tx=0x7fb60402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6364a90f3dd8f9a server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:47.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.863+0000 7fb60d59a640 1 -- 172.21.15.150:0/2711765480 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb604035020 con 0x7fb61006f400 2024-03-17T23:49:47.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.863+0000 7fb60d59a640 1 -- 172.21.15.150:0/2711765480 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb604031830 con 0x7fb61006f400 2024-03-17T23:49:47.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.863+0000 7fb60d59a640 1 -- 172.21.15.150:0/2711765480 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb604031c00 con 0x7fb61006f400 2024-03-17T23:49:47.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.863+0000 7fb614987640 1 -- 172.21.15.150:0/2711765480 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb61006f400 msgr2=0x7fb6100cdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:47.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.863+0000 7fb614987640 1 --2- 172.21.15.150:0/2711765480 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb61006f400 0x7fb6100cdf90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb6040097a0 tx=0x7fb60402b8d0 comp rx=0 tx=0).stop 2024-03-17T23:49:47.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.863+0000 7fb614987640 1 -- 172.21.15.150:0/2711765480 shutdown_connections 2024-03-17T23:49:47.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.863+0000 7fb614987640 1 --2- 172.21.15.150:0/2711765480 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb61006f400 0x7fb6100cdf90 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:47.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.863+0000 7fb614987640 1 -- 172.21.15.150:0/2711765480 >> 172.21.15.150:0/2711765480 conn(0x7fb61006b190 msgr2=0x7fb61006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:47.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.863+0000 7fb614987640 1 -- 172.21.15.150:0/2711765480 shutdown_connections 2024-03-17T23:49:47.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.863+0000 7fb614987640 1 -- 172.21.15.150:0/2711765480 wait complete. 2024-03-17T23:49:47.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.864+0000 7fb614987640 1 Processor -- start 2024-03-17T23:49:47.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.865+0000 7fb614987640 1 -- start start 2024-03-17T23:49:47.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.865+0000 7fb614987640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb61006f400 0x7fb61015b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:47.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.865+0000 7fb614987640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6100ce4d0 0x7fb610142c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:47.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.865+0000 7fb614987640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb610144bb0 0x7fb6101431c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:47.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.865+0000 7fb614987640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb61006bee0 con 0x7fb610144bb0 2024-03-17T23:49:47.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.865+0000 7fb614987640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb61006bd60 con 0x7fb6100ce4d0 2024-03-17T23:49:47.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.865+0000 7fb614987640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb610167150 con 0x7fb61006f400 2024-03-17T23:49:47.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.865+0000 7fb60ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb610144bb0 0x7fb6101431c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:47.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.865+0000 7fb60ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb610144bb0 0x7fb6101431c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60472/0 (socket says 172.21.15.150:60472) 2024-03-17T23:49:47.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.865+0000 7fb60ed9d640 1 -- 172.21.15.150:0/2074368202 learned_addr learned my addr 172.21.15.150:0/2074368202 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:47.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.865+0000 7fb60e59c640 1 --2- 172.21.15.150:0/2074368202 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb61006f400 0x7fb61015b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:47.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.865+0000 7fb60dd9b640 1 --2- 172.21.15.150:0/2074368202 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6100ce4d0 0x7fb610142c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:47.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.866+0000 7fb60ed9d640 1 -- 172.21.15.150:0/2074368202 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb61006f400 msgr2=0x7fb61015b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:47.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.866+0000 7fb60ed9d640 1 --2- 172.21.15.150:0/2074368202 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb61006f400 0x7fb61015b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:47.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.866+0000 7fb60ed9d640 1 -- 172.21.15.150:0/2074368202 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6100ce4d0 msgr2=0x7fb610142c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:47.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.866+0000 7fb60ed9d640 1 --2- 172.21.15.150:0/2074368202 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6100ce4d0 0x7fb610142c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:47.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.866+0000 7fb60ed9d640 1 -- 172.21.15.150:0/2074368202 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6101f1ad0 con 0x7fb610144bb0 2024-03-17T23:49:47.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.866+0000 7fb60ed9d640 1 --2- 172.21.15.150:0/2074368202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb610144bb0 0x7fb6101431c0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fb60800c430 tx=0x7fb60800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:47.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.866+0000 7fb5feffd640 1 -- 172.21.15.150:0/2074368202 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb608017070 con 0x7fb610144bb0 2024-03-17T23:49:47.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.866+0000 7fb5feffd640 1 -- 172.21.15.150:0/2074368202 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb608005ce0 con 0x7fb610144bb0 2024-03-17T23:49:47.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.866+0000 7fb614987640 1 -- 172.21.15.150:0/2074368202 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6101f1db0 con 0x7fb610144bb0 2024-03-17T23:49:47.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.866+0000 7fb5feffd640 1 -- 172.21.15.150:0/2074368202 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6080072f0 con 0x7fb610144bb0 2024-03-17T23:49:47.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.866+0000 7fb614987640 1 -- 172.21.15.150:0/2074368202 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb61015bc60 con 0x7fb610144bb0 2024-03-17T23:49:47.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.867+0000 7fb614987640 1 -- 172.21.15.150:0/2074368202 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6100a9de0 con 0x7fb610144bb0 2024-03-17T23:49:47.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.867+0000 7fb5feffd640 1 -- 172.21.15.150:0/2074368202 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb60801a070 con 0x7fb610144bb0 2024-03-17T23:49:47.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.868+0000 7fb5feffd640 1 --2- 172.21.15.150:0/2074368202 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb5f006bbd0 0x7fb5f006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:47.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.868+0000 7fb5feffd640 1 -- 172.21.15.150:0/2074368202 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7fb608089ff0 con 0x7fb610144bb0 2024-03-17T23:49:47.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.868+0000 7fb60e59c640 1 --2- 172.21.15.150:0/2074368202 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb5f006bbd0 0x7fb5f006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:47.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.868+0000 7fb60e59c640 1 --2- 172.21.15.150:0/2074368202 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb5f006bbd0 0x7fb5f006e090 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb61013a6f0 tx=0x7fb600009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:47.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:47.872+0000 7fb5feffd640 1 -- 172.21.15.150:0/2074368202 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb608058310 con 0x7fb610144bb0 2024-03-17T23:49:48.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.043+0000 7fb614987640 1 -- 172.21.15.150:0/2074368202 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb6100742e0 con 0x7fb610144bb0 2024-03-17T23:49:48.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.044+0000 7fb5feffd640 1 -- 172.21.15.150:0/2074368202 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) ==== 74+0+16974 (secure 0 0 0) 0x7fb6100742e0 con 0x7fb610144bb0 2024-03-17T23:49:48.045 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:48.045 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":58,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:49:12.646307+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:49:09.223630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-17T23:49:09.956422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:49:48.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.046+0000 7fb614987640 1 -- 172.21.15.150:0/2074368202 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb5f006bbd0 msgr2=0x7fb5f006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:48.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.046+0000 7fb614987640 1 --2- 172.21.15.150:0/2074368202 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb5f006bbd0 0x7fb5f006e090 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb61013a6f0 tx=0x7fb600009040 comp rx=0 tx=0).stop 2024-03-17T23:49:48.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.046+0000 7fb614987640 1 -- 172.21.15.150:0/2074368202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb610144bb0 msgr2=0x7fb6101431c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:48.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.046+0000 7fb614987640 1 --2- 172.21.15.150:0/2074368202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb610144bb0 0x7fb6101431c0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fb60800c430 tx=0x7fb60800c8f0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.046+0000 7fb614987640 1 -- 172.21.15.150:0/2074368202 shutdown_connections 2024-03-17T23:49:48.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.046+0000 7fb614987640 1 --2- 172.21.15.150:0/2074368202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb610144bb0 0x7fb6101431c0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.046+0000 7fb614987640 1 --2- 172.21.15.150:0/2074368202 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb5f006bbd0 0x7fb5f006e090 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.046+0000 7fb614987640 1 --2- 172.21.15.150:0/2074368202 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6100ce4d0 0x7fb610142c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.046+0000 7fb614987640 1 --2- 172.21.15.150:0/2074368202 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb61006f400 0x7fb61015b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.046+0000 7fb614987640 1 -- 172.21.15.150:0/2074368202 >> 172.21.15.150:0/2074368202 conn(0x7fb61006b190 msgr2=0x7fb610072560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:48.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.046+0000 7fb614987640 1 -- 172.21.15.150:0/2074368202 shutdown_connections 2024-03-17T23:49:48.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.046+0000 7fb614987640 1 -- 172.21.15.150:0/2074368202 wait complete. 2024-03-17T23:49:48.058 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-17T23:46:43.636400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-17T23:49:09.223630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-17T23:49:09.956422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-17T23:49:48.059 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-17T23:49:48.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.162+0000 7f85ae461640 1 Processor -- start 2024-03-17T23:49:48.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.162+0000 7f85ae461640 1 -- start start 2024-03-17T23:49:48.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.163+0000 7f85ae461640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f85a815ab90 0x7f85a815bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:48.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.163+0000 7f85ae461640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f85a806ecd0 0x7f85a806f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:48.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.163+0000 7f85ae461640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85a80ce790 con 0x7f85a81514f0 2024-03-17T23:49:48.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.163+0000 7f85ae461640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f85a806fba0 con 0x7f85a815ab90 2024-03-17T23:49:48.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.163+0000 7f85ae461640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f85a815b050 con 0x7f85a806ecd0 2024-03-17T23:49:48.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.163+0000 7f85a77fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f85a815ab90 0x7f85a815bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:48.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.163+0000 7f85a77fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f85a815ab90 0x7f85a815bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60484/0 (socket says 172.21.15.150:60484) 2024-03-17T23:49:48.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.163+0000 7f85a7fff640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f85a81514f0 0x7f85a806e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:56922/0 (socket says 172.21.15.150:56922) 2024-03-17T23:49:48.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.163+0000 7f85a77fe640 1 -- 172.21.15.150:0/3700389625 learned_addr learned my addr 172.21.15.150:0/3700389625 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:48.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.163+0000 7f85ac9fe640 1 --2- 172.21.15.150:0/3700389625 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f85a806ecd0 0x7f85a806f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:48.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.164+0000 7f85a77fe640 1 -- 172.21.15.150:0/3700389625 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f85a806ecd0 msgr2=0x7f85a806f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:48.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.164+0000 7f85a77fe640 1 --2- 172.21.15.150:0/3700389625 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f85a806ecd0 0x7f85a806f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.164+0000 7f85a77fe640 1 -- 172.21.15.150:0/3700389625 >> v1:172.21.15.157:6789/0 conn(0x7f85a81514f0 legacy=0x7f85a806e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:48.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.164+0000 7f85a77fe640 1 -- 172.21.15.150:0/3700389625 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a815b330 con 0x7f85a815ab90 2024-03-17T23:49:48.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.164+0000 7f85a6ffd640 1 -- 172.21.15.150:0/3700389625 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 461458394 0 0) 0x7f8598001110 con 0x7f85a81514f0 2024-03-17T23:49:48.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.164+0000 7f85a77fe640 1 --2- 172.21.15.150:0/3700389625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85a815ab90 0x7f85a815bd10 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f859c00bb00 tx=0x7f859c02df30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f98c248ad949ab3 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:48.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.165+0000 7f85a6ffd640 1 -- 172.21.15.150:0/3700389625 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859c034070 con 0x7f85a815ab90 2024-03-17T23:49:48.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.165+0000 7f85a6ffd640 1 -- 172.21.15.150:0/3700389625 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f859c02ece0 con 0x7f85a815ab90 2024-03-17T23:49:48.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.165+0000 7f85a6ffd640 1 -- 172.21.15.150:0/3700389625 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859c038620 con 0x7f85a815ab90 2024-03-17T23:49:48.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.165+0000 7f85ae461640 1 -- 172.21.15.150:0/3700389625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85a815ab90 msgr2=0x7f85a815bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:48.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.165+0000 7f85ae461640 1 --2- 172.21.15.150:0/3700389625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85a815ab90 0x7f85a815bd10 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f859c00bb00 tx=0x7f859c02df30 comp rx=0 tx=0).stop 2024-03-17T23:49:48.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.165+0000 7f85ae461640 1 -- 172.21.15.150:0/3700389625 shutdown_connections 2024-03-17T23:49:48.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.165+0000 7f85ae461640 1 --2- 172.21.15.150:0/3700389625 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f85a806ecd0 0x7f85a806f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.165+0000 7f85ae461640 1 --2- 172.21.15.150:0/3700389625 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85a815ab90 0x7f85a815bd10 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.165+0000 7f85ae461640 1 -- 172.21.15.150:0/3700389625 >> 172.21.15.150:0/3700389625 conn(0x7f85a806b190 msgr2=0x7f85a806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:48.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.165+0000 7f85ae461640 1 -- 172.21.15.150:0/3700389625 shutdown_connections 2024-03-17T23:49:48.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.165+0000 7f85ae461640 1 -- 172.21.15.150:0/3700389625 wait complete. 2024-03-17T23:49:48.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.166+0000 7f85ae461640 1 Processor -- start 2024-03-17T23:49:48.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.167+0000 7f85ae461640 1 -- start start 2024-03-17T23:49:48.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.167+0000 7f85ae461640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f85a806ecd0 0x7f85a81492b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:48.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.167+0000 7f85ae461640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85a81514f0 0x7f85a814ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:48.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.167+0000 7f85ae461640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f85a815ab90 0x7f85a814d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:48.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.167+0000 7f85ae461640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f85a80af510 con 0x7f85a81514f0 2024-03-17T23:49:48.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.167+0000 7f85ae461640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f85a815b8f0 con 0x7f85a806ecd0 2024-03-17T23:49:48.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.167+0000 7f85ae461640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f85a81518d0 con 0x7f85a815ab90 2024-03-17T23:49:48.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.167+0000 7f85a77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85a81514f0 0x7f85a814ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:48.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.167+0000 7f85a77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85a81514f0 0x7f85a814ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60494/0 (socket says 172.21.15.150:60494) 2024-03-17T23:49:48.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.167+0000 7f85a77fe640 1 -- 172.21.15.150:0/1194099534 learned_addr learned my addr 172.21.15.150:0/1194099534 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:48.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.167+0000 7f85ac9fe640 1 --2- 172.21.15.150:0/1194099534 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f85a815ab90 0x7f85a814d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:48.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.167+0000 7f85a7fff640 1 --2- 172.21.15.150:0/1194099534 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f85a806ecd0 0x7f85a81492b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:48.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.168+0000 7f85a77fe640 1 -- 172.21.15.150:0/1194099534 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f85a815ab90 msgr2=0x7f85a814d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:48.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.168+0000 7f85a77fe640 1 --2- 172.21.15.150:0/1194099534 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f85a815ab90 0x7f85a814d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.168+0000 7f85a77fe640 1 -- 172.21.15.150:0/1194099534 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f85a806ecd0 msgr2=0x7f85a81492b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:48.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.168+0000 7f85a77fe640 1 --2- 172.21.15.150:0/1194099534 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f85a806ecd0 0x7f85a81492b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.168+0000 7f85a77fe640 1 -- 172.21.15.150:0/1194099534 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a8149e50 con 0x7f85a81514f0 2024-03-17T23:49:48.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.168+0000 7f85a77fe640 1 --2- 172.21.15.150:0/1194099534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85a81514f0 0x7f85a814ab50 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f859c03a040 tx=0x7f859c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:48.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.168+0000 7f85a4ff9640 1 -- 172.21.15.150:0/1194099534 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859c006dd0 con 0x7f85a81514f0 2024-03-17T23:49:48.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.168+0000 7f85a4ff9640 1 -- 172.21.15.150:0/1194099534 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f859c004030 con 0x7f85a81514f0 2024-03-17T23:49:48.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.168+0000 7f85ae461640 1 -- 172.21.15.150:0/1194099534 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85a814e1d0 con 0x7f85a81514f0 2024-03-17T23:49:48.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.168+0000 7f85a4ff9640 1 -- 172.21.15.150:0/1194099534 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859c004440 con 0x7f85a81514f0 2024-03-17T23:49:48.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.168+0000 7f85ae461640 1 -- 172.21.15.150:0/1194099534 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85a814a070 con 0x7f85a81514f0 2024-03-17T23:49:48.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.169+0000 7f85ae461640 1 -- 172.21.15.150:0/1194099534 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85a814a840 con 0x7f85a81514f0 2024-03-17T23:49:48.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.169+0000 7f85a4ff9640 1 -- 172.21.15.150:0/1194099534 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f859c043650 con 0x7f85a81514f0 2024-03-17T23:49:48.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.170+0000 7f85a4ff9640 1 --2- 172.21.15.150:0/1194099534 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f859406bbd0 0x7f859406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:48.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.170+0000 7f85a4ff9640 1 -- 172.21.15.150:0/1194099534 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f859c0af460 con 0x7f85a81514f0 2024-03-17T23:49:48.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.170+0000 7f85a7fff640 1 --2- 172.21.15.150:0/1194099534 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f859406bbd0 0x7f859406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:48.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.170+0000 7f85a7fff640 1 --2- 172.21.15.150:0/1194099534 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f859406bbd0 0x7f859406e090 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8598000f60 tx=0x7f8598007b50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:48.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.174+0000 7f85a4ff9640 1 -- 172.21.15.150:0/1194099534 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f85a8149e50 con 0x7f85a81514f0 2024-03-17T23:49:48.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.344+0000 7f85ae461640 1 -- 172.21.15.150:0/1194099534 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f85a815e570 con 0x7f85a81514f0 2024-03-17T23:49:48.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.345+0000 7f85a4ff9640 1 -- 172.21.15.150:0/1194099534 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) ==== 93+0+10 (secure 0 0 0) 0x7f859c047020 con 0x7f85a81514f0 2024-03-17T23:49:48.346 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-03-17T23:49:48.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.347+0000 7f85ae461640 1 -- 172.21.15.150:0/1194099534 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f859406bbd0 msgr2=0x7f859406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:48.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.347+0000 7f85ae461640 1 --2- 172.21.15.150:0/1194099534 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f859406bbd0 0x7f859406e090 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8598000f60 tx=0x7f8598007b50 comp rx=0 tx=0).stop 2024-03-17T23:49:48.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.347+0000 7f85ae461640 1 -- 172.21.15.150:0/1194099534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85a81514f0 msgr2=0x7f85a814ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:48.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.347+0000 7f85ae461640 1 --2- 172.21.15.150:0/1194099534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85a81514f0 0x7f85a814ab50 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f859c03a040 tx=0x7f859c0046c0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.347+0000 7f85ae461640 1 -- 172.21.15.150:0/1194099534 shutdown_connections 2024-03-17T23:49:48.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.347+0000 7f85ae461640 1 --2- 172.21.15.150:0/1194099534 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f85a815ab90 0x7f85a814d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.347+0000 7f85ae461640 1 --2- 172.21.15.150:0/1194099534 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f85a81514f0 0x7f85a814ab50 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.347+0000 7f85ae461640 1 --2- 172.21.15.150:0/1194099534 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f859406bbd0 0x7f859406e090 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.347+0000 7f85ae461640 1 --2- 172.21.15.150:0/1194099534 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f85a806ecd0 0x7f85a81492b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.347+0000 7f85ae461640 1 -- 172.21.15.150:0/1194099534 >> 172.21.15.150:0/1194099534 conn(0x7f85a806b190 msgr2=0x7f85a815bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:48.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.347+0000 7f85ae461640 1 -- 172.21.15.150:0/1194099534 shutdown_connections 2024-03-17T23:49:48.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.347+0000 7f85ae461640 1 -- 172.21.15.150:0/1194099534 wait complete. 2024-03-17T23:49:48.359 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-17T23:49:48.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.461+0000 7f941813b640 1 Processor -- start 2024-03-17T23:49:48.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.462+0000 7f941813b640 1 -- start start 2024-03-17T23:49:48.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.462+0000 7f941813b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f94100b2bf0 0x7f94100b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:48.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.462+0000 7f941813b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9410161d30 con 0x7f94100d0ce0 2024-03-17T23:49:48.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.462+0000 7f941813b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9410162010 con 0x7f9410161520 2024-03-17T23:49:48.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.462+0000 7f941813b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f94100b9bf0 con 0x7f94100b2bf0 2024-03-17T23:49:48.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.462+0000 7f94166d8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f94100b2bf0 0x7f94100b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:48.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.462+0000 7f94166d8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f94100b2bf0 0x7f94100b96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60496/0 (socket says 172.21.15.150:60496) 2024-03-17T23:49:48.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.462+0000 7f94156d6640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f94100d0ce0 0x7f94100d1fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41866/0 (socket says 172.21.15.150:41866) 2024-03-17T23:49:48.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.462+0000 7f94166d8640 1 -- 172.21.15.150:0/4229509787 learned_addr learned my addr 172.21.15.150:0/4229509787 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:48.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.463+0000 7f94166d8640 1 -- 172.21.15.150:0/4229509787 >> v1:172.21.15.157:6789/0 conn(0x7f9410161520 legacy=0x7f94100d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:48.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.463+0000 7f94166d8640 1 -- 172.21.15.150:0/4229509787 >> v1:172.21.15.150:6789/0 conn(0x7f94100d0ce0 legacy=0x7f94100d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:48.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.463+0000 7f9414ed5640 1 -- 172.21.15.150:0/4229509787 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2592264804 0 0) 0x7f9410162010 con 0x7f9410161520 2024-03-17T23:49:48.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.463+0000 7f94166d8640 1 -- 172.21.15.150:0/4229509787 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94101622f0 con 0x7f94100b2bf0 2024-03-17T23:49:48.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.463+0000 7f94166d8640 1 --2- 172.21.15.150:0/4229509787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94100b2bf0 0x7f94100b96b0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f9408009870 tx=0x7f940802b740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e770f9833659e867 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:48.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.464+0000 7f9414ed5640 1 -- 172.21.15.150:0/4229509787 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9408037070 con 0x7f94100b2bf0 2024-03-17T23:49:48.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.464+0000 7f9414ed5640 1 -- 172.21.15.150:0/4229509787 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9408030710 con 0x7f94100b2bf0 2024-03-17T23:49:48.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.464+0000 7f9414ed5640 1 -- 172.21.15.150:0/4229509787 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9408030ae0 con 0x7f94100b2bf0 2024-03-17T23:49:48.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.464+0000 7f941813b640 1 -- 172.21.15.150:0/4229509787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94100b2bf0 msgr2=0x7f94100b96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:48.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.464+0000 7f941813b640 1 --2- 172.21.15.150:0/4229509787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94100b2bf0 0x7f94100b96b0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f9408009870 tx=0x7f940802b740 comp rx=0 tx=0).stop 2024-03-17T23:49:48.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.464+0000 7f941813b640 1 -- 172.21.15.150:0/4229509787 shutdown_connections 2024-03-17T23:49:48.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.464+0000 7f941813b640 1 --2- 172.21.15.150:0/4229509787 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94100b2bf0 0x7f94100b96b0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.464+0000 7f941813b640 1 -- 172.21.15.150:0/4229509787 >> 172.21.15.150:0/4229509787 conn(0x7f941006ca00 msgr2=0x7f941006ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:48.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.464+0000 7f941813b640 1 -- 172.21.15.150:0/4229509787 shutdown_connections 2024-03-17T23:49:48.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.464+0000 7f941813b640 1 -- 172.21.15.150:0/4229509787 wait complete. 2024-03-17T23:49:48.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.465+0000 7f941813b640 1 Processor -- start 2024-03-17T23:49:48.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.465+0000 7f941813b640 1 -- start start 2024-03-17T23:49:48.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.466+0000 7f941813b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94100b2bf0 0x7f94101f3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:48.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.466+0000 7f941813b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94100d0ce0 0x7f94101f6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:48.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.466+0000 7f941813b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9410161520 0x7f94101f7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:48.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.466+0000 7f941813b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9410161900 con 0x7f94100b2bf0 2024-03-17T23:49:48.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.466+0000 7f941813b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f9410162940 con 0x7f94100d0ce0 2024-03-17T23:49:48.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.466+0000 7f941813b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f94100d26b0 con 0x7f9410161520 2024-03-17T23:49:48.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.466+0000 7f9415ed7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94100b2bf0 0x7f94101f3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:48.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.466+0000 7f9415ed7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94100b2bf0 0x7f94101f3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60506/0 (socket says 172.21.15.150:60506) 2024-03-17T23:49:48.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.466+0000 7f9415ed7640 1 -- 172.21.15.150:0/142518597 learned_addr learned my addr 172.21.15.150:0/142518597 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:48.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.466+0000 7f94166d8640 1 --2- 172.21.15.150:0/142518597 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9410161520 0x7f94101f7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:48.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.466+0000 7f94156d6640 1 --2- 172.21.15.150:0/142518597 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94100d0ce0 0x7f94101f6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:48.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.467+0000 7f9415ed7640 1 -- 172.21.15.150:0/142518597 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9410161520 msgr2=0x7f94101f7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:48.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.467+0000 7f9415ed7640 1 --2- 172.21.15.150:0/142518597 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9410161520 0x7f94101f7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.467+0000 7f9415ed7640 1 -- 172.21.15.150:0/142518597 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94100d0ce0 msgr2=0x7f94101f6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:48.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.467+0000 7f9415ed7640 1 --2- 172.21.15.150:0/142518597 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94100d0ce0 0x7f94101f6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.467+0000 7f9415ed7640 1 -- 172.21.15.150:0/142518597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f941015f410 con 0x7f94100b2bf0 2024-03-17T23:49:48.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.467+0000 7f9415ed7640 1 --2- 172.21.15.150:0/142518597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94100b2bf0 0x7f94101f3240 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f940c00c810 tx=0x7f940c00ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:48.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.467+0000 7f94067fc640 1 -- 172.21.15.150:0/142518597 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940c017070 con 0x7f94100b2bf0 2024-03-17T23:49:48.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.467+0000 7f94067fc640 1 -- 172.21.15.150:0/142518597 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f940c003dc0 con 0x7f94100b2bf0 2024-03-17T23:49:48.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.467+0000 7f941813b640 1 -- 172.21.15.150:0/142518597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f941015f6f0 con 0x7f94100b2bf0 2024-03-17T23:49:48.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.467+0000 7f94067fc640 1 -- 172.21.15.150:0/142518597 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940c0052b0 con 0x7f94100b2bf0 2024-03-17T23:49:48.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.467+0000 7f941813b640 1 -- 172.21.15.150:0/142518597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f941015fcb0 con 0x7f94100b2bf0 2024-03-17T23:49:48.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.468+0000 7f941813b640 1 -- 172.21.15.150:0/142518597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94100681a0 con 0x7f94100b2bf0 2024-03-17T23:49:48.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.468+0000 7f94067fc640 1 -- 172.21.15.150:0/142518597 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f940c01a070 con 0x7f94100b2bf0 2024-03-17T23:49:48.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.469+0000 7f94067fc640 1 --2- 172.21.15.150:0/142518597 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f93f406bbd0 0x7f93f406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:48.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.469+0000 7f94067fc640 1 -- 172.21.15.150:0/142518597 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f940c08a240 con 0x7f94100b2bf0 2024-03-17T23:49:48.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.469+0000 7f94156d6640 1 --2- 172.21.15.150:0/142518597 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f93f406bbd0 0x7f93f406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:48.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.469+0000 7f94156d6640 1 --2- 172.21.15.150:0/142518597 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f93f406bbd0 0x7f93f406e090 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f94100c9cf0 tx=0x7f9400007e20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:48.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.473+0000 7f94067fc640 1 -- 172.21.15.150:0/142518597 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f940c0585e0 con 0x7f94100b2bf0 2024-03-17T23:49:48.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.647+0000 7f941813b640 1 -- 172.21.15.150:0/142518597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f94100d19c0 con 0x7f94100b2bf0 2024-03-17T23:49:48.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.647+0000 7f94067fc640 1 -- 172.21.15.150:0/142518597 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v58) ==== 104+0+11 (secure 0 0 0) 0x7f940c05c680 con 0x7f94100b2bf0 2024-03-17T23:49:48.648 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-03-17T23:49:48.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.649+0000 7f941813b640 1 -- 172.21.15.150:0/142518597 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f93f406bbd0 msgr2=0x7f93f406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:48.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.649+0000 7f941813b640 1 --2- 172.21.15.150:0/142518597 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f93f406bbd0 0x7f93f406e090 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f94100c9cf0 tx=0x7f9400007e20 comp rx=0 tx=0).stop 2024-03-17T23:49:48.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.649+0000 7f941813b640 1 -- 172.21.15.150:0/142518597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94100b2bf0 msgr2=0x7f94101f3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:48.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.649+0000 7f941813b640 1 --2- 172.21.15.150:0/142518597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94100b2bf0 0x7f94101f3240 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f940c00c810 tx=0x7f940c00ccd0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.649+0000 7f941813b640 1 -- 172.21.15.150:0/142518597 shutdown_connections 2024-03-17T23:49:48.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.649+0000 7f941813b640 1 --2- 172.21.15.150:0/142518597 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f9410161520 0x7f94101f7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.649+0000 7f941813b640 1 --2- 172.21.15.150:0/142518597 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f93f406bbd0 0x7f93f406e090 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.649+0000 7f941813b640 1 --2- 172.21.15.150:0/142518597 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f94100d0ce0 0x7f94101f6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.649+0000 7f941813b640 1 --2- 172.21.15.150:0/142518597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94100b2bf0 0x7f94101f3240 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.649+0000 7f941813b640 1 -- 172.21.15.150:0/142518597 >> 172.21.15.150:0/142518597 conn(0x7f941006ca00 msgr2=0x7f94100c7050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:48.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.649+0000 7f941813b640 1 -- 172.21.15.150:0/142518597 shutdown_connections 2024-03-17T23:49:48.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.650+0000 7f941813b640 1 -- 172.21.15.150:0/142518597 wait complete. 2024-03-17T23:49:48.661 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-17T23:49:48.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.765+0000 7f04744bf640 1 Processor -- start 2024-03-17T23:49:48.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.765+0000 7f04744bf640 1 -- start start 2024-03-17T23:49:48.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.766+0000 7f04744bf640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f046c1514f0 0x7f046c15bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:48.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.766+0000 7f04744bf640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f046c15ab90 0x7f046c06f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:48.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.766+0000 7f04744bf640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f046c0ce790 con 0x7f046c06e5c0 2024-03-17T23:49:48.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.766+0000 7f04744bf640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f046c06fba0 con 0x7f046c1514f0 2024-03-17T23:49:48.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.766+0000 7f04744bf640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f046c15b050 con 0x7f046c15ab90 2024-03-17T23:49:48.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.766+0000 7f047225b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f046c1514f0 0x7f046c15bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:48.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.766+0000 7f047225b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f046c1514f0 0x7f046c15bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60510/0 (socket says 172.21.15.150:60510) 2024-03-17T23:49:48.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.766+0000 7f047225b640 1 -- 172.21.15.150:0/3904021096 learned_addr learned my addr 172.21.15.150:0/3904021096 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:48.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.766+0000 7f0471a5a640 1 --2- 172.21.15.150:0/3904021096 >> v2:172.21.15.157:3300/0 conn(0x7f046c15ab90 0x7f046c06f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:48.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.767+0000 7f047225b640 1 -- 172.21.15.150:0/3904021096 >> v2:172.21.15.157:3300/0 conn(0x7f046c15ab90 msgr2=0x7f046c06f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:48.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.767+0000 7f047225b640 1 --2- 172.21.15.150:0/3904021096 >> v2:172.21.15.157:3300/0 conn(0x7f046c15ab90 0x7f046c06f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.767+0000 7f047225b640 1 -- 172.21.15.150:0/3904021096 >> v1:172.21.15.157:6789/0 conn(0x7f046c06e5c0 legacy=0x7f046c0cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:48.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.767+0000 7f047225b640 1 -- 172.21.15.150:0/3904021096 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f046c15b330 con 0x7f046c1514f0 2024-03-17T23:49:48.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.767+0000 7f047225b640 1 --2- 172.21.15.150:0/3904021096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f046c1514f0 0x7f046c15bd10 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f046c06f2d0 tx=0x7f046802d090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f73c999b7a2245f server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:48.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.768+0000 7f0471259640 1 -- 172.21.15.150:0/3904021096 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046802dc30 con 0x7f046c1514f0 2024-03-17T23:49:48.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.768+0000 7f0471259640 1 -- 172.21.15.150:0/3904021096 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f046802ddd0 con 0x7f046c1514f0 2024-03-17T23:49:48.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.768+0000 7f0471259640 1 -- 172.21.15.150:0/3904021096 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0468035610 con 0x7f046c1514f0 2024-03-17T23:49:48.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.768+0000 7f04744bf640 1 -- 172.21.15.150:0/3904021096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f046c1514f0 msgr2=0x7f046c15bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:48.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.768+0000 7f04744bf640 1 --2- 172.21.15.150:0/3904021096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f046c1514f0 0x7f046c15bd10 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f046c06f2d0 tx=0x7f046802d090 comp rx=0 tx=0).stop 2024-03-17T23:49:48.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.768+0000 7f04744bf640 1 -- 172.21.15.150:0/3904021096 shutdown_connections 2024-03-17T23:49:48.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.768+0000 7f04744bf640 1 --2- 172.21.15.150:0/3904021096 >> v2:172.21.15.157:3300/0 conn(0x7f046c15ab90 0x7f046c06f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.768+0000 7f04744bf640 1 --2- 172.21.15.150:0/3904021096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f046c1514f0 0x7f046c15bd10 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.768+0000 7f04744bf640 1 -- 172.21.15.150:0/3904021096 >> 172.21.15.150:0/3904021096 conn(0x7f046c06b190 msgr2=0x7f046c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:48.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.768+0000 7f04744bf640 1 -- 172.21.15.150:0/3904021096 shutdown_connections 2024-03-17T23:49:48.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.768+0000 7f04744bf640 1 -- 172.21.15.150:0/3904021096 wait complete. 2024-03-17T23:49:48.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.769+0000 7f04744bf640 1 Processor -- start 2024-03-17T23:49:48.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.769+0000 7f04744bf640 1 -- start start 2024-03-17T23:49:48.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.770+0000 7f04744bf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f046c06e5c0 0x7f046c166790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:48.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.770+0000 7f04744bf640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f046c1514f0 0x7f046c1436c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:48.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.770+0000 7f04744bf640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f046c15ab90 0x7f046c0b3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:48.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.770+0000 7f04744bf640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f046c0af510 con 0x7f046c06e5c0 2024-03-17T23:49:48.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.770+0000 7f04744bf640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f046c15b8f0 con 0x7f046c15ab90 2024-03-17T23:49:48.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.770+0000 7f04744bf640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f046c1518d0 con 0x7f046c1514f0 2024-03-17T23:49:48.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.770+0000 7f047225b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f046c06e5c0 0x7f046c166790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:48.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.770+0000 7f047225b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f046c06e5c0 0x7f046c166790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60526/0 (socket says 172.21.15.150:60526) 2024-03-17T23:49:48.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.770+0000 7f047225b640 1 -- 172.21.15.150:0/3414589424 learned_addr learned my addr 172.21.15.150:0/3414589424 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:48.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.770+0000 7f0471a5a640 1 --2- 172.21.15.150:0/3414589424 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f046c1514f0 0x7f046c1436c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:48.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.770+0000 7f0472a5c640 1 --2- 172.21.15.150:0/3414589424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f046c15ab90 0x7f046c0b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:48.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.770+0000 7f047225b640 1 -- 172.21.15.150:0/3414589424 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f046c1514f0 msgr2=0x7f046c1436c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:48.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.770+0000 7f047225b640 1 --2- 172.21.15.150:0/3414589424 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f046c1514f0 0x7f046c1436c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.770+0000 7f047225b640 1 -- 172.21.15.150:0/3414589424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f046c15ab90 msgr2=0x7f046c0b3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:48.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.770+0000 7f047225b640 1 --2- 172.21.15.150:0/3414589424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f046c15ab90 0x7f046c0b3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.770+0000 7f047225b640 1 -- 172.21.15.150:0/3414589424 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f046c1659d0 con 0x7f046c06e5c0 2024-03-17T23:49:48.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.771+0000 7f047225b640 1 --2- 172.21.15.150:0/3414589424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f046c06e5c0 0x7f046c166790 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f0468000f80 tx=0x7f0468004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:48.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.771+0000 7f0462ffd640 1 -- 172.21.15.150:0/3414589424 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0468009510 con 0x7f046c06e5c0 2024-03-17T23:49:48.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.771+0000 7f0462ffd640 1 -- 172.21.15.150:0/3414589424 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0468044070 con 0x7f046c06e5c0 2024-03-17T23:49:48.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.771+0000 7f0462ffd640 1 -- 172.21.15.150:0/3414589424 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046803f880 con 0x7f046c06e5c0 2024-03-17T23:49:48.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.771+0000 7f04744bf640 1 -- 172.21.15.150:0/3414589424 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f046c165cb0 con 0x7f046c06e5c0 2024-03-17T23:49:48.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.771+0000 7f04744bf640 1 -- 172.21.15.150:0/3414589424 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f046c166270 con 0x7f046c06e5c0 2024-03-17T23:49:48.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.772+0000 7f04744bf640 1 -- 172.21.15.150:0/3414589424 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f046c0681a0 con 0x7f046c06e5c0 2024-03-17T23:49:48.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.772+0000 7f0462ffd640 1 -- 172.21.15.150:0/3414589424 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f046802dc40 con 0x7f046c06e5c0 2024-03-17T23:49:48.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.773+0000 7f0462ffd640 1 --2- 172.21.15.150:0/3414589424 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f045006bc00 0x7f045006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:48.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.773+0000 7f0462ffd640 1 -- 172.21.15.150:0/3414589424 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f04680acae0 con 0x7f046c06e5c0 2024-03-17T23:49:48.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.773+0000 7f0471a5a640 1 --2- 172.21.15.150:0/3414589424 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f045006bc00 0x7f045006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:48.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.773+0000 7f0471a5a640 1 --2- 172.21.15.150:0/3414589424 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f045006bc00 0x7f045006e0c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f046c145bf0 tx=0x7f045c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:48.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.777+0000 7f0462ffd640 1 -- 172.21.15.150:0/3414589424 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f046804d020 con 0x7f046c06e5c0 2024-03-17T23:49:48.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.951+0000 7f04744bf640 1 -- 172.21.15.150:0/3414589424 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f046c06bf90 con 0x7f046c06e5c0 2024-03-17T23:49:48.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.951+0000 7f0462ffd640 1 -- 172.21.15.150:0/3414589424 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v58) ==== 100+0+11 (secure 0 0 0) 0x7f046807ae80 con 0x7f046c06e5c0 2024-03-17T23:49:48.952 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-03-17T23:49:48.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.953+0000 7f04744bf640 1 -- 172.21.15.150:0/3414589424 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f045006bc00 msgr2=0x7f045006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:48.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.953+0000 7f04744bf640 1 --2- 172.21.15.150:0/3414589424 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f045006bc00 0x7f045006e0c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f046c145bf0 tx=0x7f045c009190 comp rx=0 tx=0).stop 2024-03-17T23:49:48.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.953+0000 7f04744bf640 1 -- 172.21.15.150:0/3414589424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f046c06e5c0 msgr2=0x7f046c166790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:48.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.953+0000 7f04744bf640 1 --2- 172.21.15.150:0/3414589424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f046c06e5c0 0x7f046c166790 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f0468000f80 tx=0x7f0468004a40 comp rx=0 tx=0).stop 2024-03-17T23:49:48.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.953+0000 7f04744bf640 1 -- 172.21.15.150:0/3414589424 shutdown_connections 2024-03-17T23:49:48.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.953+0000 7f04744bf640 1 --2- 172.21.15.150:0/3414589424 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f045006bc00 0x7f045006e0c0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.953+0000 7f04744bf640 1 --2- 172.21.15.150:0/3414589424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f046c15ab90 0x7f046c0b3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.953+0000 7f04744bf640 1 --2- 172.21.15.150:0/3414589424 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f046c1514f0 0x7f046c1436c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.953+0000 7f04744bf640 1 --2- 172.21.15.150:0/3414589424 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f046c06e5c0 0x7f046c166790 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:48.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.953+0000 7f04744bf640 1 -- 172.21.15.150:0/3414589424 >> 172.21.15.150:0/3414589424 conn(0x7f046c06b190 msgr2=0x7f046c0af790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:48.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.953+0000 7f04744bf640 1 -- 172.21.15.150:0/3414589424 shutdown_connections 2024-03-17T23:49:48.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:48.953+0000 7f04744bf640 1 -- 172.21.15.150:0/3414589424 wait complete. 2024-03-17T23:49:48.965 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:49:49.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.064+0000 7f629fb53640 1 Processor -- start 2024-03-17T23:49:49.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.065+0000 7f629fb53640 1 -- start start 2024-03-17T23:49:49.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.065+0000 7f629fb53640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f629815bfd0 0x7f62980ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:49.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.065+0000 7f629fb53640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62980cf3b0 con 0x7f6298155320 2024-03-17T23:49:49.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.065+0000 7f629fb53640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62980cf690 con 0x7f62980cd250 2024-03-17T23:49:49.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.065+0000 7f629fb53640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f629815c550 con 0x7f629815bfd0 2024-03-17T23:49:49.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.066+0000 7f629d0ee640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f629815bfd0 0x7f62980ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:49.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.066+0000 7f629d0ee640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f629815bfd0 0x7f62980ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60530/0 (socket says 172.21.15.150:60530) 2024-03-17T23:49:49.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.066+0000 7f629d0ee640 1 -- 172.21.15.150:0/138491212 learned_addr learned my addr 172.21.15.150:0/138491212 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:49.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.066+0000 7f629d0ee640 1 -- 172.21.15.150:0/138491212 >> v1:172.21.15.157:6789/0 conn(0x7f62980cd250 legacy=0x7f629815ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:49.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.066+0000 7f629d0ee640 1 -- 172.21.15.150:0/138491212 >> v1:172.21.15.150:6789/0 conn(0x7f6298155320 legacy=0x7f629806dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:49.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.066+0000 7f629c8ed640 1 -- 172.21.15.150:0/138491212 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3571089906 0 0) 0x7f62980cf690 con 0x7f62980cd250 2024-03-17T23:49:49.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.066+0000 7f629d0ee640 1 -- 172.21.15.150:0/138491212 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f629815c7a0 con 0x7f629815bfd0 2024-03-17T23:49:49.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.067+0000 7f629d0ee640 1 --2- 172.21.15.150:0/138491212 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f629815bfd0 0x7f62980ccd10 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f6288009870 tx=0x7f628802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4971646541280620 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:49.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.067+0000 7f629c8ed640 1 -- 172.21.15.150:0/138491212 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6288038070 con 0x7f629815bfd0 2024-03-17T23:49:49.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.067+0000 7f629c8ed640 1 -- 172.21.15.150:0/138491212 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6288031740 con 0x7f629815bfd0 2024-03-17T23:49:49.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.067+0000 7f629c8ed640 1 -- 172.21.15.150:0/138491212 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f628802f050 con 0x7f629815bfd0 2024-03-17T23:49:49.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.067+0000 7f629fb53640 1 -- 172.21.15.150:0/138491212 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f629815bfd0 msgr2=0x7f62980ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:49.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.067+0000 7f629fb53640 1 --2- 172.21.15.150:0/138491212 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f629815bfd0 0x7f62980ccd10 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f6288009870 tx=0x7f628802b860 comp rx=0 tx=0).stop 2024-03-17T23:49:49.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.067+0000 7f629fb53640 1 -- 172.21.15.150:0/138491212 shutdown_connections 2024-03-17T23:49:49.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.067+0000 7f629fb53640 1 --2- 172.21.15.150:0/138491212 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f629815bfd0 0x7f62980ccd10 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.067+0000 7f629fb53640 1 -- 172.21.15.150:0/138491212 >> 172.21.15.150:0/138491212 conn(0x7f629806b190 msgr2=0x7f629806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:49.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.067+0000 7f629fb53640 1 -- 172.21.15.150:0/138491212 shutdown_connections 2024-03-17T23:49:49.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.067+0000 7f629fb53640 1 -- 172.21.15.150:0/138491212 wait complete. 2024-03-17T23:49:49.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.069+0000 7f629fb53640 1 Processor -- start 2024-03-17T23:49:49.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.069+0000 7f629fb53640 1 -- start start 2024-03-17T23:49:49.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.069+0000 7f629fb53640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62980cd250 0x7f629813b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:49.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.069+0000 7f629fb53640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6298155320 0x7f629814b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:49.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.069+0000 7f629fb53640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f629815bfd0 0x7f629814be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:49.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.069+0000 7f629fb53640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6298155880 con 0x7f62980cd250 2024-03-17T23:49:49.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.069+0000 7f629fb53640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6298155700 con 0x7f6298155320 2024-03-17T23:49:49.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.069+0000 7f629fb53640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f629815d3d0 con 0x7f629815bfd0 2024-03-17T23:49:49.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.069+0000 7f629d8ef640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62980cd250 0x7f629813b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:49.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.069+0000 7f629d8ef640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62980cd250 0x7f629813b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60544/0 (socket says 172.21.15.150:60544) 2024-03-17T23:49:49.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.069+0000 7f629d8ef640 1 -- 172.21.15.150:0/767872700 learned_addr learned my addr 172.21.15.150:0/767872700 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:49.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.069+0000 7f629d0ee640 1 --2- 172.21.15.150:0/767872700 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6298155320 0x7f629814b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:49.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.070+0000 7f629e0f0640 1 --2- 172.21.15.150:0/767872700 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f629815bfd0 0x7f629814be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:49.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.070+0000 7f629d8ef640 1 -- 172.21.15.150:0/767872700 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f629815bfd0 msgr2=0x7f629814be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:49.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.070+0000 7f629d8ef640 1 --2- 172.21.15.150:0/767872700 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f629815bfd0 0x7f629814be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.070+0000 7f629d8ef640 1 -- 172.21.15.150:0/767872700 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6298155320 msgr2=0x7f629814b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.070+0000 7f629d8ef640 1 --2- 172.21.15.150:0/767872700 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6298155320 0x7f629814b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.070+0000 7f629d8ef640 1 -- 172.21.15.150:0/767872700 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6298157780 con 0x7f62980cd250 2024-03-17T23:49:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.070+0000 7f629d8ef640 1 --2- 172.21.15.150:0/767872700 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62980cd250 0x7f629813b1f0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f629000cb20 tx=0x7f6290002100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.070+0000 7f628e7fc640 1 -- 172.21.15.150:0/767872700 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6290017070 con 0x7f62980cd250 2024-03-17T23:49:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.070+0000 7f628e7fc640 1 -- 172.21.15.150:0/767872700 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6290003da0 con 0x7f62980cd250 2024-03-17T23:49:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.070+0000 7f628e7fc640 1 -- 172.21.15.150:0/767872700 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f629000a950 con 0x7f62980cd250 2024-03-17T23:49:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.070+0000 7f629fb53640 1 -- 172.21.15.150:0/767872700 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6298157a60 con 0x7f62980cd250 2024-03-17T23:49:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.070+0000 7f629fb53640 1 -- 172.21.15.150:0/767872700 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6298158020 con 0x7f62980cd250 2024-03-17T23:49:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.071+0000 7f629fb53640 1 -- 172.21.15.150:0/767872700 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62980681a0 con 0x7f62980cd250 2024-03-17T23:49:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.072+0000 7f628e7fc640 1 -- 172.21.15.150:0/767872700 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f629001a070 con 0x7f62980cd250 2024-03-17T23:49:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.072+0000 7f628e7fc640 1 --2- 172.21.15.150:0/767872700 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f627c06bc00 0x7f627c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:49.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.072+0000 7f628e7fc640 1 -- 172.21.15.150:0/767872700 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f629008a390 con 0x7f62980cd250 2024-03-17T23:49:49.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.072+0000 7f629d0ee640 1 --2- 172.21.15.150:0/767872700 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f627c06bc00 0x7f627c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:49.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.072+0000 7f629d0ee640 1 --2- 172.21.15.150:0/767872700 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f627c06bc00 0x7f627c06e0c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6288035000 tx=0x7f6288008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:49.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.076+0000 7f628e7fc640 1 -- 172.21.15.150:0/767872700 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f62900586a0 con 0x7f62980cd250 2024-03-17T23:49:49.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.284+0000 7f629fb53640 1 -- 172.21.15.150:0/767872700 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f629815d000 con 0x7f62980cd250 2024-03-17T23:49:49.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.285+0000 7f628e7fc640 1 -- 172.21.15.150:0/767872700 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+5187 (secure 0 0 0) 0x7f629005c740 con 0x7f62980cd250 2024-03-17T23:49:49.285 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:49.286 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":22,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/2082413564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2082413564},{"type":"v1","addr":"172.21.15.150:6836","nonce":2082413564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":6043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/2350077845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":2350077845},{"type":"v1","addr":"172.21.15.150:6837","nonce":2350077845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":6052,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6834/137606728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":137606728},{"type":"v1","addr":"172.21.15.157:6834","nonce":137606728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-17T23:49:10.996124+0000","modified":"2024-03-17T23:49:12.994518+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6835/479821580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6833","nonce":479821580},{"type":"v1","addr":"172.21.15.157:6835","nonce":479821580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6204,"qdb_cluster":[6204]},"id":3}]} 2024-03-17T23:49:49.286 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 22 2024-03-17T23:49:49.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.287+0000 7f629fb53640 1 -- 172.21.15.150:0/767872700 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f627c06bc00 msgr2=0x7f627c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:49.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.287+0000 7f629fb53640 1 --2- 172.21.15.150:0/767872700 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f627c06bc00 0x7f627c06e0c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6288035000 tx=0x7f6288008030 comp rx=0 tx=0).stop 2024-03-17T23:49:49.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.287+0000 7f629fb53640 1 -- 172.21.15.150:0/767872700 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62980cd250 msgr2=0x7f629813b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:49.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.287+0000 7f629fb53640 1 --2- 172.21.15.150:0/767872700 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62980cd250 0x7f629813b1f0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f629000cb20 tx=0x7f6290002100 comp rx=0 tx=0).stop 2024-03-17T23:49:49.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.287+0000 7f629fb53640 1 -- 172.21.15.150:0/767872700 shutdown_connections 2024-03-17T23:49:49.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.287+0000 7f629fb53640 1 --2- 172.21.15.150:0/767872700 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f629815bfd0 0x7f629814be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.287+0000 7f629fb53640 1 --2- 172.21.15.150:0/767872700 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f627c06bc00 0x7f627c06e0c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.287+0000 7f629fb53640 1 --2- 172.21.15.150:0/767872700 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6298155320 0x7f629814b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.287+0000 7f629fb53640 1 --2- 172.21.15.150:0/767872700 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f62980cd250 0x7f629813b1f0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.287+0000 7f629fb53640 1 -- 172.21.15.150:0/767872700 >> 172.21.15.150:0/767872700 conn(0x7f629806b190 msgr2=0x7f6298158d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:49.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.287+0000 7f629fb53640 1 -- 172.21.15.150:0/767872700 shutdown_connections 2024-03-17T23:49:49.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.287+0000 7f629fb53640 1 -- 172.21.15.150:0/767872700 wait complete. 2024-03-17T23:49:49.298 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:49:49.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.400+0000 7f799cbc6640 1 Processor -- start 2024-03-17T23:49:49.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.400+0000 7f799cbc6640 1 -- start start 2024-03-17T23:49:49.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.401+0000 7f799cbc6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f79980b27b0 0x7f79980af580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:49.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.401+0000 7f799cbc6640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f7998072240 0x7f79980726d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:49.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.401+0000 7f799cbc6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79980b5270 con 0x7f799806f620 2024-03-17T23:49:49.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.401+0000 7f799cbc6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f799815e610 con 0x7f79980b27b0 2024-03-17T23:49:49.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.401+0000 7f799cbc6640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f799815e8f0 con 0x7f7998072240 2024-03-17T23:49:49.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.401+0000 7f7995d9b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f79980b27b0 0x7f79980af580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:49.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.401+0000 7f799659c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f799806f620 0x7f799806fa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41884/0 (socket says 172.21.15.150:41884) 2024-03-17T23:49:49.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.401+0000 7f799659c640 1 -- 172.21.15.150:0/757372204 learned_addr learned my addr 172.21.15.150:0/757372204 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:49.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.401+0000 7f7996d9d640 1 --2- 172.21.15.150:0/757372204 >> v2:172.21.15.157:3300/0 conn(0x7f7998072240 0x7f79980726d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:49.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.401+0000 7f7996d9d640 1 -- 172.21.15.150:0/757372204 >> v2:172.21.15.150:3300/0 conn(0x7f79980b27b0 msgr2=0x7f79980af580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:49.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.401+0000 7f7996d9d640 1 --2- 172.21.15.150:0/757372204 >> v2:172.21.15.150:3300/0 conn(0x7f79980b27b0 0x7f79980af580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.401+0000 7f7996d9d640 1 -- 172.21.15.150:0/757372204 >> v1:172.21.15.150:6789/0 conn(0x7f799806f620 legacy=0x7f799806fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:49.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.401+0000 7f7996d9d640 1 -- 172.21.15.150:0/757372204 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f799815ebd0 con 0x7f7998072240 2024-03-17T23:49:49.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.402+0000 7f7996d9d640 1 --2- 172.21.15.150:0/757372204 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7998072240 0x7f79980726d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f799809df10 tx=0x7f799002bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d2efa7399be3188 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:49.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.402+0000 7f799559a640 1 -- 172.21.15.150:0/757372204 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7990039070 con 0x7f7998072240 2024-03-17T23:49:49.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.402+0000 7f799559a640 1 -- 172.21.15.150:0/757372204 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f799002cc80 con 0x7f7998072240 2024-03-17T23:49:49.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.402+0000 7f799559a640 1 -- 172.21.15.150:0/757372204 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7990034660 con 0x7f7998072240 2024-03-17T23:49:49.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.403+0000 7f799cbc6640 1 -- 172.21.15.150:0/757372204 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7998072240 msgr2=0x7f79980726d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:49.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.403+0000 7f799cbc6640 1 --2- 172.21.15.150:0/757372204 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7998072240 0x7f79980726d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f799809df10 tx=0x7f799002bed0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.403+0000 7f799cbc6640 1 -- 172.21.15.150:0/757372204 shutdown_connections 2024-03-17T23:49:49.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.403+0000 7f799cbc6640 1 --2- 172.21.15.150:0/757372204 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7998072240 0x7f79980726d0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.403+0000 7f799cbc6640 1 --2- 172.21.15.150:0/757372204 >> v2:172.21.15.150:3300/0 conn(0x7f79980b27b0 0x7f79980af580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.403+0000 7f799cbc6640 1 -- 172.21.15.150:0/757372204 >> 172.21.15.150:0/757372204 conn(0x7f79980b1c90 msgr2=0x7f79980b20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:49.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.403+0000 7f799cbc6640 1 -- 172.21.15.150:0/757372204 shutdown_connections 2024-03-17T23:49:49.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.403+0000 7f799cbc6640 1 -- 172.21.15.150:0/757372204 wait complete. 2024-03-17T23:49:49.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.404+0000 7f799cbc6640 1 Processor -- start 2024-03-17T23:49:49.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.404+0000 7f799cbc6640 1 -- start start 2024-03-17T23:49:49.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.405+0000 7f799cbc6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f799806f620 0x7f799815a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:49.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.405+0000 7f799cbc6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7998072240 0x7f799815b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:49.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.405+0000 7f799cbc6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79980b27b0 0x7f7998157b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:49.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.405+0000 7f799cbc6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f799815e300 con 0x7f79980b27b0 2024-03-17T23:49:49.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.405+0000 7f799cbc6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f799815f190 con 0x7f799806f620 2024-03-17T23:49:49.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.405+0000 7f799cbc6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f79980cc9b0 con 0x7f7998072240 2024-03-17T23:49:49.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.405+0000 7f7996d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79980b27b0 0x7f7998157b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:49.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.405+0000 7f799659c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f799806f620 0x7f799815a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:49.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.405+0000 7f799659c640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f799806f620 0x7f799815a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:44944/0 (socket says 172.21.15.150:44944) 2024-03-17T23:49:49.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.405+0000 7f7996d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79980b27b0 0x7f7998157b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60558/0 (socket says 172.21.15.150:60558) 2024-03-17T23:49:49.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.405+0000 7f799659c640 1 -- 172.21.15.150:0/1436514549 learned_addr learned my addr 172.21.15.150:0/1436514549 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:49.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.405+0000 7f7995d9b640 1 --2- 172.21.15.150:0/1436514549 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7998072240 0x7f799815b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:49.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.405+0000 7f799659c640 1 -- 172.21.15.150:0/1436514549 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7998072240 msgr2=0x7f799815b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:49.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.405+0000 7f799659c640 1 --2- 172.21.15.150:0/1436514549 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7998072240 0x7f799815b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.405+0000 7f799659c640 1 -- 172.21.15.150:0/1436514549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79980b27b0 msgr2=0x7f7998157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:49.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.405+0000 7f799659c640 1 --2- 172.21.15.150:0/1436514549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79980b27b0 0x7f7998157b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.405+0000 7f799659c640 1 -- 172.21.15.150:0/1436514549 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f799815c040 con 0x7f799806f620 2024-03-17T23:49:49.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.405+0000 7f7996d9d640 1 --2- 172.21.15.150:0/1436514549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79980b27b0 0x7f7998157b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:49.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.405+0000 7f7995d9b640 1 --2- 172.21.15.150:0/1436514549 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7998072240 0x7f799815b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:49.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.405+0000 7f799659c640 1 --2- 172.21.15.150:0/1436514549 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f799806f620 0x7f799815a250 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f798800c400 tx=0x7f798800c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:49.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.406+0000 7f7986ffd640 1 -- 172.21.15.150:0/1436514549 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7988017070 con 0x7f799806f620 2024-03-17T23:49:49.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.406+0000 7f7986ffd640 1 -- 172.21.15.150:0/1436514549 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7988005ce0 con 0x7f799806f620 2024-03-17T23:49:49.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.406+0000 7f799cbc6640 1 -- 172.21.15.150:0/1436514549 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f799815b7a0 con 0x7f799806f620 2024-03-17T23:49:49.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.406+0000 7f7986ffd640 1 -- 172.21.15.150:0/1436514549 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79880072f0 con 0x7f799806f620 2024-03-17T23:49:49.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.406+0000 7f799cbc6640 1 -- 172.21.15.150:0/1436514549 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f799815bd60 con 0x7f799806f620 2024-03-17T23:49:49.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.407+0000 7f799cbc6640 1 -- 172.21.15.150:0/1436514549 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79980681a0 con 0x7f799806f620 2024-03-17T23:49:49.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.407+0000 7f7986ffd640 1 -- 172.21.15.150:0/1436514549 <== mon.1 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f798801a070 con 0x7f799806f620 2024-03-17T23:49:49.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.407+0000 7f7986ffd640 1 --2- 172.21.15.150:0/1436514549 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f797806bbd0 0x7f797806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:49.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.408+0000 7f7995d9b640 1 --2- 172.21.15.150:0/1436514549 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f797806bbd0 0x7f797806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:49.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.408+0000 7f7986ffd640 1 -- 172.21.15.150:0/1436514549 <== mon.1 v2:172.21.15.157:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f7988089ff0 con 0x7f799806f620 2024-03-17T23:49:49.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.408+0000 7f7995d9b640 1 --2- 172.21.15.150:0/1436514549 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f797806bbd0 0x7f797806e090 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f799806f120 tx=0x7f798c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:49.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.412+0000 7f7986ffd640 1 -- 172.21.15.150:0/1436514549 <== mon.1 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f7988058240 con 0x7f799806f620 2024-03-17T23:49:49.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.584+0000 7f799cbc6640 1 -- 172.21.15.150:0/1436514549 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7998072b10 con 0x7f799806f620 2024-03-17T23:49:49.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.585+0000 7f7986ffd640 1 -- 172.21.15.150:0/1436514549 <== mon.1 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) ==== 74+0+16974 (secure 0 0 0) 0x7f798805c2e0 con 0x7f799806f620 2024-03-17T23:49:49.586 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:49.586 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":58,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:49:12.646307+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:49:09.223630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-17T23:49:09.956422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:49:49.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.587+0000 7f799cbc6640 1 -- 172.21.15.150:0/1436514549 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f797806bbd0 msgr2=0x7f797806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:49.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.587+0000 7f799cbc6640 1 --2- 172.21.15.150:0/1436514549 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f797806bbd0 0x7f797806e090 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f799806f120 tx=0x7f798c009190 comp rx=0 tx=0).stop 2024-03-17T23:49:49.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.587+0000 7f799cbc6640 1 -- 172.21.15.150:0/1436514549 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f799806f620 msgr2=0x7f799815a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:49.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.587+0000 7f799cbc6640 1 --2- 172.21.15.150:0/1436514549 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f799806f620 0x7f799815a250 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f798800c400 tx=0x7f798800c8c0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.587+0000 7f799cbc6640 1 -- 172.21.15.150:0/1436514549 shutdown_connections 2024-03-17T23:49:49.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.587+0000 7f799cbc6640 1 --2- 172.21.15.150:0/1436514549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f79980b27b0 0x7f7998157b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.587+0000 7f799cbc6640 1 --2- 172.21.15.150:0/1436514549 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f7998072240 0x7f799815b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.587+0000 7f799cbc6640 1 --2- 172.21.15.150:0/1436514549 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f797806bbd0 0x7f797806e090 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.587+0000 7f799cbc6640 1 --2- 172.21.15.150:0/1436514549 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f799806f620 0x7f799815a250 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.587+0000 7f799cbc6640 1 -- 172.21.15.150:0/1436514549 >> 172.21.15.150:0/1436514549 conn(0x7f79980b1c90 msgr2=0x7f79980cf250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:49.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.587+0000 7f799cbc6640 1 -- 172.21.15.150:0/1436514549 shutdown_connections 2024-03-17T23:49:49.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.587+0000 7f799cbc6640 1 -- 172.21.15.150:0/1436514549 wait complete. 2024-03-17T23:49:49.599 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-17T23:49:49.599 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:49:49.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.699+0000 7f6c2cec0640 1 Processor -- start 2024-03-17T23:49:49.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.699+0000 7f6c2cec0640 1 -- start start 2024-03-17T23:49:49.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.700+0000 7f6c2cec0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c281615d0 0x7f6c280d0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:49.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.700+0000 7f6c2cec0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6c280ae3b0 0x7f6c280d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:49.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.700+0000 7f6c2cec0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c2806c6f0 con 0x7f6c2806ca10 2024-03-17T23:49:49.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.700+0000 7f6c2cec0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f6c2806c4c0 con 0x7f6c280ae3b0 2024-03-17T23:49:49.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.700+0000 7f6c2cec0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6c28072a20 con 0x7f6c281615d0 2024-03-17T23:49:49.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.700+0000 7f6c26d9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6c280ae3b0 0x7f6c280d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:49.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.700+0000 7f6c25d9b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f6c2806ca10 0x7f6c2806b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41894/0 (socket says 172.21.15.150:41894) 2024-03-17T23:49:49.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.700+0000 7f6c25d9b640 1 -- 172.21.15.150:0/2913364043 learned_addr learned my addr 172.21.15.150:0/2913364043 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:49.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.700+0000 7f6c26d9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6c280ae3b0 0x7f6c280d0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60566/0 (socket says 172.21.15.150:60566) 2024-03-17T23:49:49.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.700+0000 7f6c2659c640 1 --2- 172.21.15.150:0/2913364043 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c281615d0 0x7f6c280d0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:49.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.700+0000 7f6c26d9d640 1 -- 172.21.15.150:0/2913364043 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c281615d0 msgr2=0x7f6c280d0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:49.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.700+0000 7f6c26d9d640 1 --2- 172.21.15.150:0/2913364043 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c281615d0 0x7f6c280d0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.700+0000 7f6c26d9d640 1 -- 172.21.15.150:0/2913364043 >> v1:172.21.15.150:6789/0 conn(0x7f6c2806ca10 legacy=0x7f6c2806b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:49.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.700+0000 7f6c26d9d640 1 -- 172.21.15.150:0/2913364043 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c28072cc0 con 0x7f6c280ae3b0 2024-03-17T23:49:49.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.700+0000 7f6c2559a640 1 -- 172.21.15.150:0/2913364043 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 158536898 0 0) 0x7f6c2806c6f0 con 0x7f6c2806ca10 2024-03-17T23:49:49.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.701+0000 7f6c26d9d640 1 --2- 172.21.15.150:0/2913364043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c280ae3b0 0x7f6c280d0680 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f6c20009d90 tx=0x7f6c2002bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5cd1ba4427296d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:49.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.701+0000 7f6c2559a640 1 -- 172.21.15.150:0/2913364043 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c20039070 con 0x7f6c280ae3b0 2024-03-17T23:49:49.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.701+0000 7f6c2559a640 1 -- 172.21.15.150:0/2913364043 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c2002cb30 con 0x7f6c280ae3b0 2024-03-17T23:49:49.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.701+0000 7f6c2559a640 1 -- 172.21.15.150:0/2913364043 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c20034440 con 0x7f6c280ae3b0 2024-03-17T23:49:49.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.701+0000 7f6c2cec0640 1 -- 172.21.15.150:0/2913364043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c280ae3b0 msgr2=0x7f6c280d0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:49.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.701+0000 7f6c2cec0640 1 --2- 172.21.15.150:0/2913364043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c280ae3b0 0x7f6c280d0680 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f6c20009d90 tx=0x7f6c2002bd80 comp rx=0 tx=0).stop 2024-03-17T23:49:49.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.702+0000 7f6c2cec0640 1 -- 172.21.15.150:0/2913364043 shutdown_connections 2024-03-17T23:49:49.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.702+0000 7f6c2cec0640 1 --2- 172.21.15.150:0/2913364043 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c280ae3b0 0x7f6c280d0680 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.702+0000 7f6c2cec0640 1 --2- 172.21.15.150:0/2913364043 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c281615d0 0x7f6c280d0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.702+0000 7f6c2cec0640 1 -- 172.21.15.150:0/2913364043 >> 172.21.15.150:0/2913364043 conn(0x7f6c280b3da0 msgr2=0x7f6c280b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:49.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.702+0000 7f6c2cec0640 1 -- 172.21.15.150:0/2913364043 shutdown_connections 2024-03-17T23:49:49.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.702+0000 7f6c2cec0640 1 -- 172.21.15.150:0/2913364043 wait complete. 2024-03-17T23:49:49.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.703+0000 7f6c2cec0640 1 Processor -- start 2024-03-17T23:49:49.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.703+0000 7f6c2cec0640 1 -- start start 2024-03-17T23:49:49.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.704+0000 7f6c2cec0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c2806ca10 0x7f6c28158be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:49.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.704+0000 7f6c2cec0640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c280ae3b0 0x7f6c28146d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:49.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.704+0000 7f6c2cec0640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c281615d0 0x7f6c2815eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:49.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.704+0000 7f6c2cec0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6c281619b0 con 0x7f6c2806ca10 2024-03-17T23:49:49.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.704+0000 7f6c2cec0640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6c280d0fd0 con 0x7f6c281615d0 2024-03-17T23:49:49.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.704+0000 7f6c2cec0640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6c28067770 con 0x7f6c280ae3b0 2024-03-17T23:49:49.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.704+0000 7f6c2659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c2806ca10 0x7f6c28158be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:49.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.704+0000 7f6c2659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c2806ca10 0x7f6c28158be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60576/0 (socket says 172.21.15.150:60576) 2024-03-17T23:49:49.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.704+0000 7f6c2659c640 1 -- 172.21.15.150:0/2347634245 learned_addr learned my addr 172.21.15.150:0/2347634245 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:49.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.704+0000 7f6c25d9b640 1 --2- 172.21.15.150:0/2347634245 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c280ae3b0 0x7f6c28146d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:49.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.704+0000 7f6c26d9d640 1 --2- 172.21.15.150:0/2347634245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c281615d0 0x7f6c2815eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:49.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.704+0000 7f6c2659c640 1 -- 172.21.15.150:0/2347634245 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c280ae3b0 msgr2=0x7f6c28146d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:49.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.704+0000 7f6c2659c640 1 --2- 172.21.15.150:0/2347634245 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c280ae3b0 0x7f6c28146d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.704+0000 7f6c2659c640 1 -- 172.21.15.150:0/2347634245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c281615d0 msgr2=0x7f6c2815eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:49.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.704+0000 7f6c2659c640 1 --2- 172.21.15.150:0/2347634245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c281615d0 0x7f6c2815eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.704+0000 7f6c2659c640 1 -- 172.21.15.150:0/2347634245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c28150660 con 0x7f6c2806ca10 2024-03-17T23:49:49.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.704+0000 7f6c25d9b640 1 --2- 172.21.15.150:0/2347634245 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c280ae3b0 0x7f6c28146d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:49.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.704+0000 7f6c2659c640 1 --2- 172.21.15.150:0/2347634245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c2806ca10 0x7f6c28158be0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c00e9b0 tx=0x7f6c1c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:49.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.704+0000 7f6c16ffd640 1 -- 172.21.15.150:0/2347634245 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c017070 con 0x7f6c2806ca10 2024-03-17T23:49:49.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.704+0000 7f6c16ffd640 1 -- 172.21.15.150:0/2347634245 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c1c00c9a0 con 0x7f6c2806ca10 2024-03-17T23:49:49.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.704+0000 7f6c2cec0640 1 -- 172.21.15.150:0/2347634245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c2815de40 con 0x7f6c2806ca10 2024-03-17T23:49:49.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.705+0000 7f6c16ffd640 1 -- 172.21.15.150:0/2347634245 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c00cd90 con 0x7f6c2806ca10 2024-03-17T23:49:49.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.705+0000 7f6c2cec0640 1 -- 172.21.15.150:0/2347634245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c2815e400 con 0x7f6c2806ca10 2024-03-17T23:49:49.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.706+0000 7f6c2cec0640 1 -- 172.21.15.150:0/2347634245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c280681a0 con 0x7f6c2806ca10 2024-03-17T23:49:49.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.706+0000 7f6c16ffd640 1 -- 172.21.15.150:0/2347634245 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6c1c01a070 con 0x7f6c2806ca10 2024-03-17T23:49:49.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.706+0000 7f6c16ffd640 1 --2- 172.21.15.150:0/2347634245 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6c0806bbd0 0x7f6c0806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:49.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.706+0000 7f6c16ffd640 1 -- 172.21.15.150:0/2347634245 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f6c1c08aae0 con 0x7f6c2806ca10 2024-03-17T23:49:49.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.706+0000 7f6c25d9b640 1 --2- 172.21.15.150:0/2347634245 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6c0806bbd0 0x7f6c0806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:49.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.707+0000 7f6c25d9b640 1 --2- 172.21.15.150:0/2347634245 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6c0806bbd0 0x7f6c0806e090 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6c280b2e80 tx=0x7f6c18007de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:49.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.711+0000 7f6c16ffd640 1 -- 172.21.15.150:0/2347634245 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6c1c058e80 con 0x7f6c2806ca10 2024-03-17T23:49:49.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.922+0000 7f6c2cec0640 1 -- 172.21.15.150:0/2347634245 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c28160430 con 0x7f6c2806ca10 2024-03-17T23:49:49.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.923+0000 7f6c16ffd640 1 -- 172.21.15.150:0/2347634245 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+5187 (secure 0 0 0) 0x7f6c1c05cf20 con 0x7f6c2806ca10 2024-03-17T23:49:49.923 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:49.923 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":22,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/2082413564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2082413564},{"type":"v1","addr":"172.21.15.150:6836","nonce":2082413564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":6043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/2350077845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":2350077845},{"type":"v1","addr":"172.21.15.150:6837","nonce":2350077845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":6052,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6834/137606728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":137606728},{"type":"v1","addr":"172.21.15.157:6834","nonce":137606728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-17T23:49:10.996124+0000","modified":"2024-03-17T23:49:12.994518+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6835/479821580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6833","nonce":479821580},{"type":"v1","addr":"172.21.15.157:6835","nonce":479821580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6204,"qdb_cluster":[6204]},"id":3}]} 2024-03-17T23:49:49.924 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 22 2024-03-17T23:49:49.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.925+0000 7f6c2cec0640 1 -- 172.21.15.150:0/2347634245 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6c0806bbd0 msgr2=0x7f6c0806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:49.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.925+0000 7f6c2cec0640 1 --2- 172.21.15.150:0/2347634245 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6c0806bbd0 0x7f6c0806e090 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6c280b2e80 tx=0x7f6c18007de0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.925+0000 7f6c2cec0640 1 -- 172.21.15.150:0/2347634245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c2806ca10 msgr2=0x7f6c28158be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:49.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.925+0000 7f6c2cec0640 1 --2- 172.21.15.150:0/2347634245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c2806ca10 0x7f6c28158be0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c00e9b0 tx=0x7f6c1c00ee70 comp rx=0 tx=0).stop 2024-03-17T23:49:49.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.925+0000 7f6c2cec0640 1 -- 172.21.15.150:0/2347634245 shutdown_connections 2024-03-17T23:49:49.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.925+0000 7f6c2cec0640 1 --2- 172.21.15.150:0/2347634245 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6c0806bbd0 0x7f6c0806e090 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.925+0000 7f6c2cec0640 1 --2- 172.21.15.150:0/2347634245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6c281615d0 0x7f6c2815eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.925+0000 7f6c2cec0640 1 --2- 172.21.15.150:0/2347634245 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6c280ae3b0 0x7f6c28146d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.925+0000 7f6c2cec0640 1 --2- 172.21.15.150:0/2347634245 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6c2806ca10 0x7f6c28158be0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:49.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.925+0000 7f6c2cec0640 1 -- 172.21.15.150:0/2347634245 >> 172.21.15.150:0/2347634245 conn(0x7f6c280b3da0 msgr2=0x7f6c280aee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:49.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.925+0000 7f6c2cec0640 1 -- 172.21.15.150:0/2347634245 shutdown_connections 2024-03-17T23:49:49.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:49.925+0000 7f6c2cec0640 1 -- 172.21.15.150:0/2347634245 wait complete. 2024-03-17T23:49:49.936 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:49:50.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.035+0000 7fb33ccd1640 1 Processor -- start 2024-03-17T23:49:50.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.035+0000 7fb33ccd1640 1 -- start start 2024-03-17T23:49:50.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.036+0000 7fb33ccd1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3381619a0 0x7fb3380b9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:50.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.036+0000 7fb33ccd1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3381620a0 con 0x7fb3380cf120 2024-03-17T23:49:50.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.036+0000 7fb33ccd1640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb338162380 con 0x7fb33815de30 2024-03-17T23:49:50.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.036+0000 7fb33ccd1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb338162940 con 0x7fb3381619a0 2024-03-17T23:49:50.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.037+0000 7fb335d9b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb3380cf120 0x7fb3380d2690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41896/0 (socket says 172.21.15.150:41896) 2024-03-17T23:49:50.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.037+0000 7fb335d9b640 1 -- 172.21.15.150:0/4278629067 learned_addr learned my addr 172.21.15.150:0/4278629067 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:50.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.037+0000 7fb336d9d640 1 --2- 172.21.15.150:0/4278629067 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3381619a0 0x7fb3380b9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:50.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.037+0000 7fb33559a640 1 -- 172.21.15.150:0/4278629067 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3955991872 0 0) 0x7fb3381620a0 con 0x7fb3380cf120 2024-03-17T23:49:50.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.037+0000 7fb33559a640 1 -- 172.21.15.150:0/4278629067 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb320003520 con 0x7fb3380cf120 2024-03-17T23:49:50.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.037+0000 7fb33559a640 1 -- 172.21.15.150:0/4278629067 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 182389581 0 0) 0x7fb338162380 con 0x7fb33815de30 2024-03-17T23:49:50.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.037+0000 7fb33559a640 1 -- 172.21.15.150:0/4278629067 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3381620a0 con 0x7fb33815de30 2024-03-17T23:49:50.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.037+0000 7fb336d9d640 1 -- 172.21.15.150:0/4278629067 >> v1:172.21.15.157:6789/0 conn(0x7fb33815de30 legacy=0x7fb3380cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:50.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.037+0000 7fb336d9d640 1 -- 172.21.15.150:0/4278629067 >> v1:172.21.15.150:6789/0 conn(0x7fb3380cf120 legacy=0x7fb3380d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:50.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.037+0000 7fb336d9d640 1 -- 172.21.15.150:0/4278629067 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb338162660 con 0x7fb3381619a0 2024-03-17T23:49:50.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.038+0000 7fb336d9d640 1 --2- 172.21.15.150:0/4278629067 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3381619a0 0x7fb3380b9860 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb330009dc0 tx=0x7fb33002be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc2961a4d0f11b04 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:50.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.038+0000 7fb33559a640 1 -- 172.21.15.150:0/4278629067 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb330039070 con 0x7fb3381619a0 2024-03-17T23:49:50.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.038+0000 7fb33559a640 1 -- 172.21.15.150:0/4278629067 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb33002cc40 con 0x7fb3381619a0 2024-03-17T23:49:50.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.038+0000 7fb33559a640 1 -- 172.21.15.150:0/4278629067 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb330034440 con 0x7fb3381619a0 2024-03-17T23:49:50.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.038+0000 7fb33ccd1640 1 -- 172.21.15.150:0/4278629067 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3381619a0 msgr2=0x7fb3380b9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:50.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.038+0000 7fb33ccd1640 1 --2- 172.21.15.150:0/4278629067 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3381619a0 0x7fb3380b9860 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb330009dc0 tx=0x7fb33002be90 comp rx=0 tx=0).stop 2024-03-17T23:49:50.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.039+0000 7fb33ccd1640 1 -- 172.21.15.150:0/4278629067 shutdown_connections 2024-03-17T23:49:50.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.039+0000 7fb33ccd1640 1 --2- 172.21.15.150:0/4278629067 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb3381619a0 0x7fb3380b9860 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.039+0000 7fb33ccd1640 1 -- 172.21.15.150:0/4278629067 >> 172.21.15.150:0/4278629067 conn(0x7fb3380730b0 msgr2=0x7fb3380d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:50.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.039+0000 7fb33ccd1640 1 -- 172.21.15.150:0/4278629067 shutdown_connections 2024-03-17T23:49:50.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.039+0000 7fb33ccd1640 1 -- 172.21.15.150:0/4278629067 wait complete. 2024-03-17T23:49:50.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.040+0000 7fb33ccd1640 1 Processor -- start 2024-03-17T23:49:50.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.040+0000 7fb33ccd1640 1 -- start start 2024-03-17T23:49:50.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.041+0000 7fb33ccd1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3380cf120 0x7fb338157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:50.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.041+0000 7fb33ccd1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb33815de30 0x7fb338157be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:50.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.041+0000 7fb33ccd1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3381619a0 0x7fb338160770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:50.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.041+0000 7fb33ccd1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb33815e210 con 0x7fb3380cf120 2024-03-17T23:49:50.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.041+0000 7fb33ccd1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb338161da0 con 0x7fb3381619a0 2024-03-17T23:49:50.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.041+0000 7fb33ccd1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb3380d2da0 con 0x7fb33815de30 2024-03-17T23:49:50.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.041+0000 7fb33659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3380cf120 0x7fb338157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:50.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.041+0000 7fb33659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3380cf120 0x7fb338157180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60578/0 (socket says 172.21.15.150:60578) 2024-03-17T23:49:50.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.041+0000 7fb33659c640 1 -- 172.21.15.150:0/3653187780 learned_addr learned my addr 172.21.15.150:0/3653187780 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:50.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.041+0000 7fb335d9b640 1 --2- 172.21.15.150:0/3653187780 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb33815de30 0x7fb338157be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:50.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.041+0000 7fb336d9d640 1 --2- 172.21.15.150:0/3653187780 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3381619a0 0x7fb338160770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:50.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.041+0000 7fb33659c640 1 -- 172.21.15.150:0/3653187780 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb33815de30 msgr2=0x7fb338157be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:50.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.041+0000 7fb33659c640 1 --2- 172.21.15.150:0/3653187780 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb33815de30 0x7fb338157be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.041+0000 7fb33659c640 1 -- 172.21.15.150:0/3653187780 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3381619a0 msgr2=0x7fb338160770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:50.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.041+0000 7fb33659c640 1 --2- 172.21.15.150:0/3653187780 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3381619a0 0x7fb338160770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.041+0000 7fb33659c640 1 -- 172.21.15.150:0/3653187780 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3381584e0 con 0x7fb3380cf120 2024-03-17T23:49:50.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.041+0000 7fb33659c640 1 --2- 172.21.15.150:0/3653187780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3380cf120 0x7fb338157180 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fb32c00c530 tx=0x7fb32c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:50.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.041+0000 7fb326ffd640 1 -- 172.21.15.150:0/3653187780 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c0052d0 con 0x7fb3380cf120 2024-03-17T23:49:50.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.041+0000 7fb326ffd640 1 -- 172.21.15.150:0/3653187780 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb32c005490 con 0x7fb3380cf120 2024-03-17T23:49:50.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.041+0000 7fb33ccd1640 1 -- 172.21.15.150:0/3653187780 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3381587c0 con 0x7fb3380cf120 2024-03-17T23:49:50.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.041+0000 7fb326ffd640 1 -- 172.21.15.150:0/3653187780 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c012870 con 0x7fb3380cf120 2024-03-17T23:49:50.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.041+0000 7fb33ccd1640 1 -- 172.21.15.150:0/3653187780 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb338158d80 con 0x7fb3380cf120 2024-03-17T23:49:50.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.042+0000 7fb33ccd1640 1 -- 172.21.15.150:0/3653187780 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3380b2b00 con 0x7fb3380cf120 2024-03-17T23:49:50.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.043+0000 7fb326ffd640 1 -- 172.21.15.150:0/3653187780 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb32c012a10 con 0x7fb3380cf120 2024-03-17T23:49:50.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.043+0000 7fb326ffd640 1 --2- 172.21.15.150:0/3653187780 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb31806bbd0 0x7fb31806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:50.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.043+0000 7fb326ffd640 1 -- 172.21.15.150:0/3653187780 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7fb32c08a6d0 con 0x7fb3380cf120 2024-03-17T23:49:50.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.043+0000 7fb335d9b640 1 --2- 172.21.15.150:0/3653187780 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb31806bbd0 0x7fb31806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:50.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.044+0000 7fb335d9b640 1 --2- 172.21.15.150:0/3653187780 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb31806bbd0 0x7fb31806e090 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb338161610 tx=0x7fb328007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:50.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.048+0000 7fb326ffd640 1 -- 172.21.15.150:0/3653187780 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb3380b2b00 con 0x7fb3380cf120 2024-03-17T23:49:50.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.253+0000 7fb33ccd1640 1 -- 172.21.15.150:0/3653187780 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3380ca6a0 con 0x7fb3380cf120 2024-03-17T23:49:50.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.253+0000 7fb326ffd640 1 -- 172.21.15.150:0/3653187780 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+5187 (secure 0 0 0) 0x7fb32c0589f0 con 0x7fb3380cf120 2024-03-17T23:49:50.254 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:50.254 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":22,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/2082413564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2082413564},{"type":"v1","addr":"172.21.15.150:6836","nonce":2082413564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":6043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/2350077845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":2350077845},{"type":"v1","addr":"172.21.15.150:6837","nonce":2350077845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":6052,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6834/137606728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":137606728},{"type":"v1","addr":"172.21.15.157:6834","nonce":137606728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-17T23:49:10.996124+0000","modified":"2024-03-17T23:49:12.994518+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6204},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6204":{"gid":6204,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6835/479821580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6833","nonce":479821580},{"type":"v1","addr":"172.21.15.157:6835","nonce":479821580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6204,"qdb_cluster":[6204]},"id":3}]} 2024-03-17T23:49:50.254 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 22 2024-03-17T23:49:50.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.255+0000 7fb33ccd1640 1 -- 172.21.15.150:0/3653187780 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb31806bbd0 msgr2=0x7fb31806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:50.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.255+0000 7fb33ccd1640 1 --2- 172.21.15.150:0/3653187780 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb31806bbd0 0x7fb31806e090 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb338161610 tx=0x7fb328007d30 comp rx=0 tx=0).stop 2024-03-17T23:49:50.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.255+0000 7fb33ccd1640 1 -- 172.21.15.150:0/3653187780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3380cf120 msgr2=0x7fb338157180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:50.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.255+0000 7fb33ccd1640 1 --2- 172.21.15.150:0/3653187780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3380cf120 0x7fb338157180 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fb32c00c530 tx=0x7fb32c00c9f0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.255+0000 7fb33ccd1640 1 -- 172.21.15.150:0/3653187780 shutdown_connections 2024-03-17T23:49:50.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.255+0000 7fb33ccd1640 1 --2- 172.21.15.150:0/3653187780 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb31806bbd0 0x7fb31806e090 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.255+0000 7fb33ccd1640 1 --2- 172.21.15.150:0/3653187780 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb3381619a0 0x7fb338160770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.255+0000 7fb33ccd1640 1 --2- 172.21.15.150:0/3653187780 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb33815de30 0x7fb338157be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.255+0000 7fb33ccd1640 1 --2- 172.21.15.150:0/3653187780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3380cf120 0x7fb338157180 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.255+0000 7fb33ccd1640 1 -- 172.21.15.150:0/3653187780 >> 172.21.15.150:0/3653187780 conn(0x7fb3380730b0 msgr2=0x7fb3380d31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:50.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.255+0000 7fb33ccd1640 1 -- 172.21.15.150:0/3653187780 shutdown_connections 2024-03-17T23:49:50.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.255+0000 7fb33ccd1640 1 -- 172.21.15.150:0/3653187780 wait complete. 2024-03-17T23:49:50.267 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:49:50.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.367+0000 7f3574b1b640 1 Processor -- start 2024-03-17T23:49:50.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.367+0000 7f3574b1b640 1 -- start start 2024-03-17T23:49:50.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.368+0000 7f3574b1b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f357015de30 0x7f35701619a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:50.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.368+0000 7f3574b1b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35700cf120 0x7f35700d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:50.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.368+0000 7f3574b1b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35700b9cf0 con 0x7f35700b9860 2024-03-17T23:49:50.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.368+0000 7f3574b1b640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f3570162940 con 0x7f357015de30 2024-03-17T23:49:50.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.368+0000 7f3574b1b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f35701620a0 con 0x7f35700cf120 2024-03-17T23:49:50.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.368+0000 7f356e59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f357015de30 0x7f35701619a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:50.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.368+0000 7f356dd9b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f35700b9860 0x7f35700cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41904/0 (socket says 172.21.15.150:41904) 2024-03-17T23:49:50.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.368+0000 7f356dd9b640 1 -- 172.21.15.150:0/3487507104 learned_addr learned my addr 172.21.15.150:0/3487507104 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:50.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.369+0000 7f356ed9d640 1 --2- 172.21.15.150:0/3487507104 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35700cf120 0x7f35700d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:50.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.369+0000 7f356e59c640 1 -- 172.21.15.150:0/3487507104 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35700cf120 msgr2=0x7f35700d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:50.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.369+0000 7f356e59c640 1 --2- 172.21.15.150:0/3487507104 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35700cf120 0x7f35700d2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.369+0000 7f356e59c640 1 -- 172.21.15.150:0/3487507104 >> v1:172.21.15.150:6789/0 conn(0x7f35700b9860 legacy=0x7f35700cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:50.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.369+0000 7f356e59c640 1 -- 172.21.15.150:0/3487507104 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3570162380 con 0x7f357015de30 2024-03-17T23:49:50.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.369+0000 7f356d59a640 1 -- 172.21.15.150:0/3487507104 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1510747677 0 0) 0x7f35700b9cf0 con 0x7f35700b9860 2024-03-17T23:49:50.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.369+0000 7f356ed9d640 1 --2- 172.21.15.150:0/3487507104 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35700cf120 0x7f35700d2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:50.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.370+0000 7f356e59c640 1 --2- 172.21.15.150:0/3487507104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f357015de30 0x7f35701619a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f35680097a0 tx=0x7f356802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c837a2afd8583317 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:50.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.370+0000 7f356d59a640 1 -- 172.21.15.150:0/3487507104 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35680315a0 con 0x7f357015de30 2024-03-17T23:49:50.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.370+0000 7f356d59a640 1 -- 172.21.15.150:0/3487507104 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3568037070 con 0x7f357015de30 2024-03-17T23:49:50.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.370+0000 7f356d59a640 1 -- 172.21.15.150:0/3487507104 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3568031960 con 0x7f357015de30 2024-03-17T23:49:50.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.370+0000 7f3574b1b640 1 -- 172.21.15.150:0/3487507104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f357015de30 msgr2=0x7f35701619a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:50.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.370+0000 7f3574b1b640 1 --2- 172.21.15.150:0/3487507104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f357015de30 0x7f35701619a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f35680097a0 tx=0x7f356802b840 comp rx=0 tx=0).stop 2024-03-17T23:49:50.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.370+0000 7f3574b1b640 1 -- 172.21.15.150:0/3487507104 shutdown_connections 2024-03-17T23:49:50.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.370+0000 7f3574b1b640 1 --2- 172.21.15.150:0/3487507104 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35700cf120 0x7f35700d2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.370+0000 7f3574b1b640 1 --2- 172.21.15.150:0/3487507104 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f357015de30 0x7f35701619a0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.370+0000 7f3574b1b640 1 -- 172.21.15.150:0/3487507104 >> 172.21.15.150:0/3487507104 conn(0x7f35700730b0 msgr2=0x7f35700d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:50.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.370+0000 7f3574b1b640 1 -- 172.21.15.150:0/3487507104 shutdown_connections 2024-03-17T23:49:50.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.370+0000 7f3574b1b640 1 -- 172.21.15.150:0/3487507104 wait complete. 2024-03-17T23:49:50.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.372+0000 7f3574b1b640 1 Processor -- start 2024-03-17T23:49:50.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.372+0000 7f3574b1b640 1 -- start start 2024-03-17T23:49:50.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.372+0000 7f3574b1b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35700b9860 0x7f357015fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:50.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.372+0000 7f3574b1b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35700cf120 0x7f35701f4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:50.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.372+0000 7f3574b1b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f357015de30 0x7f35700ac140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:50.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.372+0000 7f3574b1b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f357015e210 con 0x7f35700cf120 2024-03-17T23:49:50.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.372+0000 7f3574b1b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f35700d2170 con 0x7f357015de30 2024-03-17T23:49:50.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.372+0000 7f3574b1b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f35700d2bd0 con 0x7f35700b9860 2024-03-17T23:49:50.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.372+0000 7f356dd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35700cf120 0x7f35701f4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:50.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.372+0000 7f356dd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35700cf120 0x7f35701f4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60602/0 (socket says 172.21.15.150:60602) 2024-03-17T23:49:50.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.372+0000 7f356dd9b640 1 -- 172.21.15.150:0/3669128914 learned_addr learned my addr 172.21.15.150:0/3669128914 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:50.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.373+0000 7f356e59c640 1 --2- 172.21.15.150:0/3669128914 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35700b9860 0x7f357015fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:50.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.373+0000 7f356ed9d640 1 --2- 172.21.15.150:0/3669128914 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f357015de30 0x7f35700ac140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:50.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.373+0000 7f356dd9b640 1 -- 172.21.15.150:0/3669128914 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35700b9860 msgr2=0x7f357015fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:50.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.373+0000 7f356dd9b640 1 --2- 172.21.15.150:0/3669128914 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35700b9860 0x7f357015fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.373+0000 7f356dd9b640 1 -- 172.21.15.150:0/3669128914 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f357015de30 msgr2=0x7f35700ac140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:50.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.373+0000 7f356dd9b640 1 --2- 172.21.15.150:0/3669128914 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f357015de30 0x7f35700ac140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.373+0000 7f356dd9b640 1 -- 172.21.15.150:0/3669128914 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35701f4750 con 0x7f35700cf120 2024-03-17T23:49:50.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.373+0000 7f356dd9b640 1 --2- 172.21.15.150:0/3669128914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35700cf120 0x7f35701f4210 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f355800c500 tx=0x7f355800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:50.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.373+0000 7f3566ffd640 1 -- 172.21.15.150:0/3669128914 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3558018070 con 0x7f35700cf120 2024-03-17T23:49:50.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.373+0000 7f3566ffd640 1 -- 172.21.15.150:0/3669128914 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3558007500 con 0x7f35700cf120 2024-03-17T23:49:50.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.373+0000 7f3574b1b640 1 -- 172.21.15.150:0/3669128914 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35700aaad0 con 0x7f35700cf120 2024-03-17T23:49:50.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.373+0000 7f3566ffd640 1 -- 172.21.15.150:0/3669128914 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3558004a20 con 0x7f35700cf120 2024-03-17T23:49:50.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.373+0000 7f3574b1b640 1 -- 172.21.15.150:0/3669128914 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35700ab090 con 0x7f35700cf120 2024-03-17T23:49:50.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.374+0000 7f3574b1b640 1 -- 172.21.15.150:0/3669128914 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35700681a0 con 0x7f35700cf120 2024-03-17T23:49:50.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.374+0000 7f3566ffd640 1 -- 172.21.15.150:0/3669128914 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f355801b070 con 0x7f35700cf120 2024-03-17T23:49:50.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.375+0000 7f3566ffd640 1 --2- 172.21.15.150:0/3669128914 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f355006bbd0 0x7f355006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:50.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.375+0000 7f3566ffd640 1 -- 172.21.15.150:0/3669128914 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f3558089db0 con 0x7f35700cf120 2024-03-17T23:49:50.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.375+0000 7f356e59c640 1 --2- 172.21.15.150:0/3669128914 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f355006bbd0 0x7f355006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:50.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.375+0000 7f356e59c640 1 --2- 172.21.15.150:0/3669128914 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f355006bbd0 0x7f355006e090 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3568036000 tx=0x7f356803d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:50.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.379+0000 7f3566ffd640 1 -- 172.21.15.150:0/3669128914 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f35701f4750 con 0x7f35700cf120 2024-03-17T23:49:50.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.556+0000 7f3574b1b640 1 -- 172.21.15.150:0/3669128914 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f35700b2ea0 con 0x7f35700cf120 2024-03-17T23:49:50.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.557+0000 7f3566ffd640 1 -- 172.21.15.150:0/3669128914 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) ==== 74+0+16974 (secure 0 0 0) 0x7f35580580c0 con 0x7f35700cf120 2024-03-17T23:49:50.558 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:50.559 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":58,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:49:12.646307+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-17T23:49:09.223630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-17T23:49:09.956422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:49:50.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.559+0000 7f3574b1b640 1 -- 172.21.15.150:0/3669128914 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f355006bbd0 msgr2=0x7f355006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:50.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.559+0000 7f3574b1b640 1 --2- 172.21.15.150:0/3669128914 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f355006bbd0 0x7f355006e090 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3568036000 tx=0x7f356803d040 comp rx=0 tx=0).stop 2024-03-17T23:49:50.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.559+0000 7f3574b1b640 1 -- 172.21.15.150:0/3669128914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35700cf120 msgr2=0x7f35701f4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:50.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.559+0000 7f3574b1b640 1 --2- 172.21.15.150:0/3669128914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35700cf120 0x7f35701f4210 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f355800c500 tx=0x7f355800c900 comp rx=0 tx=0).stop 2024-03-17T23:49:50.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.559+0000 7f3574b1b640 1 -- 172.21.15.150:0/3669128914 shutdown_connections 2024-03-17T23:49:50.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.559+0000 7f3574b1b640 1 --2- 172.21.15.150:0/3669128914 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f355006bbd0 0x7f355006e090 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.559+0000 7f3574b1b640 1 --2- 172.21.15.150:0/3669128914 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f357015de30 0x7f35700ac140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.559+0000 7f3574b1b640 1 --2- 172.21.15.150:0/3669128914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f35700cf120 0x7f35701f4210 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.559+0000 7f3574b1b640 1 --2- 172.21.15.150:0/3669128914 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f35700b9860 0x7f357015fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.559+0000 7f3574b1b640 1 -- 172.21.15.150:0/3669128914 >> 172.21.15.150:0/3669128914 conn(0x7f35700730b0 msgr2=0x7f35700b3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:50.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.559+0000 7f3574b1b640 1 -- 172.21.15.150:0/3669128914 shutdown_connections 2024-03-17T23:49:50.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.559+0000 7f3574b1b640 1 -- 172.21.15.150:0/3669128914 wait complete. 2024-03-17T23:49:50.571 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-17T23:49:50.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.675+0000 7f0b04b2f640 1 Processor -- start 2024-03-17T23:49:50.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.675+0000 7f0b04b2f640 1 -- start start 2024-03-17T23:49:50.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.676+0000 7f0b04b2f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b0015bb10 0x7f0b00070010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:50.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.676+0000 7f0b04b2f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b00072ad0 0x7f0b00072f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:50.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.676+0000 7f0b04b2f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b000cc5d0 con 0x7f0b0015bef0 2024-03-17T23:49:50.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.676+0000 7f0b04b2f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f0b00073460 con 0x7f0b00072ad0 2024-03-17T23:49:50.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.676+0000 7f0b04b2f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0b000ce9a0 con 0x7f0b0015bb10 2024-03-17T23:49:50.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.676+0000 7f0afed9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b00072ad0 0x7f0b00072f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:50.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.676+0000 7f0afed9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b00072ad0 0x7f0b00072f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60604/0 (socket says 172.21.15.150:60604) 2024-03-17T23:49:50.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.676+0000 7f0af7fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f0b0015bef0 0x7f0b000ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41910/0 (socket says 172.21.15.150:41910) 2024-03-17T23:49:50.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.676+0000 7f0afed9d640 1 -- 172.21.15.150:0/3686567629 learned_addr learned my addr 172.21.15.150:0/3686567629 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:50.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.676+0000 7f0afe59c640 1 --2- 172.21.15.150:0/3686567629 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b0015bb10 0x7f0b00070010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:50.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.676+0000 7f0afed9d640 1 -- 172.21.15.150:0/3686567629 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b0015bb10 msgr2=0x7f0b00070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:50.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.676+0000 7f0afed9d640 1 --2- 172.21.15.150:0/3686567629 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b0015bb10 0x7f0b00070010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.676+0000 7f0afed9d640 1 -- 172.21.15.150:0/3686567629 >> v1:172.21.15.150:6789/0 conn(0x7f0b0015bef0 legacy=0x7f0b000ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:50.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.676+0000 7f0afed9d640 1 -- 172.21.15.150:0/3686567629 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b000cec80 con 0x7f0b00072ad0 2024-03-17T23:49:50.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.676+0000 7f0afdd9b640 1 -- 172.21.15.150:0/3686567629 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1887540307 0 0) 0x7f0b000cc5d0 con 0x7f0b0015bef0 2024-03-17T23:49:50.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.677+0000 7f0afed9d640 1 --2- 172.21.15.150:0/3686567629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b00072ad0 0x7f0b00072f20 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f0af8009e50 tx=0x7f0af802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37d6be9c879deb50 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:50.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.677+0000 7f0afdd9b640 1 -- 172.21.15.150:0/3686567629 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af802cb00 con 0x7f0b00072ad0 2024-03-17T23:49:50.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.677+0000 7f0afdd9b640 1 -- 172.21.15.150:0/3686567629 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0af802cca0 con 0x7f0b00072ad0 2024-03-17T23:49:50.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.677+0000 7f0afdd9b640 1 -- 172.21.15.150:0/3686567629 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af8034610 con 0x7f0b00072ad0 2024-03-17T23:49:50.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.677+0000 7f0b04b2f640 1 -- 172.21.15.150:0/3686567629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b00072ad0 msgr2=0x7f0b00072f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:50.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.677+0000 7f0b04b2f640 1 --2- 172.21.15.150:0/3686567629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b00072ad0 0x7f0b00072f20 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f0af8009e50 tx=0x7f0af802bf60 comp rx=0 tx=0).stop 2024-03-17T23:49:50.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.678+0000 7f0b04b2f640 1 -- 172.21.15.150:0/3686567629 shutdown_connections 2024-03-17T23:49:50.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.678+0000 7f0b04b2f640 1 --2- 172.21.15.150:0/3686567629 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b00072ad0 0x7f0b00072f20 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.678+0000 7f0b04b2f640 1 --2- 172.21.15.150:0/3686567629 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b0015bb10 0x7f0b00070010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.678+0000 7f0b04b2f640 1 -- 172.21.15.150:0/3686567629 >> 172.21.15.150:0/3686567629 conn(0x7f0b00071fe0 msgr2=0x7f0b000723c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:50.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.678+0000 7f0b04b2f640 1 -- 172.21.15.150:0/3686567629 shutdown_connections 2024-03-17T23:49:50.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.678+0000 7f0b04b2f640 1 -- 172.21.15.150:0/3686567629 wait complete. 2024-03-17T23:49:50.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.679+0000 7f0b04b2f640 1 Processor -- start 2024-03-17T23:49:50.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.679+0000 7f0b04b2f640 1 -- start start 2024-03-17T23:49:50.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.680+0000 7f0b04b2f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b00072ad0 0x7f0b000b4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:50.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.680+0000 7f0b04b2f640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b0015bb10 0x7f0b0015d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:50.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.680+0000 7f0b04b2f640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b0015bef0 0x7f0b0015e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:50.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.680+0000 7f0b04b2f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0b0015c330 con 0x7f0b00072ad0 2024-03-17T23:49:50.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.680+0000 7f0b04b2f640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f0b000ccb50 con 0x7f0b0015bef0 2024-03-17T23:49:50.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.680+0000 7f0b04b2f640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f0b000cd070 con 0x7f0b0015bb10 2024-03-17T23:49:50.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.680+0000 7f0afe59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b00072ad0 0x7f0b000b4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:50.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.680+0000 7f0afe59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b00072ad0 0x7f0b000b4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60620/0 (socket says 172.21.15.150:60620) 2024-03-17T23:49:50.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.680+0000 7f0afe59c640 1 -- 172.21.15.150:0/3946020005 learned_addr learned my addr 172.21.15.150:0/3946020005 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:50.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.680+0000 7f0af7fff640 1 --2- 172.21.15.150:0/3946020005 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b0015bb10 0x7f0b0015d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:50.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.680+0000 7f0afed9d640 1 --2- 172.21.15.150:0/3946020005 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b0015bef0 0x7f0b0015e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:50.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.680+0000 7f0afe59c640 1 -- 172.21.15.150:0/3946020005 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b0015bb10 msgr2=0x7f0b0015d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:50.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.680+0000 7f0afe59c640 1 --2- 172.21.15.150:0/3946020005 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b0015bb10 0x7f0b0015d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.680+0000 7f0afe59c640 1 -- 172.21.15.150:0/3946020005 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b0015bef0 msgr2=0x7f0b0015e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:50.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.680+0000 7f0afe59c640 1 --2- 172.21.15.150:0/3946020005 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b0015bef0 0x7f0b0015e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.680+0000 7f0afe59c640 1 -- 172.21.15.150:0/3946020005 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b0015dd90 con 0x7f0b00072ad0 2024-03-17T23:49:50.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.680+0000 7f0afe59c640 1 --2- 172.21.15.150:0/3946020005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b00072ad0 0x7f0b000b4bc0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f0af000e9b0 tx=0x7f0af000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:50.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.680+0000 7f0af6ffd640 1 -- 172.21.15.150:0/3946020005 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af0017070 con 0x7f0b00072ad0 2024-03-17T23:49:50.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.680+0000 7f0af6ffd640 1 -- 172.21.15.150:0/3946020005 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0af000c9a0 con 0x7f0b00072ad0 2024-03-17T23:49:50.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.680+0000 7f0b04b2f640 1 -- 172.21.15.150:0/3946020005 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b000b3b20 con 0x7f0b00072ad0 2024-03-17T23:49:50.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.680+0000 7f0af6ffd640 1 -- 172.21.15.150:0/3946020005 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af000cd90 con 0x7f0b00072ad0 2024-03-17T23:49:50.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.680+0000 7f0b04b2f640 1 -- 172.21.15.150:0/3946020005 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b000b40e0 con 0x7f0b00072ad0 2024-03-17T23:49:50.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.681+0000 7f0b04b2f640 1 -- 172.21.15.150:0/3946020005 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b000681a0 con 0x7f0b00072ad0 2024-03-17T23:49:50.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.682+0000 7f0af6ffd640 1 -- 172.21.15.150:0/3946020005 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f0af001a070 con 0x7f0b00072ad0 2024-03-17T23:49:50.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.682+0000 7f0af6ffd640 1 --2- 172.21.15.150:0/3946020005 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0ae006bbd0 0x7f0ae006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:50.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.682+0000 7f0af7fff640 1 --2- 172.21.15.150:0/3946020005 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0ae006bbd0 0x7f0ae006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:50.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.682+0000 7f0af6ffd640 1 -- 172.21.15.150:0/3946020005 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6101+0+0 (secure 0 0 0) 0x7f0af008abc0 con 0x7f0b00072ad0 2024-03-17T23:49:50.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.683+0000 7f0af7fff640 1 --2- 172.21.15.150:0/3946020005 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0ae006bbd0 0x7f0ae006e090 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f0b0015ffe0 tx=0x7f0ae8007de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:50.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.687+0000 7f0af6ffd640 1 -- 172.21.15.150:0/3946020005 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f0af0058ed0 con 0x7f0b00072ad0 2024-03-17T23:49:50.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.902+0000 7f0b04b2f640 1 -- 172.21.15.150:0/3946020005 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f0b00067140 con 0x7f0b00072ad0 2024-03-17T23:49:50.903 INFO:tasks.ceph.mon.a.smithi150.stderr:2024-03-17T23:49:50.903+0000 7fd9a9062640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-17T23:49:50.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.904+0000 7f0af6ffd640 1 -- 172.21.15.150:0/3946020005 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v23) ==== 161+0+0 (secure 0 0 0) 0x7f0af005cf70 con 0x7f0b00072ad0 2024-03-17T23:49:50.905 INFO:teuthology.orchestra.run.smithi150.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-17T23:49:50.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.907+0000 7f0b04b2f640 1 -- 172.21.15.150:0/3946020005 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0ae006bbd0 msgr2=0x7f0ae006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:50.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.907+0000 7f0b04b2f640 1 --2- 172.21.15.150:0/3946020005 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0ae006bbd0 0x7f0ae006e090 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f0b0015ffe0 tx=0x7f0ae8007de0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.907+0000 7f0b04b2f640 1 -- 172.21.15.150:0/3946020005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b00072ad0 msgr2=0x7f0b000b4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:50.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.907+0000 7f0b04b2f640 1 --2- 172.21.15.150:0/3946020005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b00072ad0 0x7f0b000b4bc0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f0af000e9b0 tx=0x7f0af000ee70 comp rx=0 tx=0).stop 2024-03-17T23:49:50.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.907+0000 7f0b04b2f640 1 -- 172.21.15.150:0/3946020005 shutdown_connections 2024-03-17T23:49:50.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.907+0000 7f0b04b2f640 1 --2- 172.21.15.150:0/3946020005 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f0ae006bbd0 0x7f0ae006e090 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.907+0000 7f0b04b2f640 1 --2- 172.21.15.150:0/3946020005 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0b0015bef0 0x7f0b0015e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.907+0000 7f0b04b2f640 1 --2- 172.21.15.150:0/3946020005 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f0b0015bb10 0x7f0b0015d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.907+0000 7f0b04b2f640 1 --2- 172.21.15.150:0/3946020005 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b00072ad0 0x7f0b000b4bc0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:50.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.907+0000 7f0b04b2f640 1 -- 172.21.15.150:0/3946020005 >> 172.21.15.150:0/3946020005 conn(0x7f0b00071fe0 msgr2=0x7f0b0015a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:50.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.907+0000 7f0b04b2f640 1 -- 172.21.15.150:0/3946020005 shutdown_connections 2024-03-17T23:49:50.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:50.907+0000 7f0b04b2f640 1 -- 172.21.15.150:0/3946020005 wait complete. 2024-03-17T23:49:50.909 INFO:tasks.ceph.mds.b.smithi157.stderr: -3042> 2024-03-17T23:49:16.631+0000 7fa205d37640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-17T23:49:50.919 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-17T23:49:50.936 INFO:tasks.ceph.mds.b.smithi157.stdout:starting mds.b at 2024-03-17T23:49:51.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.021+0000 7f6bda545640 1 Processor -- start 2024-03-17T23:49:51.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.021+0000 7f6bda545640 1 -- start start 2024-03-17T23:49:51.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.022+0000 7f6bda545640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6bd4144bb0 0x7f6bd415ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:51.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.022+0000 7f6bda545640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f6bd40ce6a0 0x7f6bd4165a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:51.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.022+0000 7f6bda545640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bd4145170 con 0x7f6bd415d390 2024-03-17T23:49:51.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.022+0000 7f6bda545640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f6bd4145450 con 0x7f6bd4144bb0 2024-03-17T23:49:51.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.022+0000 7f6bda545640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f6bd4145730 con 0x7f6bd40ce6a0 2024-03-17T23:49:51.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.022+0000 7f6bd3fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6bd4144bb0 0x7f6bd415ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:51.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.022+0000 7f6bd3fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6bd4144bb0 0x7f6bd415ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60634/0 (socket says 172.21.15.150:60634) 2024-03-17T23:49:51.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.022+0000 7f6bd3fff640 1 -- 172.21.15.150:0/1708801414 learned_addr learned my addr 172.21.15.150:0/1708801414 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:51.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.022+0000 7f6bd3fff640 1 -- 172.21.15.150:0/1708801414 >> v2:172.21.15.157:3300/0 conn(0x7f6bd40ce6a0 msgr2=0x7f6bd4165a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:51.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.022+0000 7f6bd8ae2640 1 --2- 172.21.15.150:0/1708801414 >> v2:172.21.15.157:3300/0 conn(0x7f6bd40ce6a0 0x7f6bd4165a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:51.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.022+0000 7f6bd3fff640 1 --2- 172.21.15.150:0/1708801414 >> v2:172.21.15.157:3300/0 conn(0x7f6bd40ce6a0 0x7f6bd4165a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:51.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.022+0000 7f6bd3fff640 1 -- 172.21.15.150:0/1708801414 >> v1:172.21.15.157:6789/0 conn(0x7f6bd415d390 legacy=0x7f6bd40cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:51.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.022+0000 7f6bd3fff640 1 -- 172.21.15.150:0/1708801414 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd406bd60 con 0x7f6bd4144bb0 2024-03-17T23:49:51.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.022+0000 7f6bd8ae2640 1 --2- 172.21.15.150:0/1708801414 >> v2:172.21.15.157:3300/0 conn(0x7f6bd40ce6a0 0x7f6bd4165a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:49:51.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.023+0000 7f6bd3fff640 1 --2- 172.21.15.150:0/1708801414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bd4144bb0 0x7f6bd415ce50 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4073130 tx=0x7f6bbc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5547dbb1ec32f587 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:51.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.023+0000 7f6bd2ffd640 1 -- 172.21.15.150:0/1708801414 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc02e070 con 0x7f6bd4144bb0 2024-03-17T23:49:51.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.023+0000 7f6bd2ffd640 1 -- 172.21.15.150:0/1708801414 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bbc037070 con 0x7f6bd4144bb0 2024-03-17T23:49:51.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.023+0000 7f6bd2ffd640 1 -- 172.21.15.150:0/1708801414 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bbc031a00 con 0x7f6bd4144bb0 2024-03-17T23:49:51.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.024+0000 7f6bda545640 1 -- 172.21.15.150:0/1708801414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bd4144bb0 msgr2=0x7f6bd415ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:51.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.024+0000 7f6bda545640 1 --2- 172.21.15.150:0/1708801414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bd4144bb0 0x7f6bd415ce50 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4073130 tx=0x7f6bbc02b880 comp rx=0 tx=0).stop 2024-03-17T23:49:51.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.024+0000 7f6bda545640 1 -- 172.21.15.150:0/1708801414 shutdown_connections 2024-03-17T23:49:51.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.024+0000 7f6bda545640 1 --2- 172.21.15.150:0/1708801414 >> v2:172.21.15.157:3300/0 conn(0x7f6bd40ce6a0 0x7f6bd4165a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:51.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.024+0000 7f6bda545640 1 --2- 172.21.15.150:0/1708801414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bd4144bb0 0x7f6bd415ce50 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:51.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.024+0000 7f6bda545640 1 -- 172.21.15.150:0/1708801414 >> 172.21.15.150:0/1708801414 conn(0x7f6bd406b190 msgr2=0x7f6bd406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:51.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.024+0000 7f6bda545640 1 -- 172.21.15.150:0/1708801414 shutdown_connections 2024-03-17T23:49:51.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.024+0000 7f6bda545640 1 -- 172.21.15.150:0/1708801414 wait complete. 2024-03-17T23:49:51.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.025+0000 7f6bda545640 1 Processor -- start 2024-03-17T23:49:51.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.025+0000 7f6bda545640 1 -- start start 2024-03-17T23:49:51.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.026+0000 7f6bda545640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6bd40ce6a0 0x7f6bd413ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:51.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.026+0000 7f6bda545640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6bd4144bb0 0x7f6bd415aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:51.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.026+0000 7f6bda545640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bd415d390 0x7f6bd413eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:51.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.026+0000 7f6bda545640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd4165f70 con 0x7f6bd415d390 2024-03-17T23:49:51.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.026+0000 7f6bda545640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd406f650 con 0x7f6bd40ce6a0 2024-03-17T23:49:51.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.026+0000 7f6bda545640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6bd41660f0 con 0x7f6bd4144bb0 2024-03-17T23:49:51.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.026+0000 7f6bd8ae2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bd415d390 0x7f6bd413eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:51.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.026+0000 7f6bd8ae2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bd415d390 0x7f6bd413eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60648/0 (socket says 172.21.15.150:60648) 2024-03-17T23:49:51.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.026+0000 7f6bd8ae2640 1 -- 172.21.15.150:0/889753701 learned_addr learned my addr 172.21.15.150:0/889753701 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:51.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.026+0000 7f6bd3fff640 1 --2- 172.21.15.150:0/889753701 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6bd40ce6a0 0x7f6bd413ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:51.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.026+0000 7f6bd37fe640 1 --2- 172.21.15.150:0/889753701 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6bd4144bb0 0x7f6bd415aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:51.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.026+0000 7f6bd8ae2640 1 -- 172.21.15.150:0/889753701 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6bd4144bb0 msgr2=0x7f6bd415aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:51.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.026+0000 7f6bd8ae2640 1 --2- 172.21.15.150:0/889753701 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6bd4144bb0 0x7f6bd415aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:51.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.026+0000 7f6bd8ae2640 1 -- 172.21.15.150:0/889753701 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6bd40ce6a0 msgr2=0x7f6bd413ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:51.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.026+0000 7f6bd8ae2640 1 --2- 172.21.15.150:0/889753701 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6bd40ce6a0 0x7f6bd413ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:51.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.026+0000 7f6bd8ae2640 1 -- 172.21.15.150:0/889753701 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd415afb0 con 0x7f6bd415d390 2024-03-17T23:49:51.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.026+0000 7f6bd8ae2640 1 --2- 172.21.15.150:0/889753701 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bd415d390 0x7f6bd413eae0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00cd80 tx=0x7f6bcc00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:51.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.026+0000 7f6bd0ff9640 1 -- 172.21.15.150:0/889753701 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc00f680 con 0x7f6bd415d390 2024-03-17T23:49:51.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.026+0000 7f6bd0ff9640 1 -- 172.21.15.150:0/889753701 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bcc00f840 con 0x7f6bd415d390 2024-03-17T23:49:51.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.026+0000 7f6bd0ff9640 1 -- 172.21.15.150:0/889753701 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc00fc30 con 0x7f6bd415d390 2024-03-17T23:49:51.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.026+0000 7f6bda545640 1 -- 172.21.15.150:0/889753701 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bd40aabe0 con 0x7f6bd415d390 2024-03-17T23:49:51.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.026+0000 7f6bda545640 1 -- 172.21.15.150:0/889753701 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bd4159f90 con 0x7f6bd415d390 2024-03-17T23:49:51.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.027+0000 7f6bda545640 1 -- 172.21.15.150:0/889753701 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bd40681a0 con 0x7f6bd415d390 2024-03-17T23:49:51.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.028+0000 7f6bd0ff9640 1 -- 172.21.15.150:0/889753701 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6bcc0213f0 con 0x7f6bd415d390 2024-03-17T23:49:51.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.028+0000 7f6bd0ff9640 1 --2- 172.21.15.150:0/889753701 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6bb406bbd0 0x7f6bb406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:51.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.028+0000 7f6bd0ff9640 1 -- 172.21.15.150:0/889753701 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 6187+0+0 (secure 0 0 0) 0x7f6bcc08b850 con 0x7f6bd415d390 2024-03-17T23:49:51.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.028+0000 7f6bd3fff640 1 --2- 172.21.15.150:0/889753701 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6bb406bbd0 0x7f6bb406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:51.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.029+0000 7f6bd3fff640 1 --2- 172.21.15.150:0/889753701 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6bb406bbd0 0x7f6bb406e090 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc036000 tx=0x7f6bbc03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:51.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.032+0000 7f6bd0ff9640 1 -- 172.21.15.150:0/889753701 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6bcc014030 con 0x7f6bd415d390 2024-03-17T23:49:51.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.240+0000 7f6bda545640 1 -- 172.21.15.150:0/889753701 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f6bd40b1500 con 0x7f6bd415d390 2024-03-17T23:49:51.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.911+0000 7f6bd0ff9640 1 -- 172.21.15.150:0/889753701 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v24) ==== 104+0+0 (secure 0 0 0) 0x7f6bcc059b00 con 0x7f6bd415d390 2024-03-17T23:49:51.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.913+0000 7f6bda545640 1 -- 172.21.15.150:0/889753701 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6bb406bbd0 msgr2=0x7f6bb406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:51.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.913+0000 7f6bda545640 1 --2- 172.21.15.150:0/889753701 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6bb406bbd0 0x7f6bb406e090 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc036000 tx=0x7f6bbc03d040 comp rx=0 tx=0).stop 2024-03-17T23:49:51.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.914+0000 7f6bda545640 1 -- 172.21.15.150:0/889753701 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bd415d390 msgr2=0x7f6bd413eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:51.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.914+0000 7f6bda545640 1 --2- 172.21.15.150:0/889753701 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bd415d390 0x7f6bd413eae0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00cd80 tx=0x7f6bcc00a650 comp rx=0 tx=0).stop 2024-03-17T23:49:51.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.914+0000 7f6bda545640 1 -- 172.21.15.150:0/889753701 shutdown_connections 2024-03-17T23:49:51.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.914+0000 7f6bda545640 1 --2- 172.21.15.150:0/889753701 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6bd415d390 0x7f6bd413eae0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:51.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.914+0000 7f6bda545640 1 --2- 172.21.15.150:0/889753701 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6bd4144bb0 0x7f6bd415aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:51.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.914+0000 7f6bda545640 1 --2- 172.21.15.150:0/889753701 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6bb406bbd0 0x7f6bb406e090 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:51.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.914+0000 7f6bda545640 1 --2- 172.21.15.150:0/889753701 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6bd40ce6a0 0x7f6bd413ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:51.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.914+0000 7f6bda545640 1 -- 172.21.15.150:0/889753701 >> 172.21.15.150:0/889753701 conn(0x7f6bd406b190 msgr2=0x7f6bd415c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:51.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.914+0000 7f6bda545640 1 -- 172.21.15.150:0/889753701 shutdown_connections 2024-03-17T23:49:51.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:51.914+0000 7f6bda545640 1 -- 172.21.15.150:0/889753701 wait complete. 2024-03-17T23:49:51.927 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-17T23:49:52.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.031+0000 7f403c9b9640 1 Processor -- start 2024-03-17T23:49:52.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.031+0000 7f403c9b9640 1 -- start start 2024-03-17T23:49:52.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.032+0000 7f403c9b9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f403809a450 0x7f403815e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:52.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.032+0000 7f403c9b9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f403806d270 0x7f40380cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:52.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.032+0000 7f403c9b9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40380d1650 con 0x7f403815dfd0 2024-03-17T23:49:52.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.032+0000 7f403c9b9640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f40380b4950 con 0x7f403806d270 2024-03-17T23:49:52.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.032+0000 7f403c9b9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f403806d700 con 0x7f403809a450 2024-03-17T23:49:52.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.032+0000 7f403659c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f403815dfd0 0x7f4038099d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41914/0 (socket says 172.21.15.150:41914) 2024-03-17T23:49:52.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.032+0000 7f403659c640 1 -- 172.21.15.150:0/1221011754 learned_addr learned my addr 172.21.15.150:0/1221011754 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:52.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.033+0000 7f403559a640 1 -- 172.21.15.150:0/1221011754 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1635984722 0 0) 0x7f40380d1650 con 0x7f403815dfd0 2024-03-17T23:49:52.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.033+0000 7f4036d9d640 1 --2- 172.21.15.150:0/1221011754 >> v2:172.21.15.157:3300/0 conn(0x7f403806d270 0x7f40380cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:52.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.033+0000 7f4035d9b640 1 --2- 172.21.15.150:0/1221011754 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f403809a450 0x7f403815e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:52.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.033+0000 7f403559a640 1 -- 172.21.15.150:0/1221011754 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4020003520 con 0x7f403815dfd0 2024-03-17T23:49:52.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.033+0000 7f403559a640 1 -- 172.21.15.150:0/1221011754 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1164161959 0 0) 0x7f4020003520 con 0x7f403815dfd0 2024-03-17T23:49:52.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.033+0000 7f403559a640 1 -- 172.21.15.150:0/1221011754 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f403809a450 msgr2=0x7f403815e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:52.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.033+0000 7f403559a640 1 --2- 172.21.15.150:0/1221011754 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f403809a450 0x7f403815e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:52.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.033+0000 7f403559a640 1 -- 172.21.15.150:0/1221011754 >> v2:172.21.15.157:3300/0 conn(0x7f403806d270 msgr2=0x7f40380cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:52.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.033+0000 7f403559a640 1 --2- 172.21.15.150:0/1221011754 >> v2:172.21.15.157:3300/0 conn(0x7f403806d270 0x7f40380cfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:52.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.033+0000 7f403559a640 1 -- 172.21.15.150:0/1221011754 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40380d1380 con 0x7f403815dfd0 2024-03-17T23:49:52.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.033+0000 7f403559a640 1 -- 172.21.15.150:0/1221011754 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f40280019f0 con 0x7f403815dfd0 2024-03-17T23:49:52.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.033+0000 7f403559a640 1 -- 172.21.15.150:0/1221011754 >> v1:172.21.15.150:6789/0 conn(0x7f403815dfd0 legacy=0x7f4038099d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:52.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.033+0000 7f403559a640 1 --2- 172.21.15.150:0/1221011754 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f403809a450 0x7f40200039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:52.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.033+0000 7f403559a640 1 --2- 172.21.15.150:0/1221011754 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4020003f30 0x7f4020004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:52.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.033+0000 7f403559a640 1 --2- 172.21.15.150:0/1221011754 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4020004d40 0x7f40200051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:52.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.033+0000 7f403559a640 1 -- 172.21.15.150:0/1221011754 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f403806d700 con 0x7f4020004d40 2024-03-17T23:49:52.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.033+0000 7f403559a640 1 -- 172.21.15.150:0/1221011754 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f402000b9c0 con 0x7f4020003f30 2024-03-17T23:49:52.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.033+0000 7f403559a640 1 -- 172.21.15.150:0/1221011754 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f402000bb40 con 0x7f403809a450 2024-03-17T23:49:52.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.033+0000 7f4036d9d640 1 --2- 172.21.15.150:0/1221011754 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4020004d40 0x7f40200051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:52.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.033+0000 7f403659c640 1 --2- 172.21.15.150:0/1221011754 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f403809a450 0x7f40200039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:52.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.033+0000 7f4035d9b640 1 --2- 172.21.15.150:0/1221011754 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4020003f30 0x7f4020004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:52.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.033+0000 7f4036d9d640 1 -- 172.21.15.150:0/1221011754 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f403809a450 msgr2=0x7f40200039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:52.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.033+0000 7f4036d9d640 1 --2- 172.21.15.150:0/1221011754 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f403809a450 0x7f40200039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:52.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.033+0000 7f4036d9d640 1 -- 172.21.15.150:0/1221011754 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4020003f30 msgr2=0x7f4020004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:52.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.033+0000 7f4036d9d640 1 --2- 172.21.15.150:0/1221011754 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4020003f30 0x7f4020004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:52.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.033+0000 7f4036d9d640 1 -- 172.21.15.150:0/1221011754 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4020003520 con 0x7f4020004d40 2024-03-17T23:49:52.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.034+0000 7f4036d9d640 1 --2- 172.21.15.150:0/1221011754 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4020004d40 0x7f40200051f0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f4030009af0 tx=0x7f4020005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65b859a2bd29c10e server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:52.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.034+0000 7f403559a640 1 -- 172.21.15.150:0/1221011754 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4030039070 con 0x7f4020004d40 2024-03-17T23:49:52.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.034+0000 7f403559a640 1 -- 172.21.15.150:0/1221011754 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f403002c710 con 0x7f4020004d40 2024-03-17T23:49:52.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.034+0000 7f403559a640 1 -- 172.21.15.150:0/1221011754 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4030009690 con 0x7f4020004d40 2024-03-17T23:49:52.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.035+0000 7f403c9b9640 1 -- 172.21.15.150:0/1221011754 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4020004d40 msgr2=0x7f40200051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:52.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.035+0000 7f403c9b9640 1 --2- 172.21.15.150:0/1221011754 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4020004d40 0x7f40200051f0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f4030009af0 tx=0x7f4020005830 comp rx=0 tx=0).stop 2024-03-17T23:49:52.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.035+0000 7f4036d9d640 1 -- 172.21.15.150:0/1221011754 reap_dead start 2024-03-17T23:49:52.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.035+0000 7f403c9b9640 1 -- 172.21.15.150:0/1221011754 shutdown_connections 2024-03-17T23:49:52.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.035+0000 7f403c9b9640 1 -- 172.21.15.150:0/1221011754 >> 172.21.15.150:0/1221011754 conn(0x7f40380b35f0 msgr2=0x7f40380b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:52.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.035+0000 7f403c9b9640 1 -- 172.21.15.150:0/1221011754 shutdown_connections 2024-03-17T23:49:52.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.035+0000 7f403c9b9640 1 -- 172.21.15.150:0/1221011754 wait complete. 2024-03-17T23:49:52.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.036+0000 7f403c9b9640 1 Processor -- start 2024-03-17T23:49:52.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.036+0000 7f403c9b9640 1 -- start start 2024-03-17T23:49:52.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.037+0000 7f403c9b9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4020004d40 0x7f40380b52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:52.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.037+0000 7f403c9b9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40380b57f0 0x7f403814dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:52.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.037+0000 7f403c9b9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f403814e240 0x7f403814e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:52.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.037+0000 7f403c9b9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f40380cc8b0 con 0x7f40380b57f0 2024-03-17T23:49:52.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.037+0000 7f403c9b9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f40380cc5d0 con 0x7f4020004d40 2024-03-17T23:49:52.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.037+0000 7f403c9b9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f40380cdde0 con 0x7f403814e240 2024-03-17T23:49:52.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.037+0000 7f4035d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40380b57f0 0x7f403814dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:52.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.037+0000 7f4035d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40380b57f0 0x7f403814dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60674/0 (socket says 172.21.15.150:60674) 2024-03-17T23:49:52.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.037+0000 7f4035d9b640 1 -- 172.21.15.150:0/3164175653 learned_addr learned my addr 172.21.15.150:0/3164175653 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:52.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.037+0000 7f403659c640 1 --2- 172.21.15.150:0/3164175653 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4020004d40 0x7f40380b52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:52.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.037+0000 7f4036d9d640 1 --2- 172.21.15.150:0/3164175653 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f403814e240 0x7f403814e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:52.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.037+0000 7f4035d9b640 1 -- 172.21.15.150:0/3164175653 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f403814e240 msgr2=0x7f403814e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:52.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.037+0000 7f4035d9b640 1 --2- 172.21.15.150:0/3164175653 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f403814e240 0x7f403814e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:52.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.037+0000 7f4035d9b640 1 -- 172.21.15.150:0/3164175653 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4020004d40 msgr2=0x7f40380b52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:52.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.037+0000 7f4035d9b640 1 --2- 172.21.15.150:0/3164175653 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4020004d40 0x7f40380b52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:52.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.037+0000 7f4035d9b640 1 -- 172.21.15.150:0/3164175653 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40380b5d00 con 0x7f40380b57f0 2024-03-17T23:49:52.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.037+0000 7f403659c640 1 --2- 172.21.15.150:0/3164175653 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4020004d40 0x7f40380b52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:49:52.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.038+0000 7f4035d9b640 1 --2- 172.21.15.150:0/3164175653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40380b57f0 0x7f403814dd00 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f402c011910 tx=0x7f402c011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:52.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.038+0000 7f4026ffd640 1 -- 172.21.15.150:0/3164175653 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c00faf0 con 0x7f40380b57f0 2024-03-17T23:49:52.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.038+0000 7f4026ffd640 1 -- 172.21.15.150:0/3164175653 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f402c00fc90 con 0x7f40380b57f0 2024-03-17T23:49:52.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.038+0000 7f403c9b9640 1 -- 172.21.15.150:0/3164175653 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40380b5fe0 con 0x7f40380b57f0 2024-03-17T23:49:52.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.038+0000 7f4026ffd640 1 -- 172.21.15.150:0/3164175653 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c014440 con 0x7f40380b57f0 2024-03-17T23:49:52.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.038+0000 7f403c9b9640 1 -- 172.21.15.150:0/3164175653 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40380b65a0 con 0x7f40380b57f0 2024-03-17T23:49:52.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.039+0000 7f403c9b9640 1 -- 172.21.15.150:0/3164175653 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40380688b0 con 0x7f40380b57f0 2024-03-17T23:49:52.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.039+0000 7f4026ffd640 1 -- 172.21.15.150:0/3164175653 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f402c012040 con 0x7f40380b57f0 2024-03-17T23:49:52.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.039+0000 7f4026ffd640 1 --2- 172.21.15.150:0/3164175653 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f401806bc00 0x7f401806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:52.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.040+0000 7f4026ffd640 1 -- 172.21.15.150:0/3164175653 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 6187+0+0 (secure 0 0 0) 0x7f402c08b220 con 0x7f40380b57f0 2024-03-17T23:49:52.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.040+0000 7f403659c640 1 --2- 172.21.15.150:0/3164175653 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f401806bc00 0x7f401806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:52.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.040+0000 7f403659c640 1 --2- 172.21.15.150:0/3164175653 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f401806bc00 0x7f401806e0c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f40381f82e0 tx=0x7f4028006040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:52.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.044+0000 7f4026ffd640 1 -- 172.21.15.150:0/3164175653 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f402c019030 con 0x7f40380b57f0 2024-03-17T23:49:52.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.216+0000 7f403c9b9640 1 -- 172.21.15.150:0/3164175653 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f40380b67a0 con 0x7f40380b57f0 2024-03-17T23:49:52.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.914+0000 7f4026ffd640 1 -- 172.21.15.150:0/3164175653 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v60) ==== 181+0+0 (secure 0 0 0) 0x7f402c059bc0 con 0x7f40380b57f0 2024-03-17T23:49:52.915 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_metadata' removed 2024-03-17T23:49:52.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.917+0000 7f403c9b9640 1 -- 172.21.15.150:0/3164175653 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f401806bc00 msgr2=0x7f401806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:52.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.917+0000 7f403c9b9640 1 --2- 172.21.15.150:0/3164175653 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f401806bc00 0x7f401806e0c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f40381f82e0 tx=0x7f4028006040 comp rx=0 tx=0).stop 2024-03-17T23:49:52.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.917+0000 7f403c9b9640 1 -- 172.21.15.150:0/3164175653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40380b57f0 msgr2=0x7f403814dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:52.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.917+0000 7f403c9b9640 1 --2- 172.21.15.150:0/3164175653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40380b57f0 0x7f403814dd00 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f402c011910 tx=0x7f402c011dd0 comp rx=0 tx=0).stop 2024-03-17T23:49:52.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.917+0000 7f403c9b9640 1 -- 172.21.15.150:0/3164175653 shutdown_connections 2024-03-17T23:49:52.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.917+0000 7f403c9b9640 1 --2- 172.21.15.150:0/3164175653 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f403814e240 0x7f403814e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:52.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.917+0000 7f403c9b9640 1 --2- 172.21.15.150:0/3164175653 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40380b57f0 0x7f403814dd00 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:52.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.917+0000 7f403c9b9640 1 --2- 172.21.15.150:0/3164175653 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f401806bc00 0x7f401806e0c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:52.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.917+0000 7f403c9b9640 1 --2- 172.21.15.150:0/3164175653 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4020004d40 0x7f40380b52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:52.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.917+0000 7f403c9b9640 1 -- 172.21.15.150:0/3164175653 >> 172.21.15.150:0/3164175653 conn(0x7f40380b35f0 msgr2=0x7f4038161170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:52.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.917+0000 7f403c9b9640 1 -- 172.21.15.150:0/3164175653 shutdown_connections 2024-03-17T23:49:52.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:52.917+0000 7f403c9b9640 1 -- 172.21.15.150:0/3164175653 wait complete. 2024-03-17T23:49:52.932 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-17T23:49:53.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.039+0000 7f95ec880640 1 Processor -- start 2024-03-17T23:49:53.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.039+0000 7f95ec880640 1 -- start start 2024-03-17T23:49:53.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.040+0000 7f95ec880640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f95e80af490 0x7f95e80af870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:53.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.040+0000 7f95ec880640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f95e80d0ac0 0x7f95e806c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:53.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.040+0000 7f95ec880640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95e80afdb0 con 0x7f95e8067790 2024-03-17T23:49:53.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.040+0000 7f95ec880640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f95e80cd210 con 0x7f95e80af490 2024-03-17T23:49:53.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.040+0000 7f95ec880640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f95e806bdb0 con 0x7f95e80d0ac0 2024-03-17T23:49:53.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.040+0000 7f95e659c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f95e80af490 0x7f95e80af870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:53.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.040+0000 7f95dffff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f95e8067790 0x7f95e80ccb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41920/0 (socket says 172.21.15.150:41920) 2024-03-17T23:49:53.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.040+0000 7f95dffff640 1 -- 172.21.15.150:0/1772570122 learned_addr learned my addr 172.21.15.150:0/1772570122 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:53.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.040+0000 7f95e6d9d640 1 --2- 172.21.15.150:0/1772570122 >> v2:172.21.15.157:3300/0 conn(0x7f95e80d0ac0 0x7f95e806c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:53.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.040+0000 7f95e659c640 1 -- 172.21.15.150:0/1772570122 >> v2:172.21.15.157:3300/0 conn(0x7f95e80d0ac0 msgr2=0x7f95e806c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:53.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.040+0000 7f95e659c640 1 --2- 172.21.15.150:0/1772570122 >> v2:172.21.15.157:3300/0 conn(0x7f95e80d0ac0 0x7f95e806c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:53.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.040+0000 7f95e659c640 1 -- 172.21.15.150:0/1772570122 >> v1:172.21.15.150:6789/0 conn(0x7f95e8067790 legacy=0x7f95e80ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:53.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.040+0000 7f95e659c640 1 -- 172.21.15.150:0/1772570122 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e80b1830 con 0x7f95e80af490 2024-03-17T23:49:53.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.040+0000 7f95e5d9b640 1 -- 172.21.15.150:0/1772570122 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3554596416 0 0) 0x7f95e80afdb0 con 0x7f95e8067790 2024-03-17T23:49:53.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.041+0000 7f95e659c640 1 --2- 172.21.15.150:0/1772570122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e80af490 0x7f95e80af870 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f95e80b5880 tx=0x7f95d802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e59fc88523cfc31e server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:53.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.041+0000 7f95e5d9b640 1 -- 172.21.15.150:0/1772570122 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d8035020 con 0x7f95e80af490 2024-03-17T23:49:53.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.041+0000 7f95e5d9b640 1 -- 172.21.15.150:0/1772570122 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95d8031830 con 0x7f95e80af490 2024-03-17T23:49:53.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.041+0000 7f95e5d9b640 1 -- 172.21.15.150:0/1772570122 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d8031c00 con 0x7f95e80af490 2024-03-17T23:49:53.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.041+0000 7f95ec880640 1 -- 172.21.15.150:0/1772570122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e80af490 msgr2=0x7f95e80af870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:53.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.041+0000 7f95ec880640 1 --2- 172.21.15.150:0/1772570122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e80af490 0x7f95e80af870 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f95e80b5880 tx=0x7f95d802b8d0 comp rx=0 tx=0).stop 2024-03-17T23:49:53.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.042+0000 7f95ec880640 1 -- 172.21.15.150:0/1772570122 shutdown_connections 2024-03-17T23:49:53.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.042+0000 7f95ec880640 1 --2- 172.21.15.150:0/1772570122 >> v2:172.21.15.157:3300/0 conn(0x7f95e80d0ac0 0x7f95e806c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:53.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.042+0000 7f95ec880640 1 --2- 172.21.15.150:0/1772570122 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e80af490 0x7f95e80af870 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:53.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.042+0000 7f95ec880640 1 -- 172.21.15.150:0/1772570122 >> 172.21.15.150:0/1772570122 conn(0x7f95e8072a90 msgr2=0x7f95e80b5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:53.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.042+0000 7f95ec880640 1 -- 172.21.15.150:0/1772570122 shutdown_connections 2024-03-17T23:49:53.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.042+0000 7f95ec880640 1 -- 172.21.15.150:0/1772570122 wait complete. 2024-03-17T23:49:53.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.043+0000 7f95ec880640 1 Processor -- start 2024-03-17T23:49:53.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.043+0000 7f95ec880640 1 -- start start 2024-03-17T23:49:53.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.044+0000 7f95ec880640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95e8067790 0x7f95e8153020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:53.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.044+0000 7f95ec880640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e80af490 0x7f95e815af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:53.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.044+0000 7f95ec880640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95e80d0ac0 0x7f95e815b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:53.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.044+0000 7f95ec880640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f95e80b15d0 con 0x7f95e80af490 2024-03-17T23:49:53.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.044+0000 7f95ec880640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f95e80b1450 con 0x7f95e8067790 2024-03-17T23:49:53.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.044+0000 7f95ec880640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f95e815c120 con 0x7f95e80d0ac0 2024-03-17T23:49:53.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.044+0000 7f95dffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e80af490 0x7f95e815af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:53.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.044+0000 7f95dffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e80af490 0x7f95e815af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60682/0 (socket says 172.21.15.150:60682) 2024-03-17T23:49:53.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.044+0000 7f95dffff640 1 -- 172.21.15.150:0/3464894912 learned_addr learned my addr 172.21.15.150:0/3464894912 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:53.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.044+0000 7f95e6d9d640 1 --2- 172.21.15.150:0/3464894912 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95e80d0ac0 0x7f95e815b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:53.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.044+0000 7f95e659c640 1 --2- 172.21.15.150:0/3464894912 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95e8067790 0x7f95e8153020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:53.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.044+0000 7f95dffff640 1 -- 172.21.15.150:0/3464894912 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95e80d0ac0 msgr2=0x7f95e815b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:53.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.044+0000 7f95dffff640 1 --2- 172.21.15.150:0/3464894912 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95e80d0ac0 0x7f95e815b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:53.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.044+0000 7f95dffff640 1 -- 172.21.15.150:0/3464894912 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95e8067790 msgr2=0x7f95e8153020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:53.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.044+0000 7f95dffff640 1 --2- 172.21.15.150:0/3464894912 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95e8067790 0x7f95e8153020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:53.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.044+0000 7f95dffff640 1 -- 172.21.15.150:0/3464894912 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e8153df0 con 0x7f95e80af490 2024-03-17T23:49:53.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.044+0000 7f95dffff640 1 --2- 172.21.15.150:0/3464894912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e80af490 0x7f95e815af20 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f95d000c500 tx=0x7f95d000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:53.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.044+0000 7f95deffd640 1 -- 172.21.15.150:0/3464894912 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d0018070 con 0x7f95e80af490 2024-03-17T23:49:53.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.044+0000 7f95deffd640 1 -- 172.21.15.150:0/3464894912 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95d0007580 con 0x7f95e80af490 2024-03-17T23:49:53.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.044+0000 7f95ec880640 1 -- 172.21.15.150:0/3464894912 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95e81540d0 con 0x7f95e80af490 2024-03-17T23:49:53.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.044+0000 7f95deffd640 1 -- 172.21.15.150:0/3464894912 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d0004a20 con 0x7f95e80af490 2024-03-17T23:49:53.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.044+0000 7f95ec880640 1 -- 172.21.15.150:0/3464894912 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95e8154690 con 0x7f95e80af490 2024-03-17T23:49:53.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.046+0000 7f95ec880640 1 -- 172.21.15.150:0/3464894912 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95e80681a0 con 0x7f95e80af490 2024-03-17T23:49:53.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.046+0000 7f95deffd640 1 -- 172.21.15.150:0/3464894912 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f95d0004bc0 con 0x7f95e80af490 2024-03-17T23:49:53.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.046+0000 7f95deffd640 1 --2- 172.21.15.150:0/3464894912 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f95c806bbd0 0x7f95c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:53.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.046+0000 7f95deffd640 1 -- 172.21.15.150:0/3464894912 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 5749+0+0 (secure 0 0 0) 0x7f95d0089da0 con 0x7f95e80af490 2024-03-17T23:49:53.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.046+0000 7f95e659c640 1 --2- 172.21.15.150:0/3464894912 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f95c806bbd0 0x7f95c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:53.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.047+0000 7f95e659c640 1 --2- 172.21.15.150:0/3464894912 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f95c806bbd0 0x7f95c806e090 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f95d8036000 tx=0x7f95d8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:53.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.051+0000 7f95deffd640 1 -- 172.21.15.150:0/3464894912 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f95d0058870 con 0x7f95e80af490 2024-03-17T23:49:53.229 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.229+0000 7f95ec880640 1 -- 172.21.15.150:0/3464894912 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f95e80b4270 con 0x7f95e80af490 2024-03-17T23:49:53.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.919+0000 7f95deffd640 1 -- 172.21.15.150:0/3464894912 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v61) ==== 169+0+0 (secure 0 0 0) 0x7f95d0058170 con 0x7f95e80af490 2024-03-17T23:49:53.920 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_data' removed 2024-03-17T23:49:53.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.922+0000 7f95ec880640 1 -- 172.21.15.150:0/3464894912 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f95c806bbd0 msgr2=0x7f95c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:53.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.922+0000 7f95ec880640 1 --2- 172.21.15.150:0/3464894912 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f95c806bbd0 0x7f95c806e090 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f95d8036000 tx=0x7f95d8008030 comp rx=0 tx=0).stop 2024-03-17T23:49:53.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.922+0000 7f95ec880640 1 -- 172.21.15.150:0/3464894912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e80af490 msgr2=0x7f95e815af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:53.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.922+0000 7f95ec880640 1 --2- 172.21.15.150:0/3464894912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e80af490 0x7f95e815af20 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f95d000c500 tx=0x7f95d000c9c0 comp rx=0 tx=0).stop 2024-03-17T23:49:53.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.922+0000 7f95ec880640 1 -- 172.21.15.150:0/3464894912 shutdown_connections 2024-03-17T23:49:53.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.922+0000 7f95ec880640 1 --2- 172.21.15.150:0/3464894912 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f95e80d0ac0 0x7f95e815b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:53.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.922+0000 7f95ec880640 1 --2- 172.21.15.150:0/3464894912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95e80af490 0x7f95e815af20 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:53.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.922+0000 7f95ec880640 1 --2- 172.21.15.150:0/3464894912 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f95c806bbd0 0x7f95c806e090 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:53.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.922+0000 7f95ec880640 1 --2- 172.21.15.150:0/3464894912 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95e8067790 0x7f95e8153020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:53.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.922+0000 7f95ec880640 1 -- 172.21.15.150:0/3464894912 >> 172.21.15.150:0/3464894912 conn(0x7f95e8072a90 msgr2=0x7f95e80ce400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:53.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.923+0000 7f95ec880640 1 -- 172.21.15.150:0/3464894912 shutdown_connections 2024-03-17T23:49:53.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:53.923+0000 7f95ec880640 1 -- 172.21.15.150:0/3464894912 wait complete. 2024-03-17T23:49:53.936 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-17T23:49:54.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.042+0000 7ff9ac1e6640 1 Processor -- start 2024-03-17T23:49:54.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.042+0000 7ff9ac1e6640 1 -- start start 2024-03-17T23:49:54.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.043+0000 7ff9ac1e6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff9a415bfd0 0x7ff9a40ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:54.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.043+0000 7ff9ac1e6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9a40cd250 0x7ff9a415ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:54.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.043+0000 7ff9ac1e6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9a40cf3b0 con 0x7ff9a4155320 2024-03-17T23:49:54.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.043+0000 7ff9ac1e6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7ff9a415c550 con 0x7ff9a415bfd0 2024-03-17T23:49:54.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.043+0000 7ff9ac1e6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff9a40cf690 con 0x7ff9a40cd250 2024-03-17T23:49:54.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.043+0000 7ff9a9781640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff9a415bfd0 0x7ff9a40ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:54.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.043+0000 7ff9a9781640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff9a415bfd0 0x7ff9a40ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60698/0 (socket says 172.21.15.150:60698) 2024-03-17T23:49:54.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.043+0000 7ff9a9781640 1 -- 172.21.15.150:0/384182416 learned_addr learned my addr 172.21.15.150:0/384182416 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:54.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.043+0000 7ff9a9781640 1 -- 172.21.15.150:0/384182416 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9a40cd250 msgr2=0x7ff9a415ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:54.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.043+0000 7ff9a9781640 1 --2- 172.21.15.150:0/384182416 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9a40cd250 0x7ff9a415ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:54.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.043+0000 7ff9a9781640 1 -- 172.21.15.150:0/384182416 >> v1:172.21.15.157:6789/0 conn(0x7ff9a4155320 legacy=0x7ff9a406dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:54.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.043+0000 7ff9a9781640 1 -- 172.21.15.150:0/384182416 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a415c7a0 con 0x7ff9a415bfd0 2024-03-17T23:49:54.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.044+0000 7ff9a9781640 1 --2- 172.21.15.150:0/384182416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a415bfd0 0x7ff9a40ccd10 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7ff994009870 tx=0x7ff99402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fccb4fe49cab1ca3 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:54.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.044+0000 7ff9a8f80640 1 -- 172.21.15.150:0/384182416 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff994035020 con 0x7ff9a415bfd0 2024-03-17T23:49:54.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.044+0000 7ff9a8f80640 1 -- 172.21.15.150:0/384182416 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff994031640 con 0x7ff9a415bfd0 2024-03-17T23:49:54.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.044+0000 7ff9a8f80640 1 -- 172.21.15.150:0/384182416 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff99402d040 con 0x7ff9a415bfd0 2024-03-17T23:49:54.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.044+0000 7ff9ac1e6640 1 -- 172.21.15.150:0/384182416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a415bfd0 msgr2=0x7ff9a40ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:54.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.044+0000 7ff9ac1e6640 1 --2- 172.21.15.150:0/384182416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a415bfd0 0x7ff9a40ccd10 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7ff994009870 tx=0x7ff99402b8f0 comp rx=0 tx=0).stop 2024-03-17T23:49:54.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.044+0000 7ff9ac1e6640 1 -- 172.21.15.150:0/384182416 shutdown_connections 2024-03-17T23:49:54.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.044+0000 7ff9ac1e6640 1 --2- 172.21.15.150:0/384182416 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9a40cd250 0x7ff9a415ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:54.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.044+0000 7ff9ac1e6640 1 --2- 172.21.15.150:0/384182416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a415bfd0 0x7ff9a40ccd10 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:54.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.044+0000 7ff9ac1e6640 1 -- 172.21.15.150:0/384182416 >> 172.21.15.150:0/384182416 conn(0x7ff9a406b190 msgr2=0x7ff9a406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:54.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.045+0000 7ff9ac1e6640 1 -- 172.21.15.150:0/384182416 shutdown_connections 2024-03-17T23:49:54.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.045+0000 7ff9ac1e6640 1 -- 172.21.15.150:0/384182416 wait complete. 2024-03-17T23:49:54.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.046+0000 7ff9ac1e6640 1 Processor -- start 2024-03-17T23:49:54.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.046+0000 7ff9ac1e6640 1 -- start start 2024-03-17T23:49:54.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.046+0000 7ff9ac1e6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a40cd250 0x7ff9a414b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:54.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.046+0000 7ff9ac1e6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9a4155320 0x7ff9a415ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:54.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.046+0000 7ff9ac1e6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff9a415bfd0 0x7ff9a414bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:54.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.046+0000 7ff9ac1e6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a4155880 con 0x7ff9a40cd250 2024-03-17T23:49:54.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.046+0000 7ff9ac1e6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a4155700 con 0x7ff9a415bfd0 2024-03-17T23:49:54.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.046+0000 7ff9ac1e6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff9a415d200 con 0x7ff9a4155320 2024-03-17T23:49:54.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.047+0000 7ff9a9f82640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a40cd250 0x7ff9a414b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:54.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.047+0000 7ff9a9f82640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a40cd250 0x7ff9a414b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60704/0 (socket says 172.21.15.150:60704) 2024-03-17T23:49:54.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.047+0000 7ff9a9f82640 1 -- 172.21.15.150:0/3460465495 learned_addr learned my addr 172.21.15.150:0/3460465495 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:54.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.047+0000 7ff9a9781640 1 --2- 172.21.15.150:0/3460465495 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9a4155320 0x7ff9a415ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:54.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.047+0000 7ff9aa783640 1 --2- 172.21.15.150:0/3460465495 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff9a415bfd0 0x7ff9a414bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:54.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.047+0000 7ff9a9f82640 1 -- 172.21.15.150:0/3460465495 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9a4155320 msgr2=0x7ff9a415ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:54.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.047+0000 7ff9a9f82640 1 --2- 172.21.15.150:0/3460465495 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9a4155320 0x7ff9a415ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:54.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.047+0000 7ff9a9f82640 1 -- 172.21.15.150:0/3460465495 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff9a415bfd0 msgr2=0x7ff9a414bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:54.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.047+0000 7ff9a9f82640 1 --2- 172.21.15.150:0/3460465495 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff9a415bfd0 0x7ff9a414bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:54.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.047+0000 7ff9a9f82640 1 -- 172.21.15.150:0/3460465495 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a4158940 con 0x7ff9a40cd250 2024-03-17T23:49:54.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.047+0000 7ff9a9f82640 1 --2- 172.21.15.150:0/3460465495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a40cd250 0x7ff9a414b0b0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7ff9a000c7a0 tx=0x7ff9a000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:54.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.047+0000 7ff99a7fc640 1 -- 172.21.15.150:0/3460465495 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a0017070 con 0x7ff9a40cd250 2024-03-17T23:49:54.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.047+0000 7ff99a7fc640 1 -- 172.21.15.150:0/3460465495 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9a0003da0 con 0x7ff9a40cd250 2024-03-17T23:49:54.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.047+0000 7ff9ac1e6640 1 -- 172.21.15.150:0/3460465495 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9a414a330 con 0x7ff9a40cd250 2024-03-17T23:49:54.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.047+0000 7ff99a7fc640 1 -- 172.21.15.150:0/3460465495 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a00052b0 con 0x7ff9a40cd250 2024-03-17T23:49:54.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.047+0000 7ff9ac1e6640 1 -- 172.21.15.150:0/3460465495 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9a414a5d0 con 0x7ff9a40cd250 2024-03-17T23:49:54.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.048+0000 7ff9ac1e6640 1 -- 172.21.15.150:0/3460465495 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9a40681a0 con 0x7ff9a40cd250 2024-03-17T23:49:54.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.049+0000 7ff99a7fc640 1 -- 172.21.15.150:0/3460465495 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff9a001a070 con 0x7ff9a40cd250 2024-03-17T23:49:54.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.049+0000 7ff99a7fc640 1 --2- 172.21.15.150:0/3460465495 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff98806bbd0 0x7ff98806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:54.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.049+0000 7ff99a7fc640 1 -- 172.21.15.150:0/3460465495 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5335+0+0 (secure 0 0 0) 0x7ff9a0089f70 con 0x7ff9a40cd250 2024-03-17T23:49:54.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.049+0000 7ff9a9781640 1 --2- 172.21.15.150:0/3460465495 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff98806bbd0 0x7ff98806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:54.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.049+0000 7ff9a9781640 1 --2- 172.21.15.150:0/3460465495 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff98806bbd0 0x7ff98806e090 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff994036000 tx=0x7ff99403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:54.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.053+0000 7ff99a7fc640 1 -- 172.21.15.150:0/3460465495 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff9a0058c60 con 0x7ff9a40cd250 2024-03-17T23:49:54.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.227+0000 7ff9ac1e6640 1 -- 172.21.15.150:0/3460465495 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7ff9a406ccb0 con 0x7ff9a40cd250 2024-03-17T23:49:54.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.230+0000 7ff99a7fc640 1 -- 172.21.15.150:0/3460465495 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) ==== 104+0+0 (secure 0 0 0) 0x7ff9a0058560 con 0x7ff9a40cd250 2024-03-17T23:49:54.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.232+0000 7ff9ac1e6640 1 -- 172.21.15.150:0/3460465495 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff98806bbd0 msgr2=0x7ff98806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:54.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.232+0000 7ff9ac1e6640 1 --2- 172.21.15.150:0/3460465495 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff98806bbd0 0x7ff98806e090 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff994036000 tx=0x7ff99403c040 comp rx=0 tx=0).stop 2024-03-17T23:49:54.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.232+0000 7ff9ac1e6640 1 -- 172.21.15.150:0/3460465495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a40cd250 msgr2=0x7ff9a414b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:54.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.232+0000 7ff9ac1e6640 1 --2- 172.21.15.150:0/3460465495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a40cd250 0x7ff9a414b0b0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7ff9a000c7a0 tx=0x7ff9a000cc60 comp rx=0 tx=0).stop 2024-03-17T23:49:54.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.232+0000 7ff9ac1e6640 1 -- 172.21.15.150:0/3460465495 shutdown_connections 2024-03-17T23:49:54.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.232+0000 7ff9ac1e6640 1 --2- 172.21.15.150:0/3460465495 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff98806bbd0 0x7ff98806e090 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:54.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.232+0000 7ff9ac1e6640 1 --2- 172.21.15.150:0/3460465495 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff9a415bfd0 0x7ff9a414bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:54.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.232+0000 7ff9ac1e6640 1 --2- 172.21.15.150:0/3460465495 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff9a4155320 0x7ff9a415ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:54.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.232+0000 7ff9ac1e6640 1 --2- 172.21.15.150:0/3460465495 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9a40cd250 0x7ff9a414b0b0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:54.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.232+0000 7ff9ac1e6640 1 -- 172.21.15.150:0/3460465495 >> 172.21.15.150:0/3460465495 conn(0x7ff9a406b190 msgr2=0x7ff9a41530e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:54.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.232+0000 7ff9ac1e6640 1 -- 172.21.15.150:0/3460465495 shutdown_connections 2024-03-17T23:49:54.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.232+0000 7ff9ac1e6640 1 -- 172.21.15.150:0/3460465495 wait complete. 2024-03-17T23:49:54.243 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm' 2024-03-17T23:49:54.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.343+0000 7f24fccb2640 1 Processor -- start 2024-03-17T23:49:54.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.343+0000 7f24fccb2640 1 -- start start 2024-03-17T23:49:54.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.344+0000 7f24fccb2640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f24f806f400 0x7f24f80cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:54.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.344+0000 7f24fccb2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24f8145170 con 0x7f24f8144bb0 2024-03-17T23:49:54.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.344+0000 7f24fccb2640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24f8145450 con 0x7f24f80ce4d0 2024-03-17T23:49:54.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.344+0000 7f24fccb2640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f24f80ce940 con 0x7f24f806f400 2024-03-17T23:49:54.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.344+0000 7f24f659c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f24f8144bb0 0x7f24f815ce50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41930/0 (socket says 172.21.15.150:41930) 2024-03-17T23:49:54.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.344+0000 7f24f659c640 1 -- 172.21.15.150:0/3934666702 learned_addr learned my addr 172.21.15.150:0/3934666702 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:54.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.344+0000 7f24effff640 1 --2- 172.21.15.150:0/3934666702 >> v2:172.21.15.157:3300/0 conn(0x7f24f806f400 0x7f24f80cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:54.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.344+0000 7f24f5d9b640 1 -- 172.21.15.150:0/3934666702 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 888394168 0 0) 0x7f24f8145170 con 0x7f24f8144bb0 2024-03-17T23:49:54.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.344+0000 7f24f5d9b640 1 -- 172.21.15.150:0/3934666702 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24e4003520 con 0x7f24f8144bb0 2024-03-17T23:49:54.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.344+0000 7f24f5d9b640 1 -- 172.21.15.150:0/3934666702 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3314735443 0 0) 0x7f24e4003520 con 0x7f24f8144bb0 2024-03-17T23:49:54.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.344+0000 7f24f5d9b640 1 -- 172.21.15.150:0/3934666702 >> v2:172.21.15.157:3300/0 conn(0x7f24f806f400 msgr2=0x7f24f80cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:54.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.344+0000 7f24f5d9b640 1 --2- 172.21.15.150:0/3934666702 >> v2:172.21.15.157:3300/0 conn(0x7f24f806f400 0x7f24f80cdf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:54.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.344+0000 7f24f5d9b640 1 -- 172.21.15.150:0/3934666702 >> v1:172.21.15.157:6789/0 conn(0x7f24f80ce4d0 legacy=0x7f24f8165a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:54.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.344+0000 7f24f5d9b640 1 -- 172.21.15.150:0/3934666702 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f8145730 con 0x7f24f8144bb0 2024-03-17T23:49:54.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.344+0000 7f24f5d9b640 1 -- 172.21.15.150:0/3934666702 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7f24e8001670 con 0x7f24f8144bb0 2024-03-17T23:49:54.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.344+0000 7f24effff640 1 --2- 172.21.15.150:0/3934666702 >> v2:172.21.15.157:3300/0 conn(0x7f24f806f400 0x7f24f80cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:54.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.344+0000 7f24f5d9b640 1 -- 172.21.15.150:0/3934666702 >> v1:172.21.15.150:6789/0 conn(0x7f24f8144bb0 legacy=0x7f24f815ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:54.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.344+0000 7f24f5d9b640 1 --2- 172.21.15.150:0/3934666702 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24e40039b0 0x7f24e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:54.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.344+0000 7f24f5d9b640 1 --2- 172.21.15.150:0/3934666702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24e4004800 0x7f24e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:54.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.344+0000 7f24f5d9b640 1 --2- 172.21.15.150:0/3934666702 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24e40051c0 0x7f24e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:54.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.344+0000 7f24f5d9b640 1 -- 172.21.15.150:0/3934666702 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f24f80ce940 con 0x7f24e4004800 2024-03-17T23:49:54.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.344+0000 7f24f5d9b640 1 -- 172.21.15.150:0/3934666702 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f24e400c160 con 0x7f24e40051c0 2024-03-17T23:49:54.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.344+0000 7f24f5d9b640 1 -- 172.21.15.150:0/3934666702 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f24e400c360 con 0x7f24e40039b0 2024-03-17T23:49:54.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.344+0000 7f24effff640 1 --2- 172.21.15.150:0/3934666702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24e4004800 0x7f24e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:54.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.345+0000 7f24f659c640 1 --2- 172.21.15.150:0/3934666702 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24e40039b0 0x7f24e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:54.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.345+0000 7f24f6d9d640 1 --2- 172.21.15.150:0/3934666702 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24e40051c0 0x7f24e400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:54.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.345+0000 7f24effff640 1 -- 172.21.15.150:0/3934666702 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24e40039b0 msgr2=0x7f24e4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:54.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.345+0000 7f24effff640 1 --2- 172.21.15.150:0/3934666702 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24e40039b0 0x7f24e4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:54.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.345+0000 7f24effff640 1 -- 172.21.15.150:0/3934666702 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24e40051c0 msgr2=0x7f24e400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:54.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.345+0000 7f24effff640 1 --2- 172.21.15.150:0/3934666702 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24e40051c0 0x7f24e400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:54.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.345+0000 7f24effff640 1 -- 172.21.15.150:0/3934666702 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24e4003520 con 0x7f24e4004800 2024-03-17T23:49:54.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.345+0000 7f24f6d9d640 1 -- 172.21.15.150:0/3934666702 reap_dead start 2024-03-17T23:49:54.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.345+0000 7f24effff640 1 --2- 172.21.15.150:0/3934666702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24e4004800 0x7f24e4004c80 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f24e0009c60 tx=0x7f24e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9534170d48dcf6fe server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:54.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.346+0000 7f24f5d9b640 1 -- 172.21.15.150:0/3934666702 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e0009860 con 0x7f24e4004800 2024-03-17T23:49:54.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.346+0000 7f24f5d9b640 1 -- 172.21.15.150:0/3934666702 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24e0004020 con 0x7f24e4004800 2024-03-17T23:49:54.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.346+0000 7f24f5d9b640 1 -- 172.21.15.150:0/3934666702 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e0004430 con 0x7f24e4004800 2024-03-17T23:49:54.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.346+0000 7f24fccb2640 1 -- 172.21.15.150:0/3934666702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24e4004800 msgr2=0x7f24e4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:54.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.346+0000 7f24fccb2640 1 --2- 172.21.15.150:0/3934666702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24e4004800 0x7f24e4004c80 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f24e0009c60 tx=0x7f24e4005830 comp rx=0 tx=0).stop 2024-03-17T23:49:54.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.346+0000 7f24fccb2640 1 -- 172.21.15.150:0/3934666702 shutdown_connections 2024-03-17T23:49:54.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.346+0000 7f24fccb2640 1 --2- 172.21.15.150:0/3934666702 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24e4004800 0x7f24e4004c80 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:54.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.346+0000 7f24fccb2640 1 -- 172.21.15.150:0/3934666702 >> 172.21.15.150:0/3934666702 conn(0x7f24f806b190 msgr2=0x7f24f806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:54.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.346+0000 7f24fccb2640 1 -- 172.21.15.150:0/3934666702 shutdown_connections 2024-03-17T23:49:54.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.346+0000 7f24fccb2640 1 -- 172.21.15.150:0/3934666702 wait complete. 2024-03-17T23:49:54.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.348+0000 7f24fccb2640 1 Processor -- start 2024-03-17T23:49:54.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.348+0000 7f24fccb2640 1 -- start start 2024-03-17T23:49:54.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.348+0000 7f24fccb2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24e4004800 0x7f24f813ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:54.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.348+0000 7f24fccb2640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24f813d0c0 0x7f24f81f4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:54.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.348+0000 7f24fccb2640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24f8153f30 0x7f24f81543b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:54.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.348+0000 7f24fccb2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f24f80ce110 con 0x7f24e4004800 2024-03-17T23:49:54.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.348+0000 7f24fccb2640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f24f80cdf90 con 0x7f24f8153f30 2024-03-17T23:49:54.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.348+0000 7f24fccb2640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f24f80ce290 con 0x7f24f813d0c0 2024-03-17T23:49:54.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.348+0000 7f24f659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24e4004800 0x7f24f813ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:54.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.348+0000 7f24f659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24e4004800 0x7f24f813ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60728/0 (socket says 172.21.15.150:60728) 2024-03-17T23:49:54.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.348+0000 7f24f659c640 1 -- 172.21.15.150:0/3925520284 learned_addr learned my addr 172.21.15.150:0/3925520284 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:54.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.349+0000 7f24effff640 1 --2- 172.21.15.150:0/3925520284 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24f813d0c0 0x7f24f81f4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:54.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.349+0000 7f24f6d9d640 1 --2- 172.21.15.150:0/3925520284 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24f8153f30 0x7f24f81543b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:54.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.349+0000 7f24f659c640 1 -- 172.21.15.150:0/3925520284 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24f813d0c0 msgr2=0x7f24f81f4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:54.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.349+0000 7f24f659c640 1 --2- 172.21.15.150:0/3925520284 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24f813d0c0 0x7f24f81f4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:54.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.349+0000 7f24f659c640 1 -- 172.21.15.150:0/3925520284 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24f8153f30 msgr2=0x7f24f81543b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:54.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.349+0000 7f24f659c640 1 --2- 172.21.15.150:0/3925520284 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24f8153f30 0x7f24f81543b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:54.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.349+0000 7f24f659c640 1 -- 172.21.15.150:0/3925520284 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f81548f0 con 0x7f24e4004800 2024-03-17T23:49:54.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.349+0000 7f24effff640 1 --2- 172.21.15.150:0/3925520284 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24f813d0c0 0x7f24f81f4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:49:54.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.349+0000 7f24f6d9d640 1 --2- 172.21.15.150:0/3925520284 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24f8153f30 0x7f24f81543b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:49:54.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.349+0000 7f24f659c640 1 --2- 172.21.15.150:0/3925520284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24e4004800 0x7f24f813ae00 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f24e8007370 tx=0x7f24e8007830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:54.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.349+0000 7f24eeffd640 1 -- 172.21.15.150:0/3925520284 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e800f960 con 0x7f24e4004800 2024-03-17T23:49:54.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.349+0000 7f24eeffd640 1 -- 172.21.15.150:0/3925520284 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24e8018070 con 0x7f24e4004800 2024-03-17T23:49:54.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.349+0000 7f24fccb2640 1 -- 172.21.15.150:0/3925520284 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24f813b550 con 0x7f24e4004800 2024-03-17T23:49:54.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.349+0000 7f24eeffd640 1 -- 172.21.15.150:0/3925520284 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e800fd60 con 0x7f24e4004800 2024-03-17T23:49:54.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.349+0000 7f24fccb2640 1 -- 172.21.15.150:0/3925520284 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24f813bb10 con 0x7f24e4004800 2024-03-17T23:49:54.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.350+0000 7f24fccb2640 1 -- 172.21.15.150:0/3925520284 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24f80b4e40 con 0x7f24e4004800 2024-03-17T23:49:54.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.351+0000 7f24eeffd640 1 -- 172.21.15.150:0/3925520284 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f24e801d460 con 0x7f24e4004800 2024-03-17T23:49:54.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.351+0000 7f24eeffd640 1 --2- 172.21.15.150:0/3925520284 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f24d806bc00 0x7f24d806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:54.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.351+0000 7f24eeffd640 1 -- 172.21.15.150:0/3925520284 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5335+0+0 (secure 0 0 0) 0x7f24e808a920 con 0x7f24e4004800 2024-03-17T23:49:54.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.351+0000 7f24effff640 1 --2- 172.21.15.150:0/3925520284 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f24d806bc00 0x7f24d806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:54.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.351+0000 7f24effff640 1 --2- 172.21.15.150:0/3925520284 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f24d806bc00 0x7f24d806e0c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f24e0035000 tx=0x7f24e0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:54.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.355+0000 7f24eeffd640 1 -- 172.21.15.150:0/3925520284 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f24f80b4e40 con 0x7f24e4004800 2024-03-17T23:49:54.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.597+0000 7f24fccb2640 1 -- 172.21.15.150:0/3925520284 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]} v 0) -- 0x7f24f80b5ad0 con 0x7f24e4004800 2024-03-17T23:49:54.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.920+0000 7f24eeffd640 1 -- 172.21.15.150:0/3925520284 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]}]=0 v186) ==== 134+0+0 (secure 0 0 0) 0x7f24f80b5ad0 con 0x7f24e4004800 2024-03-17T23:49:54.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.923+0000 7f24fccb2640 1 -- 172.21.15.150:0/3925520284 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f24d806bc00 msgr2=0x7f24d806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:54.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.923+0000 7f24fccb2640 1 --2- 172.21.15.150:0/3925520284 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f24d806bc00 0x7f24d806e0c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f24e0035000 tx=0x7f24e0030000 comp rx=0 tx=0).stop 2024-03-17T23:49:54.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.923+0000 7f24fccb2640 1 -- 172.21.15.150:0/3925520284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24e4004800 msgr2=0x7f24f813ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:54.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.923+0000 7f24fccb2640 1 --2- 172.21.15.150:0/3925520284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24e4004800 0x7f24f813ae00 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f24e8007370 tx=0x7f24e8007830 comp rx=0 tx=0).stop 2024-03-17T23:49:54.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.923+0000 7f24fccb2640 1 -- 172.21.15.150:0/3925520284 shutdown_connections 2024-03-17T23:49:54.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.923+0000 7f24fccb2640 1 --2- 172.21.15.150:0/3925520284 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f24d806bc00 0x7f24d806e0c0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:54.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.923+0000 7f24fccb2640 1 --2- 172.21.15.150:0/3925520284 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24f8153f30 0x7f24f81543b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:54.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.923+0000 7f24fccb2640 1 --2- 172.21.15.150:0/3925520284 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f24f813d0c0 0x7f24f81f4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:54.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.923+0000 7f24fccb2640 1 --2- 172.21.15.150:0/3925520284 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f24e4004800 0x7f24f813ae00 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:54.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.923+0000 7f24fccb2640 1 -- 172.21.15.150:0/3925520284 >> 172.21.15.150:0/3925520284 conn(0x7f24f806b190 msgr2=0x7f24f80af7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:54.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.924+0000 7f24fccb2640 1 -- 172.21.15.150:0/3925520284 shutdown_connections 2024-03-17T23:49:54.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:54.924+0000 7f24fccb2640 1 -- 172.21.15.150:0/3925520284 wait complete. 2024-03-17T23:49:54.957 INFO:tasks.cephfs_test_runner:test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) ... ERROR 2024-03-17T23:49:54.957 INFO:tasks.cephfs_test_runner: 2024-03-17T23:49:54.957 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-17T23:49:54.957 INFO:tasks.cephfs_test_runner:ERROR: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-17T23:49:54.958 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-17T23:49:54.958 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-17T23:49:54.958 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephfs/test_pool_perm.py", line 52, in test_pool_perm 2024-03-17T23:49:54.958 INFO:tasks.cephfs_test_runner: self.mount_a.run_python(remote_script.format(path=file_path, check_read=str(True))) 2024-03-17T23:49:54.958 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephfs/mount.py", line 776, in run_python 2024-03-17T23:49:54.958 INFO:tasks.cephfs_test_runner: p.wait() 2024-03-17T23:49:54.958 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait 2024-03-17T23:49:54.958 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-03-17T23:49:54.958 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-03-17T23:49:54.959 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-03-17T23:49:54.959 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi150 with status 1: 'stdin-killer -- python3 -c \'\nimport os\nimport errno\n\nfd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR)\ntry:\n if True:\n ret = os.read(fd, 1024)\n else:\n os.write(fd, b\'"\'"\'content\'"\'"\')\nexcept OSError as e:\n if e.errno != errno.EPERM:\n raise\nelse:\n raise RuntimeError("client does not check permission of data pool")\n\'' 2024-03-17T23:49:54.959 INFO:tasks.cephfs_test_runner: 2024-03-17T23:49:54.959 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-17T23:49:54.959 INFO:tasks.cephfs_test_runner:Ran 2 tests in 137.468s 2024-03-17T23:49:54.959 INFO:tasks.cephfs_test_runner: 2024-03-17T23:49:54.959 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-03-17T23:49:54.959 INFO:tasks.cephfs_test_runner: 2024-03-17T23:49:54.960 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-17T23:49:54.960 INFO:tasks.cephfs_test_runner:ERROR: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-17T23:49:54.960 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-17T23:49:54.960 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-17T23:49:54.960 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephfs/test_pool_perm.py", line 52, in test_pool_perm 2024-03-17T23:49:54.960 INFO:tasks.cephfs_test_runner: self.mount_a.run_python(remote_script.format(path=file_path, check_read=str(True))) 2024-03-17T23:49:54.960 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephfs/mount.py", line 776, in run_python 2024-03-17T23:49:54.960 INFO:tasks.cephfs_test_runner: p.wait() 2024-03-17T23:49:54.960 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait 2024-03-17T23:49:54.961 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-03-17T23:49:54.961 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-03-17T23:49:54.961 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-03-17T23:49:54.961 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi150 with status 1: 'stdin-killer -- python3 -c \'\nimport os\nimport errno\n\nfd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR)\ntry:\n if True:\n ret = os.read(fd, 1024)\n else:\n os.write(fd, b\'"\'"\'content\'"\'"\')\nexcept OSError as e:\n if e.errno != errno.EPERM:\n raise\nelse:\n raise RuntimeError("client does not check permission of data pool")\n\'' 2024-03-17T23:49:54.961 INFO:tasks.cephfs_test_runner: 2024-03-17T23:49:54.961 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-17T23:49:55.213 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=810d613814e74b5196e016d9e432b81f Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-17T23:49:55.215 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-17T23:49:55.224 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-17T23:49:55.232 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-17T23:49:55.232 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:49:55.232 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:49:55.251 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19488 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-17T23:49:55.251 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-03-17T23:49:55.251 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:49:55.306 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19488 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-17T23:49:55.307 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:49:55.307 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:49:55.326 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=626 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-17T23:49:55.326 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-03-17T23:49:55.326 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-17T23:49:55.384 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056373,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=626 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-17T23:49:55.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:55.385 DEBUG:teuthology.orchestra.run.smithi157:> ip netns list 2024-03-17T23:49:55.442 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-17T23:49:55.443 INFO:teuthology.orchestra.run.smithi157.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-17T23:49:55.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:55.443 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-17T23:49:55.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:55.513 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-17T23:49:55.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:55.584 DEBUG:teuthology.orchestra.run.smithi157:> sudo ip link delete ceph-brx 2024-03-17T23:49:55.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:55.687 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-03-17T23:49:55.704 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-17T23:49:55.704 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-17T23:49:55.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:55.704 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-17T23:49:55.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:55.775 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-17T23:49:55.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-17T23:49:55.845 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link delete ceph-brx 2024-03-17T23:49:55.939 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-17T23:49:55.948 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-17T23:49:55.948 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-17T23:49:56.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.050+0000 7f32ac697640 1 Processor -- start 2024-03-17T23:49:56.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.050+0000 7f32ac697640 1 -- start start 2024-03-17T23:49:56.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.051+0000 7f32ac697640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f32a415bfd0 0x7f32a40ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:56.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.051+0000 7f32ac697640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32a40cd250 0x7f32a415ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:56.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.051+0000 7f32ac697640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32a40cf3b0 con 0x7f32a4155320 2024-03-17T23:49:56.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.051+0000 7f32ac697640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f32a415c550 con 0x7f32a415bfd0 2024-03-17T23:49:56.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.051+0000 7f32ac697640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f32a40cf690 con 0x7f32a40cd250 2024-03-17T23:49:56.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.051+0000 7f32aac34640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32a40cd250 0x7f32a415ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:56.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.051+0000 7f32a9c32640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f32a415bfd0 0x7f32a40ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:56.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.051+0000 7f32aa433640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f32a4155320 0x7f32a406dd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:45846/0 (socket says 172.21.15.150:45846) 2024-03-17T23:49:56.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.051+0000 7f32aa433640 1 -- 172.21.15.150:0/1650394969 learned_addr learned my addr 172.21.15.150:0/1650394969 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:56.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.051+0000 7f32a9431640 1 -- 172.21.15.150:0/1650394969 <== mon.1 v1:172.21.15.157:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 599722231 0 0) 0x7f32a40cf3b0 con 0x7f32a4155320 2024-03-17T23:49:56.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.051+0000 7f32a9431640 1 -- 172.21.15.150:0/1650394969 --> v1:172.21.15.157:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3290003520 con 0x7f32a4155320 2024-03-17T23:49:56.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.052+0000 7f32a9c32640 1 -- 172.21.15.150:0/1650394969 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32a40cd250 msgr2=0x7f32a415ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:56.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.052+0000 7f32a9c32640 1 --2- 172.21.15.150:0/1650394969 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32a40cd250 0x7f32a415ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.052+0000 7f32a9c32640 1 -- 172.21.15.150:0/1650394969 >> v1:172.21.15.157:6789/0 conn(0x7f32a4155320 legacy=0x7f32a406dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:56.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.052+0000 7f32a9c32640 1 -- 172.21.15.150:0/1650394969 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32a415c7a0 con 0x7f32a415bfd0 2024-03-17T23:49:56.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.052+0000 7f32aac34640 1 --2- 172.21.15.150:0/1650394969 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32a40cd250 0x7f32a415ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:56.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.052+0000 7f32a9c32640 1 --2- 172.21.15.150:0/1650394969 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32a415bfd0 0x7f32a40ccd10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f32940097a0 tx=0x7f329402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de20b90af429929 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:56.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.052+0000 7f32a9431640 1 -- 172.21.15.150:0/1650394969 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3294035020 con 0x7f32a415bfd0 2024-03-17T23:49:56.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.052+0000 7f32a9431640 1 -- 172.21.15.150:0/1650394969 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3294031830 con 0x7f32a415bfd0 2024-03-17T23:49:56.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.052+0000 7f32a9431640 1 -- 172.21.15.150:0/1650394969 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3294031be0 con 0x7f32a415bfd0 2024-03-17T23:49:56.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.053+0000 7f32ac697640 1 -- 172.21.15.150:0/1650394969 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32a415bfd0 msgr2=0x7f32a40ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:56.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.053+0000 7f32ac697640 1 --2- 172.21.15.150:0/1650394969 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32a415bfd0 0x7f32a40ccd10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f32940097a0 tx=0x7f329402b8d0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.053+0000 7f32ac697640 1 -- 172.21.15.150:0/1650394969 shutdown_connections 2024-03-17T23:49:56.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.053+0000 7f32ac697640 1 --2- 172.21.15.150:0/1650394969 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32a40cd250 0x7f32a415ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.053+0000 7f32ac697640 1 --2- 172.21.15.150:0/1650394969 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32a415bfd0 0x7f32a40ccd10 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.053+0000 7f32ac697640 1 -- 172.21.15.150:0/1650394969 >> 172.21.15.150:0/1650394969 conn(0x7f32a406b190 msgr2=0x7f32a406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:56.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.053+0000 7f32ac697640 1 -- 172.21.15.150:0/1650394969 shutdown_connections 2024-03-17T23:49:56.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.053+0000 7f32ac697640 1 -- 172.21.15.150:0/1650394969 wait complete. 2024-03-17T23:49:56.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.054+0000 7f32ac697640 1 Processor -- start 2024-03-17T23:49:56.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.054+0000 7f32ac697640 1 -- start start 2024-03-17T23:49:56.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.055+0000 7f32ac697640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32a40cd250 0x7f32a4148d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:56.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.055+0000 7f32ac697640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32a4155320 0x7f32a4150230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:56.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.055+0000 7f32ac697640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32a415bfd0 0x7f32a40b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:56.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.055+0000 7f32ac697640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f32a4155880 con 0x7f32a40cd250 2024-03-17T23:49:56.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.055+0000 7f32ac697640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f32a4155700 con 0x7f32a4155320 2024-03-17T23:49:56.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.055+0000 7f32ac697640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f32a415d200 con 0x7f32a415bfd0 2024-03-17T23:49:56.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.055+0000 7f32aa433640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32a40cd250 0x7f32a4148d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:56.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.055+0000 7f32aa433640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32a40cd250 0x7f32a4148d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44710/0 (socket says 172.21.15.150:44710) 2024-03-17T23:49:56.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.055+0000 7f32aa433640 1 -- 172.21.15.150:0/3842054905 learned_addr learned my addr 172.21.15.150:0/3842054905 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:56.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.055+0000 7f32aac34640 1 --2- 172.21.15.150:0/3842054905 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32a415bfd0 0x7f32a40b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:56.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.055+0000 7f32a9c32640 1 --2- 172.21.15.150:0/3842054905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32a4155320 0x7f32a4150230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:56.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.055+0000 7f32aa433640 1 -- 172.21.15.150:0/3842054905 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32a415bfd0 msgr2=0x7f32a40b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:56.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.055+0000 7f32aa433640 1 --2- 172.21.15.150:0/3842054905 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32a415bfd0 0x7f32a40b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.055+0000 7f32aa433640 1 -- 172.21.15.150:0/3842054905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32a4155320 msgr2=0x7f32a4150230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:56.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.055+0000 7f32aa433640 1 --2- 172.21.15.150:0/3842054905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32a4155320 0x7f32a4150230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.055+0000 7f32aa433640 1 -- 172.21.15.150:0/3842054905 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32a4159fc0 con 0x7f32a40cd250 2024-03-17T23:49:56.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.056+0000 7f32aa433640 1 --2- 172.21.15.150:0/3842054905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32a40cd250 0x7f32a4148d80 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f32a000c460 tx=0x7f32a000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:56.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.056+0000 7f329affd640 1 -- 172.21.15.150:0/3842054905 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a0017070 con 0x7f32a40cd250 2024-03-17T23:49:56.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.056+0000 7f329affd640 1 -- 172.21.15.150:0/3842054905 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32a0005ce0 con 0x7f32a40cd250 2024-03-17T23:49:56.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.056+0000 7f329affd640 1 -- 172.21.15.150:0/3842054905 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a00072f0 con 0x7f32a40cd250 2024-03-17T23:49:56.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.056+0000 7f32ac697640 1 -- 172.21.15.150:0/3842054905 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32a415a2a0 con 0x7f32a40cd250 2024-03-17T23:49:56.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.056+0000 7f32ac697640 1 -- 172.21.15.150:0/3842054905 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32a414a380 con 0x7f32a40cd250 2024-03-17T23:49:56.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.057+0000 7f32ac697640 1 -- 172.21.15.150:0/3842054905 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32a415a470 con 0x7f32a40cd250 2024-03-17T23:49:56.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.057+0000 7f329affd640 1 -- 172.21.15.150:0/3842054905 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f32a001a070 con 0x7f32a40cd250 2024-03-17T23:49:56.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.058+0000 7f329affd640 1 --2- 172.21.15.150:0/3842054905 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f328806bbd0 0x7f328806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:56.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.058+0000 7f329affd640 1 -- 172.21.15.150:0/3842054905 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5335+0+0 (secure 0 0 0) 0x7f32a0089bb0 con 0x7f32a40cd250 2024-03-17T23:49:56.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.058+0000 7f32a9c32640 1 --2- 172.21.15.150:0/3842054905 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f328806bbd0 0x7f328806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:56.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.058+0000 7f32a9c32640 1 --2- 172.21.15.150:0/3842054905 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f328806bbd0 0x7f328806e090 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3294036000 tx=0x7f329403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:56.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.062+0000 7f329affd640 1 -- 172.21.15.150:0/3842054905 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f32a0058250 con 0x7f32a40cd250 2024-03-17T23:49:56.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.232+0000 7f32ac697640 1 -- 172.21.15.150:0/3842054905 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f32a40681a0 con 0x7f328806bbd0 2024-03-17T23:49:56.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.233+0000 7f329affd640 1 -- 172.21.15.150:0/3842054905 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13415 (secure 0 0 0) 0x7f329404b480 con 0x7f328806bbd0 2024-03-17T23:49:56.234 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:56.234 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-03-17T23:49:56.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.235+0000 7f32ac697640 1 -- 172.21.15.150:0/3842054905 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f328806bbd0 msgr2=0x7f328806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:56.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.235+0000 7f32ac697640 1 --2- 172.21.15.150:0/3842054905 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f328806bbd0 0x7f328806e090 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3294036000 tx=0x7f329403c040 comp rx=0 tx=0).stop 2024-03-17T23:49:56.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.235+0000 7f32ac697640 1 -- 172.21.15.150:0/3842054905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32a40cd250 msgr2=0x7f32a4148d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:56.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.235+0000 7f32ac697640 1 --2- 172.21.15.150:0/3842054905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32a40cd250 0x7f32a4148d80 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f32a000c460 tx=0x7f32a000c860 comp rx=0 tx=0).stop 2024-03-17T23:49:56.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.235+0000 7f32ac697640 1 -- 172.21.15.150:0/3842054905 shutdown_connections 2024-03-17T23:49:56.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.235+0000 7f32ac697640 1 --2- 172.21.15.150:0/3842054905 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f32a415bfd0 0x7f32a40b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.235+0000 7f32ac697640 1 --2- 172.21.15.150:0/3842054905 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f328806bbd0 0x7f328806e090 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.235+0000 7f32ac697640 1 --2- 172.21.15.150:0/3842054905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f32a4155320 0x7f32a4150230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.235+0000 7f32ac697640 1 --2- 172.21.15.150:0/3842054905 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f32a40cd250 0x7f32a4148d80 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.235+0000 7f32ac697640 1 -- 172.21.15.150:0/3842054905 >> 172.21.15.150:0/3842054905 conn(0x7f32a406b190 msgr2=0x7f32a415d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:56.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.235+0000 7f32ac697640 1 -- 172.21.15.150:0/3842054905 shutdown_connections 2024-03-17T23:49:56.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.235+0000 7f32ac697640 1 -- 172.21.15.150:0/3842054905 wait complete. 2024-03-17T23:49:56.246 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2024-03-17T23:49:55.633153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239540,"kb_used_data":6724,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735180,"statfs":{"total":773094113280,"available":772848824320,"internally_reserved":0,"allocated":6885376,"data_stored":3776681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14933,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-69,"num_write_kb":-38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001170"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":242,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-17T23:49:51.485656+0000","last_change":"2024-03-17T23:46:44.768668+0000","last_active":"2024-03-17T23:49:51.485656+0000","last_peered":"2024-03-17T23:49:51.485656+0000","last_clean":"2024-03-17T23:49:51.485656+0000","last_became_active":"2024-03-17T23:46:44.767826+0000","last_became_peered":"2024-03-17T23:46:44.767826+0000","last_unstale":"2024-03-17T23:49:51.485656+0000","last_undegraded":"2024-03-17T23:49:51.485656+0000","last_fullsized":"2024-03-17T23:49:51.485656+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_clean_scrub_stamp":"2024-03-17T23:46:43.744878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:54:22.286155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":14,"seq":60129542186,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27716,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344124,"statfs":{"total":96636764160,"available":96608382976,"internally_reserved":0,"allocated":712704,"data_stored":322682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542186,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":720896,"data_stored":329021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542186,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28308,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343532,"statfs":{"total":96636764160,"available":96607776768,"internally_reserved":0,"allocated":1314816,"data_stored":919389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32456,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1306624,"data_stored":913154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574889,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":704512,"data_stored":322626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574889,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":700416,"data_stored":316805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574889,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31892,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339948,"statfs":{"total":96636764160,"available":96604106752,"internally_reserved":0,"allocated":729088,"data_stored":336761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":696320,"data_stored":316243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-17T23:49:56.247 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-17T23:49:56.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.347+0000 7f1a43036640 1 Processor -- start 2024-03-17T23:49:56.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.347+0000 7f1a43036640 1 -- start start 2024-03-17T23:49:56.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.348+0000 7f1a43036640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1a3c1619a0 0x7f1a3c0b9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:56.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.348+0000 7f1a43036640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a3c161dd0 con 0x7f1a3c15de30 2024-03-17T23:49:56.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.348+0000 7f1a43036640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a3c1620a0 con 0x7f1a3c0cf120 2024-03-17T23:49:56.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.348+0000 7f1a43036640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f1a3c162940 con 0x7f1a3c1619a0 2024-03-17T23:49:56.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.348+0000 7f1a415d3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1a3c1619a0 0x7f1a3c0b9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:56.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.348+0000 7f1a40dd2640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f1a3c15de30 0x7f1a3c0cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:58656/0 (socket says 172.21.15.150:58656) 2024-03-17T23:49:56.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.348+0000 7f1a40dd2640 1 -- 172.21.15.150:0/3579356358 learned_addr learned my addr 172.21.15.150:0/3579356358 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:56.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.348+0000 7f1a415d3640 1 -- 172.21.15.150:0/3579356358 >> v1:172.21.15.157:6789/0 conn(0x7f1a3c0cf120 legacy=0x7f1a3c0d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:56.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.348+0000 7f1a415d3640 1 -- 172.21.15.150:0/3579356358 >> v1:172.21.15.150:6789/0 conn(0x7f1a3c15de30 legacy=0x7f1a3c0cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:56.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.348+0000 7f1a415d3640 1 -- 172.21.15.150:0/3579356358 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a3c162380 con 0x7f1a3c1619a0 2024-03-17T23:49:56.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.348+0000 7f1a3b7fe640 1 -- 172.21.15.150:0/3579356358 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3513781783 0 0) 0x7f1a3c161dd0 con 0x7f1a3c15de30 2024-03-17T23:49:56.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.349+0000 7f1a415d3640 1 --2- 172.21.15.150:0/3579356358 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a3c1619a0 0x7f1a3c0b9860 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f1a34009d90 tx=0x7f1a3402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f08f2d7ecdfeae server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:56.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.349+0000 7f1a3b7fe640 1 -- 172.21.15.150:0/3579356358 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a34039070 con 0x7f1a3c1619a0 2024-03-17T23:49:56.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.349+0000 7f1a3b7fe640 1 -- 172.21.15.150:0/3579356358 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a3402cbd0 con 0x7f1a3c1619a0 2024-03-17T23:49:56.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.349+0000 7f1a3b7fe640 1 -- 172.21.15.150:0/3579356358 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a34034440 con 0x7f1a3c1619a0 2024-03-17T23:49:56.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.349+0000 7f1a43036640 1 -- 172.21.15.150:0/3579356358 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a3c1619a0 msgr2=0x7f1a3c0b9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:56.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.349+0000 7f1a43036640 1 --2- 172.21.15.150:0/3579356358 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a3c1619a0 0x7f1a3c0b9860 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f1a34009d90 tx=0x7f1a3402be20 comp rx=0 tx=0).stop 2024-03-17T23:49:56.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.349+0000 7f1a43036640 1 -- 172.21.15.150:0/3579356358 shutdown_connections 2024-03-17T23:49:56.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.349+0000 7f1a43036640 1 --2- 172.21.15.150:0/3579356358 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a3c1619a0 0x7f1a3c0b9860 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.349+0000 7f1a43036640 1 -- 172.21.15.150:0/3579356358 >> 172.21.15.150:0/3579356358 conn(0x7f1a3c0730b0 msgr2=0x7f1a3c0d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:56.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.349+0000 7f1a43036640 1 -- 172.21.15.150:0/3579356358 shutdown_connections 2024-03-17T23:49:56.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.349+0000 7f1a43036640 1 -- 172.21.15.150:0/3579356358 wait complete. 2024-03-17T23:49:56.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.351+0000 7f1a43036640 1 Processor -- start 2024-03-17T23:49:56.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.351+0000 7f1a43036640 1 -- start start 2024-03-17T23:49:56.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.351+0000 7f1a43036640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a3c0cf120 0x7f1a3c1497c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:56.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.351+0000 7f1a43036640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a3c15de30 0x7f1a3c157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:56.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.351+0000 7f1a43036640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a3c1619a0 0x7f1a3c0c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:56.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.351+0000 7f1a43036640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3c15e210 con 0x7f1a3c15de30 2024-03-17T23:49:56.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.351+0000 7f1a43036640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3c0d2170 con 0x7f1a3c0cf120 2024-03-17T23:49:56.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.351+0000 7f1a43036640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1a3c0d2da0 con 0x7f1a3c1619a0 2024-03-17T23:49:56.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.352+0000 7f1a3bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a3c15de30 0x7f1a3c157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:56.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.352+0000 7f1a3bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a3c15de30 0x7f1a3c157a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44732/0 (socket says 172.21.15.150:44732) 2024-03-17T23:49:56.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.352+0000 7f1a3bfff640 1 -- 172.21.15.150:0/2250016434 learned_addr learned my addr 172.21.15.150:0/2250016434 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:56.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.352+0000 7f1a40dd2640 1 --2- 172.21.15.150:0/2250016434 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a3c0cf120 0x7f1a3c1497c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:56.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.352+0000 7f1a415d3640 1 --2- 172.21.15.150:0/2250016434 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a3c1619a0 0x7f1a3c0c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:56.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.352+0000 7f1a3bfff640 1 -- 172.21.15.150:0/2250016434 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a3c1619a0 msgr2=0x7f1a3c0c94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:56.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.352+0000 7f1a3bfff640 1 --2- 172.21.15.150:0/2250016434 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a3c1619a0 0x7f1a3c0c94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.352+0000 7f1a3bfff640 1 -- 172.21.15.150:0/2250016434 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a3c0cf120 msgr2=0x7f1a3c1497c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:56.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.352+0000 7f1a3bfff640 1 --2- 172.21.15.150:0/2250016434 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a3c0cf120 0x7f1a3c1497c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.352+0000 7f1a3bfff640 1 -- 172.21.15.150:0/2250016434 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a3c1595e0 con 0x7f1a3c15de30 2024-03-17T23:49:56.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.352+0000 7f1a415d3640 1 --2- 172.21.15.150:0/2250016434 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a3c1619a0 0x7f1a3c0c94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:49:56.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.352+0000 7f1a3bfff640 1 --2- 172.21.15.150:0/2250016434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a3c15de30 0x7f1a3c157a50 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f1a2800c430 tx=0x7f1a2800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:56.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.352+0000 7f1a397fa640 1 -- 172.21.15.150:0/2250016434 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a28011790 con 0x7f1a3c15de30 2024-03-17T23:49:56.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.352+0000 7f1a397fa640 1 -- 172.21.15.150:0/2250016434 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a28017070 con 0x7f1a3c15de30 2024-03-17T23:49:56.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.352+0000 7f1a43036640 1 -- 172.21.15.150:0/2250016434 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a3c159860 con 0x7f1a3c15de30 2024-03-17T23:49:56.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.352+0000 7f1a397fa640 1 -- 172.21.15.150:0/2250016434 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a28011bb0 con 0x7f1a3c15de30 2024-03-17T23:49:56.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.352+0000 7f1a43036640 1 -- 172.21.15.150:0/2250016434 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a3c159e20 con 0x7f1a3c15de30 2024-03-17T23:49:56.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.353+0000 7f1a43036640 1 -- 172.21.15.150:0/2250016434 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a3c0681a0 con 0x7f1a3c15de30 2024-03-17T23:49:56.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.354+0000 7f1a397fa640 1 -- 172.21.15.150:0/2250016434 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1a2801c460 con 0x7f1a3c15de30 2024-03-17T23:49:56.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.354+0000 7f1a397fa640 1 --2- 172.21.15.150:0/2250016434 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1a2006bbd0 0x7f1a2006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:56.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.354+0000 7f1a397fa640 1 -- 172.21.15.150:0/2250016434 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5335+0+0 (secure 0 0 0) 0x7f1a2808a6d0 con 0x7f1a3c15de30 2024-03-17T23:49:56.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.354+0000 7f1a40dd2640 1 --2- 172.21.15.150:0/2250016434 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1a2006bbd0 0x7f1a2006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:56.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.355+0000 7f1a40dd2640 1 --2- 172.21.15.150:0/2250016434 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1a2006bbd0 0x7f1a2006e090 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1a30000f60 tx=0x7f1a30007d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:56.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.359+0000 7f1a397fa640 1 -- 172.21.15.150:0/2250016434 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f1a3c1595e0 con 0x7f1a3c15de30 2024-03-17T23:49:56.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.528+0000 7f1a43036640 1 -- 172.21.15.150:0/2250016434 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1a3c0d1910 con 0x7f1a2006bbd0 2024-03-17T23:49:56.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.529+0000 7f1a397fa640 1 -- 172.21.15.150:0/2250016434 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13415 (secure 0 0 0) 0x7f1a3c0d1910 con 0x7f1a2006bbd0 2024-03-17T23:49:56.530 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:56.530 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-03-17T23:49:56.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.531+0000 7f1a43036640 1 -- 172.21.15.150:0/2250016434 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1a2006bbd0 msgr2=0x7f1a2006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:56.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.531+0000 7f1a43036640 1 --2- 172.21.15.150:0/2250016434 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1a2006bbd0 0x7f1a2006e090 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1a30000f60 tx=0x7f1a30007d50 comp rx=0 tx=0).stop 2024-03-17T23:49:56.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.531+0000 7f1a43036640 1 -- 172.21.15.150:0/2250016434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a3c15de30 msgr2=0x7f1a3c157a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:56.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.531+0000 7f1a43036640 1 --2- 172.21.15.150:0/2250016434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a3c15de30 0x7f1a3c157a50 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f1a2800c430 tx=0x7f1a2800c8f0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.531+0000 7f1a43036640 1 -- 172.21.15.150:0/2250016434 shutdown_connections 2024-03-17T23:49:56.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.531+0000 7f1a43036640 1 --2- 172.21.15.150:0/2250016434 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1a3c1619a0 0x7f1a3c0c94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.531+0000 7f1a43036640 1 --2- 172.21.15.150:0/2250016434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1a3c15de30 0x7f1a3c157a50 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.531+0000 7f1a43036640 1 --2- 172.21.15.150:0/2250016434 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1a2006bbd0 0x7f1a2006e090 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.531+0000 7f1a43036640 1 --2- 172.21.15.150:0/2250016434 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a3c0cf120 0x7f1a3c1497c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.531+0000 7f1a43036640 1 -- 172.21.15.150:0/2250016434 >> 172.21.15.150:0/2250016434 conn(0x7f1a3c0730b0 msgr2=0x7f1a3c0b8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:56.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.531+0000 7f1a43036640 1 -- 172.21.15.150:0/2250016434 shutdown_connections 2024-03-17T23:49:56.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.531+0000 7f1a43036640 1 -- 172.21.15.150:0/2250016434 wait complete. 2024-03-17T23:49:56.543 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2024-03-17T23:49:55.633153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239540,"kb_used_data":6724,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735180,"statfs":{"total":773094113280,"available":772848824320,"internally_reserved":0,"allocated":6885376,"data_stored":3776681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14933,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-69,"num_write_kb":-38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001170"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":242,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-17T23:49:51.485656+0000","last_change":"2024-03-17T23:46:44.768668+0000","last_active":"2024-03-17T23:49:51.485656+0000","last_peered":"2024-03-17T23:49:51.485656+0000","last_clean":"2024-03-17T23:49:51.485656+0000","last_became_active":"2024-03-17T23:46:44.767826+0000","last_became_peered":"2024-03-17T23:46:44.767826+0000","last_unstale":"2024-03-17T23:49:51.485656+0000","last_undegraded":"2024-03-17T23:49:51.485656+0000","last_fullsized":"2024-03-17T23:49:51.485656+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_clean_scrub_stamp":"2024-03-17T23:46:43.744878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:54:22.286155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":14,"seq":60129542186,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27716,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344124,"statfs":{"total":96636764160,"available":96608382976,"internally_reserved":0,"allocated":712704,"data_stored":322682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542186,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":720896,"data_stored":329021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542186,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28308,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343532,"statfs":{"total":96636764160,"available":96607776768,"internally_reserved":0,"allocated":1314816,"data_stored":919389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32456,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1306624,"data_stored":913154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574889,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":704512,"data_stored":322626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574889,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":700416,"data_stored":316805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574889,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31892,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339948,"statfs":{"total":96636764160,"available":96604106752,"internally_reserved":0,"allocated":729088,"data_stored":336761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":696320,"data_stored":316243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-17T23:49:56.543 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-17T23:49:56.543 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-17T23:49:56.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.644+0000 7ff706916640 1 Processor -- start 2024-03-17T23:49:56.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.644+0000 7ff706916640 1 -- start start 2024-03-17T23:49:56.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.645+0000 7ff706916640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff7001514f0 0x7ff70015bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:56.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.645+0000 7ff706916640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7000ce790 con 0x7ff70015ab90 2024-03-17T23:49:56.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.645+0000 7ff706916640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff70015b050 con 0x7ff70006ecd0 2024-03-17T23:49:56.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.645+0000 7ff706916640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7ff70015b330 con 0x7ff7001514f0 2024-03-17T23:49:56.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.645+0000 7ff6fffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff7001514f0 0x7ff70015bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:56.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.645+0000 7ff6ff7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7ff70015ab90 0x7ff70006e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:58658/0 (socket says 172.21.15.150:58658) 2024-03-17T23:49:56.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.645+0000 7ff6ff7fe640 1 -- 172.21.15.150:0/1714762554 learned_addr learned my addr 172.21.15.150:0/1714762554 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:56.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.645+0000 7ff6fffff640 1 -- 172.21.15.150:0/1714762554 >> v1:172.21.15.157:6789/0 conn(0x7ff70006ecd0 legacy=0x7ff7000cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:56.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.645+0000 7ff6fffff640 1 -- 172.21.15.150:0/1714762554 >> v1:172.21.15.150:6789/0 conn(0x7ff70015ab90 legacy=0x7ff70006e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:56.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.645+0000 7ff6fffff640 1 -- 172.21.15.150:0/1714762554 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff70015b610 con 0x7ff7001514f0 2024-03-17T23:49:56.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.645+0000 7ff6feffd640 1 -- 172.21.15.150:0/1714762554 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1429454793 0 0) 0x7ff6f4001110 con 0x7ff70015ab90 2024-03-17T23:49:56.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.646+0000 7ff6fffff640 1 --2- 172.21.15.150:0/1714762554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7001514f0 0x7ff70015bd10 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7ff6f00097a0 tx=0x7ff6f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df269ae1c26de0b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:56.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.646+0000 7ff6feffd640 1 -- 172.21.15.150:0/1714762554 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f0035020 con 0x7ff7001514f0 2024-03-17T23:49:56.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.646+0000 7ff6feffd640 1 -- 172.21.15.150:0/1714762554 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6f0031830 con 0x7ff7001514f0 2024-03-17T23:49:56.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.646+0000 7ff6feffd640 1 -- 172.21.15.150:0/1714762554 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f0031c00 con 0x7ff7001514f0 2024-03-17T23:49:56.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.647+0000 7ff706916640 1 -- 172.21.15.150:0/1714762554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7001514f0 msgr2=0x7ff70015bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:56.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.647+0000 7ff706916640 1 --2- 172.21.15.150:0/1714762554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7001514f0 0x7ff70015bd10 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7ff6f00097a0 tx=0x7ff6f002b8d0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.647+0000 7ff706916640 1 -- 172.21.15.150:0/1714762554 shutdown_connections 2024-03-17T23:49:56.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.647+0000 7ff706916640 1 --2- 172.21.15.150:0/1714762554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff7001514f0 0x7ff70015bd10 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.647+0000 7ff706916640 1 -- 172.21.15.150:0/1714762554 >> 172.21.15.150:0/1714762554 conn(0x7ff70006b190 msgr2=0x7ff70006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:56.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.647+0000 7ff706916640 1 -- 172.21.15.150:0/1714762554 shutdown_connections 2024-03-17T23:49:56.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.647+0000 7ff706916640 1 -- 172.21.15.150:0/1714762554 wait complete. 2024-03-17T23:49:56.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.648+0000 7ff706916640 1 Processor -- start 2024-03-17T23:49:56.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.648+0000 7ff706916640 1 -- start start 2024-03-17T23:49:56.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.649+0000 7ff706916640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff70006ecd0 0x7ff700152640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:56.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.649+0000 7ff706916640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7001514f0 0x7ff70013f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:56.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.649+0000 7ff706916640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff70015ab90 0x7ff7000b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:56.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.649+0000 7ff706916640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff7001518d0 con 0x7ff70015ab90 2024-03-17T23:49:56.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.649+0000 7ff706916640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff7000af510 con 0x7ff70006ecd0 2024-03-17T23:49:56.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.649+0000 7ff706916640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff700151a50 con 0x7ff7001514f0 2024-03-17T23:49:56.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.649+0000 7ff704eb3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff70015ab90 0x7ff7000b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:56.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.649+0000 7ff704eb3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff70015ab90 0x7ff7000b3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44756/0 (socket says 172.21.15.150:44756) 2024-03-17T23:49:56.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.649+0000 7ff704eb3640 1 -- 172.21.15.150:0/1796009509 learned_addr learned my addr 172.21.15.150:0/1796009509 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:56.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.649+0000 7ff6fffff640 1 --2- 172.21.15.150:0/1796009509 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff70006ecd0 0x7ff700152640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:56.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.649+0000 7ff6ff7fe640 1 --2- 172.21.15.150:0/1796009509 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7001514f0 0x7ff70013f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:56.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.649+0000 7ff704eb3640 1 -- 172.21.15.150:0/1796009509 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7001514f0 msgr2=0x7ff70013f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:56.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.649+0000 7ff704eb3640 1 --2- 172.21.15.150:0/1796009509 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7001514f0 0x7ff70013f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.649+0000 7ff704eb3640 1 -- 172.21.15.150:0/1796009509 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff70006ecd0 msgr2=0x7ff700152640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:56.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.649+0000 7ff704eb3640 1 --2- 172.21.15.150:0/1796009509 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff70006ecd0 0x7ff700152640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.649+0000 7ff704eb3640 1 -- 172.21.15.150:0/1796009509 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7001534f0 con 0x7ff70015ab90 2024-03-17T23:49:56.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.649+0000 7ff6fffff640 1 --2- 172.21.15.150:0/1796009509 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff70006ecd0 0x7ff700152640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:49:56.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.650+0000 7ff704eb3640 1 --2- 172.21.15.150:0/1796009509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff70015ab90 0x7ff7000b3aa0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7ff6f800b9a0 tx=0x7ff6f800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:56.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.650+0000 7ff6fcff9640 1 -- 172.21.15.150:0/1796009509 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f8019070 con 0x7ff70015ab90 2024-03-17T23:49:56.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.650+0000 7ff6fcff9640 1 -- 172.21.15.150:0/1796009509 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6f800cba0 con 0x7ff70015ab90 2024-03-17T23:49:56.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.650+0000 7ff706916640 1 -- 172.21.15.150:0/1796009509 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7001537d0 con 0x7ff70015ab90 2024-03-17T23:49:56.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.650+0000 7ff6fcff9640 1 -- 172.21.15.150:0/1796009509 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f8014440 con 0x7ff70015ab90 2024-03-17T23:49:56.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.650+0000 7ff706916640 1 -- 172.21.15.150:0/1796009509 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff700153d90 con 0x7ff70015ab90 2024-03-17T23:49:56.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.651+0000 7ff706916640 1 -- 172.21.15.150:0/1796009509 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7000681a0 con 0x7ff70015ab90 2024-03-17T23:49:56.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.651+0000 7ff6fcff9640 1 -- 172.21.15.150:0/1796009509 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff6f801c070 con 0x7ff70015ab90 2024-03-17T23:49:56.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.652+0000 7ff6fcff9640 1 --2- 172.21.15.150:0/1796009509 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff6ec06bbd0 0x7ff6ec06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:56.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.652+0000 7ff6fffff640 1 --2- 172.21.15.150:0/1796009509 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff6ec06bbd0 0x7ff6ec06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:56.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.652+0000 7ff6fcff9640 1 -- 172.21.15.150:0/1796009509 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5335+0+0 (secure 0 0 0) 0x7ff6f808a570 con 0x7ff70015ab90 2024-03-17T23:49:56.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.652+0000 7ff6fffff640 1 --2- 172.21.15.150:0/1796009509 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff6ec06bbd0 0x7ff6ec06e090 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0036000 tx=0x7ff6f003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:56.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.656+0000 7ff6fcff9640 1 -- 172.21.15.150:0/1796009509 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7ff6f80591d0 con 0x7ff70015ab90 2024-03-17T23:49:56.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.826+0000 7ff706916640 1 -- 172.21.15.150:0/1796009509 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff7000b0e90 con 0x7ff6ec06bbd0 2024-03-17T23:49:56.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.827+0000 7ff6fcff9640 1 -- 172.21.15.150:0/1796009509 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13415 (secure 0 0 0) 0x7ff7000b0e90 con 0x7ff6ec06bbd0 2024-03-17T23:49:56.828 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:56.829 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-03-17T23:49:56.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.829+0000 7ff706916640 1 -- 172.21.15.150:0/1796009509 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff6ec06bbd0 msgr2=0x7ff6ec06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:56.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.829+0000 7ff706916640 1 --2- 172.21.15.150:0/1796009509 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff6ec06bbd0 0x7ff6ec06e090 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0036000 tx=0x7ff6f003c040 comp rx=0 tx=0).stop 2024-03-17T23:49:56.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.829+0000 7ff706916640 1 -- 172.21.15.150:0/1796009509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff70015ab90 msgr2=0x7ff7000b3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:56.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.829+0000 7ff706916640 1 --2- 172.21.15.150:0/1796009509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff70015ab90 0x7ff7000b3aa0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7ff6f800b9a0 tx=0x7ff6f800be60 comp rx=0 tx=0).stop 2024-03-17T23:49:56.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.829+0000 7ff706916640 1 -- 172.21.15.150:0/1796009509 shutdown_connections 2024-03-17T23:49:56.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.829+0000 7ff706916640 1 --2- 172.21.15.150:0/1796009509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff70015ab90 0x7ff7000b3aa0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.829+0000 7ff706916640 1 --2- 172.21.15.150:0/1796009509 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff7001514f0 0x7ff70013f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.829+0000 7ff706916640 1 --2- 172.21.15.150:0/1796009509 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff6ec06bbd0 0x7ff6ec06e090 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.829+0000 7ff706916640 1 --2- 172.21.15.150:0/1796009509 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff70006ecd0 0x7ff700152640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.829+0000 7ff706916640 1 -- 172.21.15.150:0/1796009509 >> 172.21.15.150:0/1796009509 conn(0x7ff70006b190 msgr2=0x7ff70015cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:56.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.829+0000 7ff706916640 1 -- 172.21.15.150:0/1796009509 shutdown_connections 2024-03-17T23:49:56.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.829+0000 7ff706916640 1 -- 172.21.15.150:0/1796009509 wait complete. 2024-03-17T23:49:56.840 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2024-03-17T23:49:55.633153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239540,"kb_used_data":6724,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735180,"statfs":{"total":773094113280,"available":772848824320,"internally_reserved":0,"allocated":6885376,"data_stored":3776681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14933,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-69,"num_write_kb":-38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001170"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":242,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-17T23:49:51.485656+0000","last_change":"2024-03-17T23:46:44.768668+0000","last_active":"2024-03-17T23:49:51.485656+0000","last_peered":"2024-03-17T23:49:51.485656+0000","last_clean":"2024-03-17T23:49:51.485656+0000","last_became_active":"2024-03-17T23:46:44.767826+0000","last_became_peered":"2024-03-17T23:46:44.767826+0000","last_unstale":"2024-03-17T23:49:51.485656+0000","last_undegraded":"2024-03-17T23:49:51.485656+0000","last_fullsized":"2024-03-17T23:49:51.485656+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_clean_scrub_stamp":"2024-03-17T23:46:43.744878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:54:22.286155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":14,"seq":60129542186,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27716,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344124,"statfs":{"total":96636764160,"available":96608382976,"internally_reserved":0,"allocated":712704,"data_stored":322682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542186,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":720896,"data_stored":329021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542186,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28308,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343532,"statfs":{"total":96636764160,"available":96607776768,"internally_reserved":0,"allocated":1314816,"data_stored":919389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32456,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1306624,"data_stored":913154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574889,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":704512,"data_stored":322626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574889,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":700416,"data_stored":316805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574889,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31892,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339948,"statfs":{"total":96636764160,"available":96604106752,"internally_reserved":0,"allocated":729088,"data_stored":336761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":696320,"data_stored":316243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-17T23:49:56.841 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-17T23:49:56.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.942+0000 7fb45e6e8640 1 Processor -- start 2024-03-17T23:49:56.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.942+0000 7fb45e6e8640 1 -- start start 2024-03-17T23:49:56.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.942+0000 7fb45e6e8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb45815ab90 0x7fb45815bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:56.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.942+0000 7fb45e6e8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb45806ecd0 0x7fb45806f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:56.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.942+0000 7fb45e6e8640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4580ce790 con 0x7fb4581514f0 2024-03-17T23:49:56.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.942+0000 7fb45e6e8640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fb45806fba0 con 0x7fb45806ecd0 2024-03-17T23:49:56.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.942+0000 7fb45e6e8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb45815b050 con 0x7fb45815ab90 2024-03-17T23:49:56.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.943+0000 7fb45cc85640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb45806ecd0 0x7fb45806f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:56.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.943+0000 7fb457fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb4581514f0 0x7fb45806e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:58672/0 (socket says 172.21.15.150:58672) 2024-03-17T23:49:56.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.943+0000 7fb457fff640 1 -- 172.21.15.150:0/2428057053 learned_addr learned my addr 172.21.15.150:0/2428057053 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:56.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.943+0000 7fb4577fe640 1 --2- 172.21.15.150:0/2428057053 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb45815ab90 0x7fb45815bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:56.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.943+0000 7fb45cc85640 1 -- 172.21.15.150:0/2428057053 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb45815ab90 msgr2=0x7fb45815bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:56.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.943+0000 7fb45cc85640 1 --2- 172.21.15.150:0/2428057053 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb45815ab90 0x7fb45815bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.943+0000 7fb45cc85640 1 -- 172.21.15.150:0/2428057053 >> v1:172.21.15.150:6789/0 conn(0x7fb4581514f0 legacy=0x7fb45806e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:56.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.943+0000 7fb45cc85640 1 -- 172.21.15.150:0/2428057053 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb45815b330 con 0x7fb45806ecd0 2024-03-17T23:49:56.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.944+0000 7fb45cc85640 1 --2- 172.21.15.150:0/2428057053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb45806ecd0 0x7fb45806f660 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fb450009d90 tx=0x7fb45002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7b78e87e901848a server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:56.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.944+0000 7fb456ffd640 1 -- 172.21.15.150:0/2428057053 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb450039070 con 0x7fb45806ecd0 2024-03-17T23:49:56.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.944+0000 7fb456ffd640 1 -- 172.21.15.150:0/2428057053 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb45002cc50 con 0x7fb45806ecd0 2024-03-17T23:49:56.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.944+0000 7fb456ffd640 1 -- 172.21.15.150:0/2428057053 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb450034440 con 0x7fb45806ecd0 2024-03-17T23:49:56.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.944+0000 7fb45e6e8640 1 -- 172.21.15.150:0/2428057053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb45806ecd0 msgr2=0x7fb45806f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:56.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.944+0000 7fb45e6e8640 1 --2- 172.21.15.150:0/2428057053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb45806ecd0 0x7fb45806f660 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fb450009d90 tx=0x7fb45002bea0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.944+0000 7fb45e6e8640 1 -- 172.21.15.150:0/2428057053 shutdown_connections 2024-03-17T23:49:56.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.944+0000 7fb45e6e8640 1 --2- 172.21.15.150:0/2428057053 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb45815ab90 0x7fb45815bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.944+0000 7fb45e6e8640 1 --2- 172.21.15.150:0/2428057053 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb45806ecd0 0x7fb45806f660 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.944+0000 7fb45e6e8640 1 -- 172.21.15.150:0/2428057053 >> 172.21.15.150:0/2428057053 conn(0x7fb45806b190 msgr2=0x7fb45806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:56.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.944+0000 7fb45e6e8640 1 -- 172.21.15.150:0/2428057053 shutdown_connections 2024-03-17T23:49:56.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.944+0000 7fb45e6e8640 1 -- 172.21.15.150:0/2428057053 wait complete. 2024-03-17T23:49:56.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.945+0000 7fb45e6e8640 1 Processor -- start 2024-03-17T23:49:56.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.946+0000 7fb45e6e8640 1 -- start start 2024-03-17T23:49:56.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.946+0000 7fb45e6e8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb45806ecd0 0x7fb458143ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:56.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.946+0000 7fb45e6e8640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb4581514f0 0x7fb458144d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:56.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.946+0000 7fb45e6e8640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb45815ab90 0x7fb458145280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:56.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.946+0000 7fb45e6e8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb4580af510 con 0x7fb45806ecd0 2024-03-17T23:49:56.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.946+0000 7fb45e6e8640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fb45815b8f0 con 0x7fb4581514f0 2024-03-17T23:49:56.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.946+0000 7fb45e6e8640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fb4581518d0 con 0x7fb45815ab90 2024-03-17T23:49:56.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.946+0000 7fb457fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb45806ecd0 0x7fb458143ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:56.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.946+0000 7fb457fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb45806ecd0 0x7fb458143ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44782/0 (socket says 172.21.15.150:44782) 2024-03-17T23:49:56.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.947+0000 7fb457fff640 1 -- 172.21.15.150:0/449553206 learned_addr learned my addr 172.21.15.150:0/449553206 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:56.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.947+0000 7fb4577fe640 1 --2- 172.21.15.150:0/449553206 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb4581514f0 0x7fb458144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:56.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.947+0000 7fb45cc85640 1 --2- 172.21.15.150:0/449553206 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb45815ab90 0x7fb458145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:56.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.947+0000 7fb457fff640 1 -- 172.21.15.150:0/449553206 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb45815ab90 msgr2=0x7fb458145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:56.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.947+0000 7fb457fff640 1 --2- 172.21.15.150:0/449553206 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb45815ab90 0x7fb458145280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.947+0000 7fb457fff640 1 -- 172.21.15.150:0/449553206 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb4581514f0 msgr2=0x7fb458144d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:56.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.947+0000 7fb457fff640 1 --2- 172.21.15.150:0/449553206 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb4581514f0 0x7fb458144d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:56.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.947+0000 7fb457fff640 1 -- 172.21.15.150:0/449553206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb458149710 con 0x7fb45806ecd0 2024-03-17T23:49:56.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.947+0000 7fb4577fe640 1 --2- 172.21.15.150:0/449553206 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb4581514f0 0x7fb458144d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:49:56.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.947+0000 7fb457fff640 1 --2- 172.21.15.150:0/449553206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb45806ecd0 0x7fb458143ba0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fb44800c400 tx=0x7fb44800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:56.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.947+0000 7fb454ff9640 1 -- 172.21.15.150:0/449553206 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb448017070 con 0x7fb45806ecd0 2024-03-17T23:49:56.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.947+0000 7fb454ff9640 1 -- 172.21.15.150:0/449553206 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb448005ce0 con 0x7fb45806ecd0 2024-03-17T23:49:56.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.947+0000 7fb45e6e8640 1 -- 172.21.15.150:0/449553206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4580ad2f0 con 0x7fb45806ecd0 2024-03-17T23:49:56.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.947+0000 7fb454ff9640 1 -- 172.21.15.150:0/449553206 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4480072f0 con 0x7fb45806ecd0 2024-03-17T23:49:56.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.947+0000 7fb45e6e8640 1 -- 172.21.15.150:0/449553206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4580ad8b0 con 0x7fb45806ecd0 2024-03-17T23:49:56.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.948+0000 7fb45e6e8640 1 -- 172.21.15.150:0/449553206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4580681a0 con 0x7fb45806ecd0 2024-03-17T23:49:56.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.948+0000 7fb454ff9640 1 -- 172.21.15.150:0/449553206 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fb44801a070 con 0x7fb45806ecd0 2024-03-17T23:49:56.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.949+0000 7fb454ff9640 1 --2- 172.21.15.150:0/449553206 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb44406bbd0 0x7fb44406e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:56.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.949+0000 7fb454ff9640 1 -- 172.21.15.150:0/449553206 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5335+0+0 (secure 0 0 0) 0x7fb448089ce0 con 0x7fb45806ecd0 2024-03-17T23:49:56.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.949+0000 7fb4577fe640 1 --2- 172.21.15.150:0/449553206 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb44406bbd0 0x7fb44406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:56.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.949+0000 7fb4577fe640 1 --2- 172.21.15.150:0/449553206 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb44406bbd0 0x7fb44406e090 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb44c006fd0 tx=0x7fb44c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:56.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:56.953+0000 7fb454ff9640 1 -- 172.21.15.150:0/449553206 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fb458149710 con 0x7fb45806ecd0 2024-03-17T23:49:57.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:57.126+0000 7fb45e6e8640 1 -- 172.21.15.150:0/449553206 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb45806bf90 con 0x7fb45806ecd0 2024-03-17T23:49:57.127 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:57.126+0000 7fb454ff9640 1 -- 172.21.15.150:0/449553206 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) ==== 74+0+13329 (secure 0 0 0) 0x7fb4580ce790 con 0x7fb45806ecd0 2024-03-17T23:49:57.127 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:49:57.127 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":61,"fsid":"a1c5acd3-e97f-470b-8226-335736a7795b","created":"2024-03-17T23:46:31.404460+0000","modified":"2024-03-17T23:49:53.915981+0000","last_up_change":"2024-03-17T23:46:42.740908+0000","last_in_change":"2024-03-17T23:46:33.643210+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-17T23:46:43.636400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3b0ae141-f0f1-4834-9621-bf11818f4be4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6801","nonce":1962922469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6803","nonce":1962922469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6807","nonce":1962922469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":1962922469},{"type":"v1","addr":"172.21.15.150:6805","nonce":1962922469}]},"public_addr":"172.21.15.150:6801/1962922469","cluster_addr":"172.21.15.150:6803/1962922469","heartbeat_back_addr":"172.21.15.150:6807/1962922469","heartbeat_front_addr":"172.21.15.150:6805/1962922469","state":["exists","up"]},{"osd":1,"uuid":"51e9a837-3530-419b-b5af-a30a5882ae2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6809","nonce":1014841375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6811","nonce":1014841375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6815","nonce":1014841375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":1014841375},{"type":"v1","addr":"172.21.15.150:6813","nonce":1014841375}]},"public_addr":"172.21.15.150:6809/1014841375","cluster_addr":"172.21.15.150:6811/1014841375","heartbeat_back_addr":"172.21.15.150:6815/1014841375","heartbeat_front_addr":"172.21.15.150:6813/1014841375","state":["exists","up"]},{"osd":2,"uuid":"4fc66c29-64c3-4d0f-97ec-d9a4f554eb3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6819","nonce":347494201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6821","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6823","nonce":347494201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6829","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6831","nonce":347494201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6825","nonce":347494201},{"type":"v1","addr":"172.21.15.150:6827","nonce":347494201}]},"public_addr":"172.21.15.150:6819/347494201","cluster_addr":"172.21.15.150:6823/347494201","heartbeat_back_addr":"172.21.15.150:6831/347494201","heartbeat_front_addr":"172.21.15.150:6827/347494201","state":["exists","up"]},{"osd":3,"uuid":"d28afbd0-215b-419c-9071-40c83096e9bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6818","nonce":3072406303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6820","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6822","nonce":3072406303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6828","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6830","nonce":3072406303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6824","nonce":3072406303},{"type":"v1","addr":"172.21.15.150:6826","nonce":3072406303}]},"public_addr":"172.21.15.150:6818/3072406303","cluster_addr":"172.21.15.150:6822/3072406303","heartbeat_back_addr":"172.21.15.150:6830/3072406303","heartbeat_front_addr":"172.21.15.150:6826/3072406303","state":["exists","up"]},{"osd":4,"uuid":"7e973c64-ea03-4b69-ad16-6f5e6ccfb389","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6810","nonce":4022520191}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6814","nonce":4022520191}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6822","nonce":4022520191}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":4022520191},{"type":"v1","addr":"172.21.15.157:6818","nonce":4022520191}]},"public_addr":"172.21.15.157:6810/4022520191","cluster_addr":"172.21.15.157:6814/4022520191","heartbeat_back_addr":"172.21.15.157:6822/4022520191","heartbeat_front_addr":"172.21.15.157:6818/4022520191","state":["exists","up"]},{"osd":5,"uuid":"ea4552bf-1e67-4e73-87b6-ea8f00131017","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6825","nonce":1224877787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6827","nonce":1224877787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6831","nonce":1224877787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":1224877787},{"type":"v1","addr":"172.21.15.157:6829","nonce":1224877787}]},"public_addr":"172.21.15.157:6825/1224877787","cluster_addr":"172.21.15.157:6827/1224877787","heartbeat_back_addr":"172.21.15.157:6831/1224877787","heartbeat_front_addr":"172.21.15.157:6829/1224877787","state":["exists","up"]},{"osd":6,"uuid":"c7411d26-c78e-4ccb-be71-c13a7a91235e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6809","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6811","nonce":2764449896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6813","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6815","nonce":2764449896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6821","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6823","nonce":2764449896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6817","nonce":2764449896},{"type":"v1","addr":"172.21.15.157:6819","nonce":2764449896}]},"public_addr":"172.21.15.157:6811/2764449896","cluster_addr":"172.21.15.157:6815/2764449896","heartbeat_back_addr":"172.21.15.157:6823/2764449896","heartbeat_front_addr":"172.21.15.157:6819/2764449896","state":["exists","up"]},{"osd":7,"uuid":"bb6837d5-de8b-4ded-a8eb-c17b930002ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6801","nonce":624642970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6803","nonce":624642970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6807","nonce":624642970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":624642970},{"type":"v1","addr":"172.21.15.157:6805","nonce":624642970}]},"public_addr":"172.21.15.157:6801/624642970","cluster_addr":"172.21.15.157:6803/624642970","heartbeat_back_addr":"172.21.15.157:6807/624642970","heartbeat_front_addr":"172.21.15.157:6805/624642970","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.725733+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.766488+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.849393+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.841455+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.805140+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.796560+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.777294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-17T23:46:38.734944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.157:6835/479821580":"2024-03-18T23:49:50.903831+0000","172.21.15.157:6833/479821580":"2024-03-18T23:49:50.903831+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-17T23:49:57.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:57.128+0000 7fb45e6e8640 1 -- 172.21.15.150:0/449553206 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb44406bbd0 msgr2=0x7fb44406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:57.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:57.128+0000 7fb45e6e8640 1 --2- 172.21.15.150:0/449553206 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb44406bbd0 0x7fb44406e090 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb44c006fd0 tx=0x7fb44c008040 comp rx=0 tx=0).stop 2024-03-17T23:49:57.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:57.128+0000 7fb45e6e8640 1 -- 172.21.15.150:0/449553206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb45806ecd0 msgr2=0x7fb458143ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:57.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:57.128+0000 7fb45e6e8640 1 --2- 172.21.15.150:0/449553206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb45806ecd0 0x7fb458143ba0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fb44800c400 tx=0x7fb44800c8c0 comp rx=0 tx=0).stop 2024-03-17T23:49:57.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:57.128+0000 7fb45e6e8640 1 -- 172.21.15.150:0/449553206 shutdown_connections 2024-03-17T23:49:57.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:57.128+0000 7fb45e6e8640 1 --2- 172.21.15.150:0/449553206 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fb45815ab90 0x7fb458145280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:57.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:57.128+0000 7fb45e6e8640 1 --2- 172.21.15.150:0/449553206 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fb44406bbd0 0x7fb44406e090 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:57.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:57.128+0000 7fb45e6e8640 1 --2- 172.21.15.150:0/449553206 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb4581514f0 0x7fb458144d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:57.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:57.128+0000 7fb45e6e8640 1 --2- 172.21.15.150:0/449553206 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb45806ecd0 0x7fb458143ba0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:57.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:57.128+0000 7fb45e6e8640 1 -- 172.21.15.150:0/449553206 >> 172.21.15.150:0/449553206 conn(0x7fb45806b190 msgr2=0x7fb4580ae760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:57.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:57.129+0000 7fb45e6e8640 1 -- 172.21.15.150:0/449553206 shutdown_connections 2024-03-17T23:49:57.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:57.129+0000 7fb45e6e8640 1 -- 172.21.15.150:0/449553206 wait complete. 2024-03-17T23:49:57.939 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-17T23:49:57.942 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-17T23:49:58.141 INFO:tasks.ceph:Scrubbing osd.0 2024-03-17T23:49:58.141 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-17T23:49:58.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.246+0000 7f01ed834640 1 Processor -- start 2024-03-17T23:49:58.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.246+0000 7f01ed834640 1 -- start start 2024-03-17T23:49:58.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.247+0000 7f01ed834640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f01e80d1e30 0x7f01e80b57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:58.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.247+0000 7f01ed834640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01e80b6dc0 con 0x7f01e815de20 2024-03-17T23:49:58.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.247+0000 7f01ed834640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01e80d23a0 con 0x7f01e80b5cf0 2024-03-17T23:49:58.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.247+0000 7f01ed834640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f01e80b6150 con 0x7f01e80d1e30 2024-03-17T23:49:58.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.247+0000 7f01dffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f01e80d1e30 0x7f01e80b57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:58.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.247+0000 7f01e6ffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f01e815de20 0x7f01e80d34e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:58674/0 (socket says 172.21.15.150:58674) 2024-03-17T23:49:58.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.247+0000 7f01e6ffd640 1 -- 172.21.15.150:0/1587116274 learned_addr learned my addr 172.21.15.150:0/1587116274 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:58.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.247+0000 7f01dffff640 1 -- 172.21.15.150:0/1587116274 >> v1:172.21.15.157:6789/0 conn(0x7f01e80b5cf0 legacy=0x7f01e80b27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:58.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.247+0000 7f01dffff640 1 -- 172.21.15.150:0/1587116274 >> v1:172.21.15.150:6789/0 conn(0x7f01e815de20 legacy=0x7f01e80d34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:58.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.247+0000 7f01dffff640 1 -- 172.21.15.150:0/1587116274 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e80d2680 con 0x7f01e80d1e30 2024-03-17T23:49:58.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.247+0000 7f01e67fc640 1 -- 172.21.15.150:0/1587116274 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2258379226 0 0) 0x7f01d8001110 con 0x7f01e815de20 2024-03-17T23:49:58.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.248+0000 7f01dffff640 1 --2- 172.21.15.150:0/1587116274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01e80d1e30 0x7f01e80b57b0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f01d00097a0 tx=0x7f01d002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4022c84a568d0bc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:58.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.248+0000 7f01e67fc640 1 -- 172.21.15.150:0/1587116274 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d0034070 con 0x7f01e80d1e30 2024-03-17T23:49:58.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.249+0000 7f01e67fc640 1 -- 172.21.15.150:0/1587116274 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01d0030610 con 0x7f01e80d1e30 2024-03-17T23:49:58.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.249+0000 7f01e67fc640 1 -- 172.21.15.150:0/1587116274 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d00309e0 con 0x7f01e80d1e30 2024-03-17T23:49:58.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.249+0000 7f01ed834640 1 -- 172.21.15.150:0/1587116274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01e80d1e30 msgr2=0x7f01e80b57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:58.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.249+0000 7f01ed834640 1 --2- 172.21.15.150:0/1587116274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01e80d1e30 0x7f01e80b57b0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f01d00097a0 tx=0x7f01d002b960 comp rx=0 tx=0).stop 2024-03-17T23:49:58.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.249+0000 7f01ed834640 1 -- 172.21.15.150:0/1587116274 shutdown_connections 2024-03-17T23:49:58.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.249+0000 7f01ed834640 1 --2- 172.21.15.150:0/1587116274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01e80d1e30 0x7f01e80b57b0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.249+0000 7f01ed834640 1 -- 172.21.15.150:0/1587116274 >> 172.21.15.150:0/1587116274 conn(0x7f01e80d0990 msgr2=0x7f01e80d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:58.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.249+0000 7f01ed834640 1 -- 172.21.15.150:0/1587116274 shutdown_connections 2024-03-17T23:49:58.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.249+0000 7f01ed834640 1 -- 172.21.15.150:0/1587116274 wait complete. 2024-03-17T23:49:58.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.250+0000 7f01ed834640 1 Processor -- start 2024-03-17T23:49:58.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.250+0000 7f01ed834640 1 -- start start 2024-03-17T23:49:58.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.251+0000 7f01ed834640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01e80b5cf0 0x7f01e80c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:58.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.251+0000 7f01ed834640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f01e80d1e30 0x7f01e81f3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:58.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.251+0000 7f01ed834640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01e815de20 0x7f01e80cbe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:58.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.251+0000 7f01ed834640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f01e815e200 con 0x7f01e80b5cf0 2024-03-17T23:49:58.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.251+0000 7f01ed834640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f01e80d2c40 con 0x7f01e815de20 2024-03-17T23:49:58.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.251+0000 7f01ed834640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f01e815e380 con 0x7f01e80d1e30 2024-03-17T23:49:58.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.251+0000 7f01e6ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01e80b5cf0 0x7f01e80c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:58.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.251+0000 7f01e6ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01e80b5cf0 0x7f01e80c9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44806/0 (socket says 172.21.15.150:44806) 2024-03-17T23:49:58.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.251+0000 7f01e6ffd640 1 -- 172.21.15.150:0/2764913947 learned_addr learned my addr 172.21.15.150:0/2764913947 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:58.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.251+0000 7f01dffff640 1 --2- 172.21.15.150:0/2764913947 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f01e80d1e30 0x7f01e81f3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:58.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.251+0000 7f01e77fe640 1 --2- 172.21.15.150:0/2764913947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01e815de20 0x7f01e80cbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:58.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.251+0000 7f01e6ffd640 1 -- 172.21.15.150:0/2764913947 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f01e80d1e30 msgr2=0x7f01e81f3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:58.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.251+0000 7f01e6ffd640 1 --2- 172.21.15.150:0/2764913947 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f01e80d1e30 0x7f01e81f3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.251+0000 7f01e6ffd640 1 -- 172.21.15.150:0/2764913947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01e815de20 msgr2=0x7f01e80cbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:58.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.251+0000 7f01e6ffd640 1 --2- 172.21.15.150:0/2764913947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f01e815de20 0x7f01e80cbe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.252+0000 7f01e6ffd640 1 -- 172.21.15.150:0/2764913947 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e81f4530 con 0x7f01e80b5cf0 2024-03-17T23:49:58.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.252+0000 7f01e6ffd640 1 --2- 172.21.15.150:0/2764913947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01e80b5cf0 0x7f01e80c9810 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f01d800c3f0 tx=0x7f01d800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:58.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.252+0000 7f01e4ff9640 1 -- 172.21.15.150:0/2764913947 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d80074e0 con 0x7f01e80b5cf0 2024-03-17T23:49:58.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.252+0000 7f01e4ff9640 1 -- 172.21.15.150:0/2764913947 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01d80047e0 con 0x7f01e80b5cf0 2024-03-17T23:49:58.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.252+0000 7f01ed834640 1 -- 172.21.15.150:0/2764913947 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01e81f4810 con 0x7f01e80b5cf0 2024-03-17T23:49:58.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.252+0000 7f01e4ff9640 1 -- 172.21.15.150:0/2764913947 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d8004c10 con 0x7f01e80b5cf0 2024-03-17T23:49:58.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.252+0000 7f01ed834640 1 -- 172.21.15.150:0/2764913947 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01e81f4dd0 con 0x7f01e80b5cf0 2024-03-17T23:49:58.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.253+0000 7f01ed834640 1 -- 172.21.15.150:0/2764913947 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f01e80c7ec0 con 0x7f01e80b5cf0 2024-03-17T23:49:58.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.253+0000 7f01e4ff9640 1 -- 172.21.15.150:0/2764913947 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f01d8011070 con 0x7f01e80b5cf0 2024-03-17T23:49:58.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.254+0000 7f01e4ff9640 1 --2- 172.21.15.150:0/2764913947 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f01c806bbd0 0x7f01c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:58.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.254+0000 7f01e4ff9640 1 -- 172.21.15.150:0/2764913947 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5335+0+0 (secure 0 0 0) 0x7f01d808a5b0 con 0x7f01e80b5cf0 2024-03-17T23:49:58.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.254+0000 7f01dffff640 1 --2- 172.21.15.150:0/2764913947 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f01c806bbd0 0x7f01c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:58.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.254+0000 7f01e4ff9640 1 --2- 172.21.15.150:0/2764913947 >> [v2:172.21.15.150:6800/1962922469,v1:172.21.15.150:6801/1962922469] conn(0x7f01c8072070 0x7f01c80744b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:58.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.254+0000 7f01e4ff9640 1 -- 172.21.15.150:0/2764913947 --> [v2:172.21.15.150:6800/1962922469,v1:172.21.15.150:6801/1962922469] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f01d8015e70 con 0x7f01c8072070 2024-03-17T23:49:58.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.254+0000 7f01e4ff9640 1 -- 172.21.15.150:0/2764913947 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7f01d8091050 con 0x7f01e80b5cf0 2024-03-17T23:49:58.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.254+0000 7f01e77fe640 1 --2- 172.21.15.150:0/2764913947 >> [v2:172.21.15.150:6800/1962922469,v1:172.21.15.150:6801/1962922469] conn(0x7f01c8072070 0x7f01c80744b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:58.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.254+0000 7f01dffff640 1 --2- 172.21.15.150:0/2764913947 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f01c806bbd0 0x7f01c806e090 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f01d0036000 tx=0x7f01d0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:58.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.254+0000 7f01e77fe640 1 --2- 172.21.15.150:0/2764913947 >> [v2:172.21.15.150:6800/1962922469,v1:172.21.15.150:6801/1962922469] conn(0x7f01c8072070 0x7f01c80744b0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:58.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.255+0000 7f01e4ff9640 1 -- 172.21.15.150:0/2764913947 <== osd.0 v2:172.21.15.150:6800/1962922469 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f01d8015e70 con 0x7f01c8072070 2024-03-17T23:49:58.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.280+0000 7f01ed834640 1 -- 172.21.15.150:0/2764913947 --> [v2:172.21.15.150:6800/1962922469,v1:172.21.15.150:6801/1962922469] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f01e80681a0 con 0x7f01c8072070 2024-03-17T23:49:58.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.280+0000 7f01e4ff9640 1 -- 172.21.15.150:0/2764913947 <== osd.0 v2:172.21.15.150:6800/1962922469 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f01e80681a0 con 0x7f01c8072070 2024-03-17T23:49:58.281 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-17T23:49:58.282 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-17T23:49:58.282 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-17T23:49:58.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.281+0000 7f01ed834640 1 -- 172.21.15.150:0/2764913947 >> [v2:172.21.15.150:6800/1962922469,v1:172.21.15.150:6801/1962922469] conn(0x7f01c8072070 msgr2=0x7f01c80744b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:58.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.281+0000 7f01ed834640 1 --2- 172.21.15.150:0/2764913947 >> [v2:172.21.15.150:6800/1962922469,v1:172.21.15.150:6801/1962922469] conn(0x7f01c8072070 0x7f01c80744b0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.281+0000 7f01ed834640 1 -- 172.21.15.150:0/2764913947 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f01c806bbd0 msgr2=0x7f01c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:58.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.281+0000 7f01ed834640 1 --2- 172.21.15.150:0/2764913947 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f01c806bbd0 0x7f01c806e090 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f01d0036000 tx=0x7f01d0008030 comp rx=0 tx=0).stop 2024-03-17T23:49:58.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.281+0000 7f01ed834640 1 -- 172.21.15.150:0/2764913947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01e80b5cf0 msgr2=0x7f01e80c9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:58.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.281+0000 7f01ed834640 1 --2- 172.21.15.150:0/2764913947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01e80b5cf0 0x7f01e80c9810 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f01d800c3f0 tx=0x7f01d800c8b0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.281+0000 7f01e77fe640 1 -- 172.21.15.150:0/2764913947 reap_dead start 2024-03-17T23:49:58.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.281+0000 7f01ed834640 1 -- 172.21.15.150:0/2764913947 shutdown_connections 2024-03-17T23:49:58.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.281+0000 7f01ed834640 1 -- 172.21.15.150:0/2764913947 >> 172.21.15.150:0/2764913947 conn(0x7f01e80d0990 msgr2=0x7f01e80d3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:58.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.281+0000 7f01ed834640 1 -- 172.21.15.150:0/2764913947 shutdown_connections 2024-03-17T23:49:58.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.281+0000 7f01ed834640 1 -- 172.21.15.150:0/2764913947 wait complete. 2024-03-17T23:49:58.292 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-17T23:49:58.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.395+0000 7f892d1b3640 1 Processor -- start 2024-03-17T23:49:58.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.395+0000 7f892d1b3640 1 -- start start 2024-03-17T23:49:58.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.396+0000 7f892d1b3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f892806f390 0x7f89280cdf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:58.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.396+0000 7f892d1b3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f89280ce440 0x7f8928165b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:58.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.396+0000 7f892d1b3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f892806bf60 con 0x7f8928142520 2024-03-17T23:49:58.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.396+0000 7f892d1b3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f8928142900 con 0x7f892806f390 2024-03-17T23:49:58.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.396+0000 7f892d1b3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f89280cd900 con 0x7f89280ce440 2024-03-17T23:49:58.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.396+0000 7f891ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f892806f390 0x7f89280cdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:58.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.396+0000 7f8926d9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f8928142520 0x7f89280cc510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:58676/0 (socket says 172.21.15.150:58676) 2024-03-17T23:49:58.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.396+0000 7f8926d9d640 1 -- 172.21.15.150:0/4194936414 learned_addr learned my addr 172.21.15.150:0/4194936414 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:58.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.396+0000 7f892759e640 1 --2- 172.21.15.150:0/4194936414 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f89280ce440 0x7f8928165b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:58.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.396+0000 7f891ffff640 1 -- 172.21.15.150:0/4194936414 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f89280ce440 msgr2=0x7f8928165b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:58.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.396+0000 7f891ffff640 1 --2- 172.21.15.150:0/4194936414 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f89280ce440 0x7f8928165b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.396+0000 7f891ffff640 1 -- 172.21.15.150:0/4194936414 >> v1:172.21.15.150:6789/0 conn(0x7f8928142520 legacy=0x7f89280cc510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:58.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.396+0000 7f891ffff640 1 -- 172.21.15.150:0/4194936414 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89280cd060 con 0x7f892806f390 2024-03-17T23:49:58.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.396+0000 7f892659c640 1 -- 172.21.15.150:0/4194936414 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 800950031 0 0) 0x7f892806bf60 con 0x7f8928142520 2024-03-17T23:49:58.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.396+0000 7f892759e640 1 --2- 172.21.15.150:0/4194936414 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f89280ce440 0x7f8928165b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:49:58.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.397+0000 7f891ffff640 1 --2- 172.21.15.150:0/4194936414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f892806f390 0x7f89280cdf00 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f89100097d0 tx=0x7f891002b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f6bc3ffba74107a server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:58.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.397+0000 7f892659c640 1 -- 172.21.15.150:0/4194936414 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8910031770 con 0x7f892806f390 2024-03-17T23:49:58.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.397+0000 7f892659c640 1 -- 172.21.15.150:0/4194936414 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8910037070 con 0x7f892806f390 2024-03-17T23:49:58.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.397+0000 7f892659c640 1 -- 172.21.15.150:0/4194936414 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8910031b30 con 0x7f892806f390 2024-03-17T23:49:58.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.398+0000 7f892d1b3640 1 -- 172.21.15.150:0/4194936414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f892806f390 msgr2=0x7f89280cdf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:58.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.398+0000 7f892d1b3640 1 --2- 172.21.15.150:0/4194936414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f892806f390 0x7f89280cdf00 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f89100097d0 tx=0x7f891002b810 comp rx=0 tx=0).stop 2024-03-17T23:49:58.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.398+0000 7f892d1b3640 1 -- 172.21.15.150:0/4194936414 shutdown_connections 2024-03-17T23:49:58.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.398+0000 7f892d1b3640 1 --2- 172.21.15.150:0/4194936414 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f89280ce440 0x7f8928165b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.398+0000 7f892d1b3640 1 --2- 172.21.15.150:0/4194936414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f892806f390 0x7f89280cdf00 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.398+0000 7f892d1b3640 1 -- 172.21.15.150:0/4194936414 >> 172.21.15.150:0/4194936414 conn(0x7f892806b190 msgr2=0x7f892806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:58.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.398+0000 7f892d1b3640 1 -- 172.21.15.150:0/4194936414 shutdown_connections 2024-03-17T23:49:58.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.398+0000 7f892d1b3640 1 -- 172.21.15.150:0/4194936414 wait complete. 2024-03-17T23:49:58.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.399+0000 7f892d1b3640 1 Processor -- start 2024-03-17T23:49:58.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.399+0000 7f892d1b3640 1 -- start start 2024-03-17T23:49:58.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.400+0000 7f892d1b3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f892806f390 0x7f892815b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:58.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.400+0000 7f892d1b3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89280ce440 0x7f89281e3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:58.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.400+0000 7f892d1b3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8928142520 0x7f89281e42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:58.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.400+0000 7f892d1b3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f892806de60 con 0x7f89280ce440 2024-03-17T23:49:58.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.400+0000 7f892d1b3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f89280cd620 con 0x7f8928142520 2024-03-17T23:49:58.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.400+0000 7f892d1b3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f89280ccc20 con 0x7f892806f390 2024-03-17T23:49:58.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.400+0000 7f891ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89280ce440 0x7f89281e3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:58.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.400+0000 7f891ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89280ce440 0x7f89281e3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44834/0 (socket says 172.21.15.150:44834) 2024-03-17T23:49:58.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.400+0000 7f891ffff640 1 -- 172.21.15.150:0/1333337015 learned_addr learned my addr 172.21.15.150:0/1333337015 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:58.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.400+0000 7f892759e640 1 --2- 172.21.15.150:0/1333337015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8928142520 0x7f89281e42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:58.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.400+0000 7f8926d9d640 1 --2- 172.21.15.150:0/1333337015 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f892806f390 0x7f892815b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:58.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.400+0000 7f891ffff640 1 -- 172.21.15.150:0/1333337015 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f892806f390 msgr2=0x7f892815b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:58.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.400+0000 7f891ffff640 1 --2- 172.21.15.150:0/1333337015 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f892806f390 0x7f892815b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.400+0000 7f891ffff640 1 -- 172.21.15.150:0/1333337015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8928142520 msgr2=0x7f89281e42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:58.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.400+0000 7f891ffff640 1 --2- 172.21.15.150:0/1333337015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8928142520 0x7f89281e42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.400+0000 7f891ffff640 1 -- 172.21.15.150:0/1333337015 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89281e1510 con 0x7f89280ce440 2024-03-17T23:49:58.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.400+0000 7f892759e640 1 --2- 172.21.15.150:0/1333337015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8928142520 0x7f89281e42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:49:58.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.400+0000 7f8926d9d640 1 --2- 172.21.15.150:0/1333337015 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f892806f390 0x7f892815b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:49:58.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.400+0000 7f891ffff640 1 --2- 172.21.15.150:0/1333337015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89280ce440 0x7f89281e3d70 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f89100098f0 tx=0x7f89100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:58.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.400+0000 7f891f7fe640 1 -- 172.21.15.150:0/1333337015 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8910032600 con 0x7f89280ce440 2024-03-17T23:49:58.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.401+0000 7f891f7fe640 1 -- 172.21.15.150:0/1333337015 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89100327a0 con 0x7f89280ce440 2024-03-17T23:49:58.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.401+0000 7f892d1b3640 1 -- 172.21.15.150:0/1333337015 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89281e17f0 con 0x7f89280ce440 2024-03-17T23:49:58.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.401+0000 7f891f7fe640 1 -- 172.21.15.150:0/1333337015 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8910032b90 con 0x7f89280ce440 2024-03-17T23:49:58.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.401+0000 7f892d1b3640 1 -- 172.21.15.150:0/1333337015 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89281e1db0 con 0x7f89280ce440 2024-03-17T23:49:58.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.402+0000 7f892d1b3640 1 -- 172.21.15.150:0/1333337015 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89280681a0 con 0x7f89280ce440 2024-03-17T23:49:58.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.402+0000 7f891f7fe640 1 -- 172.21.15.150:0/1333337015 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f891004c060 con 0x7f89280ce440 2024-03-17T23:49:58.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.402+0000 7f891f7fe640 1 --2- 172.21.15.150:0/1333337015 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f890806bc00 0x7f890806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:58.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.402+0000 7f8926d9d640 1 --2- 172.21.15.150:0/1333337015 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f890806bc00 0x7f890806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:58.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.402+0000 7f891f7fe640 1 -- 172.21.15.150:0/1333337015 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5335+0+0 (secure 0 0 0) 0x7f89100abdf0 con 0x7f89280ce440 2024-03-17T23:49:58.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.403+0000 7f8926d9d640 1 --2- 172.21.15.150:0/1333337015 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f890806bc00 0x7f890806e0c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f8918000f60 tx=0x7f8918009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:58.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.407+0000 7f891f7fe640 1 -- 172.21.15.150:0/1333337015 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f8910040070 con 0x7f89280ce440 2024-03-17T23:49:58.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.578+0000 7f892d1b3640 1 -- 172.21.15.150:0/1333337015 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f892806c1f0 con 0x7f890806bc00 2024-03-17T23:49:58.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.579+0000 7f891f7fe640 1 -- 172.21.15.150:0/1333337015 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f892806c1f0 con 0x7f890806bc00 2024-03-17T23:49:58.580 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 0 to deep-scrub 2024-03-17T23:49:58.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.581+0000 7f892d1b3640 1 -- 172.21.15.150:0/1333337015 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f890806bc00 msgr2=0x7f890806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:58.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.581+0000 7f892d1b3640 1 --2- 172.21.15.150:0/1333337015 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f890806bc00 0x7f890806e0c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f8918000f60 tx=0x7f8918009040 comp rx=0 tx=0).stop 2024-03-17T23:49:58.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.581+0000 7f892d1b3640 1 -- 172.21.15.150:0/1333337015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89280ce440 msgr2=0x7f89281e3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:58.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.581+0000 7f892d1b3640 1 --2- 172.21.15.150:0/1333337015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89280ce440 0x7f89281e3d70 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f89100098f0 tx=0x7f89100046c0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.581+0000 7f892d1b3640 1 -- 172.21.15.150:0/1333337015 shutdown_connections 2024-03-17T23:49:58.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.581+0000 7f892d1b3640 1 --2- 172.21.15.150:0/1333337015 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f890806bc00 0x7f890806e0c0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.581+0000 7f892d1b3640 1 --2- 172.21.15.150:0/1333337015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8928142520 0x7f89281e42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.581+0000 7f892d1b3640 1 --2- 172.21.15.150:0/1333337015 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89280ce440 0x7f89281e3d70 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.581+0000 7f892d1b3640 1 --2- 172.21.15.150:0/1333337015 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f892806f390 0x7f892815b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.581+0000 7f892d1b3640 1 -- 172.21.15.150:0/1333337015 >> 172.21.15.150:0/1333337015 conn(0x7f892806b190 msgr2=0x7f89280af610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:58.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.581+0000 7f892d1b3640 1 -- 172.21.15.150:0/1333337015 shutdown_connections 2024-03-17T23:49:58.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.581+0000 7f892d1b3640 1 -- 172.21.15.150:0/1333337015 wait complete. 2024-03-17T23:49:58.592 INFO:tasks.ceph:Scrubbing osd.1 2024-03-17T23:49:58.592 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-17T23:49:58.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.698+0000 7fa24afd3640 1 Processor -- start 2024-03-17T23:49:58.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.698+0000 7fa24afd3640 1 -- start start 2024-03-17T23:49:58.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.698+0000 7fa24afd3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa2440cd4d0 0x7fa244072070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:58.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.699+0000 7fa24afd3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa2440cf590 0x7fa2440b3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:58.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.699+0000 7fa24afd3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa24415aff0 con 0x7fa2440725b0 2024-03-17T23:49:58.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.699+0000 7fa24afd3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fa24415b260 con 0x7fa2440cd4d0 2024-03-17T23:49:58.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.699+0000 7fa24afd3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa24415b540 con 0x7fa2440cf590 2024-03-17T23:49:58.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.699+0000 7fa248d6f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa2440cd4d0 0x7fa244072070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:58.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.699+0000 7fa243fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa2440725b0 0x7fa2440cee80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:58688/0 (socket says 172.21.15.150:58688) 2024-03-17T23:49:58.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.699+0000 7fa248d6f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa2440cd4d0 0x7fa244072070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44848/0 (socket says 172.21.15.150:44848) 2024-03-17T23:49:58.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.699+0000 7fa243fff640 1 -- 172.21.15.150:0/4006778980 learned_addr learned my addr 172.21.15.150:0/4006778980 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:58.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.699+0000 7fa248d6f640 1 -- 172.21.15.150:0/4006778980 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa2440cf590 msgr2=0x7fa2440b3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:58.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.699+0000 7fa248d6f640 1 --2- 172.21.15.150:0/4006778980 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa2440cf590 0x7fa2440b3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.699+0000 7fa248d6f640 1 -- 172.21.15.150:0/4006778980 >> v1:172.21.15.150:6789/0 conn(0x7fa2440725b0 legacy=0x7fa2440cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:58.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.699+0000 7fa248d6f640 1 -- 172.21.15.150:0/4006778980 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa24415b820 con 0x7fa2440cd4d0 2024-03-17T23:49:58.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.699+0000 7fa2437fe640 1 -- 172.21.15.150:0/4006778980 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2875431809 0 0) 0x7fa24415aff0 con 0x7fa2440725b0 2024-03-17T23:49:58.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.700+0000 7fa248d6f640 1 --2- 172.21.15.150:0/4006778980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2440cd4d0 0x7fa244072070 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fa2380097a0 tx=0x7fa23802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61a3f6b9a0a7ea85 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:58.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.700+0000 7fa2437fe640 1 -- 172.21.15.150:0/4006778980 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa238035020 con 0x7fa2440cd4d0 2024-03-17T23:49:58.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.700+0000 7fa2437fe640 1 -- 172.21.15.150:0/4006778980 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa238031830 con 0x7fa2440cd4d0 2024-03-17T23:49:58.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.700+0000 7fa2437fe640 1 -- 172.21.15.150:0/4006778980 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa238031c00 con 0x7fa2440cd4d0 2024-03-17T23:49:58.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.700+0000 7fa24afd3640 1 -- 172.21.15.150:0/4006778980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2440cd4d0 msgr2=0x7fa244072070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:58.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.700+0000 7fa24afd3640 1 --2- 172.21.15.150:0/4006778980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2440cd4d0 0x7fa244072070 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fa2380097a0 tx=0x7fa23802b8d0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.700+0000 7fa24afd3640 1 -- 172.21.15.150:0/4006778980 shutdown_connections 2024-03-17T23:49:58.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.700+0000 7fa24afd3640 1 --2- 172.21.15.150:0/4006778980 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa2440cf590 0x7fa2440b3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.700+0000 7fa24afd3640 1 --2- 172.21.15.150:0/4006778980 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2440cd4d0 0x7fa244072070 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.700+0000 7fa24afd3640 1 -- 172.21.15.150:0/4006778980 >> 172.21.15.150:0/4006778980 conn(0x7fa2440b1be0 msgr2=0x7fa2440b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:58.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.700+0000 7fa24afd3640 1 -- 172.21.15.150:0/4006778980 shutdown_connections 2024-03-17T23:49:58.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.700+0000 7fa24afd3640 1 -- 172.21.15.150:0/4006778980 wait complete. 2024-03-17T23:49:58.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.702+0000 7fa24afd3640 1 Processor -- start 2024-03-17T23:49:58.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.702+0000 7fa24afd3640 1 -- start start 2024-03-17T23:49:58.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.702+0000 7fa24afd3640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa2440725b0 0x7fa24414c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:58.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.702+0000 7fa24afd3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2440cd4d0 0x7fa2440b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:58.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.702+0000 7fa24afd3640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa2440cf590 0x7fa2440b0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:58.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.702+0000 7fa24afd3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa2440cd8b0 con 0x7fa2440cd4d0 2024-03-17T23:49:58.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.702+0000 7fa24afd3640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fa24415bde0 con 0x7fa2440cf590 2024-03-17T23:49:58.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.702+0000 7fa24afd3640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fa2440b3e40 con 0x7fa2440725b0 2024-03-17T23:49:58.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.703+0000 7fa243fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2440cd4d0 0x7fa2440b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:58.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.703+0000 7fa243fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2440cd4d0 0x7fa2440b07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44850/0 (socket says 172.21.15.150:44850) 2024-03-17T23:49:58.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.703+0000 7fa243fff640 1 -- 172.21.15.150:0/4038065185 learned_addr learned my addr 172.21.15.150:0/4038065185 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:58.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.703+0000 7fa243fff640 1 -- 172.21.15.150:0/4038065185 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa2440725b0 msgr2=0x7fa24414c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:58.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.703+0000 7fa243fff640 1 --2- 172.21.15.150:0/4038065185 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fa2440725b0 0x7fa24414c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.703+0000 7fa243fff640 1 -- 172.21.15.150:0/4038065185 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa2440cf590 msgr2=0x7fa2440b0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:58.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.703+0000 7fa243fff640 1 --2- 172.21.15.150:0/4038065185 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa2440cf590 0x7fa2440b0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.703+0000 7fa243fff640 1 -- 172.21.15.150:0/4038065185 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2440ad9b0 con 0x7fa2440cd4d0 2024-03-17T23:49:58.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.703+0000 7fa243fff640 1 --2- 172.21.15.150:0/4038065185 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2440cd4d0 0x7fa2440b07f0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fa23000c4d0 tx=0x7fa23000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:58.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.703+0000 7fa2417fa640 1 -- 172.21.15.150:0/4038065185 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa230018070 con 0x7fa2440cd4d0 2024-03-17T23:49:58.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.703+0000 7fa2417fa640 1 -- 172.21.15.150:0/4038065185 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa230007500 con 0x7fa2440cd4d0 2024-03-17T23:49:58.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.703+0000 7fa24afd3640 1 -- 172.21.15.150:0/4038065185 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2440aa880 con 0x7fa2440cd4d0 2024-03-17T23:49:58.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.703+0000 7fa2417fa640 1 -- 172.21.15.150:0/4038065185 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa230004a20 con 0x7fa2440cd4d0 2024-03-17T23:49:58.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.703+0000 7fa24afd3640 1 -- 172.21.15.150:0/4038065185 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2440aae40 con 0x7fa2440cd4d0 2024-03-17T23:49:58.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.704+0000 7fa24afd3640 1 -- 172.21.15.150:0/4038065185 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa2440681a0 con 0x7fa2440cd4d0 2024-03-17T23:49:58.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.704+0000 7fa2417fa640 1 -- 172.21.15.150:0/4038065185 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fa23001b070 con 0x7fa2440cd4d0 2024-03-17T23:49:58.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.705+0000 7fa2417fa640 1 --2- 172.21.15.150:0/4038065185 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa22806bbd0 0x7fa22806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:58.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.705+0000 7fa2417fa640 1 -- 172.21.15.150:0/4038065185 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5335+0+0 (secure 0 0 0) 0x7fa230089ba0 con 0x7fa2440cd4d0 2024-03-17T23:49:58.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.705+0000 7fa248d6f640 1 --2- 172.21.15.150:0/4038065185 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa22806bbd0 0x7fa22806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:58.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.705+0000 7fa2417fa640 1 --2- 172.21.15.150:0/4038065185 >> [v2:172.21.15.150:6808/1014841375,v1:172.21.15.150:6809/1014841375] conn(0x7fa228072070 0x7fa2280744b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:58.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.705+0000 7fa2417fa640 1 -- 172.21.15.150:0/4038065185 --> [v2:172.21.15.150:6808/1014841375,v1:172.21.15.150:6809/1014841375] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa230017ec0 con 0x7fa228072070 2024-03-17T23:49:58.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.705+0000 7fa2417fa640 1 -- 172.21.15.150:0/4038065185 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7fa23008e2c0 con 0x7fa2440cd4d0 2024-03-17T23:49:58.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.705+0000 7fa249570640 1 --2- 172.21.15.150:0/4038065185 >> [v2:172.21.15.150:6808/1014841375,v1:172.21.15.150:6809/1014841375] conn(0x7fa228072070 0x7fa2280744b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:58.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.705+0000 7fa248d6f640 1 --2- 172.21.15.150:0/4038065185 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa22806bbd0 0x7fa22806e090 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa238036000 tx=0x7fa238030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:58.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.706+0000 7fa249570640 1 --2- 172.21.15.150:0/4038065185 >> [v2:172.21.15.150:6808/1014841375,v1:172.21.15.150:6809/1014841375] conn(0x7fa228072070 0x7fa2280744b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:58.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.706+0000 7fa2417fa640 1 -- 172.21.15.150:0/4038065185 <== osd.1 v2:172.21.15.150:6808/1014841375 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa230017ec0 con 0x7fa228072070 2024-03-17T23:49:58.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.719+0000 7fa24afd3640 1 -- 172.21.15.150:0/4038065185 --> [v2:172.21.15.150:6808/1014841375,v1:172.21.15.150:6809/1014841375] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa24415a960 con 0x7fa228072070 2024-03-17T23:49:58.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.719+0000 7fa2417fa640 1 -- 172.21.15.150:0/4038065185 <== osd.1 v2:172.21.15.150:6808/1014841375 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fa24415a960 con 0x7fa228072070 2024-03-17T23:49:58.721 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-17T23:49:58.721 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-17T23:49:58.721 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-17T23:49:58.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.720+0000 7fa24afd3640 1 -- 172.21.15.150:0/4038065185 >> [v2:172.21.15.150:6808/1014841375,v1:172.21.15.150:6809/1014841375] conn(0x7fa228072070 msgr2=0x7fa2280744b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:58.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.720+0000 7fa24afd3640 1 --2- 172.21.15.150:0/4038065185 >> [v2:172.21.15.150:6808/1014841375,v1:172.21.15.150:6809/1014841375] conn(0x7fa228072070 0x7fa2280744b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.720+0000 7fa24afd3640 1 -- 172.21.15.150:0/4038065185 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa22806bbd0 msgr2=0x7fa22806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:58.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.720+0000 7fa24afd3640 1 --2- 172.21.15.150:0/4038065185 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fa22806bbd0 0x7fa22806e090 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa238036000 tx=0x7fa238030000 comp rx=0 tx=0).stop 2024-03-17T23:49:58.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.720+0000 7fa24afd3640 1 -- 172.21.15.150:0/4038065185 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2440cd4d0 msgr2=0x7fa2440b07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:58.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.720+0000 7fa24afd3640 1 --2- 172.21.15.150:0/4038065185 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2440cd4d0 0x7fa2440b07f0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fa23000c4d0 tx=0x7fa23000c990 comp rx=0 tx=0).stop 2024-03-17T23:49:58.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.720+0000 7fa249570640 1 -- 172.21.15.150:0/4038065185 reap_dead start 2024-03-17T23:49:58.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.720+0000 7fa24afd3640 1 -- 172.21.15.150:0/4038065185 shutdown_connections 2024-03-17T23:49:58.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.720+0000 7fa24afd3640 1 -- 172.21.15.150:0/4038065185 >> 172.21.15.150:0/4038065185 conn(0x7fa2440b1be0 msgr2=0x7fa2440b1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:58.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.720+0000 7fa24afd3640 1 -- 172.21.15.150:0/4038065185 shutdown_connections 2024-03-17T23:49:58.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.720+0000 7fa24afd3640 1 -- 172.21.15.150:0/4038065185 wait complete. 2024-03-17T23:49:58.730 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-17T23:49:58.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.832+0000 7fab530c6640 1 Processor -- start 2024-03-17T23:49:58.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.832+0000 7fab530c6640 1 -- start start 2024-03-17T23:49:58.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.833+0000 7fab530c6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fab4c15a4f0 0x7fab4c0cce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:58.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.833+0000 7fab530c6640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fab4c06f420 0x7fab4c06e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:58.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.833+0000 7fab530c6640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab4c0cebc0 con 0x7fab4c06eb00 2024-03-17T23:49:58.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.833+0000 7fab530c6640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fab4c0ceea0 con 0x7fab4c06f420 2024-03-17T23:49:58.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.833+0000 7fab530c6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fab4c0cf180 con 0x7fab4c15a4f0 2024-03-17T23:49:58.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.833+0000 7fab50e62640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fab4c15a4f0 0x7fab4c0cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:58.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.833+0000 7fab4bfff640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fab4c06f420 0x7fab4c06e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:58.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.833+0000 7fab51663640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7fab4c06eb00 0x7fab4c15dfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:45876/0 (socket says 172.21.15.150:45876) 2024-03-17T23:49:58.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.833+0000 7fab51663640 1 -- 172.21.15.150:0/285132420 learned_addr learned my addr 172.21.15.150:0/285132420 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:58.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.833+0000 7fab50e62640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fab4c15a4f0 0x7fab4c0cce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:57244/0 (socket says 172.21.15.150:57244) 2024-03-17T23:49:58.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.833+0000 7fab50e62640 1 -- 172.21.15.150:0/285132420 >> v2:172.21.15.157:3300/0 conn(0x7fab4c06f420 msgr2=0x7fab4c06e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:58.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.833+0000 7fab50e62640 1 --2- 172.21.15.150:0/285132420 >> v2:172.21.15.157:3300/0 conn(0x7fab4c06f420 0x7fab4c06e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.833+0000 7fab50e62640 1 -- 172.21.15.150:0/285132420 >> v1:172.21.15.157:6789/0 conn(0x7fab4c06eb00 legacy=0x7fab4c15dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:58.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.833+0000 7fab50e62640 1 -- 172.21.15.150:0/285132420 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab4c0cf460 con 0x7fab4c15a4f0 2024-03-17T23:49:58.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.833+0000 7fab4bfff640 1 --2- 172.21.15.150:0/285132420 >> v2:172.21.15.157:3300/0 conn(0x7fab4c06f420 0x7fab4c06e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:58.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.834+0000 7fab50e62640 1 --2- 172.21.15.150:0/285132420 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fab4c15a4f0 0x7fab4c0cce30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fab4000a940 tx=0x7fab4002cd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15db8d6d2ad6423a server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:58.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.834+0000 7fab4b7fe640 1 -- 172.21.15.150:0/285132420 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4003a070 con 0x7fab4c15a4f0 2024-03-17T23:49:58.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.834+0000 7fab4b7fe640 1 -- 172.21.15.150:0/285132420 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab4002db30 con 0x7fab4c15a4f0 2024-03-17T23:49:58.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.834+0000 7fab4b7fe640 1 -- 172.21.15.150:0/285132420 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4000b040 con 0x7fab4c15a4f0 2024-03-17T23:49:58.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.835+0000 7fab530c6640 1 -- 172.21.15.150:0/285132420 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fab4c15a4f0 msgr2=0x7fab4c0cce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:58.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.835+0000 7fab530c6640 1 --2- 172.21.15.150:0/285132420 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fab4c15a4f0 0x7fab4c0cce30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fab4000a940 tx=0x7fab4002cd80 comp rx=0 tx=0).stop 2024-03-17T23:49:58.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.835+0000 7fab530c6640 1 -- 172.21.15.150:0/285132420 shutdown_connections 2024-03-17T23:49:58.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.835+0000 7fab530c6640 1 --2- 172.21.15.150:0/285132420 >> v2:172.21.15.157:3300/0 conn(0x7fab4c06f420 0x7fab4c06e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.835+0000 7fab530c6640 1 --2- 172.21.15.150:0/285132420 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fab4c15a4f0 0x7fab4c0cce30 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.835+0000 7fab530c6640 1 -- 172.21.15.150:0/285132420 >> 172.21.15.150:0/285132420 conn(0x7fab4c06b190 msgr2=0x7fab4c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:58.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.835+0000 7fab530c6640 1 -- 172.21.15.150:0/285132420 shutdown_connections 2024-03-17T23:49:58.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.835+0000 7fab530c6640 1 -- 172.21.15.150:0/285132420 wait complete. 2024-03-17T23:49:58.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.836+0000 7fab530c6640 1 Processor -- start 2024-03-17T23:49:58.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.836+0000 7fab530c6640 1 -- start start 2024-03-17T23:49:58.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.837+0000 7fab530c6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab4c06eb00 0x7fab4c1477d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:58.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.837+0000 7fab530c6640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fab4c06f420 0x7fab4c145090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:58.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.837+0000 7fab530c6640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fab4c15a4f0 0x7fab4c150340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:58.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.837+0000 7fab530c6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fab4c0ce9a0 con 0x7fab4c06eb00 2024-03-17T23:49:58.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.837+0000 7fab530c6640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fab4c0ce820 con 0x7fab4c15a4f0 2024-03-17T23:49:58.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.837+0000 7fab530c6640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fab4c0b3b40 con 0x7fab4c06f420 2024-03-17T23:49:58.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.837+0000 7fab50e62640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab4c06eb00 0x7fab4c1477d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:58.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.837+0000 7fab50e62640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab4c06eb00 0x7fab4c1477d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44866/0 (socket says 172.21.15.150:44866) 2024-03-17T23:49:58.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.837+0000 7fab50e62640 1 -- 172.21.15.150:0/3229547036 learned_addr learned my addr 172.21.15.150:0/3229547036 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:58.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.837+0000 7fab51663640 1 --2- 172.21.15.150:0/3229547036 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fab4c15a4f0 0x7fab4c150340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:58.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.837+0000 7fab4bfff640 1 --2- 172.21.15.150:0/3229547036 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fab4c06f420 0x7fab4c145090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:58.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.837+0000 7fab50e62640 1 -- 172.21.15.150:0/3229547036 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fab4c06f420 msgr2=0x7fab4c145090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:58.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.837+0000 7fab50e62640 1 --2- 172.21.15.150:0/3229547036 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fab4c06f420 0x7fab4c145090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.837+0000 7fab50e62640 1 -- 172.21.15.150:0/3229547036 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fab4c15a4f0 msgr2=0x7fab4c150340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:58.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.837+0000 7fab50e62640 1 --2- 172.21.15.150:0/3229547036 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fab4c15a4f0 0x7fab4c150340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:58.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.838+0000 7fab50e62640 1 -- 172.21.15.150:0/3229547036 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab4c147d10 con 0x7fab4c06eb00 2024-03-17T23:49:58.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.838+0000 7fab50e62640 1 --2- 172.21.15.150:0/3229547036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab4c06eb00 0x7fab4c1477d0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fab40000c00 tx=0x7fab400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:58.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.838+0000 7fab497fa640 1 -- 172.21.15.150:0/3229547036 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4003a040 con 0x7fab4c06eb00 2024-03-17T23:49:58.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.838+0000 7fab497fa640 1 -- 172.21.15.150:0/3229547036 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab40043070 con 0x7fab4c06eb00 2024-03-17T23:49:58.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.838+0000 7fab530c6640 1 -- 172.21.15.150:0/3229547036 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab4c149540 con 0x7fab4c06eb00 2024-03-17T23:49:58.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.838+0000 7fab497fa640 1 -- 172.21.15.150:0/3229547036 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4003e650 con 0x7fab4c06eb00 2024-03-17T23:49:58.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.838+0000 7fab530c6640 1 -- 172.21.15.150:0/3229547036 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab4c149b00 con 0x7fab4c06eb00 2024-03-17T23:49:58.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.839+0000 7fab530c6640 1 -- 172.21.15.150:0/3229547036 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab4c0681a0 con 0x7fab4c06eb00 2024-03-17T23:49:58.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.839+0000 7fab497fa640 1 -- 172.21.15.150:0/3229547036 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fab4003e7f0 con 0x7fab4c06eb00 2024-03-17T23:49:58.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.840+0000 7fab497fa640 1 --2- 172.21.15.150:0/3229547036 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fab3006bbd0 0x7fab3006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:58.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.840+0000 7fab4bfff640 1 --2- 172.21.15.150:0/3229547036 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fab3006bbd0 0x7fab3006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:58.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.840+0000 7fab497fa640 1 -- 172.21.15.150:0/3229547036 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5335+0+0 (secure 0 0 0) 0x7fab400ac6e0 con 0x7fab4c06eb00 2024-03-17T23:49:58.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.840+0000 7fab4bfff640 1 --2- 172.21.15.150:0/3229547036 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fab3006bbd0 0x7fab3006e090 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fab38009560 tx=0x7fab38009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:58.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:58.844+0000 7fab497fa640 1 -- 172.21.15.150:0/3229547036 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fab4007ac70 con 0x7fab4c06eb00 2024-03-17T23:49:59.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.022+0000 7fab530c6640 1 -- 172.21.15.150:0/3229547036 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fab4c159940 con 0x7fab3006bbd0 2024-03-17T23:49:59.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.023+0000 7fab497fa640 1 -- 172.21.15.150:0/3229547036 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fab4c159940 con 0x7fab3006bbd0 2024-03-17T23:49:59.023 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 1 to deep-scrub 2024-03-17T23:49:59.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.025+0000 7fab530c6640 1 -- 172.21.15.150:0/3229547036 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fab3006bbd0 msgr2=0x7fab3006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.025+0000 7fab530c6640 1 --2- 172.21.15.150:0/3229547036 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fab3006bbd0 0x7fab3006e090 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fab38009560 tx=0x7fab38009190 comp rx=0 tx=0).stop 2024-03-17T23:49:59.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.025+0000 7fab530c6640 1 -- 172.21.15.150:0/3229547036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab4c06eb00 msgr2=0x7fab4c1477d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.025+0000 7fab530c6640 1 --2- 172.21.15.150:0/3229547036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab4c06eb00 0x7fab4c1477d0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fab40000c00 tx=0x7fab400046c0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.025+0000 7fab530c6640 1 -- 172.21.15.150:0/3229547036 shutdown_connections 2024-03-17T23:49:59.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.025+0000 7fab530c6640 1 --2- 172.21.15.150:0/3229547036 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fab3006bbd0 0x7fab3006e090 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.025+0000 7fab530c6640 1 --2- 172.21.15.150:0/3229547036 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fab4c15a4f0 0x7fab4c150340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.025+0000 7fab530c6640 1 --2- 172.21.15.150:0/3229547036 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fab4c06f420 0x7fab4c145090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.025+0000 7fab530c6640 1 --2- 172.21.15.150:0/3229547036 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fab4c06eb00 0x7fab4c1477d0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.025+0000 7fab530c6640 1 -- 172.21.15.150:0/3229547036 >> 172.21.15.150:0/3229547036 conn(0x7fab4c06b190 msgr2=0x7fab4c14e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:59.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.025+0000 7fab530c6640 1 -- 172.21.15.150:0/3229547036 shutdown_connections 2024-03-17T23:49:59.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.025+0000 7fab530c6640 1 -- 172.21.15.150:0/3229547036 wait complete. 2024-03-17T23:49:59.036 INFO:tasks.ceph:Scrubbing osd.2 2024-03-17T23:49:59.036 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-17T23:49:59.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.139+0000 7f1ff3d7e640 1 Processor -- start 2024-03-17T23:49:59.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.139+0000 7f1ff3d7e640 1 -- start start 2024-03-17T23:49:59.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.140+0000 7f1ff3d7e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1fec15de20 0x7f1fec0d36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:59.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.140+0000 7f1ff3d7e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1fec0b99c0 0x7f1fec0d1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:59.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.140+0000 7f1ff3d7e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fec0d2650 con 0x7f1fec0b57b0 2024-03-17T23:49:59.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.140+0000 7f1ff3d7e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f1fec0d2930 con 0x7f1fec15de20 2024-03-17T23:49:59.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.140+0000 7f1ff3d7e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1fec0d2c10 con 0x7f1fec0b99c0 2024-03-17T23:49:59.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.140+0000 7f1ff1b1a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1fec15de20 0x7f1fec0d36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:59.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.140+0000 7f1ff1b1a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1fec15de20 0x7f1fec0d36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44882/0 (socket says 172.21.15.150:44882) 2024-03-17T23:49:59.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.140+0000 7f1ff1b1a640 1 -- 172.21.15.150:0/3357698917 learned_addr learned my addr 172.21.15.150:0/3357698917 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:59.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.141+0000 7f1ff1b1a640 1 -- 172.21.15.150:0/3357698917 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1fec0b99c0 msgr2=0x7f1fec0d1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:59.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.141+0000 7f1ff1b1a640 1 --2- 172.21.15.150:0/3357698917 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1fec0b99c0 0x7f1fec0d1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.141+0000 7f1ff1b1a640 1 -- 172.21.15.150:0/3357698917 >> v1:172.21.15.157:6789/0 conn(0x7f1fec0b57b0 legacy=0x7f1fec06c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:59.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.141+0000 7f1ff1b1a640 1 -- 172.21.15.150:0/3357698917 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fec0d2ef0 con 0x7f1fec15de20 2024-03-17T23:49:59.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.141+0000 7f1ff1b1a640 1 --2- 172.21.15.150:0/3357698917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fec15de20 0x7f1fec0d36f0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8009e50 tx=0x7f1fe802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b154a47c92a60493 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:59.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.142+0000 7f1ff0b18640 1 -- 172.21.15.150:0/3357698917 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe802cb00 con 0x7f1fec15de20 2024-03-17T23:49:59.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.142+0000 7f1ff0b18640 1 -- 172.21.15.150:0/3357698917 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fe802cca0 con 0x7f1fec15de20 2024-03-17T23:49:59.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.142+0000 7f1ff0b18640 1 -- 172.21.15.150:0/3357698917 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe8034610 con 0x7f1fec15de20 2024-03-17T23:49:59.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.142+0000 7f1ff3d7e640 1 -- 172.21.15.150:0/3357698917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fec15de20 msgr2=0x7f1fec0d36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.142+0000 7f1ff3d7e640 1 --2- 172.21.15.150:0/3357698917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fec15de20 0x7f1fec0d36f0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8009e50 tx=0x7f1fe802bf60 comp rx=0 tx=0).stop 2024-03-17T23:49:59.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.142+0000 7f1ff3d7e640 1 -- 172.21.15.150:0/3357698917 shutdown_connections 2024-03-17T23:49:59.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.142+0000 7f1ff3d7e640 1 --2- 172.21.15.150:0/3357698917 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1fec0b99c0 0x7f1fec0d1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.142+0000 7f1ff3d7e640 1 --2- 172.21.15.150:0/3357698917 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fec15de20 0x7f1fec0d36f0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.142+0000 7f1ff3d7e640 1 -- 172.21.15.150:0/3357698917 >> 172.21.15.150:0/3357698917 conn(0x7f1fec0731d0 msgr2=0x7f1fec0d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:59.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.142+0000 7f1ff3d7e640 1 -- 172.21.15.150:0/3357698917 shutdown_connections 2024-03-17T23:49:59.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.142+0000 7f1ff3d7e640 1 -- 172.21.15.150:0/3357698917 wait complete. 2024-03-17T23:49:59.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.143+0000 7f1ff3d7e640 1 Processor -- start 2024-03-17T23:49:59.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.143+0000 7f1ff3d7e640 1 -- start start 2024-03-17T23:49:59.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.144+0000 7f1ff3d7e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1fec0b57b0 0x7f1fec1528a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:59.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.144+0000 7f1ff3d7e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fec0b99c0 0x7f1fec15e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:59.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.144+0000 7f1ff3d7e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1fec15de20 0x7f1fec15e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:59.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.144+0000 7f1ff3d7e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec0d2290 con 0x7f1fec0b99c0 2024-03-17T23:49:59.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.144+0000 7f1ff3d7e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f1fec15e200 con 0x7f1fec15de20 2024-03-17T23:49:59.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.144+0000 7f1ff3d7e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f1fec0d2410 con 0x7f1fec0b57b0 2024-03-17T23:49:59.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.144+0000 7f1ff1319640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fec0b99c0 0x7f1fec15e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:59.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.144+0000 7f1ff1319640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fec0b99c0 0x7f1fec15e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44898/0 (socket says 172.21.15.150:44898) 2024-03-17T23:49:59.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.144+0000 7f1ff1319640 1 -- 172.21.15.150:0/3886482783 learned_addr learned my addr 172.21.15.150:0/3886482783 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:59.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.145+0000 7f1ff1b1a640 1 --2- 172.21.15.150:0/3886482783 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1fec0b57b0 0x7f1fec1528a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:59.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.145+0000 7f1ff231b640 1 --2- 172.21.15.150:0/3886482783 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1fec15de20 0x7f1fec15e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:59.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.145+0000 7f1ff1319640 1 -- 172.21.15.150:0/3886482783 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1fec0b57b0 msgr2=0x7f1fec1528a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.145+0000 7f1ff1319640 1 --2- 172.21.15.150:0/3886482783 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f1fec0b57b0 0x7f1fec1528a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.145+0000 7f1ff1319640 1 -- 172.21.15.150:0/3886482783 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1fec15de20 msgr2=0x7f1fec15e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.145+0000 7f1ff1319640 1 --2- 172.21.15.150:0/3886482783 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1fec15de20 0x7f1fec15e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.145+0000 7f1ff1319640 1 -- 172.21.15.150:0/3886482783 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fec0cae00 con 0x7f1fec0b99c0 2024-03-17T23:49:59.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.145+0000 7f1ff1319640 1 --2- 172.21.15.150:0/3886482783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fec0b99c0 0x7f1fec15e410 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc00b4a0 tx=0x7f1fdc00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:59.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.145+0000 7f1fe27fc640 1 -- 172.21.15.150:0/3886482783 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc014070 con 0x7f1fec0b99c0 2024-03-17T23:49:59.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.145+0000 7f1fe27fc640 1 -- 172.21.15.150:0/3886482783 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fdc010610 con 0x7f1fec0b99c0 2024-03-17T23:49:59.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.145+0000 7f1ff3d7e640 1 -- 172.21.15.150:0/3886482783 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fec15f970 con 0x7f1fec0b99c0 2024-03-17T23:49:59.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.145+0000 7f1fe27fc640 1 -- 172.21.15.150:0/3886482783 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc010a00 con 0x7f1fec0b99c0 2024-03-17T23:49:59.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.145+0000 7f1ff3d7e640 1 -- 172.21.15.150:0/3886482783 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fec0cb0c0 con 0x7f1fec0b99c0 2024-03-17T23:49:59.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.146+0000 7f1ff3d7e640 1 -- 172.21.15.150:0/3886482783 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1fec0688b0 con 0x7f1fec0b99c0 2024-03-17T23:49:59.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.146+0000 7f1fe27fc640 1 -- 172.21.15.150:0/3886482783 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f1fdc017070 con 0x7f1fec0b99c0 2024-03-17T23:49:59.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.147+0000 7f1fe27fc640 1 --2- 172.21.15.150:0/3886482783 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1fd006bbd0 0x7f1fd006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:59.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.147+0000 7f1ff1b1a640 1 --2- 172.21.15.150:0/3886482783 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1fd006bbd0 0x7f1fd006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:59.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.147+0000 7f1fe27fc640 1 -- 172.21.15.150:0/3886482783 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5335+0+0 (secure 0 0 0) 0x7f1fdc08a260 con 0x7f1fec0b99c0 2024-03-17T23:49:59.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.147+0000 7f1fe27fc640 1 --2- 172.21.15.150:0/3886482783 >> [v2:172.21.15.150:6817/347494201,v1:172.21.15.150:6819/347494201] conn(0x7f1fd0072070 0x7f1fd00744b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:59.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.147+0000 7f1fe27fc640 1 -- 172.21.15.150:0/3886482783 --> [v2:172.21.15.150:6817/347494201,v1:172.21.15.150:6819/347494201] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1fdc00ee50 con 0x7f1fd0072070 2024-03-17T23:49:59.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.147+0000 7f1fe27fc640 1 -- 172.21.15.150:0/3886482783 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7f1fdc091050 con 0x7f1fec0b99c0 2024-03-17T23:49:59.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.147+0000 7f1ff231b640 1 --2- 172.21.15.150:0/3886482783 >> [v2:172.21.15.150:6817/347494201,v1:172.21.15.150:6819/347494201] conn(0x7f1fd0072070 0x7f1fd00744b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:59.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.147+0000 7f1ff1b1a640 1 --2- 172.21.15.150:0/3886482783 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1fd006bbd0 0x7f1fd006e090 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8035000 tx=0x7f1fe8030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:59.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.147+0000 7f1ff231b640 1 --2- 172.21.15.150:0/3886482783 >> [v2:172.21.15.150:6817/347494201,v1:172.21.15.150:6819/347494201] conn(0x7f1fd0072070 0x7f1fd00744b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:59.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.148+0000 7f1fe27fc640 1 -- 172.21.15.150:0/3886482783 <== osd.2 v2:172.21.15.150:6817/347494201 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1fdc00ee50 con 0x7f1fd0072070 2024-03-17T23:49:59.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.161+0000 7f1ff3d7e640 1 -- 172.21.15.150:0/3886482783 --> [v2:172.21.15.150:6817/347494201,v1:172.21.15.150:6819/347494201] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f1fec0cb290 con 0x7f1fd0072070 2024-03-17T23:49:59.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.161+0000 7f1fe27fc640 1 -- 172.21.15.150:0/3886482783 <== osd.2 v2:172.21.15.150:6817/347494201 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f1fec0cb290 con 0x7f1fd0072070 2024-03-17T23:49:59.162 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-17T23:49:59.162 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-17T23:49:59.162 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-17T23:49:59.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.161+0000 7f1ff3d7e640 1 -- 172.21.15.150:0/3886482783 >> [v2:172.21.15.150:6817/347494201,v1:172.21.15.150:6819/347494201] conn(0x7f1fd0072070 msgr2=0x7f1fd00744b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.161+0000 7f1ff3d7e640 1 --2- 172.21.15.150:0/3886482783 >> [v2:172.21.15.150:6817/347494201,v1:172.21.15.150:6819/347494201] conn(0x7f1fd0072070 0x7f1fd00744b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.162+0000 7f1ff3d7e640 1 -- 172.21.15.150:0/3886482783 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1fd006bbd0 msgr2=0x7f1fd006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.162+0000 7f1ff3d7e640 1 --2- 172.21.15.150:0/3886482783 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f1fd006bbd0 0x7f1fd006e090 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8035000 tx=0x7f1fe8030000 comp rx=0 tx=0).stop 2024-03-17T23:49:59.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.162+0000 7f1ff3d7e640 1 -- 172.21.15.150:0/3886482783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fec0b99c0 msgr2=0x7f1fec15e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.162+0000 7f1ff3d7e640 1 --2- 172.21.15.150:0/3886482783 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fec0b99c0 0x7f1fec15e410 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc00b4a0 tx=0x7f1fdc00b960 comp rx=0 tx=0).stop 2024-03-17T23:49:59.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.162+0000 7f1ff231b640 1 -- 172.21.15.150:0/3886482783 reap_dead start 2024-03-17T23:49:59.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.162+0000 7f1ff3d7e640 1 -- 172.21.15.150:0/3886482783 shutdown_connections 2024-03-17T23:49:59.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.162+0000 7f1ff3d7e640 1 -- 172.21.15.150:0/3886482783 >> 172.21.15.150:0/3886482783 conn(0x7f1fec0731d0 msgr2=0x7f1fec0b73d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:59.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.162+0000 7f1ff3d7e640 1 -- 172.21.15.150:0/3886482783 shutdown_connections 2024-03-17T23:49:59.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.162+0000 7f1ff3d7e640 1 -- 172.21.15.150:0/3886482783 wait complete. 2024-03-17T23:49:59.172 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-17T23:49:59.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.274+0000 7f2c5b9d4640 1 Processor -- start 2024-03-17T23:49:59.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.274+0000 7f2c5b9d4640 1 -- start start 2024-03-17T23:49:59.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.275+0000 7f2c5b9d4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2c540ccf90 0x7f2c540cf100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:59.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.275+0000 7f2c5b9d4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c5415bce0 0x7f2c540cd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:59.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.275+0000 7f2c5b9d4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c540cd8b0 con 0x7f2c5415e830 2024-03-17T23:49:59.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.275+0000 7f2c5b9d4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f2c5406e420 con 0x7f2c540ccf90 2024-03-17T23:49:59.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.275+0000 7f2c5b9d4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2c540cf8a0 con 0x7f2c5415bce0 2024-03-17T23:49:59.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.275+0000 7f2c59770640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2c540ccf90 0x7f2c540cf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:59.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.275+0000 7f2c59f71640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f2c5415e830 0x7f2c5415ecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:58702/0 (socket says 172.21.15.150:58702) 2024-03-17T23:49:59.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.275+0000 7f2c59f71640 1 -- 172.21.15.150:0/3726151539 learned_addr learned my addr 172.21.15.150:0/3726151539 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:59.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.275+0000 7f2c59770640 1 -- 172.21.15.150:0/3726151539 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c5415bce0 msgr2=0x7f2c540cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:59.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.275+0000 7f2c59770640 1 --2- 172.21.15.150:0/3726151539 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c5415bce0 0x7f2c540cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.275+0000 7f2c59770640 1 -- 172.21.15.150:0/3726151539 >> v1:172.21.15.150:6789/0 conn(0x7f2c5415e830 legacy=0x7f2c5415ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.275+0000 7f2c4bfff640 1 -- 172.21.15.150:0/3726151539 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1644923120 0 0) 0x7f2c540cd8b0 con 0x7f2c5415e830 2024-03-17T23:49:59.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.275+0000 7f2c59770640 1 -- 172.21.15.150:0/3726151539 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c5406de60 con 0x7f2c540ccf90 2024-03-17T23:49:59.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.276+0000 7f2c59770640 1 --2- 172.21.15.150:0/3726151539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c540ccf90 0x7f2c540cf100 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f2c500097a0 tx=0x7f2c5002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a179836adb622f1b server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:59.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.276+0000 7f2c4bfff640 1 -- 172.21.15.150:0/3726151539 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c500315a0 con 0x7f2c540ccf90 2024-03-17T23:49:59.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.276+0000 7f2c4bfff640 1 -- 172.21.15.150:0/3726151539 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c50037070 con 0x7f2c540ccf90 2024-03-17T23:49:59.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.276+0000 7f2c4bfff640 1 -- 172.21.15.150:0/3726151539 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c50031960 con 0x7f2c540ccf90 2024-03-17T23:49:59.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.276+0000 7f2c5b9d4640 1 -- 172.21.15.150:0/3726151539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c540ccf90 msgr2=0x7f2c540cf100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.276+0000 7f2c5b9d4640 1 --2- 172.21.15.150:0/3726151539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c540ccf90 0x7f2c540cf100 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f2c500097a0 tx=0x7f2c5002b840 comp rx=0 tx=0).stop 2024-03-17T23:49:59.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.277+0000 7f2c5b9d4640 1 -- 172.21.15.150:0/3726151539 shutdown_connections 2024-03-17T23:49:59.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.277+0000 7f2c5b9d4640 1 --2- 172.21.15.150:0/3726151539 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c5415bce0 0x7f2c540cd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.277+0000 7f2c5b9d4640 1 --2- 172.21.15.150:0/3726151539 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c540ccf90 0x7f2c540cf100 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.277+0000 7f2c5b9d4640 1 -- 172.21.15.150:0/3726151539 >> 172.21.15.150:0/3726151539 conn(0x7f2c5406b190 msgr2=0x7f2c5406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:59.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.277+0000 7f2c5b9d4640 1 -- 172.21.15.150:0/3726151539 shutdown_connections 2024-03-17T23:49:59.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.277+0000 7f2c5b9d4640 1 -- 172.21.15.150:0/3726151539 wait complete. 2024-03-17T23:49:59.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.278+0000 7f2c5b9d4640 1 Processor -- start 2024-03-17T23:49:59.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.278+0000 7f2c5b9d4640 1 -- start start 2024-03-17T23:49:59.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.278+0000 7f2c5b9d4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c540ccf90 0x7f2c54142a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:59.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.278+0000 7f2c5b9d4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c5415bce0 0x7f2c540b3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:59.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.278+0000 7f2c5b9d4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c5415e830 0x7f2c540b2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:59.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.278+0000 7f2c5b9d4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2c5415c5a0 con 0x7f2c540ccf90 2024-03-17T23:49:59.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.278+0000 7f2c5b9d4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f2c540cf640 con 0x7f2c5415bce0 2024-03-17T23:49:59.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.278+0000 7f2c5b9d4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f2c5415c160 con 0x7f2c5415e830 2024-03-17T23:49:59.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.279+0000 7f2c59770640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c540ccf90 0x7f2c54142a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:59.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.279+0000 7f2c59770640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c540ccf90 0x7f2c54142a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44922/0 (socket says 172.21.15.150:44922) 2024-03-17T23:49:59.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.279+0000 7f2c59770640 1 -- 172.21.15.150:0/1895717087 learned_addr learned my addr 172.21.15.150:0/1895717087 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:59.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.279+0000 7f2c59770640 1 -- 172.21.15.150:0/1895717087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c5415e830 msgr2=0x7f2c540b2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.279+0000 7f2c59770640 1 --2- 172.21.15.150:0/1895717087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c5415e830 0x7f2c540b2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.279+0000 7f2c59770640 1 -- 172.21.15.150:0/1895717087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c5415bce0 msgr2=0x7f2c540b3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.279+0000 7f2c59770640 1 --2- 172.21.15.150:0/1895717087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c5415bce0 0x7f2c540b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.279+0000 7f2c59770640 1 -- 172.21.15.150:0/1895717087 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c540b4060 con 0x7f2c540ccf90 2024-03-17T23:49:59.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.279+0000 7f2c59770640 1 --2- 172.21.15.150:0/1895717087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c540ccf90 0x7f2c54142a90 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f2c500098c0 tx=0x7f2c500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:59.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.279+0000 7f2c49ffb640 1 -- 172.21.15.150:0/1895717087 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c50032840 con 0x7f2c540ccf90 2024-03-17T23:49:59.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.279+0000 7f2c49ffb640 1 -- 172.21.15.150:0/1895717087 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c500329e0 con 0x7f2c540ccf90 2024-03-17T23:49:59.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.279+0000 7f2c49ffb640 1 -- 172.21.15.150:0/1895717087 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c50032dd0 con 0x7f2c540ccf90 2024-03-17T23:49:59.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.279+0000 7f2c5b9d4640 1 -- 172.21.15.150:0/1895717087 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c541670c0 con 0x7f2c540ccf90 2024-03-17T23:49:59.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.279+0000 7f2c5b9d4640 1 -- 172.21.15.150:0/1895717087 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c54167680 con 0x7f2c540ccf90 2024-03-17T23:49:59.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.280+0000 7f2c5b9d4640 1 -- 172.21.15.150:0/1895717087 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c540681a0 con 0x7f2c540ccf90 2024-03-17T23:49:59.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.281+0000 7f2c49ffb640 1 -- 172.21.15.150:0/1895717087 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f2c5004d3f0 con 0x7f2c540ccf90 2024-03-17T23:49:59.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.281+0000 7f2c49ffb640 1 --2- 172.21.15.150:0/1895717087 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2c3806bbd0 0x7f2c3806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:59.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.281+0000 7f2c49ffb640 1 -- 172.21.15.150:0/1895717087 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5335+0+0 (secure 0 0 0) 0x7f2c5002f030 con 0x7f2c540ccf90 2024-03-17T23:49:59.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.281+0000 7f2c58f6f640 1 --2- 172.21.15.150:0/1895717087 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2c3806bbd0 0x7f2c3806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:59.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.281+0000 7f2c58f6f640 1 --2- 172.21.15.150:0/1895717087 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2c3806bbd0 0x7f2c3806e090 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2c541f75f0 tx=0x7f2c44006c90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:59.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.285+0000 7f2c49ffb640 1 -- 172.21.15.150:0/1895717087 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f2c50040070 con 0x7f2c540ccf90 2024-03-17T23:49:59.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.457+0000 7f2c5b9d4640 1 -- 172.21.15.150:0/1895717087 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f2c5406fff0 con 0x7f2c3806bbd0 2024-03-17T23:49:59.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.457+0000 7f2c49ffb640 1 -- 172.21.15.150:0/1895717087 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f2c5406fff0 con 0x7f2c3806bbd0 2024-03-17T23:49:59.458 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 2 to deep-scrub 2024-03-17T23:49:59.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.459+0000 7f2c5b9d4640 1 -- 172.21.15.150:0/1895717087 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2c3806bbd0 msgr2=0x7f2c3806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.459+0000 7f2c5b9d4640 1 --2- 172.21.15.150:0/1895717087 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2c3806bbd0 0x7f2c3806e090 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2c541f75f0 tx=0x7f2c44006c90 comp rx=0 tx=0).stop 2024-03-17T23:49:59.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.459+0000 7f2c5b9d4640 1 -- 172.21.15.150:0/1895717087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c540ccf90 msgr2=0x7f2c54142a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.459+0000 7f2c5b9d4640 1 --2- 172.21.15.150:0/1895717087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c540ccf90 0x7f2c54142a90 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f2c500098c0 tx=0x7f2c500046c0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.459+0000 7f2c5b9d4640 1 -- 172.21.15.150:0/1895717087 shutdown_connections 2024-03-17T23:49:59.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.459+0000 7f2c5b9d4640 1 --2- 172.21.15.150:0/1895717087 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f2c5415e830 0x7f2c540b2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.459+0000 7f2c5b9d4640 1 --2- 172.21.15.150:0/1895717087 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f2c3806bbd0 0x7f2c3806e090 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.459+0000 7f2c5b9d4640 1 --2- 172.21.15.150:0/1895717087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c5415bce0 0x7f2c540b3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.459+0000 7f2c5b9d4640 1 --2- 172.21.15.150:0/1895717087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2c540ccf90 0x7f2c54142a90 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.459+0000 7f2c5b9d4640 1 -- 172.21.15.150:0/1895717087 >> 172.21.15.150:0/1895717087 conn(0x7f2c5406b190 msgr2=0x7f2c5415a280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:59.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.459+0000 7f2c5b9d4640 1 -- 172.21.15.150:0/1895717087 shutdown_connections 2024-03-17T23:49:59.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.459+0000 7f2c5b9d4640 1 -- 172.21.15.150:0/1895717087 wait complete. 2024-03-17T23:49:59.471 INFO:tasks.ceph:Scrubbing osd.3 2024-03-17T23:49:59.471 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-17T23:49:59.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.575+0000 7ff3cc2f1640 1 Processor -- start 2024-03-17T23:49:59.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.575+0000 7ff3cc2f1640 1 -- start start 2024-03-17T23:49:59.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.576+0000 7ff3cc2f1640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff3c40cd4d0 0x7ff3c4072070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:59.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.576+0000 7ff3cc2f1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3c40cf590 0x7ff3c40b3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:59.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.576+0000 7ff3cc2f1640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3c415aff0 con 0x7ff3c40725b0 2024-03-17T23:49:59.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.576+0000 7ff3cc2f1640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7ff3c415b260 con 0x7ff3c40cd4d0 2024-03-17T23:49:59.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.576+0000 7ff3cc2f1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff3c415b540 con 0x7ff3c40cf590 2024-03-17T23:49:59.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.576+0000 7ff3ca88e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3c40cf590 0x7ff3c40b3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:59.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.576+0000 7ff3ca08d640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7ff3c40cd4d0 0x7ff3c4072070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:59.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.576+0000 7ff3ca88e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3c40cf590 0x7ff3c40b3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:57290/0 (socket says 172.21.15.150:57290) 2024-03-17T23:49:59.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.576+0000 7ff3ca88e640 1 -- 172.21.15.150:0/4111129731 learned_addr learned my addr 172.21.15.150:0/4111129731 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:59.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.576+0000 7ff3c988c640 1 --1- 172.21.15.150:0/4111129731 >> v1:172.21.15.157:6789/0 conn(0x7ff3c40725b0 0x7ff3c40cee80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:45900/0 (socket says 172.21.15.150:45900) 2024-03-17T23:49:59.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.577+0000 7ff3ca88e640 1 -- 172.21.15.150:0/4111129731 >> v2:172.21.15.157:3300/0 conn(0x7ff3c40cd4d0 msgr2=0x7ff3c4072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:59.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.577+0000 7ff3ca88e640 1 --2- 172.21.15.150:0/4111129731 >> v2:172.21.15.157:3300/0 conn(0x7ff3c40cd4d0 0x7ff3c4072070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.577+0000 7ff3ca88e640 1 -- 172.21.15.150:0/4111129731 >> v1:172.21.15.157:6789/0 conn(0x7ff3c40725b0 legacy=0x7ff3c40cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.577+0000 7ff3ca88e640 1 -- 172.21.15.150:0/4111129731 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3c415b820 con 0x7ff3c40cf590 2024-03-17T23:49:59.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.577+0000 7ff3ca08d640 1 --2- 172.21.15.150:0/4111129731 >> v2:172.21.15.157:3300/0 conn(0x7ff3c40cd4d0 0x7ff3c4072070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:49:59.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.577+0000 7ff3ca88e640 1 --2- 172.21.15.150:0/4111129731 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3c40cf590 0x7ff3c40b3900 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009870 tx=0x7ff3bc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=189ef7177664ce33 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:59.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.578+0000 7ff3c908b640 1 -- 172.21.15.150:0/4111129731 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc034070 con 0x7ff3c40cf590 2024-03-17T23:49:59.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.578+0000 7ff3c908b640 1 -- 172.21.15.150:0/4111129731 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3bc030840 con 0x7ff3c40cf590 2024-03-17T23:49:59.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.578+0000 7ff3c908b640 1 -- 172.21.15.150:0/4111129731 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc030c10 con 0x7ff3c40cf590 2024-03-17T23:49:59.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.578+0000 7ff3cc2f1640 1 -- 172.21.15.150:0/4111129731 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3c40cf590 msgr2=0x7ff3c40b3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.578+0000 7ff3cc2f1640 1 --2- 172.21.15.150:0/4111129731 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3c40cf590 0x7ff3c40b3900 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009870 tx=0x7ff3bc02b980 comp rx=0 tx=0).stop 2024-03-17T23:49:59.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.578+0000 7ff3cc2f1640 1 -- 172.21.15.150:0/4111129731 shutdown_connections 2024-03-17T23:49:59.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.578+0000 7ff3cc2f1640 1 --2- 172.21.15.150:0/4111129731 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3c40cf590 0x7ff3c40b3900 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.578+0000 7ff3cc2f1640 1 --2- 172.21.15.150:0/4111129731 >> v2:172.21.15.157:3300/0 conn(0x7ff3c40cd4d0 0x7ff3c4072070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.578+0000 7ff3cc2f1640 1 -- 172.21.15.150:0/4111129731 >> 172.21.15.150:0/4111129731 conn(0x7ff3c40b1be0 msgr2=0x7ff3c40b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:59.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.578+0000 7ff3cc2f1640 1 -- 172.21.15.150:0/4111129731 shutdown_connections 2024-03-17T23:49:59.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.578+0000 7ff3cc2f1640 1 -- 172.21.15.150:0/4111129731 wait complete. 2024-03-17T23:49:59.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.579+0000 7ff3cc2f1640 1 Processor -- start 2024-03-17T23:49:59.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.579+0000 7ff3cc2f1640 1 -- start start 2024-03-17T23:49:59.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.580+0000 7ff3cc2f1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3c40725b0 0x7ff3c4150540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:59.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.580+0000 7ff3cc2f1640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3c40cd4d0 0x7ff3c406e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:59.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.580+0000 7ff3cc2f1640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3c40cf590 0x7ff3c414fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:59.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.580+0000 7ff3cc2f1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c40cd8b0 con 0x7ff3c40725b0 2024-03-17T23:49:59.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.580+0000 7ff3cc2f1640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c415bde0 con 0x7ff3c40cd4d0 2024-03-17T23:49:59.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.580+0000 7ff3cc2f1640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7ff3c40b3e40 con 0x7ff3c40cf590 2024-03-17T23:49:59.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.580+0000 7ff3ca08d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3c40725b0 0x7ff3c4150540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:59.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.580+0000 7ff3ca08d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3c40725b0 0x7ff3c4150540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44928/0 (socket says 172.21.15.150:44928) 2024-03-17T23:49:59.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.580+0000 7ff3ca08d640 1 -- 172.21.15.150:0/4054771526 learned_addr learned my addr 172.21.15.150:0/4054771526 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:59.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.580+0000 7ff3ca88e640 1 --2- 172.21.15.150:0/4054771526 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3c40cf590 0x7ff3c414fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:59.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.580+0000 7ff3c988c640 1 --2- 172.21.15.150:0/4054771526 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3c40cd4d0 0x7ff3c406e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:59.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.581+0000 7ff3ca08d640 1 -- 172.21.15.150:0/4054771526 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3c40cf590 msgr2=0x7ff3c414fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.581+0000 7ff3ca08d640 1 --2- 172.21.15.150:0/4054771526 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7ff3c40cf590 0x7ff3c414fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.581+0000 7ff3ca08d640 1 -- 172.21.15.150:0/4054771526 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3c40cd4d0 msgr2=0x7ff3c406e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.581+0000 7ff3ca08d640 1 --2- 172.21.15.150:0/4054771526 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3c40cd4d0 0x7ff3c406e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.581+0000 7ff3ca08d640 1 -- 172.21.15.150:0/4054771526 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3c40ada20 con 0x7ff3c40725b0 2024-03-17T23:49:59.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.581+0000 7ff3c988c640 1 --2- 172.21.15.150:0/4054771526 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3c40cd4d0 0x7ff3c406e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:49:59.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.581+0000 7ff3ca08d640 1 --2- 172.21.15.150:0/4054771526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3c40725b0 0x7ff3c4150540 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff3c000efc0 tx=0x7ff3c000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:59.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.581+0000 7ff3baffd640 1 -- 172.21.15.150:0/4054771526 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0018020 con 0x7ff3c40725b0 2024-03-17T23:49:59.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.581+0000 7ff3baffd640 1 -- 172.21.15.150:0/4054771526 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3c0014680 con 0x7ff3c40725b0 2024-03-17T23:49:59.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.581+0000 7ff3cc2f1640 1 -- 172.21.15.150:0/4054771526 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3c40add00 con 0x7ff3c40725b0 2024-03-17T23:49:59.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.581+0000 7ff3baffd640 1 -- 172.21.15.150:0/4054771526 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0014a50 con 0x7ff3c40725b0 2024-03-17T23:49:59.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.581+0000 7ff3cc2f1640 1 -- 172.21.15.150:0/4054771526 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3c40ae2c0 con 0x7ff3c40725b0 2024-03-17T23:49:59.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.582+0000 7ff3cc2f1640 1 -- 172.21.15.150:0/4054771526 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff3c4156900 con 0x7ff3c40725b0 2024-03-17T23:49:59.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.582+0000 7ff3baffd640 1 -- 172.21.15.150:0/4054771526 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7ff3c0005d10 con 0x7ff3c40725b0 2024-03-17T23:49:59.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.583+0000 7ff3baffd640 1 --2- 172.21.15.150:0/4054771526 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff3a806bbd0 0x7ff3a806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:59.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.583+0000 7ff3baffd640 1 -- 172.21.15.150:0/4054771526 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5335+0+0 (secure 0 0 0) 0x7ff3c008b810 con 0x7ff3c40725b0 2024-03-17T23:49:59.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.583+0000 7ff3c988c640 1 --2- 172.21.15.150:0/4054771526 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff3a806bbd0 0x7ff3a806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:59.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.583+0000 7ff3baffd640 1 --2- 172.21.15.150:0/4054771526 >> [v2:172.21.15.150:6816/3072406303,v1:172.21.15.150:6818/3072406303] conn(0x7ff3a8072070 0x7ff3a80744b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:59.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.583+0000 7ff3baffd640 1 -- 172.21.15.150:0/4054771526 --> [v2:172.21.15.150:6816/3072406303,v1:172.21.15.150:6818/3072406303] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff3c000be50 con 0x7ff3a8072070 2024-03-17T23:49:59.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.583+0000 7ff3baffd640 1 -- 172.21.15.150:0/4054771526 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7ff3c00902c0 con 0x7ff3c40725b0 2024-03-17T23:49:59.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.583+0000 7ff3ca88e640 1 --2- 172.21.15.150:0/4054771526 >> [v2:172.21.15.150:6816/3072406303,v1:172.21.15.150:6818/3072406303] conn(0x7ff3a8072070 0x7ff3a80744b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:59.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.583+0000 7ff3c988c640 1 --2- 172.21.15.150:0/4054771526 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff3a806bbd0 0x7ff3a806e090 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ff3c40b3d00 tx=0x7ff3b4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:59.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.583+0000 7ff3ca88e640 1 --2- 172.21.15.150:0/4054771526 >> [v2:172.21.15.150:6816/3072406303,v1:172.21.15.150:6818/3072406303] conn(0x7ff3a8072070 0x7ff3a80744b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:59.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.584+0000 7ff3baffd640 1 -- 172.21.15.150:0/4054771526 <== osd.3 v2:172.21.15.150:6816/3072406303 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff3c000be50 con 0x7ff3a8072070 2024-03-17T23:49:59.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.597+0000 7ff3cc2f1640 1 -- 172.21.15.150:0/4054771526 --> [v2:172.21.15.150:6816/3072406303,v1:172.21.15.150:6818/3072406303] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff3c40681a0 con 0x7ff3a8072070 2024-03-17T23:49:59.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.597+0000 7ff3baffd640 1 -- 172.21.15.150:0/4054771526 <== osd.3 v2:172.21.15.150:6816/3072406303 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff3c40681a0 con 0x7ff3a8072070 2024-03-17T23:49:59.598 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-17T23:49:59.598 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-17T23:49:59.598 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-17T23:49:59.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.597+0000 7ff3cc2f1640 1 -- 172.21.15.150:0/4054771526 >> [v2:172.21.15.150:6816/3072406303,v1:172.21.15.150:6818/3072406303] conn(0x7ff3a8072070 msgr2=0x7ff3a80744b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.597+0000 7ff3cc2f1640 1 --2- 172.21.15.150:0/4054771526 >> [v2:172.21.15.150:6816/3072406303,v1:172.21.15.150:6818/3072406303] conn(0x7ff3a8072070 0x7ff3a80744b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.597+0000 7ff3cc2f1640 1 -- 172.21.15.150:0/4054771526 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff3a806bbd0 msgr2=0x7ff3a806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.597+0000 7ff3cc2f1640 1 --2- 172.21.15.150:0/4054771526 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7ff3a806bbd0 0x7ff3a806e090 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ff3c40b3d00 tx=0x7ff3b4009040 comp rx=0 tx=0).stop 2024-03-17T23:49:59.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.597+0000 7ff3cc2f1640 1 -- 172.21.15.150:0/4054771526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3c40725b0 msgr2=0x7ff3c4150540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.597+0000 7ff3cc2f1640 1 --2- 172.21.15.150:0/4054771526 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff3c40725b0 0x7ff3c4150540 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff3c000efc0 tx=0x7ff3c000c490 comp rx=0 tx=0).stop 2024-03-17T23:49:59.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.597+0000 7ff3ca88e640 1 -- 172.21.15.150:0/4054771526 reap_dead start 2024-03-17T23:49:59.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.597+0000 7ff3cc2f1640 1 -- 172.21.15.150:0/4054771526 shutdown_connections 2024-03-17T23:49:59.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.597+0000 7ff3cc2f1640 1 -- 172.21.15.150:0/4054771526 >> 172.21.15.150:0/4054771526 conn(0x7ff3c40b1be0 msgr2=0x7ff3c406f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:59.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.598+0000 7ff3cc2f1640 1 -- 172.21.15.150:0/4054771526 shutdown_connections 2024-03-17T23:49:59.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.598+0000 7ff3cc2f1640 1 -- 172.21.15.150:0/4054771526 wait complete. 2024-03-17T23:49:59.608 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-17T23:49:59.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.710+0000 7fe7cf909640 1 Processor -- start 2024-03-17T23:49:59.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.710+0000 7fe7cf909640 1 -- start start 2024-03-17T23:49:59.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.711+0000 7fe7cf909640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe7c815bce0 0x7fe7c815d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:59.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.711+0000 7fe7cf909640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7c806de60 con 0x7fe7c80cf100 2024-03-17T23:49:59.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.711+0000 7fe7cf909640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7c806e140 con 0x7fe7c80ccf90 2024-03-17T23:49:59.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.711+0000 7fe7cf909640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fe7c806e420 con 0x7fe7c815bce0 2024-03-17T23:49:59.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.711+0000 7fe7ccea4640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fe7c80cf100 0x7fe7c815cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:58708/0 (socket says 172.21.15.150:58708) 2024-03-17T23:49:59.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.711+0000 7fe7cdea6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe7c815bce0 0x7fe7c815d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:59.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.711+0000 7fe7ccea4640 1 -- 172.21.15.150:0/2870440052 learned_addr learned my addr 172.21.15.150:0/2870440052 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:49:59.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.711+0000 7fe7cdea6640 1 -- 172.21.15.150:0/2870440052 >> v1:172.21.15.157:6789/0 conn(0x7fe7c80ccf90 legacy=0x7fe7c80cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:49:59.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.711+0000 7fe7cdea6640 1 -- 172.21.15.150:0/2870440052 >> v1:172.21.15.150:6789/0 conn(0x7fe7c80cf100 legacy=0x7fe7c815cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.711+0000 7fe7cdea6640 1 -- 172.21.15.150:0/2870440052 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7c815c5a0 con 0x7fe7c815bce0 2024-03-17T23:49:59.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.711+0000 7fe7bffff640 1 -- 172.21.15.150:0/2870440052 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1586770433 0 0) 0x7fe7c806de60 con 0x7fe7c80cf100 2024-03-17T23:49:59.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.712+0000 7fe7cdea6640 1 --2- 172.21.15.150:0/2870440052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7c815bce0 0x7fe7c815d1d0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fe7c00097a0 tx=0x7fe7c002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d21e252b29bf2772 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:59.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.712+0000 7fe7bffff640 1 -- 172.21.15.150:0/2870440052 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c00315a0 con 0x7fe7c815bce0 2024-03-17T23:49:59.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.712+0000 7fe7bffff640 1 -- 172.21.15.150:0/2870440052 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7c0037070 con 0x7fe7c815bce0 2024-03-17T23:49:59.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.712+0000 7fe7bffff640 1 -- 172.21.15.150:0/2870440052 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c0031960 con 0x7fe7c815bce0 2024-03-17T23:49:59.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.713+0000 7fe7cf909640 1 -- 172.21.15.150:0/2870440052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7c815bce0 msgr2=0x7fe7c815d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.713+0000 7fe7cf909640 1 --2- 172.21.15.150:0/2870440052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7c815bce0 0x7fe7c815d1d0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fe7c00097a0 tx=0x7fe7c002b840 comp rx=0 tx=0).stop 2024-03-17T23:49:59.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.713+0000 7fe7cf909640 1 -- 172.21.15.150:0/2870440052 shutdown_connections 2024-03-17T23:49:59.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.713+0000 7fe7cf909640 1 --2- 172.21.15.150:0/2870440052 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7c815bce0 0x7fe7c815d1d0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.713+0000 7fe7cf909640 1 -- 172.21.15.150:0/2870440052 >> 172.21.15.150:0/2870440052 conn(0x7fe7c806b190 msgr2=0x7fe7c806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:59.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.713+0000 7fe7cf909640 1 -- 172.21.15.150:0/2870440052 shutdown_connections 2024-03-17T23:49:59.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.713+0000 7fe7cf909640 1 -- 172.21.15.150:0/2870440052 wait complete. 2024-03-17T23:49:59.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.714+0000 7fe7cf909640 1 Processor -- start 2024-03-17T23:49:59.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.714+0000 7fe7cf909640 1 -- start start 2024-03-17T23:49:59.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.715+0000 7fe7cf909640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7c80ccf90 0x7fe7c813d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:59.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.715+0000 7fe7cf909640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7c80cf100 0x7fe7c814de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:59.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.715+0000 7fe7cf909640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7c815bce0 0x7fe7c814e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:59.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.715+0000 7fe7cf909640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c815c170 con 0x7fe7c815bce0 2024-03-17T23:49:59.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.715+0000 7fe7cf909640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c80cf8a0 con 0x7fe7c80cf100 2024-03-17T23:49:59.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.715+0000 7fe7cf909640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fe7c8155530 con 0x7fe7c80ccf90 2024-03-17T23:49:59.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.715+0000 7fe7cdea6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7c815bce0 0x7fe7c814e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:59.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.715+0000 7fe7cdea6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7c815bce0 0x7fe7c814e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44946/0 (socket says 172.21.15.150:44946) 2024-03-17T23:49:59.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.715+0000 7fe7cdea6640 1 -- 172.21.15.150:0/45804523 learned_addr learned my addr 172.21.15.150:0/45804523 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:49:59.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.715+0000 7fe7ccea4640 1 --2- 172.21.15.150:0/45804523 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7c80cf100 0x7fe7c814de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:59.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.715+0000 7fe7cd6a5640 1 --2- 172.21.15.150:0/45804523 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7c80ccf90 0x7fe7c813d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:59.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.715+0000 7fe7cdea6640 1 -- 172.21.15.150:0/45804523 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7c80ccf90 msgr2=0x7fe7c813d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.715+0000 7fe7cdea6640 1 --2- 172.21.15.150:0/45804523 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7c80ccf90 0x7fe7c813d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.715+0000 7fe7cdea6640 1 -- 172.21.15.150:0/45804523 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7c80cf100 msgr2=0x7fe7c814de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.715+0000 7fe7cdea6640 1 --2- 172.21.15.150:0/45804523 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7c80cf100 0x7fe7c814de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.715+0000 7fe7cdea6640 1 -- 172.21.15.150:0/45804523 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7c80b3610 con 0x7fe7c815bce0 2024-03-17T23:49:59.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.715+0000 7fe7cd6a5640 1 --2- 172.21.15.150:0/45804523 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7c80ccf90 0x7fe7c813d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:49:59.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.715+0000 7fe7ccea4640 1 --2- 172.21.15.150:0/45804523 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7c80cf100 0x7fe7c814de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:49:59.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.715+0000 7fe7cdea6640 1 --2- 172.21.15.150:0/45804523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7c815bce0 0x7fe7c814e3b0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fe7c00097a0 tx=0x7fe7c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:59.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.715+0000 7fe7bdffb640 1 -- 172.21.15.150:0/45804523 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c0004230 con 0x7fe7c815bce0 2024-03-17T23:49:59.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.716+0000 7fe7bdffb640 1 -- 172.21.15.150:0/45804523 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7c00043f0 con 0x7fe7c815bce0 2024-03-17T23:49:59.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.716+0000 7fe7cf909640 1 -- 172.21.15.150:0/45804523 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7c80b38f0 con 0x7fe7c815bce0 2024-03-17T23:49:59.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.716+0000 7fe7bdffb640 1 -- 172.21.15.150:0/45804523 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c0003850 con 0x7fe7c815bce0 2024-03-17T23:49:59.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.716+0000 7fe7cf909640 1 -- 172.21.15.150:0/45804523 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7c80b3eb0 con 0x7fe7c815bce0 2024-03-17T23:49:59.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.717+0000 7fe7cf909640 1 -- 172.21.15.150:0/45804523 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7c80681a0 con 0x7fe7c815bce0 2024-03-17T23:49:59.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.717+0000 7fe7bdffb640 1 -- 172.21.15.150:0/45804523 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fe7c00039f0 con 0x7fe7c815bce0 2024-03-17T23:49:59.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.718+0000 7fe7bdffb640 1 --2- 172.21.15.150:0/45804523 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe7ac06bbd0 0x7fe7ac06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:49:59.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.718+0000 7fe7cd6a5640 1 --2- 172.21.15.150:0/45804523 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe7ac06bbd0 0x7fe7ac06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:49:59.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.718+0000 7fe7bdffb640 1 -- 172.21.15.150:0/45804523 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5335+0+0 (secure 0 0 0) 0x7fe7c002f030 con 0x7fe7c815bce0 2024-03-17T23:49:59.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.718+0000 7fe7cd6a5640 1 --2- 172.21.15.150:0/45804523 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe7ac06bbd0 0x7fe7ac06e090 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe7c81f3340 tx=0x7fe7c4009570 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:49:59.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.722+0000 7fe7bdffb640 1 -- 172.21.15.150:0/45804523 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fe7c0041070 con 0x7fe7c815bce0 2024-03-17T23:49:59.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.894+0000 7fe7cf909640 1 -- 172.21.15.150:0/45804523 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fe7c806fff0 con 0x7fe7ac06bbd0 2024-03-17T23:49:59.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.895+0000 7fe7bdffb640 1 -- 172.21.15.150:0/45804523 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe7c806fff0 con 0x7fe7ac06bbd0 2024-03-17T23:49:59.896 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 3 to deep-scrub 2024-03-17T23:49:59.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.897+0000 7fe7cf909640 1 -- 172.21.15.150:0/45804523 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe7ac06bbd0 msgr2=0x7fe7ac06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.897+0000 7fe7cf909640 1 --2- 172.21.15.150:0/45804523 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe7ac06bbd0 0x7fe7ac06e090 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fe7c81f3340 tx=0x7fe7c4009570 comp rx=0 tx=0).stop 2024-03-17T23:49:59.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.897+0000 7fe7cf909640 1 -- 172.21.15.150:0/45804523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7c815bce0 msgr2=0x7fe7c814e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:49:59.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.897+0000 7fe7cf909640 1 --2- 172.21.15.150:0/45804523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7c815bce0 0x7fe7c814e3b0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fe7c00097a0 tx=0x7fe7c00046c0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.897+0000 7fe7cf909640 1 -- 172.21.15.150:0/45804523 shutdown_connections 2024-03-17T23:49:59.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.897+0000 7fe7cf909640 1 --2- 172.21.15.150:0/45804523 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe7c815bce0 0x7fe7c814e3b0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.897+0000 7fe7cf909640 1 --2- 172.21.15.150:0/45804523 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fe7ac06bbd0 0x7fe7ac06e090 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.897+0000 7fe7cf909640 1 --2- 172.21.15.150:0/45804523 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7c80cf100 0x7fe7c814de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.897+0000 7fe7cf909640 1 --2- 172.21.15.150:0/45804523 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fe7c80ccf90 0x7fe7c813d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:49:59.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.897+0000 7fe7cf909640 1 -- 172.21.15.150:0/45804523 >> 172.21.15.150:0/45804523 conn(0x7fe7c806b190 msgr2=0x7fe7c815a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:49:59.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.897+0000 7fe7cf909640 1 -- 172.21.15.150:0/45804523 shutdown_connections 2024-03-17T23:49:59.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:49:59.897+0000 7fe7cf909640 1 -- 172.21.15.150:0/45804523 wait complete. 2024-03-17T23:49:59.908 INFO:tasks.ceph:Scrubbing osd.4 2024-03-17T23:49:59.908 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-17T23:50:00.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.012+0000 7f723baff640 1 Processor -- start 2024-03-17T23:50:00.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.013+0000 7f723baff640 1 -- start start 2024-03-17T23:50:00.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.014+0000 7f723baff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f723413d340 0x7f723415b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.014+0000 7f723baff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f72340cc440 0x7f72340cc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.014+0000 7f723baff640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72340cce00 con 0x7f723415de80 2024-03-17T23:50:00.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.014+0000 7f723baff640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f723415b850 con 0x7f72340cc440 2024-03-17T23:50:00.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.014+0000 7f723baff640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f72340ceab0 con 0x7f723413d340 2024-03-17T23:50:00.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.014+0000 7f723909a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f72340cc440 0x7f72340cc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.014+0000 7f723909a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f72340cc440 0x7f72340cc8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44950/0 (socket says 172.21.15.150:44950) 2024-03-17T23:50:00.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.014+0000 7f723909a640 1 -- 172.21.15.150:0/3905488673 learned_addr learned my addr 172.21.15.150:0/3905488673 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:50:00.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.014+0000 7f723909a640 1 -- 172.21.15.150:0/3905488673 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f723413d340 msgr2=0x7f723415b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:50:00.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.014+0000 7f723989b640 1 --2- 172.21.15.150:0/3905488673 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f723413d340 0x7f723415b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.014+0000 7f723909a640 1 --2- 172.21.15.150:0/3905488673 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f723413d340 0x7f723415b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.014+0000 7f723909a640 1 -- 172.21.15.150:0/3905488673 >> v1:172.21.15.157:6789/0 conn(0x7f723415de80 legacy=0x7f723415e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.014+0000 7f723909a640 1 -- 172.21.15.150:0/3905488673 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72340ced90 con 0x7f72340cc440 2024-03-17T23:50:00.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.014+0000 7f723989b640 1 --2- 172.21.15.150:0/3905488673 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f723413d340 0x7f723415b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:50:00.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.015+0000 7f723909a640 1 --2- 172.21.15.150:0/3905488673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72340cc440 0x7f72340cc8c0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f7230009870 tx=0x7f723002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b86b2c63a6b4cb44 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:00.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.015+0000 7f7238899640 1 -- 172.21.15.150:0/3905488673 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7230034070 con 0x7f72340cc440 2024-03-17T23:50:00.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.015+0000 7f7238899640 1 -- 172.21.15.150:0/3905488673 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7230030840 con 0x7f72340cc440 2024-03-17T23:50:00.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.015+0000 7f7238899640 1 -- 172.21.15.150:0/3905488673 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7230030c10 con 0x7f72340cc440 2024-03-17T23:50:00.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.015+0000 7f723baff640 1 -- 172.21.15.150:0/3905488673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72340cc440 msgr2=0x7f72340cc8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.015+0000 7f723baff640 1 --2- 172.21.15.150:0/3905488673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72340cc440 0x7f72340cc8c0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f7230009870 tx=0x7f723002b980 comp rx=0 tx=0).stop 2024-03-17T23:50:00.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.016+0000 7f723baff640 1 -- 172.21.15.150:0/3905488673 shutdown_connections 2024-03-17T23:50:00.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.016+0000 7f723baff640 1 --2- 172.21.15.150:0/3905488673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72340cc440 0x7f72340cc8c0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.016+0000 7f723baff640 1 --2- 172.21.15.150:0/3905488673 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f723413d340 0x7f723415b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.016+0000 7f723baff640 1 -- 172.21.15.150:0/3905488673 >> 172.21.15.150:0/3905488673 conn(0x7f72340b1be0 msgr2=0x7f72340b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:00.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.016+0000 7f723baff640 1 -- 172.21.15.150:0/3905488673 shutdown_connections 2024-03-17T23:50:00.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.016+0000 7f723baff640 1 -- 172.21.15.150:0/3905488673 wait complete. 2024-03-17T23:50:00.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.017+0000 7f723baff640 1 Processor -- start 2024-03-17T23:50:00.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.017+0000 7f723baff640 1 -- start start 2024-03-17T23:50:00.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.017+0000 7f723baff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72340cc440 0x7f7234135810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.017+0000 7f723baff640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f723413d340 0x7f72340ca590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.017+0000 7f723baff640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f723415de80 0x7f7234158dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.017+0000 7f723baff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f72340cf630 con 0x7f72340cc440 2024-03-17T23:50:00.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.017+0000 7f723baff640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f72340cf350 con 0x7f723415de80 2024-03-17T23:50:00.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.017+0000 7f723baff640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f723413d720 con 0x7f723413d340 2024-03-17T23:50:00.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.018+0000 7f723989b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72340cc440 0x7f7234135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.018+0000 7f723989b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72340cc440 0x7f7234135810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44952/0 (socket says 172.21.15.150:44952) 2024-03-17T23:50:00.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.018+0000 7f723989b640 1 -- 172.21.15.150:0/2854774704 learned_addr learned my addr 172.21.15.150:0/2854774704 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:50:00.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.018+0000 7f723a09c640 1 --2- 172.21.15.150:0/2854774704 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f723415de80 0x7f7234158dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.018+0000 7f723909a640 1 --2- 172.21.15.150:0/2854774704 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f723413d340 0x7f72340ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.018+0000 7f723989b640 1 -- 172.21.15.150:0/2854774704 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f723413d340 msgr2=0x7f72340ca590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.018+0000 7f723989b640 1 --2- 172.21.15.150:0/2854774704 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f723413d340 0x7f72340ca590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.018+0000 7f723989b640 1 -- 172.21.15.150:0/2854774704 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f723415de80 msgr2=0x7f7234158dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.018+0000 7f723989b640 1 --2- 172.21.15.150:0/2854774704 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f723415de80 0x7f7234158dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.018+0000 7f723989b640 1 -- 172.21.15.150:0/2854774704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7234134a50 con 0x7f72340cc440 2024-03-17T23:50:00.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.018+0000 7f723909a640 1 --2- 172.21.15.150:0/2854774704 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f723413d340 0x7f72340ca590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:50:00.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.018+0000 7f723989b640 1 --2- 172.21.15.150:0/2854774704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72340cc440 0x7f7234135810 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f722400c7f0 tx=0x7f722400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:00.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.018+0000 7f722a7fc640 1 -- 172.21.15.150:0/2854774704 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7224015070 con 0x7f72340cc440 2024-03-17T23:50:00.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.018+0000 7f722a7fc640 1 -- 172.21.15.150:0/2854774704 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7224007a30 con 0x7f72340cc440 2024-03-17T23:50:00.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.018+0000 7f723baff640 1 -- 172.21.15.150:0/2854774704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7234134d30 con 0x7f72340cc440 2024-03-17T23:50:00.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.018+0000 7f722a7fc640 1 -- 172.21.15.150:0/2854774704 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7224007e00 con 0x7f72340cc440 2024-03-17T23:50:00.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.018+0000 7f723baff640 1 -- 172.21.15.150:0/2854774704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72341352f0 con 0x7f72340cc440 2024-03-17T23:50:00.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.019+0000 7f723baff640 1 -- 172.21.15.150:0/2854774704 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f72340681a0 con 0x7f72340cc440 2024-03-17T23:50:00.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.019+0000 7f722a7fc640 1 -- 172.21.15.150:0/2854774704 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f7224018070 con 0x7f72340cc440 2024-03-17T23:50:00.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.020+0000 7f722a7fc640 1 --2- 172.21.15.150:0/2854774704 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f721806bc00 0x7f721806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.020+0000 7f722a7fc640 1 -- 172.21.15.150:0/2854774704 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5335+0+0 (secure 0 0 0) 0x7f722408a790 con 0x7f72340cc440 2024-03-17T23:50:00.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.020+0000 7f723909a640 1 --2- 172.21.15.150:0/2854774704 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f721806bc00 0x7f721806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.020+0000 7f722a7fc640 1 --2- 172.21.15.150:0/2854774704 >> [v2:172.21.15.157:6808/4022520191,v1:172.21.15.157:6810/4022520191] conn(0x7f72180720a0 0x7f72180744e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.020+0000 7f722a7fc640 1 -- 172.21.15.150:0/2854774704 --> [v2:172.21.15.157:6808/4022520191,v1:172.21.15.157:6810/4022520191] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7218074bb0 con 0x7f72180720a0 2024-03-17T23:50:00.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.020+0000 7f722a7fc640 1 -- 172.21.15.150:0/2854774704 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7f722408f2c0 con 0x7f72340cc440 2024-03-17T23:50:00.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.020+0000 7f723a09c640 1 --2- 172.21.15.150:0/2854774704 >> [v2:172.21.15.157:6808/4022520191,v1:172.21.15.157:6810/4022520191] conn(0x7f72180720a0 0x7f72180744e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.020+0000 7f723909a640 1 --2- 172.21.15.150:0/2854774704 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f721806bc00 0x7f721806e0c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7230036000 tx=0x7f723003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:00.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.021+0000 7f723a09c640 1 --2- 172.21.15.150:0/2854774704 >> [v2:172.21.15.157:6808/4022520191,v1:172.21.15.157:6810/4022520191] conn(0x7f72180720a0 0x7f72180744e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:00.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.021+0000 7f722a7fc640 1 -- 172.21.15.150:0/2854774704 <== osd.4 v2:172.21.15.157:6808/4022520191 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7218074bb0 con 0x7f72180720a0 2024-03-17T23:50:00.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.035+0000 7f723baff640 1 -- 172.21.15.150:0/2854774704 --> [v2:172.21.15.157:6808/4022520191,v1:172.21.15.157:6810/4022520191] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f723406e400 con 0x7f72180720a0 2024-03-17T23:50:00.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.035+0000 7f722a7fc640 1 -- 172.21.15.150:0/2854774704 <== osd.4 v2:172.21.15.157:6808/4022520191 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f723406e400 con 0x7f72180720a0 2024-03-17T23:50:00.036 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-17T23:50:00.036 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-17T23:50:00.036 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-17T23:50:00.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.035+0000 7f723baff640 1 -- 172.21.15.150:0/2854774704 >> [v2:172.21.15.157:6808/4022520191,v1:172.21.15.157:6810/4022520191] conn(0x7f72180720a0 msgr2=0x7f72180744e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.035+0000 7f723baff640 1 --2- 172.21.15.150:0/2854774704 >> [v2:172.21.15.157:6808/4022520191,v1:172.21.15.157:6810/4022520191] conn(0x7f72180720a0 0x7f72180744e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.036+0000 7f723baff640 1 -- 172.21.15.150:0/2854774704 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f721806bc00 msgr2=0x7f721806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.036+0000 7f723baff640 1 --2- 172.21.15.150:0/2854774704 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f721806bc00 0x7f721806e0c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7230036000 tx=0x7f723003b040 comp rx=0 tx=0).stop 2024-03-17T23:50:00.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.036+0000 7f723baff640 1 -- 172.21.15.150:0/2854774704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72340cc440 msgr2=0x7f7234135810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.036+0000 7f723baff640 1 --2- 172.21.15.150:0/2854774704 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72340cc440 0x7f7234135810 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f722400c7f0 tx=0x7f722400cbf0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.036+0000 7f723a09c640 1 -- 172.21.15.150:0/2854774704 reap_dead start 2024-03-17T23:50:00.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.036+0000 7f723baff640 1 -- 172.21.15.150:0/2854774704 shutdown_connections 2024-03-17T23:50:00.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.036+0000 7f723baff640 1 -- 172.21.15.150:0/2854774704 >> 172.21.15.150:0/2854774704 conn(0x7f72340b1be0 msgr2=0x7f72340b3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:00.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.036+0000 7f723baff640 1 -- 172.21.15.150:0/2854774704 shutdown_connections 2024-03-17T23:50:00.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.036+0000 7f723baff640 1 -- 172.21.15.150:0/2854774704 wait complete. 2024-03-17T23:50:00.046 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-17T23:50:00.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.149+0000 7fbb73105640 1 Processor -- start 2024-03-17T23:50:00.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.149+0000 7fbb73105640 1 -- start start 2024-03-17T23:50:00.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.150+0000 7fbb73105640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbb6c15bce0 0x7fbb6c15d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.150+0000 7fbb73105640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb6c06de60 con 0x7fbb6c0ccf90 2024-03-17T23:50:00.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.150+0000 7fbb73105640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb6c06e140 con 0x7fbb6c0cf100 2024-03-17T23:50:00.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.150+0000 7fbb73105640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fbb6c06e420 con 0x7fbb6c15bce0 2024-03-17T23:50:00.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.150+0000 7fbb716a2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbb6c15bce0 0x7fbb6c15d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.150+0000 7fbb70ea1640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fbb6c0ccf90 0x7fbb6c0cd370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:58718/0 (socket says 172.21.15.150:58718) 2024-03-17T23:50:00.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.150+0000 7fbb70ea1640 1 -- 172.21.15.150:0/3503090789 learned_addr learned my addr 172.21.15.150:0/3503090789 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:50:00.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.150+0000 7fbb716a2640 1 -- 172.21.15.150:0/3503090789 >> v1:172.21.15.157:6789/0 conn(0x7fbb6c0cf100 legacy=0x7fbb6c15cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.150+0000 7fbb716a2640 1 -- 172.21.15.150:0/3503090789 >> v1:172.21.15.150:6789/0 conn(0x7fbb6c0ccf90 legacy=0x7fbb6c0cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.150+0000 7fbb716a2640 1 -- 172.21.15.150:0/3503090789 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb6c15c5a0 con 0x7fbb6c15bce0 2024-03-17T23:50:00.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.150+0000 7fbb6b7fe640 1 -- 172.21.15.150:0/3503090789 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3588253166 0 0) 0x7fbb6c06de60 con 0x7fbb6c0ccf90 2024-03-17T23:50:00.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.151+0000 7fbb716a2640 1 --2- 172.21.15.150:0/3503090789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb6c15bce0 0x7fbb6c15d1d0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fbb64009d90 tx=0x7fbb6402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38bc3a64384b0c0c server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:00.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.151+0000 7fbb6b7fe640 1 -- 172.21.15.150:0/3503090789 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb64039070 con 0x7fbb6c15bce0 2024-03-17T23:50:00.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.151+0000 7fbb6b7fe640 1 -- 172.21.15.150:0/3503090789 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb6402cc60 con 0x7fbb6c15bce0 2024-03-17T23:50:00.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.151+0000 7fbb6b7fe640 1 -- 172.21.15.150:0/3503090789 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb64034660 con 0x7fbb6c15bce0 2024-03-17T23:50:00.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.151+0000 7fbb73105640 1 -- 172.21.15.150:0/3503090789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb6c15bce0 msgr2=0x7fbb6c15d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.151+0000 7fbb73105640 1 --2- 172.21.15.150:0/3503090789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb6c15bce0 0x7fbb6c15d1d0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fbb64009d90 tx=0x7fbb6402beb0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.152+0000 7fbb73105640 1 -- 172.21.15.150:0/3503090789 shutdown_connections 2024-03-17T23:50:00.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.152+0000 7fbb73105640 1 --2- 172.21.15.150:0/3503090789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb6c15bce0 0x7fbb6c15d1d0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.152+0000 7fbb73105640 1 -- 172.21.15.150:0/3503090789 >> 172.21.15.150:0/3503090789 conn(0x7fbb6c06b190 msgr2=0x7fbb6c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:00.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.152+0000 7fbb73105640 1 -- 172.21.15.150:0/3503090789 shutdown_connections 2024-03-17T23:50:00.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.152+0000 7fbb73105640 1 -- 172.21.15.150:0/3503090789 wait complete. 2024-03-17T23:50:00.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.153+0000 7fbb73105640 1 Processor -- start 2024-03-17T23:50:00.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.153+0000 7fbb73105640 1 -- start start 2024-03-17T23:50:00.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.154+0000 7fbb73105640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb6c0ccf90 0x7fbb6c156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.154+0000 7fbb73105640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb6c0cf100 0x7fbb6c143270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.154+0000 7fbb73105640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb6c15bce0 0x7fbb6c1580d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.154+0000 7fbb73105640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbb6c15c170 con 0x7fbb6c0cf100 2024-03-17T23:50:00.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.154+0000 7fbb73105640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fbb6c0cf8a0 con 0x7fbb6c15bce0 2024-03-17T23:50:00.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.154+0000 7fbb73105640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fbb6c155530 con 0x7fbb6c0ccf90 2024-03-17T23:50:00.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.154+0000 7fbb6bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb6c0cf100 0x7fbb6c143270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.154+0000 7fbb6bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb6c0cf100 0x7fbb6c143270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44970/0 (socket says 172.21.15.150:44970) 2024-03-17T23:50:00.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.154+0000 7fbb6bfff640 1 -- 172.21.15.150:0/2856530211 learned_addr learned my addr 172.21.15.150:0/2856530211 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:50:00.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.154+0000 7fbb716a2640 1 --2- 172.21.15.150:0/2856530211 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb6c15bce0 0x7fbb6c1580d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.154+0000 7fbb70ea1640 1 --2- 172.21.15.150:0/2856530211 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb6c0ccf90 0x7fbb6c156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.154+0000 7fbb6bfff640 1 -- 172.21.15.150:0/2856530211 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb6c0ccf90 msgr2=0x7fbb6c156df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.154+0000 7fbb6bfff640 1 --2- 172.21.15.150:0/2856530211 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb6c0ccf90 0x7fbb6c156df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.154+0000 7fbb6bfff640 1 -- 172.21.15.150:0/2856530211 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb6c15bce0 msgr2=0x7fbb6c1580d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.154+0000 7fbb6bfff640 1 --2- 172.21.15.150:0/2856530211 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb6c15bce0 0x7fbb6c1580d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.154+0000 7fbb6bfff640 1 -- 172.21.15.150:0/2856530211 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb6c1e5a40 con 0x7fbb6c0cf100 2024-03-17T23:50:00.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.154+0000 7fbb70ea1640 1 --2- 172.21.15.150:0/2856530211 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb6c0ccf90 0x7fbb6c156df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:50:00.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.154+0000 7fbb6bfff640 1 --2- 172.21.15.150:0/2856530211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb6c0cf100 0x7fbb6c143270 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fbb5800c400 tx=0x7fbb5800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:00.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.154+0000 7fbb697fa640 1 -- 172.21.15.150:0/2856530211 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb58017070 con 0x7fbb6c0cf100 2024-03-17T23:50:00.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.154+0000 7fbb697fa640 1 -- 172.21.15.150:0/2856530211 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb580052d0 con 0x7fbb6c0cf100 2024-03-17T23:50:00.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.154+0000 7fbb73105640 1 -- 172.21.15.150:0/2856530211 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb6c1e5d20 con 0x7fbb6c0cf100 2024-03-17T23:50:00.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.154+0000 7fbb697fa640 1 -- 172.21.15.150:0/2856530211 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb58003830 con 0x7fbb6c0cf100 2024-03-17T23:50:00.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.154+0000 7fbb73105640 1 -- 172.21.15.150:0/2856530211 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb6c1562e0 con 0x7fbb6c0cf100 2024-03-17T23:50:00.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.155+0000 7fbb73105640 1 -- 172.21.15.150:0/2856530211 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb6c0681a0 con 0x7fbb6c0cf100 2024-03-17T23:50:00.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.156+0000 7fbb697fa640 1 -- 172.21.15.150:0/2856530211 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fbb5801a070 con 0x7fbb6c0cf100 2024-03-17T23:50:00.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.156+0000 7fbb697fa640 1 --2- 172.21.15.150:0/2856530211 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbb5006bbd0 0x7fbb5006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.156+0000 7fbb697fa640 1 -- 172.21.15.150:0/2856530211 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5335+0+0 (secure 0 0 0) 0x7fbb58089bc0 con 0x7fbb6c0cf100 2024-03-17T23:50:00.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.156+0000 7fbb70ea1640 1 --2- 172.21.15.150:0/2856530211 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbb5006bbd0 0x7fbb5006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.157+0000 7fbb70ea1640 1 --2- 172.21.15.150:0/2856530211 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbb5006bbd0 0x7fbb5006e090 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fbb60007fd0 tx=0x7fbb60007de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:00.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.161+0000 7fbb697fa640 1 -- 172.21.15.150:0/2856530211 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7fbb580581d0 con 0x7fbb6c0cf100 2024-03-17T23:50:00.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.333+0000 7fbb73105640 1 -- 172.21.15.150:0/2856530211 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7fbb6c06bf90 con 0x7fbb5006bbd0 2024-03-17T23:50:00.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.334+0000 7fbb697fa640 1 -- 172.21.15.150:0/2856530211 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fbb6c06bf90 con 0x7fbb5006bbd0 2024-03-17T23:50:00.334 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 4 to deep-scrub 2024-03-17T23:50:00.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.336+0000 7fbb73105640 1 -- 172.21.15.150:0/2856530211 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbb5006bbd0 msgr2=0x7fbb5006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.336+0000 7fbb73105640 1 --2- 172.21.15.150:0/2856530211 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbb5006bbd0 0x7fbb5006e090 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fbb60007fd0 tx=0x7fbb60007de0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.336+0000 7fbb73105640 1 -- 172.21.15.150:0/2856530211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb6c0cf100 msgr2=0x7fbb6c143270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.336+0000 7fbb73105640 1 --2- 172.21.15.150:0/2856530211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb6c0cf100 0x7fbb6c143270 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fbb5800c400 tx=0x7fbb5800c8c0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.336+0000 7fbb73105640 1 -- 172.21.15.150:0/2856530211 shutdown_connections 2024-03-17T23:50:00.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.336+0000 7fbb73105640 1 --2- 172.21.15.150:0/2856530211 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fbb5006bbd0 0x7fbb5006e090 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.336+0000 7fbb73105640 1 --2- 172.21.15.150:0/2856530211 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb6c15bce0 0x7fbb6c1580d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.336+0000 7fbb73105640 1 --2- 172.21.15.150:0/2856530211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb6c0cf100 0x7fbb6c143270 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.336+0000 7fbb73105640 1 --2- 172.21.15.150:0/2856530211 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fbb6c0ccf90 0x7fbb6c156df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.336+0000 7fbb73105640 1 -- 172.21.15.150:0/2856530211 >> 172.21.15.150:0/2856530211 conn(0x7fbb6c06b190 msgr2=0x7fbb6c15a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:00.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.336+0000 7fbb73105640 1 -- 172.21.15.150:0/2856530211 shutdown_connections 2024-03-17T23:50:00.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.336+0000 7fbb73105640 1 -- 172.21.15.150:0/2856530211 wait complete. 2024-03-17T23:50:00.347 INFO:tasks.ceph:Scrubbing osd.5 2024-03-17T23:50:00.347 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-17T23:50:00.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.452+0000 7fed77273640 1 Processor -- start 2024-03-17T23:50:00.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.452+0000 7fed77273640 1 -- start start 2024-03-17T23:50:00.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.453+0000 7fed77273640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7fed700cd4d0 0x7fed70072070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.453+0000 7fed77273640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fed700cf590 0x7fed700b3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.453+0000 7fed77273640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed7015aff0 con 0x7fed700725b0 2024-03-17T23:50:00.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.453+0000 7fed77273640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7fed7015b260 con 0x7fed700cd4d0 2024-03-17T23:50:00.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.453+0000 7fed77273640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fed7015b540 con 0x7fed700cf590 2024-03-17T23:50:00.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.453+0000 7fed7480e640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fed700725b0 0x7fed700cee80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:58734/0 (socket says 172.21.15.150:58734) 2024-03-17T23:50:00.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.453+0000 7fed7480e640 1 -- 172.21.15.150:0/496485441 learned_addr learned my addr 172.21.15.150:0/496485441 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:50:00.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.453+0000 7fed75810640 1 --2- 172.21.15.150:0/496485441 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fed700cf590 0x7fed700b3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.453+0000 7fed7500f640 1 --2- 172.21.15.150:0/496485441 >> v2:172.21.15.157:3300/0 conn(0x7fed700cd4d0 0x7fed70072070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.453+0000 7fed67fff640 1 -- 172.21.15.150:0/496485441 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 556803463 0 0) 0x7fed7015aff0 con 0x7fed700725b0 2024-03-17T23:50:00.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.453+0000 7fed67fff640 1 -- 172.21.15.150:0/496485441 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed5c003520 con 0x7fed700725b0 2024-03-17T23:50:00.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.453+0000 7fed67fff640 1 -- 172.21.15.150:0/496485441 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1936710466 0 0) 0x7fed5c003520 con 0x7fed700725b0 2024-03-17T23:50:00.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.453+0000 7fed67fff640 1 -- 172.21.15.150:0/496485441 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fed700cf590 msgr2=0x7fed700b3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:50:00.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.453+0000 7fed67fff640 1 --2- 172.21.15.150:0/496485441 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fed700cf590 0x7fed700b3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.453+0000 7fed67fff640 1 -- 172.21.15.150:0/496485441 >> v2:172.21.15.157:3300/0 conn(0x7fed700cd4d0 msgr2=0x7fed70072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:50:00.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.453+0000 7fed67fff640 1 --2- 172.21.15.150:0/496485441 >> v2:172.21.15.157:3300/0 conn(0x7fed700cd4d0 0x7fed70072070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.453+0000 7fed67fff640 1 -- 172.21.15.150:0/496485441 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed7015b820 con 0x7fed700725b0 2024-03-17T23:50:00.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.453+0000 7fed67fff640 1 -- 172.21.15.150:0/496485441 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2618233945 0 0) 0x7fed60001670 con 0x7fed700725b0 2024-03-17T23:50:00.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.453+0000 7fed7500f640 1 --2- 172.21.15.150:0/496485441 >> v2:172.21.15.157:3300/0 conn(0x7fed700cd4d0 0x7fed70072070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:50:00.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.453+0000 7fed67fff640 1 -- 172.21.15.150:0/496485441 >> v1:172.21.15.150:6789/0 conn(0x7fed700725b0 legacy=0x7fed700cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.453+0000 7fed67fff640 1 --2- 172.21.15.150:0/496485441 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fed700cf590 0x7fed5c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.453+0000 7fed67fff640 1 --2- 172.21.15.150:0/496485441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed5c003f30 0x7fed5c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.453+0000 7fed67fff640 1 --2- 172.21.15.150:0/496485441 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed5c004d40 0x7fed5c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.453+0000 7fed67fff640 1 -- 172.21.15.150:0/496485441 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fed7015b260 con 0x7fed5c003f30 2024-03-17T23:50:00.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.453+0000 7fed67fff640 1 -- 172.21.15.150:0/496485441 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fed7015b540 con 0x7fed5c004d40 2024-03-17T23:50:00.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.453+0000 7fed67fff640 1 -- 172.21.15.150:0/496485441 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fed5c00dbd0 con 0x7fed700cf590 2024-03-17T23:50:00.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.454+0000 7fed7480e640 1 --2- 172.21.15.150:0/496485441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed5c003f30 0x7fed5c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.454+0000 7fed7500f640 1 --2- 172.21.15.150:0/496485441 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fed700cf590 0x7fed5c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.454+0000 7fed75810640 1 --2- 172.21.15.150:0/496485441 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed5c004d40 0x7fed5c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.454+0000 7fed7480e640 1 -- 172.21.15.150:0/496485441 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fed700cf590 msgr2=0x7fed5c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:50:00.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.454+0000 7fed7480e640 1 --2- 172.21.15.150:0/496485441 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fed700cf590 0x7fed5c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.454+0000 7fed7480e640 1 -- 172.21.15.150:0/496485441 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed5c004d40 msgr2=0x7fed5c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:50:00.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.454+0000 7fed7480e640 1 --2- 172.21.15.150:0/496485441 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed5c004d40 0x7fed5c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.454+0000 7fed7480e640 1 -- 172.21.15.150:0/496485441 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed5c003520 con 0x7fed5c003f30 2024-03-17T23:50:00.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.455+0000 7fed7480e640 1 --2- 172.21.15.150:0/496485441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed5c003f30 0x7fed5c004800 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fed60007fd0 tx=0x7fed5c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d1be832b1e49dd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:00.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.455+0000 7fed67fff640 1 -- 172.21.15.150:0/496485441 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed60007d50 con 0x7fed5c003f30 2024-03-17T23:50:00.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.455+0000 7fed67fff640 1 -- 172.21.15.150:0/496485441 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed60030070 con 0x7fed5c003f30 2024-03-17T23:50:00.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.455+0000 7fed67fff640 1 -- 172.21.15.150:0/496485441 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6002cd20 con 0x7fed5c003f30 2024-03-17T23:50:00.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.455+0000 7fed77273640 1 -- 172.21.15.150:0/496485441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed5c003f30 msgr2=0x7fed5c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.455+0000 7fed77273640 1 --2- 172.21.15.150:0/496485441 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed5c003f30 0x7fed5c004800 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fed60007fd0 tx=0x7fed5c005830 comp rx=0 tx=0).stop 2024-03-17T23:50:00.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.455+0000 7fed75810640 1 -- 172.21.15.150:0/496485441 reap_dead start 2024-03-17T23:50:00.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.455+0000 7fed77273640 1 -- 172.21.15.150:0/496485441 shutdown_connections 2024-03-17T23:50:00.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.455+0000 7fed77273640 1 -- 172.21.15.150:0/496485441 >> 172.21.15.150:0/496485441 conn(0x7fed700b1be0 msgr2=0x7fed700b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:00.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.455+0000 7fed77273640 1 -- 172.21.15.150:0/496485441 shutdown_connections 2024-03-17T23:50:00.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.455+0000 7fed77273640 1 -- 172.21.15.150:0/496485441 wait complete. 2024-03-17T23:50:00.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.456+0000 7fed77273640 1 Processor -- start 2024-03-17T23:50:00.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.456+0000 7fed77273640 1 -- start start 2024-03-17T23:50:00.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.457+0000 7fed77273640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed5c003f30 0x7fed70143430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.457+0000 7fed77273640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed70152250 0x7fed70152fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.457+0000 7fed77273640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fed701534f0 0x7fed7015d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.457+0000 7fed77273640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fed700cd8b0 con 0x7fed5c003f30 2024-03-17T23:50:00.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.457+0000 7fed77273640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7fed7015bde0 con 0x7fed70152250 2024-03-17T23:50:00.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.457+0000 7fed77273640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7fed7006d9d0 con 0x7fed701534f0 2024-03-17T23:50:00.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.457+0000 7fed7500f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed5c003f30 0x7fed70143430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.457+0000 7fed7500f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed5c003f30 0x7fed70143430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44982/0 (socket says 172.21.15.150:44982) 2024-03-17T23:50:00.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.457+0000 7fed7500f640 1 -- 172.21.15.150:0/1694534194 learned_addr learned my addr 172.21.15.150:0/1694534194 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:50:00.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.457+0000 7fed7480e640 1 --2- 172.21.15.150:0/1694534194 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed70152250 0x7fed70152fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.457+0000 7fed75810640 1 --2- 172.21.15.150:0/1694534194 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fed701534f0 0x7fed7015d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.458+0000 7fed7500f640 1 -- 172.21.15.150:0/1694534194 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fed701534f0 msgr2=0x7fed7015d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.458+0000 7fed7500f640 1 --2- 172.21.15.150:0/1694534194 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7fed701534f0 0x7fed7015d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.458+0000 7fed7500f640 1 -- 172.21.15.150:0/1694534194 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed70152250 msgr2=0x7fed70152fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.458+0000 7fed7500f640 1 --2- 172.21.15.150:0/1694534194 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed70152250 0x7fed70152fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.458+0000 7fed7500f640 1 -- 172.21.15.150:0/1694534194 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed701511b0 con 0x7fed5c003f30 2024-03-17T23:50:00.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.458+0000 7fed7500f640 1 --2- 172.21.15.150:0/1694534194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed5c003f30 0x7fed70143430 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fed6c011cd0 tx=0x7fed6c00f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:00.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.458+0000 7fed65ffb640 1 -- 172.21.15.150:0/1694534194 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6c00fcc0 con 0x7fed5c003f30 2024-03-17T23:50:00.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.458+0000 7fed65ffb640 1 -- 172.21.15.150:0/1694534194 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed6c00fe60 con 0x7fed5c003f30 2024-03-17T23:50:00.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.458+0000 7fed77273640 1 -- 172.21.15.150:0/1694534194 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed70151490 con 0x7fed5c003f30 2024-03-17T23:50:00.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.458+0000 7fed65ffb640 1 -- 172.21.15.150:0/1694534194 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6c004240 con 0x7fed5c003f30 2024-03-17T23:50:00.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.458+0000 7fed77273640 1 -- 172.21.15.150:0/1694534194 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed70151a50 con 0x7fed5c003f30 2024-03-17T23:50:00.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.459+0000 7fed77273640 1 -- 172.21.15.150:0/1694534194 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fed700688b0 con 0x7fed5c003f30 2024-03-17T23:50:00.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.459+0000 7fed65ffb640 1 -- 172.21.15.150:0/1694534194 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7fed6c012040 con 0x7fed5c003f30 2024-03-17T23:50:00.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.460+0000 7fed65ffb640 1 --2- 172.21.15.150:0/1694534194 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fed5406bc00 0x7fed5406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.460+0000 7fed65ffb640 1 -- 172.21.15.150:0/1694534194 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5335+0+0 (secure 0 0 0) 0x7fed6c08ace0 con 0x7fed5c003f30 2024-03-17T23:50:00.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.460+0000 7fed7480e640 1 --2- 172.21.15.150:0/1694534194 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fed5406bc00 0x7fed5406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.460+0000 7fed65ffb640 1 --2- 172.21.15.150:0/1694534194 >> [v2:172.21.15.157:6824/1224877787,v1:172.21.15.157:6825/1224877787] conn(0x7fed540720a0 0x7fed540744e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.460+0000 7fed65ffb640 1 -- 172.21.15.150:0/1694534194 --> [v2:172.21.15.157:6824/1224877787,v1:172.21.15.157:6825/1224877787] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fed6c015e50 con 0x7fed540720a0 2024-03-17T23:50:00.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.460+0000 7fed65ffb640 1 -- 172.21.15.150:0/1694534194 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7fed6c08d3e0 con 0x7fed5c003f30 2024-03-17T23:50:00.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.460+0000 7fed75810640 1 --2- 172.21.15.150:0/1694534194 >> [v2:172.21.15.157:6824/1224877787,v1:172.21.15.157:6825/1224877787] conn(0x7fed540720a0 0x7fed540744e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.460+0000 7fed7480e640 1 --2- 172.21.15.150:0/1694534194 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fed5406bc00 0x7fed5406e0c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fed60034000 tx=0x7fed60007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:00.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.461+0000 7fed75810640 1 --2- 172.21.15.150:0/1694534194 >> [v2:172.21.15.157:6824/1224877787,v1:172.21.15.157:6825/1224877787] conn(0x7fed540720a0 0x7fed540744e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:00.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.461+0000 7fed65ffb640 1 -- 172.21.15.150:0/1694534194 <== osd.5 v2:172.21.15.157:6824/1224877787 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed6c015e50 con 0x7fed540720a0 2024-03-17T23:50:00.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.474+0000 7fed77273640 1 -- 172.21.15.150:0/1694534194 --> [v2:172.21.15.157:6824/1224877787,v1:172.21.15.157:6825/1224877787] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fed700681a0 con 0x7fed540720a0 2024-03-17T23:50:00.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.475+0000 7fed65ffb640 1 -- 172.21.15.150:0/1694534194 <== osd.5 v2:172.21.15.157:6824/1224877787 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fed700681a0 con 0x7fed540720a0 2024-03-17T23:50:00.475 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-17T23:50:00.476 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-17T23:50:00.476 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-17T23:50:00.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.475+0000 7fed77273640 1 -- 172.21.15.150:0/1694534194 >> [v2:172.21.15.157:6824/1224877787,v1:172.21.15.157:6825/1224877787] conn(0x7fed540720a0 msgr2=0x7fed540744e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.475+0000 7fed77273640 1 --2- 172.21.15.150:0/1694534194 >> [v2:172.21.15.157:6824/1224877787,v1:172.21.15.157:6825/1224877787] conn(0x7fed540720a0 0x7fed540744e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.475+0000 7fed77273640 1 -- 172.21.15.150:0/1694534194 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fed5406bc00 msgr2=0x7fed5406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.475+0000 7fed77273640 1 --2- 172.21.15.150:0/1694534194 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7fed5406bc00 0x7fed5406e0c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fed60034000 tx=0x7fed60007000 comp rx=0 tx=0).stop 2024-03-17T23:50:00.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.475+0000 7fed77273640 1 -- 172.21.15.150:0/1694534194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed5c003f30 msgr2=0x7fed70143430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.475+0000 7fed77273640 1 --2- 172.21.15.150:0/1694534194 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fed5c003f30 0x7fed70143430 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fed6c011cd0 tx=0x7fed6c00f610 comp rx=0 tx=0).stop 2024-03-17T23:50:00.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.475+0000 7fed75810640 1 -- 172.21.15.150:0/1694534194 reap_dead start 2024-03-17T23:50:00.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.475+0000 7fed77273640 1 -- 172.21.15.150:0/1694534194 shutdown_connections 2024-03-17T23:50:00.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.475+0000 7fed77273640 1 -- 172.21.15.150:0/1694534194 >> 172.21.15.150:0/1694534194 conn(0x7fed700b1be0 msgr2=0x7fed7015aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:00.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.475+0000 7fed77273640 1 -- 172.21.15.150:0/1694534194 shutdown_connections 2024-03-17T23:50:00.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.476+0000 7fed77273640 1 -- 172.21.15.150:0/1694534194 wait complete. 2024-03-17T23:50:00.486 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-17T23:50:00.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.587+0000 7f6d70c9a640 1 Processor -- start 2024-03-17T23:50:00.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.587+0000 7f6d70c9a640 1 -- start start 2024-03-17T23:50:00.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.588+0000 7f6d70c9a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6d6c147830 0x7f6d6c099a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.588+0000 7f6d70c9a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6d6c0b59e0 0x7f6d6c0d1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.588+0000 7f6d70c9a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d6c0d19d0 con 0x7f6d6c0d1580 2024-03-17T23:50:00.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.588+0000 7f6d70c9a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f6d6c06fe90 con 0x7f6d6c0b59e0 2024-03-17T23:50:00.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.588+0000 7f6d70c9a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6d6c15e440 con 0x7f6d6c147830 2024-03-17T23:50:00.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.588+0000 7f6d69d9b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6d6c0b59e0 0x7f6d6c0d1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.588+0000 7f6d6ad9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f6d6c0d1580 0x7f6d6c06b190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:58736/0 (socket says 172.21.15.150:58736) 2024-03-17T23:50:00.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.588+0000 7f6d6ad9d640 1 -- 172.21.15.150:0/2223692939 learned_addr learned my addr 172.21.15.150:0/2223692939 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:50:00.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.589+0000 7f6d6a59c640 1 --2- 172.21.15.150:0/2223692939 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6d6c147830 0x7f6d6c099a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.589+0000 7f6d69d9b640 1 -- 172.21.15.150:0/2223692939 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6d6c147830 msgr2=0x7f6d6c099a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:50:00.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.589+0000 7f6d69d9b640 1 --2- 172.21.15.150:0/2223692939 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6d6c147830 0x7f6d6c099a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.589+0000 7f6d69d9b640 1 -- 172.21.15.150:0/2223692939 >> v1:172.21.15.150:6789/0 conn(0x7f6d6c0d1580 legacy=0x7f6d6c06b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.589+0000 7f6d69d9b640 1 -- 172.21.15.150:0/2223692939 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d6c09a310 con 0x7f6d6c0b59e0 2024-03-17T23:50:00.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.589+0000 7f6d6959a640 1 -- 172.21.15.150:0/2223692939 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3406264808 0 0) 0x7f6d6c0d19d0 con 0x7f6d6c0d1580 2024-03-17T23:50:00.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.589+0000 7f6d6a59c640 1 --2- 172.21.15.150:0/2223692939 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6d6c147830 0x7f6d6c099a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:50:00.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.589+0000 7f6d69d9b640 1 --2- 172.21.15.150:0/2223692939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d6c0b59e0 0x7f6d6c0d1040 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f6d600097a0 tx=0x7f6d6002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e365b5e4477dfb4b server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:00.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.589+0000 7f6d6959a640 1 -- 172.21.15.150:0/2223692939 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d600315a0 con 0x7f6d6c0b59e0 2024-03-17T23:50:00.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.590+0000 7f6d6959a640 1 -- 172.21.15.150:0/2223692939 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d60037070 con 0x7f6d6c0b59e0 2024-03-17T23:50:00.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.590+0000 7f6d6959a640 1 -- 172.21.15.150:0/2223692939 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d60031960 con 0x7f6d6c0b59e0 2024-03-17T23:50:00.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.590+0000 7f6d70c9a640 1 -- 172.21.15.150:0/2223692939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d6c0b59e0 msgr2=0x7f6d6c0d1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.590+0000 7f6d70c9a640 1 --2- 172.21.15.150:0/2223692939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d6c0b59e0 0x7f6d6c0d1040 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f6d600097a0 tx=0x7f6d6002b840 comp rx=0 tx=0).stop 2024-03-17T23:50:00.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.590+0000 7f6d70c9a640 1 -- 172.21.15.150:0/2223692939 shutdown_connections 2024-03-17T23:50:00.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.590+0000 7f6d70c9a640 1 --2- 172.21.15.150:0/2223692939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d6c0b59e0 0x7f6d6c0d1040 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.590+0000 7f6d70c9a640 1 --2- 172.21.15.150:0/2223692939 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6d6c147830 0x7f6d6c099a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.590+0000 7f6d70c9a640 1 -- 172.21.15.150:0/2223692939 >> 172.21.15.150:0/2223692939 conn(0x7f6d6c0b5ec0 msgr2=0x7f6d6c0b62d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:00.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.590+0000 7f6d70c9a640 1 -- 172.21.15.150:0/2223692939 shutdown_connections 2024-03-17T23:50:00.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.590+0000 7f6d70c9a640 1 -- 172.21.15.150:0/2223692939 wait complete. 2024-03-17T23:50:00.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.591+0000 7f6d70c9a640 1 Processor -- start 2024-03-17T23:50:00.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.592+0000 7f6d70c9a640 1 -- start start 2024-03-17T23:50:00.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.592+0000 7f6d70c9a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d6c0b59e0 0x7f6d6c13dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.592+0000 7f6d70c9a640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6d6c0d1580 0x7f6d6c15d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.592+0000 7f6d70c9a640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d6c147830 0x7f6d6c1609d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.592+0000 7f6d70c9a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6c15ecc0 con 0x7f6d6c0b59e0 2024-03-17T23:50:00.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.592+0000 7f6d70c9a640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6c15e9e0 con 0x7f6d6c147830 2024-03-17T23:50:00.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.592+0000 7f6d70c9a640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f6d6c099fb0 con 0x7f6d6c0d1580 2024-03-17T23:50:00.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.592+0000 7f6d6a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d6c0b59e0 0x7f6d6c13dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.592+0000 7f6d6a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d6c0b59e0 0x7f6d6c13dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45002/0 (socket says 172.21.15.150:45002) 2024-03-17T23:50:00.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.592+0000 7f6d6a59c640 1 -- 172.21.15.150:0/2415285018 learned_addr learned my addr 172.21.15.150:0/2415285018 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:50:00.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.593+0000 7f6d69d9b640 1 --2- 172.21.15.150:0/2415285018 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6d6c0d1580 0x7f6d6c15d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.593+0000 7f6d6ad9d640 1 --2- 172.21.15.150:0/2415285018 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d6c147830 0x7f6d6c1609d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.593+0000 7f6d6a59c640 1 -- 172.21.15.150:0/2415285018 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6d6c0d1580 msgr2=0x7f6d6c15d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.593+0000 7f6d6a59c640 1 --2- 172.21.15.150:0/2415285018 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6d6c0d1580 0x7f6d6c15d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.593+0000 7f6d6a59c640 1 -- 172.21.15.150:0/2415285018 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d6c147830 msgr2=0x7f6d6c1609d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.593+0000 7f6d6a59c640 1 --2- 172.21.15.150:0/2415285018 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d6c147830 0x7f6d6c1609d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.593+0000 7f6d6a59c640 1 -- 172.21.15.150:0/2415285018 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d6c15d5a0 con 0x7f6d6c0b59e0 2024-03-17T23:50:00.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.593+0000 7f6d6a59c640 1 --2- 172.21.15.150:0/2415285018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d6c0b59e0 0x7f6d6c13dbf0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c00cf60 tx=0x7f6d5c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:00.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.593+0000 7f6d5affd640 1 -- 172.21.15.150:0/2415285018 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c01a070 con 0x7f6d6c0b59e0 2024-03-17T23:50:00.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.593+0000 7f6d5affd640 1 -- 172.21.15.150:0/2415285018 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d5c00d840 con 0x7f6d6c0b59e0 2024-03-17T23:50:00.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.593+0000 7f6d70c9a640 1 -- 172.21.15.150:0/2415285018 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d6c15f930 con 0x7f6d6c0b59e0 2024-03-17T23:50:00.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.593+0000 7f6d5affd640 1 -- 172.21.15.150:0/2415285018 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c00dc10 con 0x7f6d6c0b59e0 2024-03-17T23:50:00.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.593+0000 7f6d70c9a640 1 -- 172.21.15.150:0/2415285018 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d6c15fef0 con 0x7f6d6c0b59e0 2024-03-17T23:50:00.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.594+0000 7f6d70c9a640 1 -- 172.21.15.150:0/2415285018 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d6c0681a0 con 0x7f6d6c0b59e0 2024-03-17T23:50:00.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.594+0000 7f6d5affd640 1 -- 172.21.15.150:0/2415285018 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f6d5c01d070 con 0x7f6d6c0b59e0 2024-03-17T23:50:00.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.595+0000 7f6d5affd640 1 --2- 172.21.15.150:0/2415285018 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6d4c06bbd0 0x7f6d4c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.595+0000 7f6d69d9b640 1 --2- 172.21.15.150:0/2415285018 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6d4c06bbd0 0x7f6d4c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.595+0000 7f6d5affd640 1 -- 172.21.15.150:0/2415285018 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5335+0+0 (secure 0 0 0) 0x7f6d5c08b030 con 0x7f6d6c0b59e0 2024-03-17T23:50:00.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.595+0000 7f6d69d9b640 1 --2- 172.21.15.150:0/2415285018 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6d4c06bbd0 0x7f6d4c06e090 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6d60036000 tx=0x7f6d60008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:00.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.599+0000 7f6d5affd640 1 -- 172.21.15.150:0/2415285018 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f6d5c0596d0 con 0x7f6d6c0b59e0 2024-03-17T23:50:00.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.774+0000 7f6d70c9a640 1 -- 172.21.15.150:0/2415285018 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f6d6c0008d0 con 0x7f6d4c06bbd0 2024-03-17T23:50:00.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.774+0000 7f6d5affd640 1 -- 172.21.15.150:0/2415285018 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f6d6c0008d0 con 0x7f6d4c06bbd0 2024-03-17T23:50:00.775 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 5 to deep-scrub 2024-03-17T23:50:00.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.776+0000 7f6d70c9a640 1 -- 172.21.15.150:0/2415285018 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6d4c06bbd0 msgr2=0x7f6d4c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.776+0000 7f6d70c9a640 1 --2- 172.21.15.150:0/2415285018 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6d4c06bbd0 0x7f6d4c06e090 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6d60036000 tx=0x7f6d60008030 comp rx=0 tx=0).stop 2024-03-17T23:50:00.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.776+0000 7f6d70c9a640 1 -- 172.21.15.150:0/2415285018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d6c0b59e0 msgr2=0x7f6d6c13dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.776+0000 7f6d70c9a640 1 --2- 172.21.15.150:0/2415285018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d6c0b59e0 0x7f6d6c13dbf0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c00cf60 tx=0x7f6d5c00cf90 comp rx=0 tx=0).stop 2024-03-17T23:50:00.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.776+0000 7f6d70c9a640 1 -- 172.21.15.150:0/2415285018 shutdown_connections 2024-03-17T23:50:00.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.776+0000 7f6d70c9a640 1 --2- 172.21.15.150:0/2415285018 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f6d4c06bbd0 0x7f6d4c06e090 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.776+0000 7f6d70c9a640 1 --2- 172.21.15.150:0/2415285018 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d6c147830 0x7f6d6c1609d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.776+0000 7f6d70c9a640 1 --2- 172.21.15.150:0/2415285018 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f6d6c0d1580 0x7f6d6c15d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.776+0000 7f6d70c9a640 1 --2- 172.21.15.150:0/2415285018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d6c0b59e0 0x7f6d6c13dbf0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.776+0000 7f6d70c9a640 1 -- 172.21.15.150:0/2415285018 >> 172.21.15.150:0/2415285018 conn(0x7f6d6c0b5ec0 msgr2=0x7f6d6c06ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:00.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.776+0000 7f6d70c9a640 1 -- 172.21.15.150:0/2415285018 shutdown_connections 2024-03-17T23:50:00.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.776+0000 7f6d70c9a640 1 -- 172.21.15.150:0/2415285018 wait complete. 2024-03-17T23:50:00.788 INFO:tasks.ceph:Scrubbing osd.6 2024-03-17T23:50:00.788 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-17T23:50:00.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.891+0000 7f804f9a9640 1 Processor -- start 2024-03-17T23:50:00.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.892+0000 7f804f9a9640 1 -- start start 2024-03-17T23:50:00.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.892+0000 7f804f9a9640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f804813d340 0x7f804815b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.892+0000 7f804f9a9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f804815b850 con 0x7f804806dc10 2024-03-17T23:50:00.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.893+0000 7f804f9a9640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80480ceab0 con 0x7f80480cc440 2024-03-17T23:50:00.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.893+0000 7f804f9a9640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f80480ced90 con 0x7f804813d340 2024-03-17T23:50:00.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.893+0000 7f804df46640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f804806dc10 0x7f804806e0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:58744/0 (socket says 172.21.15.150:58744) 2024-03-17T23:50:00.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.893+0000 7f804df46640 1 -- 172.21.15.150:0/3178815003 learned_addr learned my addr 172.21.15.150:0/3178815003 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:50:00.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.893+0000 7f804d745640 1 --2- 172.21.15.150:0/3178815003 >> v2:172.21.15.157:3300/0 conn(0x7f804813d340 0x7f804815b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.893+0000 7f803ffff640 1 -- 172.21.15.150:0/3178815003 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3059624837 0 0) 0x7f8040001490 con 0x7f804806dc10 2024-03-17T23:50:00.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.893+0000 7f803ffff640 1 -- 172.21.15.150:0/3178815003 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8034003520 con 0x7f804806dc10 2024-03-17T23:50:00.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.893+0000 7f804d745640 1 -- 172.21.15.150:0/3178815003 >> v1:172.21.15.157:6789/0 conn(0x7f80480cc440 legacy=0x7f80480cc8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.893+0000 7f803ffff640 1 -- 172.21.15.150:0/3178815003 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 976920249 0 0) 0x7f8034003520 con 0x7f804806dc10 2024-03-17T23:50:00.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.893+0000 7f804d745640 1 -- 172.21.15.150:0/3178815003 >> v1:172.21.15.150:6789/0 conn(0x7f804806dc10 legacy=0x7f804806e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.893+0000 7f804d745640 1 -- 172.21.15.150:0/3178815003 --> v2:172.21.15.157:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80480cf070 con 0x7f804813d340 2024-03-17T23:50:00.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.894+0000 7f804d745640 1 --2- 172.21.15.150:0/3178815003 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f804813d340 0x7f804815b310 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f80440097a0 tx=0x7f804402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2fbf8712cfa6872 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:00.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.894+0000 7f803ffff640 1 -- 172.21.15.150:0/3178815003 <== mon.1 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8044038070 con 0x7f804813d340 2024-03-17T23:50:00.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.894+0000 7f803ffff640 1 -- 172.21.15.150:0/3178815003 <== mon.1 v2:172.21.15.157:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8044031500 con 0x7f804813d340 2024-03-17T23:50:00.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.894+0000 7f803ffff640 1 -- 172.21.15.150:0/3178815003 <== mon.1 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8040001bc0 con 0x7f804813d340 2024-03-17T23:50:00.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.895+0000 7f804f9a9640 1 -- 172.21.15.150:0/3178815003 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f804813d340 msgr2=0x7f804815b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.895+0000 7f804f9a9640 1 --2- 172.21.15.150:0/3178815003 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f804813d340 0x7f804815b310 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f80440097a0 tx=0x7f804402b820 comp rx=0 tx=0).stop 2024-03-17T23:50:00.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.895+0000 7f804f9a9640 1 -- 172.21.15.150:0/3178815003 shutdown_connections 2024-03-17T23:50:00.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.895+0000 7f804f9a9640 1 --2- 172.21.15.150:0/3178815003 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f804813d340 0x7f804815b310 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.895+0000 7f804f9a9640 1 -- 172.21.15.150:0/3178815003 >> 172.21.15.150:0/3178815003 conn(0x7f80480b1be0 msgr2=0x7f80480b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:00.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.895+0000 7f804f9a9640 1 -- 172.21.15.150:0/3178815003 shutdown_connections 2024-03-17T23:50:00.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.895+0000 7f804f9a9640 1 -- 172.21.15.150:0/3178815003 wait complete. 2024-03-17T23:50:00.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.896+0000 7f804f9a9640 1 Processor -- start 2024-03-17T23:50:00.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.896+0000 7f804f9a9640 1 -- start start 2024-03-17T23:50:00.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.897+0000 7f804f9a9640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f804806dc10 0x7f8048152710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.897+0000 7f804f9a9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80480cc440 0x7f80481f25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.897+0000 7f804f9a9640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f804813d340 0x7f80480b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.897+0000 7f804f9a9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f804813d720 con 0x7f80480cc440 2024-03-17T23:50:00.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.897+0000 7f804f9a9640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f80480cf630 con 0x7f804813d340 2024-03-17T23:50:00.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.897+0000 7f804f9a9640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f80480cbe70 con 0x7f804806dc10 2024-03-17T23:50:00.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.897+0000 7f804cf44640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80480cc440 0x7f80481f25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.897+0000 7f804cf44640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80480cc440 0x7f80481f25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45014/0 (socket says 172.21.15.150:45014) 2024-03-17T23:50:00.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.897+0000 7f804cf44640 1 -- 172.21.15.150:0/490833095 learned_addr learned my addr 172.21.15.150:0/490833095 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:50:00.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.897+0000 7f804df46640 1 --2- 172.21.15.150:0/490833095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f804813d340 0x7f80480b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.897+0000 7f804d745640 1 --2- 172.21.15.150:0/490833095 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f804806dc10 0x7f8048152710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.897+0000 7f804cf44640 1 -- 172.21.15.150:0/490833095 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f804806dc10 msgr2=0x7f8048152710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.897+0000 7f804cf44640 1 --2- 172.21.15.150:0/490833095 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f804806dc10 0x7f8048152710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.897+0000 7f804cf44640 1 -- 172.21.15.150:0/490833095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f804813d340 msgr2=0x7f80480b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.897+0000 7f804cf44640 1 --2- 172.21.15.150:0/490833095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f804813d340 0x7f80480b3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.897+0000 7f804cf44640 1 -- 172.21.15.150:0/490833095 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80480b40c0 con 0x7f80480cc440 2024-03-17T23:50:00.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.897+0000 7f804df46640 1 --2- 172.21.15.150:0/490833095 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f804813d340 0x7f80480b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:50:00.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.897+0000 7f804cf44640 1 --2- 172.21.15.150:0/490833095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80480cc440 0x7f80481f25b0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f803800c450 tx=0x7f803800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:00.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.897+0000 7f803dffb640 1 -- 172.21.15.150:0/490833095 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80380052d0 con 0x7f80480cc440 2024-03-17T23:50:00.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.897+0000 7f803dffb640 1 -- 172.21.15.150:0/490833095 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8038005470 con 0x7f80480cc440 2024-03-17T23:50:00.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.897+0000 7f804f9a9640 1 -- 172.21.15.150:0/490833095 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80480b4340 con 0x7f80480cc440 2024-03-17T23:50:00.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.897+0000 7f803dffb640 1 -- 172.21.15.150:0/490833095 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8038012870 con 0x7f80480cc440 2024-03-17T23:50:00.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.898+0000 7f804f9a9640 1 -- 172.21.15.150:0/490833095 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80480b4900 con 0x7f80480cc440 2024-03-17T23:50:00.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.898+0000 7f804f9a9640 1 -- 172.21.15.150:0/490833095 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f80480681a0 con 0x7f80480cc440 2024-03-17T23:50:00.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.899+0000 7f803dffb640 1 -- 172.21.15.150:0/490833095 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f8038012a10 con 0x7f80480cc440 2024-03-17T23:50:00.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.899+0000 7f803dffb640 1 --2- 172.21.15.150:0/490833095 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f802c06bbd0 0x7f802c06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.899+0000 7f803dffb640 1 -- 172.21.15.150:0/490833095 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5335+0+0 (secure 0 0 0) 0x7f803808a530 con 0x7f80480cc440 2024-03-17T23:50:00.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.899+0000 7f804d745640 1 --2- 172.21.15.150:0/490833095 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f802c06bbd0 0x7f802c06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.899+0000 7f803dffb640 1 --2- 172.21.15.150:0/490833095 >> [v2:172.21.15.157:6809/2764449896,v1:172.21.15.157:6811/2764449896] conn(0x7f802c072070 0x7f802c0744b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:00.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.899+0000 7f803dffb640 1 -- 172.21.15.150:0/490833095 --> [v2:172.21.15.157:6809/2764449896,v1:172.21.15.157:6811/2764449896] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f802c074b80 con 0x7f802c072070 2024-03-17T23:50:00.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.899+0000 7f803dffb640 1 -- 172.21.15.150:0/490833095 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7f8038091050 con 0x7f80480cc440 2024-03-17T23:50:00.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.900+0000 7f804df46640 1 --2- 172.21.15.150:0/490833095 >> [v2:172.21.15.157:6809/2764449896,v1:172.21.15.157:6811/2764449896] conn(0x7f802c072070 0x7f802c0744b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:00.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.900+0000 7f804d745640 1 --2- 172.21.15.150:0/490833095 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f802c06bbd0 0x7f802c06e090 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f8044035000 tx=0x7f8044008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:00.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.900+0000 7f804df46640 1 --2- 172.21.15.150:0/490833095 >> [v2:172.21.15.157:6809/2764449896,v1:172.21.15.157:6811/2764449896] conn(0x7f802c072070 0x7f802c0744b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:00.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.901+0000 7f803dffb640 1 -- 172.21.15.150:0/490833095 <== osd.6 v2:172.21.15.157:6809/2764449896 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f802c074b80 con 0x7f802c072070 2024-03-17T23:50:00.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.914+0000 7f804f9a9640 1 -- 172.21.15.150:0/490833095 --> [v2:172.21.15.157:6809/2764449896,v1:172.21.15.157:6811/2764449896] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f80480b4d20 con 0x7f802c072070 2024-03-17T23:50:00.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.914+0000 7f803dffb640 1 -- 172.21.15.150:0/490833095 <== osd.6 v2:172.21.15.157:6809/2764449896 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f80480b4d20 con 0x7f802c072070 2024-03-17T23:50:00.915 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-17T23:50:00.915 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-17T23:50:00.916 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-17T23:50:00.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.915+0000 7f804f9a9640 1 -- 172.21.15.150:0/490833095 >> [v2:172.21.15.157:6809/2764449896,v1:172.21.15.157:6811/2764449896] conn(0x7f802c072070 msgr2=0x7f802c0744b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.915+0000 7f804f9a9640 1 --2- 172.21.15.150:0/490833095 >> [v2:172.21.15.157:6809/2764449896,v1:172.21.15.157:6811/2764449896] conn(0x7f802c072070 0x7f802c0744b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:00.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.915+0000 7f804f9a9640 1 -- 172.21.15.150:0/490833095 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f802c06bbd0 msgr2=0x7f802c06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.915+0000 7f804f9a9640 1 --2- 172.21.15.150:0/490833095 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f802c06bbd0 0x7f802c06e090 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f8044035000 tx=0x7f8044008030 comp rx=0 tx=0).stop 2024-03-17T23:50:00.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.915+0000 7f804f9a9640 1 -- 172.21.15.150:0/490833095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80480cc440 msgr2=0x7f80481f25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:00.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.915+0000 7f804f9a9640 1 --2- 172.21.15.150:0/490833095 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80480cc440 0x7f80481f25b0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f803800c450 tx=0x7f803800c910 comp rx=0 tx=0).stop 2024-03-17T23:50:00.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.915+0000 7f804df46640 1 -- 172.21.15.150:0/490833095 reap_dead start 2024-03-17T23:50:00.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.915+0000 7f804f9a9640 1 -- 172.21.15.150:0/490833095 shutdown_connections 2024-03-17T23:50:00.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.915+0000 7f804f9a9640 1 -- 172.21.15.150:0/490833095 >> 172.21.15.150:0/490833095 conn(0x7f80480b1be0 msgr2=0x7f8048153370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:00.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.915+0000 7f804f9a9640 1 -- 172.21.15.150:0/490833095 shutdown_connections 2024-03-17T23:50:00.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:00.915+0000 7f804f9a9640 1 -- 172.21.15.150:0/490833095 wait complete. 2024-03-17T23:50:00.926 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-17T23:50:01.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.027+0000 7faedac80640 1 Processor -- start 2024-03-17T23:50:01.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.027+0000 7faedac80640 1 -- start start 2024-03-17T23:50:01.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.028+0000 7faedac80640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7faed406ccb0 0x7faed409a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:01.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.028+0000 7faedac80640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faed406d090 0x7faed415e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:01.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.028+0000 7faedac80640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7faed4161d10 0x7faed41621a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:01.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.028+0000 7faedac80640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7faed40d0030 con 0x7faed4161d10 2024-03-17T23:50:01.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.028+0000 7faedac80640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7faed406c750 con 0x7faed406ccb0 2024-03-17T23:50:01.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.028+0000 7faedac80640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7faed415edf0 con 0x7faed406d090 2024-03-17T23:50:01.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.028+0000 7faed921d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7faed4161d10 0x7faed41621a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:01.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.028+0000 7faed921d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7faed4161d10 0x7faed41621a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45028/0 (socket says 172.21.15.150:45028) 2024-03-17T23:50:01.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.028+0000 7faed921d640 1 -- 172.21.15.150:0/1048687921 learned_addr learned my addr 172.21.15.150:0/1048687921 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:50:01.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.028+0000 7faed8a1c640 1 --2- 172.21.15.150:0/1048687921 >> v2:172.21.15.157:3300/0 conn(0x7faed406ccb0 0x7faed409a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:01.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.028+0000 7faed3fff640 1 --2- 172.21.15.150:0/1048687921 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faed406d090 0x7faed415e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:01.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.029+0000 7faed921d640 1 -- 172.21.15.150:0/1048687921 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faed406d090 msgr2=0x7faed415e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:50:01.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.029+0000 7faed921d640 1 --2- 172.21.15.150:0/1048687921 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faed406d090 0x7faed415e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.029+0000 7faed921d640 1 -- 172.21.15.150:0/1048687921 >> v2:172.21.15.157:3300/0 conn(0x7faed406ccb0 msgr2=0x7faed409a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:50:01.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.029+0000 7faed921d640 1 --2- 172.21.15.150:0/1048687921 >> v2:172.21.15.157:3300/0 conn(0x7faed406ccb0 0x7faed409a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.029+0000 7faed921d640 1 -- 172.21.15.150:0/1048687921 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed415f0d0 con 0x7faed4161d10 2024-03-17T23:50:01.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.029+0000 7faed921d640 1 --2- 172.21.15.150:0/1048687921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faed4161d10 0x7faed41621a0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7faed4073550 tx=0x7faecc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d194223ee25b8aad server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:01.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.030+0000 7faed37fe640 1 -- 172.21.15.150:0/1048687921 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faecc02ca70 con 0x7faed4161d10 2024-03-17T23:50:01.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.030+0000 7faed37fe640 1 -- 172.21.15.150:0/1048687921 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faecc02cc10 con 0x7faed4161d10 2024-03-17T23:50:01.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.030+0000 7faed37fe640 1 -- 172.21.15.150:0/1048687921 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faecc0343f0 con 0x7faed4161d10 2024-03-17T23:50:01.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.030+0000 7faedac80640 1 -- 172.21.15.150:0/1048687921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faed4161d10 msgr2=0x7faed41621a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:01.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.030+0000 7faedac80640 1 --2- 172.21.15.150:0/1048687921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faed4161d10 0x7faed41621a0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7faed4073550 tx=0x7faecc02bed0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.030+0000 7faedac80640 1 -- 172.21.15.150:0/1048687921 shutdown_connections 2024-03-17T23:50:01.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.030+0000 7faedac80640 1 --2- 172.21.15.150:0/1048687921 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faed4161d10 0x7faed41621a0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.030+0000 7faedac80640 1 --2- 172.21.15.150:0/1048687921 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faed406d090 0x7faed415e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.030+0000 7faedac80640 1 --2- 172.21.15.150:0/1048687921 >> v2:172.21.15.157:3300/0 conn(0x7faed406ccb0 0x7faed409a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.030+0000 7faedac80640 1 -- 172.21.15.150:0/1048687921 >> 172.21.15.150:0/1048687921 conn(0x7faed40b36b0 msgr2=0x7faed40b3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:01.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.030+0000 7faedac80640 1 -- 172.21.15.150:0/1048687921 shutdown_connections 2024-03-17T23:50:01.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.030+0000 7faedac80640 1 -- 172.21.15.150:0/1048687921 wait complete. 2024-03-17T23:50:01.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.031+0000 7faedac80640 1 Processor -- start 2024-03-17T23:50:01.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.031+0000 7faedac80640 1 -- start start 2024-03-17T23:50:01.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.032+0000 7faedac80640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faed406ccb0 0x7faed40b5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:01.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.032+0000 7faedac80640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faed406d090 0x7faed41f5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:01.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.032+0000 7faedac80640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faed4161d10 0x7faed409a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:01.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.032+0000 7faedac80640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7faed406d4f0 con 0x7faed406d090 2024-03-17T23:50:01.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.032+0000 7faedac80640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7faed415f690 con 0x7faed4161d10 2024-03-17T23:50:01.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.032+0000 7faedac80640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7faed415ea10 con 0x7faed406ccb0 2024-03-17T23:50:01.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.032+0000 7faed3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faed406d090 0x7faed41f5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:01.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.032+0000 7faed3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faed406d090 0x7faed41f5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45038/0 (socket says 172.21.15.150:45038) 2024-03-17T23:50:01.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.032+0000 7faed3fff640 1 -- 172.21.15.150:0/2496296384 learned_addr learned my addr 172.21.15.150:0/2496296384 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:50:01.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.032+0000 7faed921d640 1 --2- 172.21.15.150:0/2496296384 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faed4161d10 0x7faed409a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:01.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.033+0000 7faed8a1c640 1 --2- 172.21.15.150:0/2496296384 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faed406ccb0 0x7faed40b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:01.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.033+0000 7faed3fff640 1 -- 172.21.15.150:0/2496296384 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faed406ccb0 msgr2=0x7faed40b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:01.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.033+0000 7faed3fff640 1 --2- 172.21.15.150:0/2496296384 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faed406ccb0 0x7faed40b5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.033+0000 7faed3fff640 1 -- 172.21.15.150:0/2496296384 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faed4161d10 msgr2=0x7faed409a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:01.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.033+0000 7faed3fff640 1 --2- 172.21.15.150:0/2496296384 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faed4161d10 0x7faed409a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.033+0000 7faed3fff640 1 -- 172.21.15.150:0/2496296384 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed415a220 con 0x7faed406d090 2024-03-17T23:50:01.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.033+0000 7faed8a1c640 1 --2- 172.21.15.150:0/2496296384 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faed406ccb0 0x7faed40b5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:50:01.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.033+0000 7faed3fff640 1 --2- 172.21.15.150:0/2496296384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faed406d090 0x7faed41f5040 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7faec800e9b0 tx=0x7faec800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:01.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.033+0000 7faed17fa640 1 -- 172.21.15.150:0/2496296384 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec8017070 con 0x7faed406d090 2024-03-17T23:50:01.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.033+0000 7faed17fa640 1 -- 172.21.15.150:0/2496296384 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faec800c9a0 con 0x7faed406d090 2024-03-17T23:50:01.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.033+0000 7faedac80640 1 -- 172.21.15.150:0/2496296384 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faed415a500 con 0x7faed406d090 2024-03-17T23:50:01.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.033+0000 7faed17fa640 1 -- 172.21.15.150:0/2496296384 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec800cd90 con 0x7faed406d090 2024-03-17T23:50:01.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.033+0000 7faedac80640 1 -- 172.21.15.150:0/2496296384 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faed415aac0 con 0x7faed406d090 2024-03-17T23:50:01.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.034+0000 7faedac80640 1 -- 172.21.15.150:0/2496296384 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faed40681a0 con 0x7faed406d090 2024-03-17T23:50:01.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.034+0000 7faed17fa640 1 -- 172.21.15.150:0/2496296384 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7faec801a070 con 0x7faed406d090 2024-03-17T23:50:01.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.035+0000 7faed17fa640 1 --2- 172.21.15.150:0/2496296384 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7faec006bbd0 0x7faec006e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:01.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.035+0000 7faed8a1c640 1 --2- 172.21.15.150:0/2496296384 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7faec006bbd0 0x7faec006e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:01.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.035+0000 7faed17fa640 1 -- 172.21.15.150:0/2496296384 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5335+0+0 (secure 0 0 0) 0x7faec808aba0 con 0x7faed406d090 2024-03-17T23:50:01.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.035+0000 7faed8a1c640 1 --2- 172.21.15.150:0/2496296384 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7faec006bbd0 0x7faec006e090 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7faed4071280 tx=0x7faec4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:01.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.039+0000 7faed17fa640 1 -- 172.21.15.150:0/2496296384 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7faec8059240 con 0x7faed406d090 2024-03-17T23:50:01.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.212+0000 7faedac80640 1 -- 172.21.15.150:0/2496296384 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7faed40b4550 con 0x7faec006bbd0 2024-03-17T23:50:01.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.212+0000 7faed17fa640 1 -- 172.21.15.150:0/2496296384 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7faed40b4550 con 0x7faec006bbd0 2024-03-17T23:50:01.213 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 6 to deep-scrub 2024-03-17T23:50:01.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.214+0000 7faedac80640 1 -- 172.21.15.150:0/2496296384 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7faec006bbd0 msgr2=0x7faec006e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:01.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.214+0000 7faedac80640 1 --2- 172.21.15.150:0/2496296384 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7faec006bbd0 0x7faec006e090 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7faed4071280 tx=0x7faec4008040 comp rx=0 tx=0).stop 2024-03-17T23:50:01.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.214+0000 7faedac80640 1 -- 172.21.15.150:0/2496296384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faed406d090 msgr2=0x7faed41f5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:01.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.214+0000 7faedac80640 1 --2- 172.21.15.150:0/2496296384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faed406d090 0x7faed41f5040 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7faec800e9b0 tx=0x7faec800ee70 comp rx=0 tx=0).stop 2024-03-17T23:50:01.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.214+0000 7faedac80640 1 -- 172.21.15.150:0/2496296384 shutdown_connections 2024-03-17T23:50:01.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.214+0000 7faedac80640 1 --2- 172.21.15.150:0/2496296384 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7faec006bbd0 0x7faec006e090 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.214+0000 7faedac80640 1 --2- 172.21.15.150:0/2496296384 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faed4161d10 0x7faed409a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.214+0000 7faedac80640 1 --2- 172.21.15.150:0/2496296384 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faed406d090 0x7faed41f5040 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.214+0000 7faedac80640 1 --2- 172.21.15.150:0/2496296384 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7faed406ccb0 0x7faed40b5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.214+0000 7faedac80640 1 -- 172.21.15.150:0/2496296384 >> 172.21.15.150:0/2496296384 conn(0x7faed40b36b0 msgr2=0x7faed40b0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:01.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.214+0000 7faedac80640 1 -- 172.21.15.150:0/2496296384 shutdown_connections 2024-03-17T23:50:01.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.214+0000 7faedac80640 1 -- 172.21.15.150:0/2496296384 wait complete. 2024-03-17T23:50:01.225 INFO:tasks.ceph:Scrubbing osd.7 2024-03-17T23:50:01.225 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-17T23:50:01.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.330+0000 7f930cf3b640 1 Processor -- start 2024-03-17T23:50:01.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.330+0000 7f930cf3b640 1 -- start start 2024-03-17T23:50:01.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.331+0000 7f930cf3b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f930813d340 0x7f930815b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:01.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.331+0000 7f930cf3b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f93080cc440 0x7f93080cc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:01.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.331+0000 7f930cf3b640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93080cce00 con 0x7f930815de80 2024-03-17T23:50:01.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.331+0000 7f930cf3b640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f930815b850 con 0x7f93080cc440 2024-03-17T23:50:01.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.331+0000 7f930cf3b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f93080ceab0 con 0x7f930813d340 2024-03-17T23:50:01.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.331+0000 7f930659c640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f930813d340 0x7f930815b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:01.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.331+0000 7f9305d9b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f93080cc440 0x7f93080cc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:01.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.331+0000 7f9305d9b640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f93080cc440 0x7f93080cc8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:59670/0 (socket says 172.21.15.150:59670) 2024-03-17T23:50:01.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.331+0000 7f9306d9d640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f930815de80 0x7f930815e2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:45930/0 (socket says 172.21.15.150:45930) 2024-03-17T23:50:01.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.331+0000 7f9305d9b640 1 -- 172.21.15.150:0/2912431473 learned_addr learned my addr 172.21.15.150:0/2912431473 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:50:01.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.331+0000 7f930659c640 1 -- 172.21.15.150:0/2912431473 >> v2:172.21.15.157:3300/0 conn(0x7f93080cc440 msgr2=0x7f93080cc8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:50:01.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.331+0000 7f930659c640 1 --2- 172.21.15.150:0/2912431473 >> v2:172.21.15.157:3300/0 conn(0x7f93080cc440 0x7f93080cc8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.331+0000 7f930659c640 1 -- 172.21.15.150:0/2912431473 >> v1:172.21.15.157:6789/0 conn(0x7f930815de80 legacy=0x7f930815e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:01.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.331+0000 7f930659c640 1 -- 172.21.15.150:0/2912431473 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93080ced90 con 0x7f930813d340 2024-03-17T23:50:01.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.331+0000 7f9305d9b640 1 --2- 172.21.15.150:0/2912431473 >> v2:172.21.15.157:3300/0 conn(0x7f93080cc440 0x7f93080cc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:50:01.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.332+0000 7f930659c640 1 --2- 172.21.15.150:0/2912431473 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f930813d340 0x7f930815b310 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f92f80097a0 tx=0x7f92f802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1c487bd14484ae1 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:01.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.332+0000 7f930559a640 1 -- 172.21.15.150:0/2912431473 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f8035020 con 0x7f930813d340 2024-03-17T23:50:01.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.332+0000 7f930559a640 1 -- 172.21.15.150:0/2912431473 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92f8031830 con 0x7f930813d340 2024-03-17T23:50:01.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.332+0000 7f930559a640 1 -- 172.21.15.150:0/2912431473 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f8031c00 con 0x7f930813d340 2024-03-17T23:50:01.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.333+0000 7f930cf3b640 1 -- 172.21.15.150:0/2912431473 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f930813d340 msgr2=0x7f930815b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:01.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.333+0000 7f930cf3b640 1 --2- 172.21.15.150:0/2912431473 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f930813d340 0x7f930815b310 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f92f80097a0 tx=0x7f92f802b8d0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.333+0000 7f930cf3b640 1 -- 172.21.15.150:0/2912431473 shutdown_connections 2024-03-17T23:50:01.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.333+0000 7f930cf3b640 1 --2- 172.21.15.150:0/2912431473 >> v2:172.21.15.157:3300/0 conn(0x7f93080cc440 0x7f93080cc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.333+0000 7f930cf3b640 1 --2- 172.21.15.150:0/2912431473 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f930813d340 0x7f930815b310 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.333+0000 7f930cf3b640 1 -- 172.21.15.150:0/2912431473 >> 172.21.15.150:0/2912431473 conn(0x7f93080b1be0 msgr2=0x7f93080b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:01.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.333+0000 7f930cf3b640 1 -- 172.21.15.150:0/2912431473 shutdown_connections 2024-03-17T23:50:01.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.333+0000 7f930cf3b640 1 -- 172.21.15.150:0/2912431473 wait complete. 2024-03-17T23:50:01.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.334+0000 7f930cf3b640 1 Processor -- start 2024-03-17T23:50:01.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.334+0000 7f930cf3b640 1 -- start start 2024-03-17T23:50:01.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.335+0000 7f930cf3b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93080cc440 0x7f9308159630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:01.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.335+0000 7f930cf3b640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f930813d340 0x7f93081ef760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:01.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.335+0000 7f930cf3b640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f930815de80 0x7f93080caa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:01.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.335+0000 7f930cf3b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f93080cf630 con 0x7f93080cc440 2024-03-17T23:50:01.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.335+0000 7f930cf3b640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f93080cf350 con 0x7f930815de80 2024-03-17T23:50:01.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.335+0000 7f930cf3b640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f930813d720 con 0x7f930813d340 2024-03-17T23:50:01.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.335+0000 7f930659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93080cc440 0x7f9308159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:01.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.335+0000 7f930659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93080cc440 0x7f9308159630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45044/0 (socket says 172.21.15.150:45044) 2024-03-17T23:50:01.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.335+0000 7f930659c640 1 -- 172.21.15.150:0/2706482766 learned_addr learned my addr 172.21.15.150:0/2706482766 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:50:01.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.335+0000 7f9306d9d640 1 --2- 172.21.15.150:0/2706482766 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f930815de80 0x7f93080caa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:01.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.335+0000 7f9305d9b640 1 --2- 172.21.15.150:0/2706482766 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f930813d340 0x7f93081ef760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:01.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.336+0000 7f930659c640 1 -- 172.21.15.150:0/2706482766 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f930813d340 msgr2=0x7f93081ef760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:01.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.336+0000 7f930659c640 1 --2- 172.21.15.150:0/2706482766 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f930813d340 0x7f93081ef760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.336+0000 7f930659c640 1 -- 172.21.15.150:0/2706482766 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f930815de80 msgr2=0x7f93080caa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:01.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.336+0000 7f930659c640 1 --2- 172.21.15.150:0/2706482766 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f930815de80 0x7f93080caa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.336+0000 7f930659c640 1 -- 172.21.15.150:0/2706482766 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93080c5c00 con 0x7f93080cc440 2024-03-17T23:50:01.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.336+0000 7f930659c640 1 --2- 172.21.15.150:0/2706482766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93080cc440 0x7f9308159630 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f92f8000c00 tx=0x7f92f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:01.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.336+0000 7f92f6ffd640 1 -- 172.21.15.150:0/2706482766 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f803b980 con 0x7f93080cc440 2024-03-17T23:50:01.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.336+0000 7f92f6ffd640 1 -- 172.21.15.150:0/2706482766 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92f803bb40 con 0x7f93080cc440 2024-03-17T23:50:01.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.336+0000 7f92f6ffd640 1 -- 172.21.15.150:0/2706482766 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f80453f0 con 0x7f93080cc440 2024-03-17T23:50:01.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.336+0000 7f930cf3b640 1 -- 172.21.15.150:0/2706482766 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93080c5ee0 con 0x7f93080cc440 2024-03-17T23:50:01.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.336+0000 7f930cf3b640 1 -- 172.21.15.150:0/2706482766 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93080c64a0 con 0x7f93080cc440 2024-03-17T23:50:01.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.337+0000 7f930cf3b640 1 -- 172.21.15.150:0/2706482766 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f93080681a0 con 0x7f93080cc440 2024-03-17T23:50:01.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.337+0000 7f92f6ffd640 1 -- 172.21.15.150:0/2706482766 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f92f804c060 con 0x7f93080cc440 2024-03-17T23:50:01.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.338+0000 7f92f6ffd640 1 --2- 172.21.15.150:0/2706482766 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f92e806bc00 0x7f92e806e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:01.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.338+0000 7f92f6ffd640 1 -- 172.21.15.150:0/2706482766 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5335+0+0 (secure 0 0 0) 0x7f92f80ac390 con 0x7f93080cc440 2024-03-17T23:50:01.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.338+0000 7f9305d9b640 1 --2- 172.21.15.150:0/2706482766 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f92e806bc00 0x7f92e806e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:01.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.338+0000 7f92f6ffd640 1 --2- 172.21.15.150:0/2706482766 >> [v2:172.21.15.157:6800/624642970,v1:172.21.15.157:6801/624642970] conn(0x7f92e80720a0 0x7f92e80744e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:01.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.338+0000 7f92f6ffd640 1 -- 172.21.15.150:0/2706482766 --> [v2:172.21.15.157:6800/624642970,v1:172.21.15.157:6801/624642970] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f92e8074bb0 con 0x7f92e80720a0 2024-03-17T23:50:01.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.338+0000 7f92f6ffd640 1 -- 172.21.15.150:0/2706482766 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7f92f80b3050 con 0x7f93080cc440 2024-03-17T23:50:01.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.338+0000 7f9306d9d640 1 --2- 172.21.15.150:0/2706482766 >> [v2:172.21.15.157:6800/624642970,v1:172.21.15.157:6801/624642970] conn(0x7f92e80720a0 0x7f92e80744e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:01.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.338+0000 7f9305d9b640 1 --2- 172.21.15.150:0/2706482766 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f92e806bc00 0x7f92e806e0c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f930815d780 tx=0x7f92fc009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:01.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.339+0000 7f9306d9d640 1 --2- 172.21.15.150:0/2706482766 >> [v2:172.21.15.157:6800/624642970,v1:172.21.15.157:6801/624642970] conn(0x7f92e80720a0 0x7f92e80744e0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:01.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.339+0000 7f92f6ffd640 1 -- 172.21.15.150:0/2706482766 <== osd.7 v2:172.21.15.157:6800/624642970 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f92e8074bb0 con 0x7f92e80720a0 2024-03-17T23:50:01.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.352+0000 7f930cf3b640 1 -- 172.21.15.150:0/2706482766 --> [v2:172.21.15.157:6800/624642970,v1:172.21.15.157:6801/624642970] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f93081efef0 con 0x7f92e80720a0 2024-03-17T23:50:01.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.353+0000 7f92f6ffd640 1 -- 172.21.15.150:0/2706482766 <== osd.7 v2:172.21.15.157:6800/624642970 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f93081efef0 con 0x7f92e80720a0 2024-03-17T23:50:01.354 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-03-17T23:50:01.354 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-17T23:50:01.355 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-03-17T23:50:01.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.353+0000 7f930cf3b640 1 -- 172.21.15.150:0/2706482766 >> [v2:172.21.15.157:6800/624642970,v1:172.21.15.157:6801/624642970] conn(0x7f92e80720a0 msgr2=0x7f92e80744e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:01.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.353+0000 7f930cf3b640 1 --2- 172.21.15.150:0/2706482766 >> [v2:172.21.15.157:6800/624642970,v1:172.21.15.157:6801/624642970] conn(0x7f92e80720a0 0x7f92e80744e0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.353+0000 7f930cf3b640 1 -- 172.21.15.150:0/2706482766 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f92e806bc00 msgr2=0x7f92e806e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:01.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.353+0000 7f930cf3b640 1 --2- 172.21.15.150:0/2706482766 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f92e806bc00 0x7f92e806e0c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f930815d780 tx=0x7f92fc009240 comp rx=0 tx=0).stop 2024-03-17T23:50:01.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.353+0000 7f930cf3b640 1 -- 172.21.15.150:0/2706482766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93080cc440 msgr2=0x7f9308159630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:01.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.353+0000 7f930cf3b640 1 --2- 172.21.15.150:0/2706482766 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93080cc440 0x7f9308159630 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f92f8000c00 tx=0x7f92f80046c0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.353+0000 7f9306d9d640 1 -- 172.21.15.150:0/2706482766 reap_dead start 2024-03-17T23:50:01.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.353+0000 7f930cf3b640 1 -- 172.21.15.150:0/2706482766 shutdown_connections 2024-03-17T23:50:01.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.353+0000 7f930cf3b640 1 -- 172.21.15.150:0/2706482766 >> 172.21.15.150:0/2706482766 conn(0x7f93080b1be0 msgr2=0x7f93080b43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:01.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.354+0000 7f930cf3b640 1 -- 172.21.15.150:0/2706482766 shutdown_connections 2024-03-17T23:50:01.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.354+0000 7f930cf3b640 1 -- 172.21.15.150:0/2706482766 wait complete. 2024-03-17T23:50:01.364 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-17T23:50:01.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.465+0000 7f08f099e640 1 Processor -- start 2024-03-17T23:50:01.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.465+0000 7f08f099e640 1 -- start start 2024-03-17T23:50:01.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.466+0000 7f08f099e640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f08ec0b97f0 0x7f08ec162160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:01.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.466+0000 7f08f099e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f08ec1626a0 0x7f08ec0d2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:01.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.466+0000 7f08f099e640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08ec15ec90 con 0x7f08ec15df40 2024-03-17T23:50:01.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.466+0000 7f08f099e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f08ec0b6d40 con 0x7f08ec1626a0 2024-03-17T23:50:01.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.466+0000 7f08f099e640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f08ec15ea30 con 0x7f08ec0b97f0 2024-03-17T23:50:01.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.466+0000 7f08ead9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f08ec1626a0 0x7f08ec0d2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:01.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.466+0000 7f08ead9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f08ec1626a0 0x7f08ec0d2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45046/0 (socket says 172.21.15.150:45046) 2024-03-17T23:50:01.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.466+0000 7f08ead9d640 1 -- 172.21.15.150:0/22623321 learned_addr learned my addr 172.21.15.150:0/22623321 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:50:01.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.467+0000 7f08e9d9b640 1 --2- 172.21.15.150:0/22623321 >> v2:172.21.15.157:3300/0 conn(0x7f08ec0b97f0 0x7f08ec162160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:01.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.467+0000 7f08ead9d640 1 -- 172.21.15.150:0/22623321 >> v2:172.21.15.157:3300/0 conn(0x7f08ec0b97f0 msgr2=0x7f08ec162160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:50:01.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.467+0000 7f08ead9d640 1 --2- 172.21.15.150:0/22623321 >> v2:172.21.15.157:3300/0 conn(0x7f08ec0b97f0 0x7f08ec162160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.467+0000 7f08ead9d640 1 -- 172.21.15.150:0/22623321 >> v1:172.21.15.157:6789/0 conn(0x7f08ec15df40 legacy=0x7f08ec15e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:50:01.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.467+0000 7f08ead9d640 1 -- 172.21.15.150:0/22623321 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08ec15ef70 con 0x7f08ec1626a0 2024-03-17T23:50:01.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.467+0000 7f08ead9d640 1 --2- 172.21.15.150:0/22623321 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08ec1626a0 0x7f08ec0d2100 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f08ec06d470 tx=0x7f08e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6087064dff4367ce server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:01.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.468+0000 7f08e959a640 1 -- 172.21.15.150:0/22623321 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e4035020 con 0x7f08ec1626a0 2024-03-17T23:50:01.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.468+0000 7f08e959a640 1 -- 172.21.15.150:0/22623321 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08e4031870 con 0x7f08ec1626a0 2024-03-17T23:50:01.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.468+0000 7f08e959a640 1 -- 172.21.15.150:0/22623321 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e4031c40 con 0x7f08ec1626a0 2024-03-17T23:50:01.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.468+0000 7f08f099e640 1 -- 172.21.15.150:0/22623321 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08ec1626a0 msgr2=0x7f08ec0d2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:01.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.468+0000 7f08f099e640 1 --2- 172.21.15.150:0/22623321 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08ec1626a0 0x7f08ec0d2100 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f08ec06d470 tx=0x7f08e402b910 comp rx=0 tx=0).stop 2024-03-17T23:50:01.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.468+0000 7f08f099e640 1 -- 172.21.15.150:0/22623321 shutdown_connections 2024-03-17T23:50:01.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.468+0000 7f08f099e640 1 --2- 172.21.15.150:0/22623321 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08ec1626a0 0x7f08ec0d2100 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.468+0000 7f08f099e640 1 --2- 172.21.15.150:0/22623321 >> v2:172.21.15.157:3300/0 conn(0x7f08ec0b97f0 0x7f08ec162160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.468+0000 7f08f099e640 1 -- 172.21.15.150:0/22623321 >> 172.21.15.150:0/22623321 conn(0x7f08ec072ef0 msgr2=0x7f08ec0d3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:01.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.468+0000 7f08f099e640 1 -- 172.21.15.150:0/22623321 shutdown_connections 2024-03-17T23:50:01.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.468+0000 7f08f099e640 1 -- 172.21.15.150:0/22623321 wait complete. 2024-03-17T23:50:01.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.469+0000 7f08f099e640 1 Processor -- start 2024-03-17T23:50:01.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.470+0000 7f08f099e640 1 -- start start 2024-03-17T23:50:01.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.470+0000 7f08f099e640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08ec0b97f0 0x7f08ec0cad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:01.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.470+0000 7f08f099e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08ec15df40 0x7f08ec151150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:01.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.470+0000 7f08f099e640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08ec1626a0 0x7f08ec15f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:01.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.470+0000 7f08f099e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f08ec161c40 con 0x7f08ec15df40 2024-03-17T23:50:01.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.470+0000 7f08f099e640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f08ec0b9ca0 con 0x7f08ec1626a0 2024-03-17T23:50:01.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.470+0000 7f08f099e640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f08ec0d2640 con 0x7f08ec0b97f0 2024-03-17T23:50:01.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.470+0000 7f08e9d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08ec15df40 0x7f08ec151150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:01.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.470+0000 7f08e9d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08ec15df40 0x7f08ec151150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45048/0 (socket says 172.21.15.150:45048) 2024-03-17T23:50:01.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.470+0000 7f08e9d9b640 1 -- 172.21.15.150:0/375492135 learned_addr learned my addr 172.21.15.150:0/375492135 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:50:01.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.471+0000 7f08ead9d640 1 --2- 172.21.15.150:0/375492135 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08ec1626a0 0x7f08ec15f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:01.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.471+0000 7f08e9d9b640 1 -- 172.21.15.150:0/375492135 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08ec0b97f0 msgr2=0x7f08ec0cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:01.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.471+0000 7f08e9d9b640 1 --2- 172.21.15.150:0/375492135 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08ec0b97f0 0x7f08ec0cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.471+0000 7f08e9d9b640 1 -- 172.21.15.150:0/375492135 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08ec1626a0 msgr2=0x7f08ec15f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:01.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.471+0000 7f08e9d9b640 1 --2- 172.21.15.150:0/375492135 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08ec1626a0 0x7f08ec15f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.471+0000 7f08e9d9b640 1 -- 172.21.15.150:0/375492135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08ec0cb620 con 0x7f08ec15df40 2024-03-17T23:50:01.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.471+0000 7f08ead9d640 1 --2- 172.21.15.150:0/375492135 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08ec1626a0 0x7f08ec15f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:50:01.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.471+0000 7f08e9d9b640 1 --2- 172.21.15.150:0/375492135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08ec15df40 0x7f08ec151150 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f08e000d830 tx=0x7f08e000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:01.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.471+0000 7f08daffd640 1 -- 172.21.15.150:0/375492135 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e000b6c0 con 0x7f08ec15df40 2024-03-17T23:50:01.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.471+0000 7f08daffd640 1 -- 172.21.15.150:0/375492135 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08e0019070 con 0x7f08ec15df40 2024-03-17T23:50:01.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.471+0000 7f08f099e640 1 -- 172.21.15.150:0/375492135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08ec0cb900 con 0x7f08ec15df40 2024-03-17T23:50:01.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.471+0000 7f08daffd640 1 -- 172.21.15.150:0/375492135 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e000bac0 con 0x7f08ec15df40 2024-03-17T23:50:01.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.471+0000 7f08f099e640 1 -- 172.21.15.150:0/375492135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08ec0cc1a0 con 0x7f08ec15df40 2024-03-17T23:50:01.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.472+0000 7f08f099e640 1 -- 172.21.15.150:0/375492135 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08ec0681a0 con 0x7f08ec15df40 2024-03-17T23:50:01.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.473+0000 7f08daffd640 1 -- 172.21.15.150:0/375492135 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f08e001e460 con 0x7f08ec15df40 2024-03-17T23:50:01.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.473+0000 7f08daffd640 1 --2- 172.21.15.150:0/375492135 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f08cc06bbd0 0x7f08cc06e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:01.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.473+0000 7f08daffd640 1 -- 172.21.15.150:0/375492135 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5335+0+0 (secure 0 0 0) 0x7f08e008b020 con 0x7f08ec15df40 2024-03-17T23:50:01.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.473+0000 7f08ea59c640 1 --2- 172.21.15.150:0/375492135 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f08cc06bbd0 0x7f08cc06e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:01.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.473+0000 7f08ea59c640 1 --2- 172.21.15.150:0/375492135 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f08cc06bbd0 0x7f08cc06e090 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f08dc0043c0 tx=0x7f08dc007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:01.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.477+0000 7f08daffd640 1 -- 172.21.15.150:0/375492135 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f08e0059c80 con 0x7f08ec15df40 2024-03-17T23:50:01.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.652+0000 7f08f099e640 1 -- 172.21.15.150:0/375492135 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f08ec0cc370 con 0x7f08cc06bbd0 2024-03-17T23:50:01.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.652+0000 7f08daffd640 1 -- 172.21.15.150:0/375492135 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f08ec0cc370 con 0x7f08cc06bbd0 2024-03-17T23:50:01.653 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 7 to deep-scrub 2024-03-17T23:50:01.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.654+0000 7f08f099e640 1 -- 172.21.15.150:0/375492135 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f08cc06bbd0 msgr2=0x7f08cc06e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:01.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.654+0000 7f08f099e640 1 --2- 172.21.15.150:0/375492135 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f08cc06bbd0 0x7f08cc06e090 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f08dc0043c0 tx=0x7f08dc007d30 comp rx=0 tx=0).stop 2024-03-17T23:50:01.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.654+0000 7f08f099e640 1 -- 172.21.15.150:0/375492135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08ec15df40 msgr2=0x7f08ec151150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:01.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.654+0000 7f08f099e640 1 --2- 172.21.15.150:0/375492135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08ec15df40 0x7f08ec151150 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f08e000d830 tx=0x7f08e000dcf0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.654+0000 7f08f099e640 1 -- 172.21.15.150:0/375492135 shutdown_connections 2024-03-17T23:50:01.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.654+0000 7f08f099e640 1 --2- 172.21.15.150:0/375492135 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f08cc06bbd0 0x7f08cc06e090 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.654+0000 7f08f099e640 1 --2- 172.21.15.150:0/375492135 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08ec1626a0 0x7f08ec15f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.654+0000 7f08f099e640 1 --2- 172.21.15.150:0/375492135 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f08ec15df40 0x7f08ec151150 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.654+0000 7f08f099e640 1 --2- 172.21.15.150:0/375492135 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f08ec0b97f0 0x7f08ec0cad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.654+0000 7f08f099e640 1 -- 172.21.15.150:0/375492135 >> 172.21.15.150:0/375492135 conn(0x7f08ec072ef0 msgr2=0x7f08ec0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:01.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.655+0000 7f08f099e640 1 -- 172.21.15.150:0/375492135 shutdown_connections 2024-03-17T23:50:01.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.655+0000 7f08f099e640 1 -- 172.21.15.150:0/375492135 wait complete. 2024-03-17T23:50:01.666 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-17T23:50:01.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.766+0000 7f87e2510640 1 Processor -- start 2024-03-17T23:50:01.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.766+0000 7f87e2510640 1 -- start start 2024-03-17T23:50:01.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.767+0000 7f87e2510640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f87dc15e060 0x7f87dc0d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:01.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.767+0000 7f87e2510640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87dc06d500 con 0x7f87dc0b5c40 2024-03-17T23:50:01.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.767+0000 7f87e2510640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87dc0d2890 con 0x7f87dc0b57d0 2024-03-17T23:50:01.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.767+0000 7f87e2510640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f87dc0d2b70 con 0x7f87dc15e060 2024-03-17T23:50:01.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.767+0000 7f87dbfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f87dc15e060 0x7f87dc0d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:01.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.767+0000 7f87e0aad640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f87dc0b5c40 0x7f87dc06cdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:58750/0 (socket says 172.21.15.150:58750) 2024-03-17T23:50:01.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.767+0000 7f87e0aad640 1 -- 172.21.15.150:0/1062662201 learned_addr learned my addr 172.21.15.150:0/1062662201 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:50:01.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.768+0000 7f87dbfff640 1 -- 172.21.15.150:0/1062662201 >> v1:172.21.15.157:6789/0 conn(0x7f87dc0b57d0 legacy=0x7f87dc06c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:01.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.768+0000 7f87daffd640 1 -- 172.21.15.150:0/1062662201 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4262345015 0 0) 0x7f87dc06d500 con 0x7f87dc0b5c40 2024-03-17T23:50:01.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.768+0000 7f87dbfff640 1 -- 172.21.15.150:0/1062662201 >> v1:172.21.15.150:6789/0 conn(0x7f87dc0b5c40 legacy=0x7f87dc06cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:01.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.768+0000 7f87dbfff640 1 -- 172.21.15.150:0/1062662201 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87dc0d2e50 con 0x7f87dc15e060 2024-03-17T23:50:01.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.768+0000 7f87dbfff640 1 --2- 172.21.15.150:0/1062662201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87dc15e060 0x7f87dc0d1f20 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f87cc0097a0 tx=0x7f87cc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93063e4731aeb7e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:01.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.768+0000 7f87daffd640 1 -- 172.21.15.150:0/1062662201 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc035020 con 0x7f87dc15e060 2024-03-17T23:50:01.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.769+0000 7f87daffd640 1 -- 172.21.15.150:0/1062662201 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87cc031830 con 0x7f87dc15e060 2024-03-17T23:50:01.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.769+0000 7f87daffd640 1 -- 172.21.15.150:0/1062662201 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc031c00 con 0x7f87dc15e060 2024-03-17T23:50:01.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.769+0000 7f87e2510640 1 -- 172.21.15.150:0/1062662201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87dc15e060 msgr2=0x7f87dc0d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:01.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.769+0000 7f87e2510640 1 --2- 172.21.15.150:0/1062662201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87dc15e060 0x7f87dc0d1f20 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f87cc0097a0 tx=0x7f87cc02b8d0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.769+0000 7f87e2510640 1 -- 172.21.15.150:0/1062662201 shutdown_connections 2024-03-17T23:50:01.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.769+0000 7f87e2510640 1 --2- 172.21.15.150:0/1062662201 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87dc15e060 0x7f87dc0d1f20 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.769+0000 7f87e2510640 1 -- 172.21.15.150:0/1062662201 >> 172.21.15.150:0/1062662201 conn(0x7f87dc073200 msgr2=0x7f87dc0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:01.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.769+0000 7f87e2510640 1 -- 172.21.15.150:0/1062662201 shutdown_connections 2024-03-17T23:50:01.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.769+0000 7f87e2510640 1 -- 172.21.15.150:0/1062662201 wait complete. 2024-03-17T23:50:01.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.770+0000 7f87e2510640 1 Processor -- start 2024-03-17T23:50:01.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.770+0000 7f87e2510640 1 -- start start 2024-03-17T23:50:01.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.771+0000 7f87e2510640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87dc0b57d0 0x7f87dc1573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:01.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.771+0000 7f87e2510640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87dc0b5c40 0x7f87dc0b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:01.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.771+0000 7f87e2510640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87dc15e060 0x7f87dc0b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:01.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.771+0000 7f87e2510640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f87dc0b7f00 con 0x7f87dc0b5c40 2024-03-17T23:50:01.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.771+0000 7f87e2510640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f87dc0d3410 con 0x7f87dc0b57d0 2024-03-17T23:50:01.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.771+0000 7f87e2510640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f87dc0b60b0 con 0x7f87dc15e060 2024-03-17T23:50:01.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.771+0000 7f87db7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87dc0b5c40 0x7f87dc0b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:01.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.771+0000 7f87db7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87dc0b5c40 0x7f87dc0b3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45064/0 (socket says 172.21.15.150:45064) 2024-03-17T23:50:01.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.771+0000 7f87db7fe640 1 -- 172.21.15.150:0/1082916304 learned_addr learned my addr 172.21.15.150:0/1082916304 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:50:01.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.771+0000 7f87dbfff640 1 --2- 172.21.15.150:0/1082916304 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87dc0b57d0 0x7f87dc1573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:01.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.771+0000 7f87e0aad640 1 --2- 172.21.15.150:0/1082916304 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87dc15e060 0x7f87dc0b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:01.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.772+0000 7f87db7fe640 1 -- 172.21.15.150:0/1082916304 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87dc15e060 msgr2=0x7f87dc0b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:01.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.772+0000 7f87db7fe640 1 --2- 172.21.15.150:0/1082916304 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87dc15e060 0x7f87dc0b41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.772+0000 7f87db7fe640 1 -- 172.21.15.150:0/1082916304 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87dc0b57d0 msgr2=0x7f87dc1573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:01.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.772+0000 7f87db7fe640 1 --2- 172.21.15.150:0/1082916304 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87dc0b57d0 0x7f87dc1573b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.772+0000 7f87db7fe640 1 -- 172.21.15.150:0/1082916304 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87dc1f48f0 con 0x7f87dc0b5c40 2024-03-17T23:50:01.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.772+0000 7f87db7fe640 1 --2- 172.21.15.150:0/1082916304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87dc0b5c40 0x7f87dc0b3c90 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f87d000c450 tx=0x7f87d000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:01.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.772+0000 7f87d8ff9640 1 -- 172.21.15.150:0/1082916304 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d00052d0 con 0x7f87dc0b5c40 2024-03-17T23:50:01.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.772+0000 7f87d8ff9640 1 -- 172.21.15.150:0/1082916304 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87d0005470 con 0x7f87dc0b5c40 2024-03-17T23:50:01.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.772+0000 7f87e2510640 1 -- 172.21.15.150:0/1082916304 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87dc15e440 con 0x7f87dc0b5c40 2024-03-17T23:50:01.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.772+0000 7f87d8ff9640 1 -- 172.21.15.150:0/1082916304 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d0012870 con 0x7f87dc0b5c40 2024-03-17T23:50:01.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.772+0000 7f87e2510640 1 -- 172.21.15.150:0/1082916304 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87dc15ea00 con 0x7f87dc0b5c40 2024-03-17T23:50:01.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.773+0000 7f87e2510640 1 -- 172.21.15.150:0/1082916304 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87dc0688b0 con 0x7f87dc0b5c40 2024-03-17T23:50:01.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.773+0000 7f87d8ff9640 1 -- 172.21.15.150:0/1082916304 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f87d0012a10 con 0x7f87dc0b5c40 2024-03-17T23:50:01.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.774+0000 7f87d8ff9640 1 --2- 172.21.15.150:0/1082916304 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f87c806bbd0 0x7f87c806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:01.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.774+0000 7f87d8ff9640 1 -- 172.21.15.150:0/1082916304 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5335+0+0 (secure 0 0 0) 0x7f87d008a350 con 0x7f87dc0b5c40 2024-03-17T23:50:01.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.774+0000 7f87dbfff640 1 --2- 172.21.15.150:0/1082916304 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f87c806bbd0 0x7f87c806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:01.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.774+0000 7f87dbfff640 1 --2- 172.21.15.150:0/1082916304 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f87c806bbd0 0x7f87c806e090 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f87cc036000 tx=0x7f87cc03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:01.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.778+0000 7f87d8ff9640 1 -- 172.21.15.150:0/1082916304 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f87d00589f0 con 0x7f87dc0b5c40 2024-03-17T23:50:01.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.951+0000 7f87e2510640 1 -- 172.21.15.150:0/1082916304 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f87dc0b7b50 con 0x7f87c806bbd0 2024-03-17T23:50:01.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.952+0000 7f87d8ff9640 1 -- 172.21.15.150:0/1082916304 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13442 (secure 0 0 0) 0x7f87cc04b4a0 con 0x7f87c806bbd0 2024-03-17T23:50:01.953 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:50:01.953 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-03-17T23:50:01.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.954+0000 7f87e2510640 1 -- 172.21.15.150:0/1082916304 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f87c806bbd0 msgr2=0x7f87c806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:01.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.954+0000 7f87e2510640 1 --2- 172.21.15.150:0/1082916304 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f87c806bbd0 0x7f87c806e090 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f87cc036000 tx=0x7f87cc03c040 comp rx=0 tx=0).stop 2024-03-17T23:50:01.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.954+0000 7f87e2510640 1 -- 172.21.15.150:0/1082916304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87dc0b5c40 msgr2=0x7f87dc0b3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:01.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.954+0000 7f87e2510640 1 --2- 172.21.15.150:0/1082916304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87dc0b5c40 0x7f87dc0b3c90 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f87d000c450 tx=0x7f87d000c910 comp rx=0 tx=0).stop 2024-03-17T23:50:01.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.954+0000 7f87e2510640 1 -- 172.21.15.150:0/1082916304 shutdown_connections 2024-03-17T23:50:01.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.954+0000 7f87e2510640 1 --2- 172.21.15.150:0/1082916304 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f87dc15e060 0x7f87dc0b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.954+0000 7f87e2510640 1 --2- 172.21.15.150:0/1082916304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f87dc0b5c40 0x7f87dc0b3c90 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.954+0000 7f87e2510640 1 --2- 172.21.15.150:0/1082916304 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f87c806bbd0 0x7f87c806e090 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.954+0000 7f87e2510640 1 --2- 172.21.15.150:0/1082916304 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f87dc0b57d0 0x7f87dc1573b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:01.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.954+0000 7f87e2510640 1 -- 172.21.15.150:0/1082916304 >> 172.21.15.150:0/1082916304 conn(0x7f87dc073200 msgr2=0x7f87dc06c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:01.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.954+0000 7f87e2510640 1 -- 172.21.15.150:0/1082916304 shutdown_connections 2024-03-17T23:50:01.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:01.954+0000 7f87e2510640 1 -- 172.21.15.150:0/1082916304 wait complete. 2024-03-17T23:50:01.965 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-03-17T23:50:01.634119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":19,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":239720,"kb_used_data":6920,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735000,"statfs":{"total":773094113280,"available":772848640000,"internally_reserved":0,"allocated":7086080,"data_stored":3860560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.715854"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":246,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-17T23:49:54.887071+0000","last_change":"2024-03-17T23:46:44.768668+0000","last_active":"2024-03-17T23:49:54.887071+0000","last_peered":"2024-03-17T23:49:54.887071+0000","last_clean":"2024-03-17T23:49:54.887071+0000","last_became_active":"2024-03-17T23:46:44.767826+0000","last_became_peered":"2024-03-17T23:46:44.767826+0000","last_unstale":"2024-03-17T23:49:54.887071+0000","last_undegraded":"2024-03-17T23:49:54.887071+0000","last_fullsized":"2024-03-17T23:49:54.887071+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-17T23:46:43.744878+0000","last_clean_scrub_stamp":"2024-03-17T23:46:43.744878+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:54:22.286155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":14,"seq":60129542187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27740,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344100,"statfs":{"total":96636764160,"available":96608358400,"internally_reserved":0,"allocated":737280,"data_stored":334978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27740,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344100,"statfs":{"total":96636764160,"available":96608358400,"internally_reserved":0,"allocated":737280,"data_stored":334978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542187,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28320,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343520,"statfs":{"total":96636764160,"available":96607764480,"internally_reserved":0,"allocated":1331200,"data_stored":925346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32480,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339360,"statfs":{"total":96636764160,"available":96603504640,"internally_reserved":0,"allocated":1331200,"data_stored":925346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31900,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339940,"statfs":{"total":96636764160,"available":96604098560,"internally_reserved":0,"allocated":737280,"data_stored":334978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31900,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339940,"statfs":{"total":96636764160,"available":96604098560,"internally_reserved":0,"allocated":737280,"data_stored":334978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31900,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339940,"statfs":{"total":96636764160,"available":96604098560,"internally_reserved":0,"allocated":737280,"data_stored":334978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27740,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344100,"statfs":{"total":96636764160,"available":96608358400,"internally_reserved":0,"allocated":737280,"data_stored":334978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-17T23:50:01.968 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-17T23:46:43.744878+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=17, tm_hour=23, tm_min=46, tm_sec=43, tm_wday=6, tm_yday=77, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=17, tm_hour=23, tm_min=49, tm_sec=57, tm_wday=6, tm_yday=77, tm_isdst=0) 2024-03-17T23:50:01.968 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-17T23:50:21.969 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-17T23:50:22.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.082+0000 7f4dfe3f4640 1 Processor -- start 2024-03-17T23:50:22.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.082+0000 7f4dfe3f4640 1 -- start start 2024-03-17T23:50:22.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.083+0000 7f4dfe3f4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4df8144bb0 0x7f4df815ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:22.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.083+0000 7f4dfe3f4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4df815d390 0x7f4df80cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:22.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.083+0000 7f4dfe3f4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4df8145170 con 0x7f4df80ce4d0 2024-03-17T23:50:22.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.083+0000 7f4dfe3f4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f4df80ce920 con 0x7f4df8144bb0 2024-03-17T23:50:22.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.083+0000 7f4dfe3f4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4df8145450 con 0x7f4df815d390 2024-03-17T23:50:22.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.083+0000 7f4df7fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4df8144bb0 0x7f4df815ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:22.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.083+0000 7f4dfc991640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f4df80ce4d0 0x7f4df806e3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34140/0 (socket says 172.21.15.150:34140) 2024-03-17T23:50:22.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.083+0000 7f4dfc991640 1 -- 172.21.15.150:0/3817467520 learned_addr learned my addr 172.21.15.150:0/3817467520 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:50:22.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.083+0000 7f4df7fff640 1 --2- 172.21.15.150:0/3817467520 >> v2:172.21.15.150:3300/0 conn(0x7f4df8144bb0 0x7f4df815ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34470/0 (socket says 172.21.15.150:34470) 2024-03-17T23:50:22.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.083+0000 7f4df7fff640 1 -- 172.21.15.150:0/3817467520 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4df815d390 msgr2=0x7f4df80cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:50:22.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.084+0000 7f4df7fff640 1 --2- 172.21.15.150:0/3817467520 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4df815d390 0x7f4df80cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.084+0000 7f4df7fff640 1 -- 172.21.15.150:0/3817467520 >> v1:172.21.15.150:6789/0 conn(0x7f4df80ce4d0 legacy=0x7f4df806e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:22.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.084+0000 7f4df7fff640 1 -- 172.21.15.150:0/3817467520 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df8145730 con 0x7f4df8144bb0 2024-03-17T23:50:22.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.084+0000 7f4df6ffd640 1 -- 172.21.15.150:0/3817467520 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 533874946 0 0) 0x7f4df8145170 con 0x7f4df80ce4d0 2024-03-17T23:50:22.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.084+0000 7f4df7fff640 1 --2- 172.21.15.150:0/3817467520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4df8144bb0 0x7f4df815ce50 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f4de80097a0 tx=0x7f4de802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee6eb12c10ac816d server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:22.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.084+0000 7f4df6ffd640 1 -- 172.21.15.150:0/3817467520 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de80315a0 con 0x7f4df8144bb0 2024-03-17T23:50:22.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.084+0000 7f4df6ffd640 1 -- 172.21.15.150:0/3817467520 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4de8037070 con 0x7f4df8144bb0 2024-03-17T23:50:22.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.085+0000 7f4df6ffd640 1 -- 172.21.15.150:0/3817467520 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de8031960 con 0x7f4df8144bb0 2024-03-17T23:50:22.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.085+0000 7f4dfe3f4640 1 -- 172.21.15.150:0/3817467520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4df8144bb0 msgr2=0x7f4df815ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:22.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.085+0000 7f4dfe3f4640 1 --2- 172.21.15.150:0/3817467520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4df8144bb0 0x7f4df815ce50 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f4de80097a0 tx=0x7f4de802b840 comp rx=0 tx=0).stop 2024-03-17T23:50:22.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.085+0000 7f4dfe3f4640 1 -- 172.21.15.150:0/3817467520 shutdown_connections 2024-03-17T23:50:22.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.085+0000 7f4dfe3f4640 1 --2- 172.21.15.150:0/3817467520 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4df815d390 0x7f4df80cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.085+0000 7f4dfe3f4640 1 --2- 172.21.15.150:0/3817467520 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4df8144bb0 0x7f4df815ce50 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.085+0000 7f4dfe3f4640 1 -- 172.21.15.150:0/3817467520 >> 172.21.15.150:0/3817467520 conn(0x7f4df806b190 msgr2=0x7f4df806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:22.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.085+0000 7f4dfe3f4640 1 -- 172.21.15.150:0/3817467520 shutdown_connections 2024-03-17T23:50:22.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.085+0000 7f4dfe3f4640 1 -- 172.21.15.150:0/3817467520 wait complete. 2024-03-17T23:50:22.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.086+0000 7f4dfe3f4640 1 Processor -- start 2024-03-17T23:50:22.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.086+0000 7f4dfe3f4640 1 -- start start 2024-03-17T23:50:22.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.087+0000 7f4dfe3f4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4df80ce4d0 0x7f4df815b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:22.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.087+0000 7f4dfe3f4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4df8144bb0 0x7f4df81f1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:22.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.087+0000 7f4dfe3f4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4df815d390 0x7f4df80ad030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:22.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.087+0000 7f4dfe3f4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4df806ec80 con 0x7f4df80ce4d0 2024-03-17T23:50:22.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.087+0000 7f4dfe3f4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f4df806eb00 con 0x7f4df815d390 2024-03-17T23:50:22.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.087+0000 7f4dfe3f4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f4df806ee00 con 0x7f4df8144bb0 2024-03-17T23:50:22.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.087+0000 7f4df7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4df80ce4d0 0x7f4df815b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:22.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.087+0000 7f4df7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4df80ce4d0 0x7f4df815b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34484/0 (socket says 172.21.15.150:34484) 2024-03-17T23:50:22.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.087+0000 7f4df7fff640 1 -- 172.21.15.150:0/599034033 learned_addr learned my addr 172.21.15.150:0/599034033 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:50:22.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.088+0000 7f4dfc991640 1 --2- 172.21.15.150:0/599034033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4df815d390 0x7f4df80ad030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:22.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.088+0000 7f4df7fff640 1 -- 172.21.15.150:0/599034033 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4df8144bb0 msgr2=0x7f4df81f1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:22.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.088+0000 7f4df77fe640 1 --2- 172.21.15.150:0/599034033 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4df8144bb0 0x7f4df81f1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:22.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.088+0000 7f4df7fff640 1 --2- 172.21.15.150:0/599034033 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4df8144bb0 0x7f4df81f1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.088+0000 7f4df7fff640 1 -- 172.21.15.150:0/599034033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4df815d390 msgr2=0x7f4df80ad030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:22.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.088+0000 7f4df7fff640 1 --2- 172.21.15.150:0/599034033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4df815d390 0x7f4df80ad030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.088+0000 7f4df7fff640 1 -- 172.21.15.150:0/599034033 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df80ac710 con 0x7f4df80ce4d0 2024-03-17T23:50:22.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.088+0000 7f4dfc991640 1 --2- 172.21.15.150:0/599034033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4df815d390 0x7f4df80ad030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-17T23:50:22.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.088+0000 7f4df7fff640 1 --2- 172.21.15.150:0/599034033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4df80ce4d0 0x7f4df815b4a0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f4de8000c00 tx=0x7f4de80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:22.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.088+0000 7f4df4ff9640 1 -- 172.21.15.150:0/599034033 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de8004230 con 0x7f4df80ce4d0 2024-03-17T23:50:22.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.088+0000 7f4df4ff9640 1 -- 172.21.15.150:0/599034033 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4de80043d0 con 0x7f4df80ce4d0 2024-03-17T23:50:22.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.088+0000 7f4dfe3f4640 1 -- 172.21.15.150:0/599034033 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4df8141330 con 0x7f4df80ce4d0 2024-03-17T23:50:22.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.088+0000 7f4df4ff9640 1 -- 172.21.15.150:0/599034033 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de8032880 con 0x7f4df80ce4d0 2024-03-17T23:50:22.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.088+0000 7f4dfe3f4640 1 -- 172.21.15.150:0/599034033 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4df81f2080 con 0x7f4df80ce4d0 2024-03-17T23:50:22.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.089+0000 7f4dfe3f4640 1 -- 172.21.15.150:0/599034033 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4df80681a0 con 0x7f4df80ce4d0 2024-03-17T23:50:22.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.089+0000 7f4df4ff9640 1 -- 172.21.15.150:0/599034033 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f4de8032a20 con 0x7f4df80ce4d0 2024-03-17T23:50:22.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.090+0000 7f4df4ff9640 1 --2- 172.21.15.150:0/599034033 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4de406bc00 0x7f4de406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:22.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.090+0000 7f4df4ff9640 1 -- 172.21.15.150:0/599034033 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5335+0+0 (secure 0 0 0) 0x7f4de802f030 con 0x7f4df80ce4d0 2024-03-17T23:50:22.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.090+0000 7f4df77fe640 1 --2- 172.21.15.150:0/599034033 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4de406bc00 0x7f4de406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:22.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.090+0000 7f4df77fe640 1 --2- 172.21.15.150:0/599034033 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4de406bc00 0x7f4de406e0c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4df80aaff0 tx=0x7f4dec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:22.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.094+0000 7f4df4ff9640 1 -- 172.21.15.150:0/599034033 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f4de8041070 con 0x7f4df80ce4d0 2024-03-17T23:50:22.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.263+0000 7f4dfe3f4640 1 -- 172.21.15.150:0/599034033 --> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4df81432a0 con 0x7f4de406bc00 2024-03-17T23:50:22.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.264+0000 7f4df4ff9640 1 -- 172.21.15.150:0/599034033 <== mgr.4106 v2:172.21.15.150:6832/4239152602 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13359 (secure 0 0 0) 0x7f4df81432a0 con 0x7f4de406bc00 2024-03-17T23:50:22.266 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:50:22.266 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-03-17T23:50:22.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.266+0000 7f4dfe3f4640 1 -- 172.21.15.150:0/599034033 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4de406bc00 msgr2=0x7f4de406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:22.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.266+0000 7f4dfe3f4640 1 --2- 172.21.15.150:0/599034033 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4de406bc00 0x7f4de406e0c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4df80aaff0 tx=0x7f4dec008040 comp rx=0 tx=0).stop 2024-03-17T23:50:22.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.266+0000 7f4dfe3f4640 1 -- 172.21.15.150:0/599034033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4df80ce4d0 msgr2=0x7f4df815b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:22.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.266+0000 7f4dfe3f4640 1 --2- 172.21.15.150:0/599034033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4df80ce4d0 0x7f4df815b4a0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f4de8000c00 tx=0x7f4de80046c0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.266+0000 7f4dfe3f4640 1 -- 172.21.15.150:0/599034033 shutdown_connections 2024-03-17T23:50:22.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.266+0000 7f4dfe3f4640 1 --2- 172.21.15.150:0/599034033 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f4de406bc00 0x7f4de406e0c0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.266+0000 7f4dfe3f4640 1 --2- 172.21.15.150:0/599034033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4df815d390 0x7f4df80ad030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.266+0000 7f4dfe3f4640 1 --2- 172.21.15.150:0/599034033 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f4df8144bb0 0x7f4df81f1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.266+0000 7f4dfe3f4640 1 --2- 172.21.15.150:0/599034033 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4df80ce4d0 0x7f4df815b4a0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.266+0000 7f4dfe3f4640 1 -- 172.21.15.150:0/599034033 >> 172.21.15.150:0/599034033 conn(0x7f4df806b190 msgr2=0x7f4df806f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:22.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.266+0000 7f4dfe3f4640 1 -- 172.21.15.150:0/599034033 shutdown_connections 2024-03-17T23:50:22.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.266+0000 7f4dfe3f4640 1 -- 172.21.15.150:0/599034033 wait complete. 2024-03-17T23:50:22.278 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-03-17T23:50:21.637936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":239712,"kb_used_data":6920,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735008,"statfs":{"total":773094113280,"available":772848648192,"internally_reserved":0,"allocated":7086080,"data_stored":3860560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002451"},"pg_stats":[{"pgid":"1.0","version":"61'119","reported_seq":257,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-03-17T23:50:01.927188+0000","last_change":"2024-03-17T23:50:01.927222+0000","last_active":"2024-03-17T23:50:01.927188+0000","last_peered":"2024-03-17T23:50:01.927188+0000","last_clean":"2024-03-17T23:50:01.927188+0000","last_became_active":"2024-03-17T23:46:44.767826+0000","last_became_peered":"2024-03-17T23:46:44.767826+0000","last_unstale":"2024-03-17T23:50:01.927188+0000","last_undegraded":"2024-03-17T23:50:01.927188+0000","last_fullsized":"2024-03-17T23:50:01.927188+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'119","last_scrub_stamp":"2024-03-17T23:50:01.927097+0000","last_deep_scrub":"61'119","last_deep_scrub_stamp":"2024-03-17T23:50:01.927097+0000","last_clean_scrub_stamp":"2024-03-17T23:50:01.927097+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:46:53.247323+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":14,"seq":60129542191,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27740,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344100,"statfs":{"total":96636764160,"available":96608358400,"internally_reserved":0,"allocated":737280,"data_stored":334978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542191,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27740,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344100,"statfs":{"total":96636764160,"available":96608358400,"internally_reserved":0,"allocated":737280,"data_stored":334978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542191,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28316,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343524,"statfs":{"total":96636764160,"available":96607768576,"internally_reserved":0,"allocated":1331200,"data_stored":925346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":13,"seq":55834574894,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32476,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339364,"statfs":{"total":96636764160,"available":96603508736,"internally_reserved":0,"allocated":1331200,"data_stored":925346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31900,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339940,"statfs":{"total":96636764160,"available":96604098560,"internally_reserved":0,"allocated":737280,"data_stored":334978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31900,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339940,"statfs":{"total":96636764160,"available":96604098560,"internally_reserved":0,"allocated":737280,"data_stored":334978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31900,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339940,"statfs":{"total":96636764160,"available":96604098560,"internally_reserved":0,"allocated":737280,"data_stored":334978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27740,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344100,"statfs":{"total":96636764160,"available":96608358400,"internally_reserved":0,"allocated":737280,"data_stored":334978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-17T23:50:22.279 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-17T23:50:22.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.377+0000 7f5fe33f4640 1 Processor -- start 2024-03-17T23:50:22.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.378+0000 7f5fe33f4640 1 -- start start 2024-03-17T23:50:22.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.378+0000 7f5fe33f4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5fdc0d0ce0 0x7f5fdc0b96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:22.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.378+0000 7f5fe33f4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fdc161d30 con 0x7f5fdc0b2bf0 2024-03-17T23:50:22.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.378+0000 7f5fe33f4640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fdc162010 con 0x7f5fdc161520 2024-03-17T23:50:22.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.378+0000 7f5fe33f4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f5fdc0b9bf0 con 0x7f5fdc0d0ce0 2024-03-17T23:50:22.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.378+0000 7f5fe098f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5fdc0d0ce0 0x7f5fdc0b96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:22.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.378+0000 7f5fe1991640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f5fdc0b2bf0 0x7f5fdc0d1fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34144/0 (socket says 172.21.15.150:34144) 2024-03-17T23:50:22.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.378+0000 7f5fe1991640 1 -- 172.21.15.150:0/1117660655 learned_addr learned my addr 172.21.15.150:0/1117660655 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-03-17T23:50:22.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.379+0000 7f5fe098f640 1 -- 172.21.15.150:0/1117660655 >> v1:172.21.15.157:6789/0 conn(0x7f5fdc161520 legacy=0x7f5fdc0d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:50:22.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.379+0000 7f5fe098f640 1 -- 172.21.15.150:0/1117660655 >> v1:172.21.15.150:6789/0 conn(0x7f5fdc0b2bf0 legacy=0x7f5fdc0d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:22.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.379+0000 7f5fd3fff640 1 -- 172.21.15.150:0/1117660655 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2611468173 0 0) 0x7f5fdc161d30 con 0x7f5fdc0b2bf0 2024-03-17T23:50:22.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.379+0000 7f5fe098f640 1 -- 172.21.15.150:0/1117660655 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fdc1622f0 con 0x7f5fdc0d0ce0 2024-03-17T23:50:22.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.379+0000 7f5fe098f640 1 --2- 172.21.15.150:0/1117660655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fdc0d0ce0 0x7f5fdc0b96b0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc0097a0 tx=0x7f5fcc02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9c4e120721f80a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:22.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.380+0000 7f5fd3fff640 1 -- 172.21.15.150:0/1117660655 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc038070 con 0x7f5fdc0d0ce0 2024-03-17T23:50:22.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.380+0000 7f5fd3fff640 1 -- 172.21.15.150:0/1117660655 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fcc031700 con 0x7f5fdc0d0ce0 2024-03-17T23:50:22.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.380+0000 7f5fd3fff640 1 -- 172.21.15.150:0/1117660655 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc031ad0 con 0x7f5fdc0d0ce0 2024-03-17T23:50:22.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.380+0000 7f5fe33f4640 1 -- 172.21.15.150:0/1117660655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fdc0d0ce0 msgr2=0x7f5fdc0b96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:22.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.380+0000 7f5fe33f4640 1 --2- 172.21.15.150:0/1117660655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fdc0d0ce0 0x7f5fdc0b96b0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc0097a0 tx=0x7f5fcc02b7a0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.380+0000 7f5fe33f4640 1 -- 172.21.15.150:0/1117660655 shutdown_connections 2024-03-17T23:50:22.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.380+0000 7f5fe33f4640 1 --2- 172.21.15.150:0/1117660655 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fdc0d0ce0 0x7f5fdc0b96b0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.380+0000 7f5fe33f4640 1 -- 172.21.15.150:0/1117660655 >> 172.21.15.150:0/1117660655 conn(0x7f5fdc06ca00 msgr2=0x7f5fdc06ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:22.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.380+0000 7f5fe33f4640 1 -- 172.21.15.150:0/1117660655 shutdown_connections 2024-03-17T23:50:22.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.380+0000 7f5fe33f4640 1 -- 172.21.15.150:0/1117660655 wait complete. 2024-03-17T23:50:22.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.381+0000 7f5fe33f4640 1 Processor -- start 2024-03-17T23:50:22.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.381+0000 7f5fe33f4640 1 -- start start 2024-03-17T23:50:22.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.382+0000 7f5fe33f4640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5fdc0b2bf0 0x7f5fdc15dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:22.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.382+0000 7f5fe33f4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fdc0d0ce0 0x7f5fdc160720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:22.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.382+0000 7f5fe33f4640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5fdc161520 0x7f5fdc1f3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:22.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.382+0000 7f5fe33f4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5fdc161900 con 0x7f5fdc0d0ce0 2024-03-17T23:50:22.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.382+0000 7f5fe33f4640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f5fdc162940 con 0x7f5fdc161520 2024-03-17T23:50:22.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.382+0000 7f5fe33f4640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f5fdc0d26b0 con 0x7f5fdc0b2bf0 2024-03-17T23:50:22.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.382+0000 7f5fe098f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fdc0d0ce0 0x7f5fdc160720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:22.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.382+0000 7f5fe098f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fdc0d0ce0 0x7f5fdc160720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34502/0 (socket says 172.21.15.150:34502) 2024-03-17T23:50:22.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.382+0000 7f5fe098f640 1 -- 172.21.15.150:0/1653419930 learned_addr learned my addr 172.21.15.150:0/1653419930 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:50:22.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.382+0000 7f5fe1190640 1 --2- 172.21.15.150:0/1653419930 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5fdc0b2bf0 0x7f5fdc15dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:22.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.382+0000 7f5fe1991640 1 --2- 172.21.15.150:0/1653419930 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5fdc161520 0x7f5fdc1f3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:22.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.383+0000 7f5fe098f640 1 -- 172.21.15.150:0/1653419930 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5fdc0b2bf0 msgr2=0x7f5fdc15dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:22.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.383+0000 7f5fe098f640 1 --2- 172.21.15.150:0/1653419930 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5fdc0b2bf0 0x7f5fdc15dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.383+0000 7f5fe098f640 1 -- 172.21.15.150:0/1653419930 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5fdc161520 msgr2=0x7f5fdc1f3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:22.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.383+0000 7f5fe098f640 1 --2- 172.21.15.150:0/1653419930 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5fdc161520 0x7f5fdc1f3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.383+0000 7f5fe098f640 1 -- 172.21.15.150:0/1653419930 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fdc160c60 con 0x7f5fdc0d0ce0 2024-03-17T23:50:22.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.383+0000 7f5fe098f640 1 --2- 172.21.15.150:0/1653419930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fdc0d0ce0 0x7f5fdc160720 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc0097a0 tx=0x7f5fcc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:22.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.383+0000 7f5fd1ffb640 1 -- 172.21.15.150:0/1653419930 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc038040 con 0x7f5fdc0d0ce0 2024-03-17T23:50:22.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.383+0000 7f5fd1ffb640 1 -- 172.21.15.150:0/1653419930 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fcc033a30 con 0x7f5fdc0d0ce0 2024-03-17T23:50:22.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.383+0000 7f5fd1ffb640 1 -- 172.21.15.150:0/1653419930 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc031ed0 con 0x7f5fdc0d0ce0 2024-03-17T23:50:22.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.383+0000 7f5fe33f4640 1 -- 172.21.15.150:0/1653419930 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fdc15e220 con 0x7f5fdc0d0ce0 2024-03-17T23:50:22.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.383+0000 7f5fe33f4640 1 -- 172.21.15.150:0/1653419930 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fdc15ff20 con 0x7f5fdc0d0ce0 2024-03-17T23:50:22.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.384+0000 7f5fe33f4640 1 -- 172.21.15.150:0/1653419930 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fdc0681a0 con 0x7f5fdc0d0ce0 2024-03-17T23:50:22.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.385+0000 7f5fd1ffb640 1 -- 172.21.15.150:0/1653419930 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f5fcc004030 con 0x7f5fdc0d0ce0 2024-03-17T23:50:22.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.385+0000 7f5fd1ffb640 1 --2- 172.21.15.150:0/1653419930 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5fc006bc00 0x7f5fc006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:22.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.385+0000 7f5fd1ffb640 1 -- 172.21.15.150:0/1653419930 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5335+0+0 (secure 0 0 0) 0x7f5fcc0ac5a0 con 0x7f5fdc0d0ce0 2024-03-17T23:50:22.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.385+0000 7f5fe1190640 1 --2- 172.21.15.150:0/1653419930 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5fc006bc00 0x7f5fc006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:22.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.386+0000 7f5fe1190640 1 --2- 172.21.15.150:0/1653419930 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5fc006bc00 0x7f5fc006e0c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f5fd800b450 tx=0x7f5fd800d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:22.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.390+0000 7f5fd1ffb640 1 -- 172.21.15.150:0/1653419930 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f5fcc043020 con 0x7f5fdc0d0ce0 2024-03-17T23:50:22.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.565+0000 7f5fe33f4640 1 -- 172.21.15.150:0/1653419930 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f5fdc0d19c0 con 0x7f5fdc0d0ce0 2024-03-17T23:50:22.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.568+0000 7f5fd1ffb640 1 -- 172.21.15.150:0/1653419930 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v14)=0 v14) ==== 123+0+0 (secure 0 0 0) 0x7f5fcc030070 con 0x7f5fdc0d0ce0 2024-03-17T23:50:22.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.569+0000 7f5fd1ffb640 1 -- 172.21.15.150:0/1653419930 <== mon.0 v2:172.21.15.150:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f5fcc031700 con 0x7f5fdc0d0ce0 2024-03-17T23:50:22.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.571+0000 7f5fe33f4640 1 -- 172.21.15.150:0/1653419930 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5fc006bc00 msgr2=0x7f5fc006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:22.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.571+0000 7f5fe33f4640 1 --2- 172.21.15.150:0/1653419930 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5fc006bc00 0x7f5fc006e0c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f5fd800b450 tx=0x7f5fd800d040 comp rx=0 tx=0).stop 2024-03-17T23:50:22.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.571+0000 7f5fe33f4640 1 -- 172.21.15.150:0/1653419930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fdc0d0ce0 msgr2=0x7f5fdc160720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:22.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.571+0000 7f5fe33f4640 1 --2- 172.21.15.150:0/1653419930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fdc0d0ce0 0x7f5fdc160720 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc0097a0 tx=0x7f5fcc0046c0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.571+0000 7f5fe33f4640 1 -- 172.21.15.150:0/1653419930 shutdown_connections 2024-03-17T23:50:22.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.571+0000 7f5fe33f4640 1 --2- 172.21.15.150:0/1653419930 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f5fc006bc00 0x7f5fc006e0c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.571+0000 7f5fe33f4640 1 --2- 172.21.15.150:0/1653419930 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5fdc161520 0x7f5fdc1f3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.571+0000 7f5fe33f4640 1 --2- 172.21.15.150:0/1653419930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5fdc0d0ce0 0x7f5fdc160720 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.571+0000 7f5fe33f4640 1 --2- 172.21.15.150:0/1653419930 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f5fdc0b2bf0 0x7f5fdc15dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.571+0000 7f5fe33f4640 1 -- 172.21.15.150:0/1653419930 >> 172.21.15.150:0/1653419930 conn(0x7f5fdc06ca00 msgr2=0x7f5fdc0d2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:22.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.571+0000 7f5fe33f4640 1 -- 172.21.15.150:0/1653419930 shutdown_connections 2024-03-17T23:50:22.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.571+0000 7f5fe33f4640 1 -- 172.21.15.150:0/1653419930 wait complete. 2024-03-17T23:50:22.583 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-17T23:50:22.583 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-17T23:50:22.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.682+0000 7f3efbe12640 1 Processor -- start 2024-03-17T23:50:22.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.682+0000 7f3efbe12640 1 -- start start 2024-03-17T23:50:22.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.683+0000 7f3efbe12640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ef415ab90 0x7f3ef415bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:22.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.683+0000 7f3efbe12640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3ef406ecd0 0x7f3ef406f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:22.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.683+0000 7f3efbe12640 1 -- --> v1:172.21.15.157:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ef40ce790 con 0x7f3ef41514f0 2024-03-17T23:50:22.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.683+0000 7f3efbe12640 1 -- --> v2:172.21.15.157:3300/0 -- mon_getmap magic: 0 -- 0x7f3ef406fba0 con 0x7f3ef406ecd0 2024-03-17T23:50:22.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.683+0000 7f3efbe12640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3ef415b050 con 0x7f3ef415ab90 2024-03-17T23:50:22.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.683+0000 7f3ef93ad640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ef415ab90 0x7f3ef415bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:22.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.683+0000 7f3efa3af640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3ef406ecd0 0x7f3ef406f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:22.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.683+0000 7f3ef93ad640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ef415ab90 0x7f3ef415bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3301/0 says I am v2:172.21.15.150:50630/0 (socket says 172.21.15.150:50630) 2024-03-17T23:50:22.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.683+0000 7f3ef9bae640 1 --1- >> v1:172.21.15.157:6789/0 conn(0x7f3ef41514f0 0x7f3ef406e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.157:6789/0 says I am v1:172.21.15.150:34188/0 (socket says 172.21.15.150:34188) 2024-03-17T23:50:22.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.683+0000 7f3efa3af640 1 --2- >> v2:172.21.15.157:3300/0 conn(0x7f3ef406ecd0 0x7f3ef406f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.150:53986/0 (socket says 172.21.15.150:53986) 2024-03-17T23:50:22.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.683+0000 7f3ef93ad640 1 -- 172.21.15.150:0/3983996524 learned_addr learned my addr 172.21.15.150:0/3983996524 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:50:22.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.683+0000 7f3ef93ad640 1 -- 172.21.15.150:0/3983996524 >> v2:172.21.15.157:3300/0 conn(0x7f3ef406ecd0 msgr2=0x7f3ef406f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-17T23:50:22.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.683+0000 7f3ef93ad640 1 --2- 172.21.15.150:0/3983996524 >> v2:172.21.15.157:3300/0 conn(0x7f3ef406ecd0 0x7f3ef406f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.683+0000 7f3ef93ad640 1 -- 172.21.15.150:0/3983996524 >> v1:172.21.15.157:6789/0 conn(0x7f3ef41514f0 legacy=0x7f3ef406e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:22.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.683+0000 7f3ef93ad640 1 -- 172.21.15.150:0/3983996524 --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ef415b330 con 0x7f3ef415ab90 2024-03-17T23:50:22.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.683+0000 7f3efa3af640 1 --2- 172.21.15.150:0/3983996524 >> v2:172.21.15.157:3300/0 conn(0x7f3ef406ecd0 0x7f3ef406f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-17T23:50:22.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.684+0000 7f3ef93ad640 1 --2- 172.21.15.150:0/3983996524 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ef415ab90 0x7f3ef415bd10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4009870 tx=0x7f3ee402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fbf388d87386bdc server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:22.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.684+0000 7f3ef8bac640 1 -- 172.21.15.150:0/3983996524 <== mon.2 v2:172.21.15.157:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee4034070 con 0x7f3ef415ab90 2024-03-17T23:50:22.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.684+0000 7f3ef8bac640 1 -- 172.21.15.150:0/3983996524 <== mon.2 v2:172.21.15.157:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f3ee4030840 con 0x7f3ef415ab90 2024-03-17T23:50:22.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.684+0000 7f3ef8bac640 1 -- 172.21.15.150:0/3983996524 <== mon.2 v2:172.21.15.157:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee4030c30 con 0x7f3ef415ab90 2024-03-17T23:50:22.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.685+0000 7f3efbe12640 1 -- 172.21.15.150:0/3983996524 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ef415ab90 msgr2=0x7f3ef415bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:22.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.685+0000 7f3efbe12640 1 --2- 172.21.15.150:0/3983996524 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ef415ab90 0x7f3ef415bd10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3ee4009870 tx=0x7f3ee402b980 comp rx=0 tx=0).stop 2024-03-17T23:50:22.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.685+0000 7f3efbe12640 1 -- 172.21.15.150:0/3983996524 shutdown_connections 2024-03-17T23:50:22.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.685+0000 7f3efbe12640 1 --2- 172.21.15.150:0/3983996524 >> v2:172.21.15.157:3300/0 conn(0x7f3ef406ecd0 0x7f3ef406f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.685+0000 7f3efbe12640 1 --2- 172.21.15.150:0/3983996524 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ef415ab90 0x7f3ef415bd10 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.685+0000 7f3efbe12640 1 -- 172.21.15.150:0/3983996524 >> 172.21.15.150:0/3983996524 conn(0x7f3ef406b190 msgr2=0x7f3ef406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:22.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.685+0000 7f3efbe12640 1 -- 172.21.15.150:0/3983996524 shutdown_connections 2024-03-17T23:50:22.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.685+0000 7f3efbe12640 1 -- 172.21.15.150:0/3983996524 wait complete. 2024-03-17T23:50:22.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.686+0000 7f3efbe12640 1 Processor -- start 2024-03-17T23:50:22.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.687+0000 7f3efbe12640 1 -- start start 2024-03-17T23:50:22.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.687+0000 7f3efbe12640 1 --2- >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ef406ecd0 0x7f3ef4154390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:22.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.687+0000 7f3efbe12640 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ef41514f0 0x7f3ef413fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:22.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.687+0000 7f3efbe12640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ef415ab90 0x7f3ef414ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:22.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.687+0000 7f3efbe12640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef40af510 con 0x7f3ef415ab90 2024-03-17T23:50:22.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.687+0000 7f3efbe12640 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef415b8f0 con 0x7f3ef41514f0 2024-03-17T23:50:22.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.687+0000 7f3efbe12640 1 -- --> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] -- mon_getmap magic: 0 -- 0x7f3ef41518d0 con 0x7f3ef406ecd0 2024-03-17T23:50:22.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.687+0000 7f3efa3af640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ef415ab90 0x7f3ef414ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:22.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.687+0000 7f3efa3af640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ef415ab90 0x7f3ef414ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34512/0 (socket says 172.21.15.150:34512) 2024-03-17T23:50:22.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.687+0000 7f3efa3af640 1 -- 172.21.15.150:0/3976442165 learned_addr learned my addr 172.21.15.150:0/3976442165 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-03-17T23:50:22.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.688+0000 7f3ef9bae640 1 --2- 172.21.15.150:0/3976442165 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ef406ecd0 0x7f3ef4154390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:22.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.688+0000 7f3ef93ad640 1 --2- 172.21.15.150:0/3976442165 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ef41514f0 0x7f3ef413fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:22.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.688+0000 7f3efa3af640 1 -- 172.21.15.150:0/3976442165 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ef406ecd0 msgr2=0x7f3ef4154390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:22.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.688+0000 7f3efa3af640 1 --2- 172.21.15.150:0/3976442165 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ef406ecd0 0x7f3ef4154390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.688+0000 7f3efa3af640 1 -- 172.21.15.150:0/3976442165 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ef41514f0 msgr2=0x7f3ef413fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:22.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.688+0000 7f3efa3af640 1 --2- 172.21.15.150:0/3976442165 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ef41514f0 0x7f3ef413fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.688+0000 7f3efa3af640 1 -- 172.21.15.150:0/3976442165 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ef4140230 con 0x7f3ef415ab90 2024-03-17T23:50:22.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.688+0000 7f3ef93ad640 1 --2- 172.21.15.150:0/3976442165 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ef41514f0 0x7f3ef413fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-17T23:50:22.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.688+0000 7f3efa3af640 1 --2- 172.21.15.150:0/3976442165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ef415ab90 0x7f3ef414ecb0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f3eec00efc0 tx=0x7f3eec00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:22.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.688+0000 7f3eea7fc640 1 -- 172.21.15.150:0/3976442165 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eec018030 con 0x7f3ef415ab90 2024-03-17T23:50:22.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.688+0000 7f3eea7fc640 1 -- 172.21.15.150:0/3976442165 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f3eec00cdf0 con 0x7f3ef415ab90 2024-03-17T23:50:22.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.688+0000 7f3efbe12640 1 -- 172.21.15.150:0/3976442165 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ef41532f0 con 0x7f3ef415ab90 2024-03-17T23:50:22.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.688+0000 7f3eea7fc640 1 -- 172.21.15.150:0/3976442165 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eec003bd0 con 0x7f3ef415ab90 2024-03-17T23:50:22.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.688+0000 7f3efbe12640 1 -- 172.21.15.150:0/3976442165 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ef41538b0 con 0x7f3ef415ab90 2024-03-17T23:50:22.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.689+0000 7f3efbe12640 1 -- 172.21.15.150:0/3976442165 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ef40681a0 con 0x7f3ef415ab90 2024-03-17T23:50:22.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.689+0000 7f3eea7fc640 1 -- 172.21.15.150:0/3976442165 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 90136+0+0 (secure 0 0 0) 0x7f3eec005d10 con 0x7f3ef415ab90 2024-03-17T23:50:22.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.690+0000 7f3eea7fc640 1 --2- 172.21.15.150:0/3976442165 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3ed806bbd0 0x7f3ed806e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-17T23:50:22.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.690+0000 7f3eea7fc640 1 -- 172.21.15.150:0/3976442165 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5335+0+0 (secure 0 0 0) 0x7f3eec08b270 con 0x7f3ef415ab90 2024-03-17T23:50:22.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.690+0000 7f3ef9bae640 1 --2- 172.21.15.150:0/3976442165 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3ed806bbd0 0x7f3ed806e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-17T23:50:22.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.690+0000 7f3ef9bae640 1 --2- 172.21.15.150:0/3976442165 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3ed806bbd0 0x7f3ed806e090 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f3ef0007fd0 tx=0x7f3ef0007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-17T23:50:22.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.694+0000 7f3eea7fc640 1 -- 172.21.15.150:0/3976442165 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185025 (secure 0 0 0) 0x7f3eec022070 con 0x7f3ef415ab90 2024-03-17T23:50:22.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.904+0000 7f3efbe12640 1 -- 172.21.15.150:0/3976442165 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ef406bf90 con 0x7f3ef415ab90 2024-03-17T23:50:22.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.904+0000 7f3eea7fc640 1 -- 172.21.15.150:0/3976442165 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+3878 (secure 0 0 0) 0x7f3eec010040 con 0x7f3ef415ab90 2024-03-17T23:50:22.906 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-03-17T23:50:22.906 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":24,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5555,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/2082413564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":2082413564},{"type":"v1","addr":"172.21.15.150:6836","nonce":2082413564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":5885,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/3661802657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6833","nonce":3661802657},{"type":"v1","addr":"172.21.15.157:6835","nonce":3661802657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24},{"gid":6043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/2350077845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":2350077845},{"type":"v1","addr":"172.21.15.150:6837","nonce":2350077845}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":6052,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6834/137606728","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":137606728},{"type":"v1","addr":"172.21.15.157:6834","nonce":137606728}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[]} 2024-03-17T23:50:22.907 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 24 2024-03-17T23:50:22.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.906+0000 7f3efbe12640 1 -- 172.21.15.150:0/3976442165 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3ed806bbd0 msgr2=0x7f3ed806e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:22.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.906+0000 7f3efbe12640 1 --2- 172.21.15.150:0/3976442165 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3ed806bbd0 0x7f3ed806e090 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f3ef0007fd0 tx=0x7f3ef0007e80 comp rx=0 tx=0).stop 2024-03-17T23:50:22.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.906+0000 7f3efbe12640 1 -- 172.21.15.150:0/3976442165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ef415ab90 msgr2=0x7f3ef414ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-17T23:50:22.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.906+0000 7f3efbe12640 1 --2- 172.21.15.150:0/3976442165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ef415ab90 0x7f3ef414ecb0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f3eec00efc0 tx=0x7f3eec00c490 comp rx=0 tx=0).stop 2024-03-17T23:50:22.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.906+0000 7f3efbe12640 1 -- 172.21.15.150:0/3976442165 shutdown_connections 2024-03-17T23:50:22.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.906+0000 7f3efbe12640 1 --2- 172.21.15.150:0/3976442165 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3ef415ab90 0x7f3ef414ecb0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.906+0000 7f3efbe12640 1 --2- 172.21.15.150:0/3976442165 >> [v2:172.21.15.150:6832/4239152602,v1:172.21.15.150:6833/4239152602] conn(0x7f3ed806bbd0 0x7f3ed806e090 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.906+0000 7f3efbe12640 1 --2- 172.21.15.150:0/3976442165 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ef41514f0 0x7f3ef413fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.906+0000 7f3efbe12640 1 --2- 172.21.15.150:0/3976442165 >> [v2:172.21.15.157:3301/0,v1:172.21.15.157:6790/0] conn(0x7f3ef406ecd0 0x7f3ef4154390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-17T23:50:22.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.906+0000 7f3efbe12640 1 -- 172.21.15.150:0/3976442165 >> 172.21.15.150:0/3976442165 conn(0x7f3ef406b190 msgr2=0x7f3ef40b0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-17T23:50:22.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.906+0000 7f3efbe12640 1 -- 172.21.15.150:0/3976442165 shutdown_connections 2024-03-17T23:50:22.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-03-17T23:50:22.906+0000 7f3efbe12640 1 -- 172.21.15.150:0/3976442165 wait complete. 2024-03-17T23:50:22.919 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-17T23:50:22.919 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-17T23:50:22.919 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-17T23:50:22.919 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:50:22.928 INFO:tasks.ceph.mds.a:Stopped 2024-03-17T23:50:22.929 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-17T23:50:22.929 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:50:22.979 INFO:tasks.ceph.mds.c:Stopped 2024-03-17T23:50:22.979 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-17T23:50:22.980 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:50:22.992 INFO:tasks.ceph.mds.b:Stopped 2024-03-17T23:50:22.992 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-17T23:50:22.992 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:50:23.044 INFO:tasks.ceph.mds.d:Stopped 2024-03-17T23:50:23.045 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-17T23:50:23.045 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-17T23:50:23.045 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:50:23.076 INFO:tasks.ceph.osd.0:Stopped 2024-03-17T23:50:23.076 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-17T23:50:23.076 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:50:23.144 INFO:tasks.ceph.osd.1:Stopped 2024-03-17T23:50:23.144 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-17T23:50:23.144 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:50:23.211 INFO:tasks.ceph.osd.2:Stopped 2024-03-17T23:50:23.212 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-17T23:50:23.212 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:50:23.279 INFO:tasks.ceph.osd.3:Stopped 2024-03-17T23:50:23.279 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-17T23:50:23.279 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:50:23.314 INFO:tasks.ceph.osd.4:Stopped 2024-03-17T23:50:23.315 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-17T23:50:23.315 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:50:23.381 INFO:tasks.ceph.osd.5:Stopped 2024-03-17T23:50:23.381 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-17T23:50:23.381 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:50:23.451 INFO:tasks.ceph.osd.6:Stopped 2024-03-17T23:50:23.452 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-17T23:50:23.452 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:50:23.522 INFO:tasks.ceph.osd.7:Stopped 2024-03-17T23:50:23.522 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-17T23:50:23.523 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-17T23:50:23.523 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:50:23.549 INFO:tasks.ceph.mgr.y:Stopped 2024-03-17T23:50:23.550 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-17T23:50:23.550 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:50:23.591 INFO:tasks.ceph.mgr.x:Stopped 2024-03-17T23:50:23.591 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-17T23:50:23.592 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-17T23:50:23.592 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:50:23.603 INFO:tasks.ceph.mon.a:Stopped 2024-03-17T23:50:23.603 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-17T23:50:23.603 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:50:23.644 INFO:tasks.ceph.mon.b:Stopped 2024-03-17T23:50:23.645 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-17T23:50:23.645 INFO:teuthology.orchestra.run:waiting for 300 2024-03-17T23:50:23.697 INFO:tasks.ceph.mon.c:Stopped 2024-03-17T23:50:23.697 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-17T23:50:23.698 DEBUG:teuthology.orchestra.run.smithi150:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'mons down, quorum' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-17T23:50:23.731 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi150.front.sepia.ceph.com 2024-03-17T23:50:23.731 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-17T23:50:23.932 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi150.front.sepia.ceph.com 2024-03-17T23:50:23.933 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-17T23:50:24.061 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi150.front.sepia.ceph.com 2024-03-17T23:50:24.061 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-17T23:50:24.128 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi150.front.sepia.ceph.com 2024-03-17T23:50:24.128 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-17T23:50:24.185 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-17T23:50:24.186 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-17T23:50:24.331 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-17T23:50:24.331 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-17T23:50:24.399 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-17T23:50:24.400 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-17T23:50:24.449 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi157.front.sepia.ceph.com 2024-03-17T23:50:24.449 DEBUG:teuthology.orchestra.run.smithi157:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-17T23:50:24.500 INFO:tasks.ceph:Archiving mon data... 2024-03-17T23:50:24.501 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608488/data/mon.a.tgz 2024-03-17T23:50:24.501 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2024-03-17T23:50:24.516 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.KonFPIGkob 2024-03-17T23:50:24.516 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.KonFPIGkob 2024-03-17T23:50:24.663 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.KonFPIGkob 2024-03-17T23:50:24.748 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.KonFPIGkob is 359KB 2024-03-17T23:50:24.844 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.KonFPIGkob 2024-03-17T23:50:24.858 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608488/data/mon.b.tgz 2024-03-17T23:50:24.858 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2024-03-17T23:50:24.875 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.wV7EcN4veY 2024-03-17T23:50:24.875 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.wV7EcN4veY 2024-03-17T23:50:25.071 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.wV7EcN4veY 2024-03-17T23:50:25.157 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.wV7EcN4veY is 476KB 2024-03-17T23:50:25.257 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.wV7EcN4veY 2024-03-17T23:50:25.272 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608488/data/mon.c.tgz 2024-03-17T23:50:25.272 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2024-03-17T23:50:25.325 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.ifdF1QQ4aW 2024-03-17T23:50:25.326 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.ifdF1QQ4aW 2024-03-17T23:50:25.480 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.ifdF1QQ4aW 2024-03-17T23:50:25.564 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.ifdF1QQ4aW is 476KB 2024-03-17T23:50:25.665 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.ifdF1QQ4aW 2024-03-17T23:50:25.680 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-17T23:50:25.680 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-17T23:50:25.682 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-17T23:50:25.767 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-17T23:50:25.801 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-17T23:50:25.801 INFO:tasks.ceph:Archiving crash dumps... 2024-03-17T23:50:25.803 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608488/remote/smithi150/crash 2024-03-17T23:50:25.804 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-17T23:50:25.836 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608488/remote/smithi157/crash 2024-03-17T23:50:25.837 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-17T23:50:25.867 INFO:tasks.ceph:Shutting down logrotate 2024-03-17T23:50:25.867 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-17T23:50:25.881 DEBUG:teuthology.orchestra.run.smithi157:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-17T23:50:26.180 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-17T23:50:26.209 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-17T23:50:26.236 INFO:tasks.ceph:Compressing logs... 2024-03-17T23:50:26.236 DEBUG:teuthology.orchestra.run.smithi150:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-17T23:50:26.252 DEBUG:teuthology.orchestra.run.smithi157:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-17T23:50:26.277 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53222.log 2024-03-17T23:50:26.278 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47543.log 2024-03-17T23:50:26.278 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50154.log 2024-03-17T23:50:26.279 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47417.log 2024-03-17T23:50:26.280 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53222.log.gz 2024-03-17T23:50:26.280 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50154.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47543.log.gz 2024-03-17T23:50:26.280 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59807.log 2024-03-17T23:50:26.280 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47417.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50154.log.gz 2024-03-17T23:50:26.328 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-17T23:50:26.329 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37474.log 2024-03-17T23:50:26.329 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37554.log 2024-03-17T23:50:26.329 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.37474.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-17T23:50:26.329 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.37554.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37474.log.gz 2024-03-17T23:50:26.329 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-17T23:50:26.329 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-17T23:50:26.329 INFO:teuthology.orchestra.run.smithi157.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37554.log.gz 2024-03-17T23:50:26.329 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-17T23:50:26.330 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-17T23:50:26.330 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.43779.log 2024-03-17T23:50:26.330 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-17T23:50:26.330 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.3.43779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37434.log 2024-03-17T23:50:26.330 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-mgr.x.log: 91.2% -- replaced with /var/log/ceph/ceph-client.3.43779.log.gz 2024-03-17T23:50:26.330 INFO:teuthology.orchestra.run.smithi157.stderr: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-17T23:50:26.330 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43322.log 2024-03-17T23:50:26.330 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.37434.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-17T23:50:26.331 INFO:teuthology.orchestra.run.smithi157.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37434.log.gz 2024-03-17T23:50:26.331 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.1.43322.log: 89.9% -- replaced with /var/log/ceph/ceph-client.1.43322.log.gz 2024-03-17T23:50:26.331 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37354.log 2024-03-17T23:50:26.331 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37394.log 2024-03-17T23:50:26.331 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.37354.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37354.log.gz 2024-03-17T23:50:26.331 INFO:teuthology.orchestra.run.smithi157.stderr: 91.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-17T23:50:26.331 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-17T23:50:26.332 INFO:teuthology.orchestra.run.smithi157.stderr: 93.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-17T23:50:26.332 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-17T23:50:26.332 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.37394.log: /var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37274.log 2024-03-17T23:50:26.332 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-mds.d.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37394.log.gz 2024-03-17T23:50:26.332 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37514.log 2024-03-17T23:50:26.332 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.37274.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37274.log.gz 2024-03-17T23:50:26.332 INFO:teuthology.orchestra.run.smithi157.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-17T23:50:26.332 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-17T23:50:26.333 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37314.log 2024-03-17T23:50:26.333 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.37514.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37514.log.gz 2024-03-17T23:50:26.333 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.37314.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37314.log.gz 2024-03-17T23:50:26.369 INFO:teuthology.orchestra.run.smithi157.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-17T23:50:26.436 INFO:teuthology.orchestra.run.smithi157.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-17T23:50:26.461 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47249.log 2024-03-17T23:50:26.461 INFO:teuthology.orchestra.run.smithi150.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.47417.log.gz 2024-03-17T23:50:26.461 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.59807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63030.log 2024-03-17T23:50:26.461 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51563.log 2024-03-17T23:50:26.461 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59807.log.gz 2024-03-17T23:50:26.462 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.63030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51663.log 2024-03-17T23:50:26.462 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47249.log.gz 2024-03-17T23:50:26.462 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63030.log.gz 2024-03-17T23:50:26.462 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64557.log 2024-03-17T23:50:26.462 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51663.log: gzip -5 88.6% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.51563.log.gz 2024-03-17T23:50:26.462 INFO:teuthology.orchestra.run.smithi150.stderr: /var/log/ceph/ceph-mon.a.log 2024-03-17T23:50:26.462 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4%/var/log/ceph/ceph-client.admin.64557.log: -- replaced with /var/log/ceph/ceph-client.admin.51663.log.gz 2024-03-17T23:50:26.462 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49188.log 2024-03-17T23:50:26.463 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mon.a.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64557.log.gz 2024-03-17T23:50:26.463 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59131.log 2024-03-17T23:50:26.463 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49818.log 2024-03-17T23:50:26.463 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.59131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43582.log 2024-03-17T23:50:26.463 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49188.log.gz 2024-03-17T23:50:26.463 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49818.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59131.log.gz 2024-03-17T23:50:26.463 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63114.log 2024-03-17T23:50:26.463 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49818.log.gz 2024-03-17T23:50:26.463 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56161.log 2024-03-17T23:50:26.464 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.63114.log: gzip -5 89.2% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.43582.log.gz /var/log/ceph/ceph-client.admin.49986.log 2024-03-17T23:50:26.464 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-03-17T23:50:26.464 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63114.log.gz 2024-03-17T23:50:26.464 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46158.log 2024-03-17T23:50:26.464 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54290.log 2024-03-17T23:50:26.464 INFO:teuthology.orchestra.run.smithi150.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.56161.log.gz 2024-03-17T23:50:26.464 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46158.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49986.log.gz 2024-03-17T23:50:26.464 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62946.log 2024-03-17T23:50:26.465 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46158.log.gz 2024-03-17T23:50:26.465 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46482.log 2024-03-17T23:50:26.465 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7%/var/log/ceph/ceph-client.admin.62946.log: -- replaced with /var/log/ceph/ceph-client.admin.54290.log.gz 2024-03-17T23:50:26.465 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47669.log 2024-03-17T23:50:26.465 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46482.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62946.log.gz 2024-03-17T23:50:26.465 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55119.log 2024-03-17T23:50:26.465 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47669.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46482.log.gz 2024-03-17T23:50:26.465 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57494.log 2024-03-17T23:50:26.465 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55119.log: gzip 87.8% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.47669.log.gz -- /var/log/ceph/ceph-client.admin.57997.log 2024-03-17T23:50:26.466 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-03-17T23:50:26.466 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57494.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.55119.log.gz 2024-03-17T23:50:26.466 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44237.log 2024-03-17T23:50:26.466 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57494.log.gz 2024-03-17T23:50:26.466 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47459.log 2024-03-17T23:50:26.466 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47081.log 2024-03-17T23:50:26.466 INFO:teuthology.orchestra.run.smithi150.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57997.log.gz 2024-03-17T23:50:26.466 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47459.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44237.log.gz 2024-03-17T23:50:26.467 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46485.log 2024-03-17T23:50:26.467 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47459.log.gz 2024-03-17T23:50:26.467 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45211.log 2024-03-17T23:50:26.467 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46485.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47081.log.gz 2024-03-17T23:50:26.467 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55531.log 2024-03-17T23:50:26.467 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46485.log.gz 2024-03-17T23:50:26.467 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53141.log 2024-03-17T23:50:26.467 INFO:teuthology.orchestra.run.smithi150.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.45211.log.gz 2024-03-17T23:50:26.467 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50249.log 2024-03-17T23:50:26.468 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59173.log 2024-03-17T23:50:26.468 INFO:teuthology.orchestra.run.smithi150.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55531.log.gz 2024-03-17T23:50:26.468 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50249.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53141.log.gz 2024-03-17T23:50:26.468 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54248.log 2024-03-17T23:50:26.468 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50249.log.gz 2024-03-17T23:50:26.468 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.59173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45683.log 2024-03-17T23:50:26.468 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54248.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59173.log.gz 2024-03-17T23:50:26.468 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48394.log 2024-03-17T23:50:26.468 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54248.log.gz 2024-03-17T23:50:26.469 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54080.log 2024-03-17T23:50:26.469 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45683.log.gz 2024-03-17T23:50:26.469 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46702.log 2024-03-17T23:50:26.469 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54080.log: 89.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.48394.log.gz 2024-03-17T23:50:26.469 INFO:teuthology.orchestra.run.smithi150.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51705.log 2024-03-17T23:50:26.469 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54080.log.gz 2024-03-17T23:50:26.469 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44387.log 2024-03-17T23:50:26.469 INFO:teuthology.orchestra.run.smithi150.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46702.log.gz 2024-03-17T23:50:26.470 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56717.log 2024-03-17T23:50:26.470 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44387.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51705.log.gz 2024-03-17T23:50:26.470 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60843.log 2024-03-17T23:50:26.470 INFO:teuthology.orchestra.run.smithi150.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44387.log.gz 2024-03-17T23:50:26.470 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56555.log 2024-03-17T23:50:26.470 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56717.log.gz 2024-03-17T23:50:26.470 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.60843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52935.log 2024-03-17T23:50:26.470 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56555.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60843.log.gz 2024-03-17T23:50:26.470 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44279.log 2024-03-17T23:50:26.471 INFO:teuthology.orchestra.run.smithi150.stderr: 89.6%/var/log/ceph/ceph-client.admin.52935.log: -- replaced with /var/log/ceph/ceph-client.admin.56555.log.gz 2024-03-17T23:50:26.471 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57268.log 2024-03-17T23:50:26.471 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52935.log.gz 2024-03-17T23:50:26.471 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.61283.log 2024-03-17T23:50:26.471 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57268.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44279.log.gz 2024-03-17T23:50:26.471 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62778.log 2024-03-17T23:50:26.471 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57268.log.gz 2024-03-17T23:50:26.471 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.0.61283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45767.log 2024-03-17T23:50:26.471 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.62778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46150.log 2024-03-17T23:50:26.472 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62778.log.gz 2024-03-17T23:50:26.472 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57436.log 2024-03-17T23:50:26.472 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- 90.0% /var/log/ceph/ceph-client.admin.64307.log 2024-03-17T23:50:26.472 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.0.61283.log.gz 2024-03-17T23:50:26.472 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46150.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45767.log.gz 2024-03-17T23:50:26.472 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43751.log 2024-03-17T23:50:26.472 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57436.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46150.log.gz 2024-03-17T23:50:26.473 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.64307.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.57436.log.gz 2024-03-17T23:50:26.473 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53954.log 2024-03-17T23:50:26.473 INFO:teuthology.orchestra.run.smithi150.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.64307.log.gz 2024-03-17T23:50:26.473 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50459.log 2024-03-17T23:50:26.473 INFO:teuthology.orchestra.run.smithi150.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43751.log.gz 2024-03-17T23:50:26.473 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45127.log 2024-03-17T23:50:26.473 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50459.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53954.log.gz 2024-03-17T23:50:26.473 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51024.log 2024-03-17T23:50:26.473 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3%/var/log/ceph/ceph-client.admin.45127.log: -- replaced with /var/log/ceph/ceph-client.admin.50459.log.gz 2024-03-17T23:50:26.474 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58790.log 2024-03-17T23:50:26.474 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45127.log.gz 2024-03-17T23:50:26.474 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57664.log 2024-03-17T23:50:26.474 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.58790.log: 87.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.51024.log.gz 2024-03-17T23:50:26.474 INFO:teuthology.orchestra.run.smithi150.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.57622.log 2024-03-17T23:50:26.474 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57664.log: gzip 89.3% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.58790.log.gz 2024-03-17T23:50:26.474 INFO:teuthology.orchestra.run.smithi150.stderr: -- /var/log/ceph/ceph-client.admin.50601.log 2024-03-17T23:50:26.474 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57664.log.gz 2024-03-17T23:50:26.475 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51153.log 2024-03-17T23:50:26.475 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50601.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57622.log.gz 2024-03-17T23:50:26.475 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56471.log 2024-03-17T23:50:26.475 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51153.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50601.log.gz 2024-03-17T23:50:26.475 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45893.log 2024-03-17T23:50:26.475 INFO:teuthology.orchestra.run.smithi150.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51153.log.gz 2024-03-17T23:50:26.475 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58706.log 2024-03-17T23:50:26.475 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45893.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56471.log.gz 2024-03-17T23:50:26.475 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55615.log 2024-03-17T23:50:26.476 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5%/var/log/ceph/ceph-client.admin.58706.log: -- replaced with /var/log/ceph/ceph-client.admin.45893.log.gz 2024-03-17T23:50:26.476 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50417.log 2024-03-17T23:50:26.476 INFO:teuthology.orchestra.run.smithi150.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.58706.log.gz 2024-03-17T23:50:26.476 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55287.log 2024-03-17T23:50:26.476 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50417.log: gzip -5 89.1% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.55615.log.gz 2024-03-17T23:50:26.476 INFO:teuthology.orchestra.run.smithi150.stderr: /var/log/ceph/ceph-client.admin.63646.log 2024-03-17T23:50:26.476 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50417.log.gz 2024-03-17T23:50:26.477 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59765.log 2024-03-17T23:50:26.477 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55287.log.gz 2024-03-17T23:50:26.477 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.63646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50974.log 2024-03-17T23:50:26.477 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.59765.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.63646.log.gz 2024-03-17T23:50:26.477 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47039.log 2024-03-17T23:50:26.477 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7%/var/log/ceph/ceph-client.admin.50974.log: -- replaced with /var/log/ceph/ceph-client.admin.59765.log.gz 2024-03-17T23:50:26.477 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53824.log 2024-03-17T23:50:26.477 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50974.log.gz 2024-03-17T23:50:26.477 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64139.log 2024-03-17T23:50:26.478 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53824.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.47039.log.gz 2024-03-17T23:50:26.478 INFO:teuthology.orchestra.run.smithi150.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50643.log 2024-03-17T23:50:26.478 INFO:teuthology.orchestra.run.smithi150.stderr: 89.5%/var/log/ceph/ceph-client.admin.64139.log: -- replaced with /var/log/ceph/ceph-client.admin.53824.log.gz 2024-03-17T23:50:26.478 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53517.log 2024-03-17T23:50:26.478 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64139.log.gz 2024-03-17T23:50:26.478 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46843.log 2024-03-17T23:50:26.478 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53517.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50643.log.gz 2024-03-17T23:50:26.478 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46495.log 2024-03-17T23:50:26.479 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46843.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53517.log.gz 2024-03-17T23:50:26.479 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63240.log 2024-03-17T23:50:26.479 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46843.log.gz 2024-03-17T23:50:26.479 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59007.log 2024-03-17T23:50:26.479 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.63240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52641.log 2024-03-17T23:50:26.479 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46495.log.gz 2024-03-17T23:50:26.479 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63240.log.gz 2024-03-17T23:50:26.479 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.59007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45599.log 2024-03-17T23:50:26.479 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59007.log.gz 2024-03-17T23:50:26.480 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52893.log 2024-03-17T23:50:26.480 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45599.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52641.log.gz 2024-03-17T23:50:26.480 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63604.log 2024-03-17T23:50:26.480 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52893.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45599.log.gz 2024-03-17T23:50:26.480 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.62247.log 2024-03-17T23:50:26.480 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52893.log.gz 2024-03-17T23:50:26.480 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53912.log 2024-03-17T23:50:26.480 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.63604.log: /var/log/ceph/ceph-client.0.62247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43923.log 2024-03-17T23:50:26.480 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.63604.log.gz 2024-03-17T23:50:26.481 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55447.log 2024-03-17T23:50:26.481 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53912.log.gz 2024-03-17T23:50:26.481 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46156.log 2024-03-17T23:50:26.481 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43923.log.gz 2024-03-17T23:50:26.481 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59387.log 2024-03-17T23:50:26.481 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55447.log: 90.0% -- replaced with /var/log/ceph/ceph-client.0.62247.log.gz 2024-03-17T23:50:26.481 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49356.log 2024-03-17T23:50:26.481 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55447.log.gz 2024-03-17T23:50:26.481 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.59387.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46156.log.gz 2024-03-17T23:50:26.482 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43667.log 2024-03-17T23:50:26.482 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8%/var/log/ceph/ceph-client.admin.49356.log: -- replaced with /var/log/ceph/ceph-client.admin.59387.log.gz 2024-03-17T23:50:26.482 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62668.log 2024-03-17T23:50:26.482 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43667.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49356.log.gz 2024-03-17T23:50:26.482 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58393.log 2024-03-17T23:50:26.482 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.62668.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43667.log.gz 2024-03-17T23:50:26.482 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37719.log 2024-03-17T23:50:26.482 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.58393.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62668.log.gz 2024-03-17T23:50:26.483 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50727.log 2024-03-17T23:50:26.483 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58393.log.gz 2024-03-17T23:50:26.483 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61683.log 2024-03-17T23:50:26.483 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50727.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37719.log.gz 2024-03-17T23:50:26.483 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49566.log 2024-03-17T23:50:26.483 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.61683.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50727.log.gz 2024-03-17T23:50:26.483 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63928.log 2024-03-17T23:50:26.483 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61683.log.gz 2024-03-17T23:50:26.484 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47501.log 2024-03-17T23:50:26.484 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.63928.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49566.log.gz 2024-03-17T23:50:26.484 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49146.log 2024-03-17T23:50:26.484 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47501.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63928.log.gz 2024-03-17T23:50:26.484 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45809.log 2024-03-17T23:50:26.484 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47501.log.gz 2024-03-17T23:50:26.484 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49776.log 2024-03-17T23:50:26.484 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45809.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49146.log.gz 2024-03-17T23:50:26.484 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51747.log 2024-03-17T23:50:26.485 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49776.log: gzip 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45809.log.gz -5 2024-03-17T23:50:26.485 INFO:teuthology.orchestra.run.smithi150.stderr: --verbose -- /var/log/ceph/ceph-client.admin.58081.log 2024-03-17T23:50:26.485 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51747.log: gzip 88.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.49776.log.gz --verbose 2024-03-17T23:50:26.485 INFO:teuthology.orchestra.run.smithi150.stderr: -- /var/log/ceph/ceph-client.admin.43271.log 2024-03-17T23:50:26.485 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.58081.log: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.51747.log.gz -5 2024-03-17T23:50:26.485 INFO:teuthology.orchestra.run.smithi150.stderr: --verbose -- /var/log/ceph/ceph-client.admin.58267.log 2024-03-17T23:50:26.485 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43271.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58081.log.gz 2024-03-17T23:50:26.485 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52413.log 2024-03-17T23:50:26.485 INFO:teuthology.orchestra.run.smithi150.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43271.log.gz 2024-03-17T23:50:26.486 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.58267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52329.log 2024-03-17T23:50:26.486 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52413.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58267.log.gz 2024-03-17T23:50:26.486 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57706.log 2024-03-17T23:50:26.486 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52329.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52413.log.gz 2024-03-17T23:50:26.486 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63759.log 2024-03-17T23:50:26.486 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52329.log.gz 2024-03-17T23:50:26.486 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64012.log 2024-03-17T23:50:26.486 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.63759.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.57706.log.gz 2024-03-17T23:50:26.487 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59429.log 2024-03-17T23:50:26.487 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.64012.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.63759.log.gz 2024-03-17T23:50:26.487 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45085.log 2024-03-17T23:50:26.487 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64012.log.gz 2024-03-17T23:50:26.487 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.59429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46815.log 2024-03-17T23:50:26.487 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45085.log: gzip -5 --verbose 89.2% -- /var/log/ceph/ceph-client.admin.59933.log -- replaced with /var/log/ceph/ceph-client.admin.59429.log.gz 2024-03-17T23:50:26.487 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-03-17T23:50:26.487 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45085.log.gz 2024-03-17T23:50:26.488 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44111.log 2024-03-17T23:50:26.488 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46815.log.gz 2024-03-17T23:50:26.488 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.59933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54985.log 2024-03-17T23:50:26.488 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59933.log.gz 2024-03-17T23:50:26.488 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57784.log 2024-03-17T23:50:26.488 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7%/var/log/ceph/ceph-client.admin.54985.log: -- replaced with /var/log/ceph/ceph-client.admin.44111.log.gz 2024-03-17T23:50:26.488 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47333.log 2024-03-17T23:50:26.488 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57784.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54985.log.gz 2024-03-17T23:50:26.488 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48952.log 2024-03-17T23:50:26.489 INFO:teuthology.orchestra.run.smithi150.stderr: 89.3%/var/log/ceph/ceph-client.admin.47333.log: -- replaced with /var/log/ceph/ceph-client.admin.57784.log.gz 2024-03-17T23:50:26.489 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62584.log 2024-03-17T23:50:26.489 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48952.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47333.log.gz 2024-03-17T23:50:26.489 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54038.log 2024-03-17T23:50:26.489 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48952.log.gz 2024-03-17T23:50:26.489 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.62584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62820.log 2024-03-17T23:50:26.489 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54038.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.62584.log.gz 2024-03-17T23:50:26.489 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52539.log 2024-03-17T23:50:26.490 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0%/var/log/ceph/ceph-client.admin.62820.log: -- replaced with /var/log/ceph/ceph-client.admin.54038.log.gz 2024-03-17T23:50:26.490 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54857.log 2024-03-17T23:50:26.490 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7%/var/log/ceph/ceph-client.admin.52539.log: -- replaced with /var/log/ceph/ceph-client.admin.62820.log.gz 2024-03-17T23:50:26.490 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56371.log 2024-03-17T23:50:26.490 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54857.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52539.log.gz 2024-03-17T23:50:26.490 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57226.log 2024-03-17T23:50:26.490 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54857.log.gz 2024-03-17T23:50:26.490 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.60301.log 2024-03-17T23:50:26.491 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57226.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56371.log.gz 2024-03-17T23:50:26.491 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46822.log 2024-03-17T23:50:26.491 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.0.60301.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57226.log.gz 2024-03-17T23:50:26.491 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54440.log 2024-03-17T23:50:26.491 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46818.log 2024-03-17T23:50:26.491 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46822.log.gz 2024-03-17T23:50:26.491 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62862.log 2024-03-17T23:50:26.491 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54440.log.gz 2024-03-17T23:50:26.492 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47207.log 2024-03-17T23:50:26.492 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46818.log: /var/log/ceph/ceph-client.admin.62862.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-17T23:50:26.492 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46818.log.gz 2024-03-17T23:50:26.492 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62862.log.gz 2024-03-17T23:50:26.492 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46153.log 2024-03-17T23:50:26.492 INFO:teuthology.orchestra.run.smithi150.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.0.60301.log.gz 2024-03-17T23:50:26.492 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64349.log 2024-03-17T23:50:26.492 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7%/var/log/ceph/ceph.audit.log: -- replaced with /var/log/ceph/ceph-client.admin.47207.log.gz 2024-03-17T23:50:26.493 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64433.log 2024-03-17T23:50:26.493 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.64349.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46153.log.gz 2024-03-17T23:50:26.493 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44069.log 2024-03-17T23:50:26.493 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64349.log.gz 2024-03-17T23:50:26.493 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57579.log 2024-03-17T23:50:26.493 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.64433.log: /var/log/ceph/ceph-client.admin.44069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55573.log 2024-03-17T23:50:26.493 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64433.log.gz 2024-03-17T23:50:26.493 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57579.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44069.log.gz 2024-03-17T23:50:26.493 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53433.log 2024-03-17T23:50:26.494 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57579.log.gz 2024-03-17T23:50:26.494 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59215.log 2024-03-17T23:50:26.494 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55573.log: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-17T23:50:26.494 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.57019.log 2024-03-17T23:50:26.494 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55573.log.gz 2024-03-17T23:50:26.494 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.59215.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53433.log.gz 2024-03-17T23:50:26.494 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58165.log 2024-03-17T23:50:26.495 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.0.57019.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59215.log.gz 2024-03-17T23:50:26.495 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60617.log 2024-03-17T23:50:26.495 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.58165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64054.log 2024-03-17T23:50:26.495 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.60617.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58165.log.gz 2024-03-17T23:50:26.495 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59345.log 2024-03-17T23:50:26.495 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60617.log.gz 2024-03-17T23:50:26.495 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52977.log 2024-03-17T23:50:26.495 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.64054.log: /var/log/ceph/ceph-client.admin.59345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46149.log 2024-03-17T23:50:26.496 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64054.log.gz 2024-03-17T23:50:26.496 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52977.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59345.log.gz 2024-03-17T23:50:26.496 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45479.log 2024-03-17T23:50:26.496 INFO:teuthology.orchestra.run.smithi150.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.0.57019.log.gz 2024-03-17T23:50:26.496 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63408.log 2024-03-17T23:50:26.496 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52977.log.gz 2024-03-17T23:50:26.496 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46149.log: /var/log/ceph/ceph-client.admin.45479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56329.log 2024-03-17T23:50:26.496 INFO:teuthology.orchestra.run.smithi150.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46149.log.gz 2024-03-17T23:50:26.497 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8%/var/log/ceph/ceph-client.admin.63408.log: -- replaced with /var/log/ceph/ceph-client.admin.45479.log.gz 2024-03-17T23:50:26.497 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57955.log 2024-03-17T23:50:26.497 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56329.log: gzip 89.1% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.63408.log.gz -- 2024-03-17T23:50:26.497 INFO:teuthology.orchestra.run.smithi150.stderr: /var/log/ceph/ceph-client.admin.37819.log 2024-03-17T23:50:26.497 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56329.log.gz 2024-03-17T23:50:26.497 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52077.log 2024-03-17T23:50:26.497 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37819.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57955.log.gz 2024-03-17T23:50:26.497 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58832.log 2024-03-17T23:50:26.497 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52077.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37819.log.gz 2024-03-17T23:50:26.498 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54374.log 2024-03-17T23:50:26.498 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52077.log.gz 2024-03-17T23:50:26.498 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.58832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53019.log 2024-03-17T23:50:26.498 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54374.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58832.log.gz 2024-03-17T23:50:26.498 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58664.log 2024-03-17T23:50:26.498 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52287.log 2024-03-17T23:50:26.498 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53019.log.gz 2024-03-17T23:50:26.498 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.58664.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.54374.log.gz 2024-03-17T23:50:26.499 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51521.log 2024-03-17T23:50:26.499 INFO:teuthology.orchestra.run.smithi150.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58664.log.gz 2024-03-17T23:50:26.499 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51279.log 2024-03-17T23:50:26.499 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51521.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52287.log.gz 2024-03-17T23:50:26.499 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52683.log 2024-03-17T23:50:26.499 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51521.log.gz 2024-03-17T23:50:26.499 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45725.log 2024-03-17T23:50:26.499 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52683.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.51279.log.gz -5 2024-03-17T23:50:26.500 INFO:teuthology.orchestra.run.smithi150.stderr: --verbose -- /var/log/ceph/ceph-client.admin.57352.log 2024-03-17T23:50:26.500 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45725.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52683.log.gz 2024-03-17T23:50:26.500 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59849.log 2024-03-17T23:50:26.500 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45725.log.gz 2024-03-17T23:50:26.500 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61741.log 2024-03-17T23:50:26.500 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.59849.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57352.log.gz 2024-03-17T23:50:26.500 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50375.log 2024-03-17T23:50:26.500 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.61741.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.59849.log.gz 2024-03-17T23:50:26.500 INFO:teuthology.orchestra.run.smithi150.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52245.log 2024-03-17T23:50:26.501 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61741.log.gz 2024-03-17T23:50:26.501 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50559.log 2024-03-17T23:50:26.501 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52245.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50375.log.gz 2024-03-17T23:50:26.501 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51479.log 2024-03-17T23:50:26.501 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50559.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52245.log.gz 2024-03-17T23:50:26.501 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60801.log 2024-03-17T23:50:26.501 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50559.log.gz 2024-03-17T23:50:26.501 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52767.log 2024-03-17T23:50:26.502 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.60801.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51479.log.gz 2024-03-17T23:50:26.502 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47165.log 2024-03-17T23:50:26.502 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9%/var/log/ceph/ceph-client.admin.52767.log: -- replaced with /var/log/ceph/ceph-client.admin.60801.log.gz 2024-03-17T23:50:26.502 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46157.log 2024-03-17T23:50:26.502 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52767.log.gz 2024-03-17T23:50:26.502 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52371.log 2024-03-17T23:50:26.502 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46157.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47165.log.gz 2024-03-17T23:50:26.502 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54901.log 2024-03-17T23:50:26.503 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52371.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46157.log.gz 2024-03-17T23:50:26.503 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45311.log 2024-03-17T23:50:26.503 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52371.log.gz 2024-03-17T23:50:26.503 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58622.log 2024-03-17T23:50:26.503 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45311.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54901.log.gz 2024-03-17T23:50:26.503 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49944.log 2024-03-17T23:50:26.503 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.58622.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45311.log.gz 2024-03-17T23:50:26.503 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59513.log 2024-03-17T23:50:26.503 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58622.log.gz 2024-03-17T23:50:26.504 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58207.log 2024-03-17T23:50:26.504 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.59513.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49944.log.gz 2024-03-17T23:50:26.504 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50112.log 2024-03-17T23:50:26.504 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.58207.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59513.log.gz 2024-03-17T23:50:26.504 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58453.log 2024-03-17T23:50:26.504 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58207.log.gz 2024-03-17T23:50:26.504 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46494.log 2024-03-17T23:50:26.504 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.58453.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50112.log.gz 2024-03-17T23:50:26.505 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52497.log 2024-03-17T23:50:26.505 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46494.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58453.log.gz 2024-03-17T23:50:26.505 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58748.log 2024-03-17T23:50:26.505 INFO:teuthology.orchestra.run.smithi150.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46494.log.gz 2024-03-17T23:50:26.505 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52161.log 2024-03-17T23:50:26.505 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.58748.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52497.log.gz 2024-03-17T23:50:26.505 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-17T23:50:26.505 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58748.log.gz/var/log/ceph/ceph-client.admin.52161.log: 2024-03-17T23:50:26.505 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50333.log 2024-03-17T23:50:26.506 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52161.log.gz 2024-03-17T23:50:26.506 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59555.log 2024-03-17T23:50:26.506 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47291.log 2024-03-17T23:50:26.506 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.59555.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50333.log.gz 2024-03-17T23:50:26.506 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53866.log 2024-03-17T23:50:26.506 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59555.log.gz 2024-03-17T23:50:26.506 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44321.log 2024-03-17T23:50:26.506 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47291.log.gz 2024-03-17T23:50:26.507 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50070.log 2024-03-17T23:50:26.507 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53866.log.gz 2024-03-17T23:50:26.507 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57310.log 2024-03-17T23:50:26.507 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44321.log: /var/log/ceph/ceph-client.admin.50070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50517.log 2024-03-17T23:50:26.507 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57310.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50070.log.gz 2024-03-17T23:50:26.507 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58123.log 2024-03-17T23:50:26.507 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57310.log.gz 2024-03-17T23:50:26.507 INFO:teuthology.orchestra.run.smithi150.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.44321.log.gz 2024-03-17T23:50:26.507 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62421.log 2024-03-17T23:50:26.508 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49860.log 2024-03-17T23:50:26.508 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.58123.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.50517.log.gz 2024-03-17T23:50:26.508 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.62421.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58123.log.gz 2024-03-17T23:50:26.508 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59723.log 2024-03-17T23:50:26.508 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58351.log 2024-03-17T23:50:26.508 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49860.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62421.log.gz 2024-03-17T23:50:26.508 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.59723.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49860.log.gz 2024-03-17T23:50:26.509 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60659.log 2024-03-17T23:50:26.509 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59723.log.gz 2024-03-17T23:50:26.509 INFO:teuthology.orchestra.run.smithi150.stderr:gzip/var/log/ceph/ceph-client.admin.58351.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53643.log 2024-03-17T23:50:26.509 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.60659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46819.log 2024-03-17T23:50:26.509 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58351.log.gz 2024-03-17T23:50:26.509 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60659.log.gz 2024-03-17T23:50:26.509 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49314.log 2024-03-17T23:50:26.509 INFO:teuthology.orchestra.run.smithi150.stderr:gzip 88.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.53643.log.gz --verbose 2024-03-17T23:50:26.510 INFO:teuthology.orchestra.run.smithi150.stderr: -- /var/log/ceph/ceph-client.admin.64391.log 2024-03-17T23:50:26.510 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46819.log: /var/log/ceph/ceph-client.admin.49314.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46819.log.gz 2024-03-17T23:50:26.510 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46493.log 2024-03-17T23:50:26.510 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49314.log.gz 2024-03-17T23:50:26.510 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55161.log 2024-03-17T23:50:26.510 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.64391.log: /var/log/ceph/ceph-client.admin.46493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56597.log 2024-03-17T23:50:26.510 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64391.log.gz 2024-03-17T23:50:26.511 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46493.log.gz 2024-03-17T23:50:26.511 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53306.log 2024-03-17T23:50:26.511 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56597.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.55161.log.gz 2024-03-17T23:50:26.511 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63844.log 2024-03-17T23:50:26.511 INFO:teuthology.orchestra.run.smithi150.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.56597.log.gz 2024-03-17T23:50:26.511 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50028.log 2024-03-17T23:50:26.511 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53306.log.gz 2024-03-17T23:50:26.511 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.63844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62988.log 2024-03-17T23:50:26.511 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.63844.log.gz 2024-03-17T23:50:26.512 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45043.log 2024-03-17T23:50:26.512 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.62988.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50028.log.gz 2024-03-17T23:50:26.512 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58580.log 2024-03-17T23:50:26.512 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62988.log.gz 2024-03-17T23:50:26.512 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61987.log 2024-03-17T23:50:26.512 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45043.log: /var/log/ceph/ceph-client.admin.58580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44195.log 2024-03-17T23:50:26.512 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45043.log.gz 2024-03-17T23:50:26.513 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58580.log.gz 2024-03-17T23:50:26.513 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.61987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54332.log 2024-03-17T23:50:26.513 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45169.log 2024-03-17T23:50:26.513 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44195.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61987.log.gz 2024-03-17T23:50:26.513 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54332.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44195.log.gz 2024-03-17T23:50:26.513 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56429.log 2024-03-17T23:50:26.513 INFO:teuthology.orchestra.run.smithi150.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54332.log.gz 2024-03-17T23:50:26.513 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62626.log 2024-03-17T23:50:26.513 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45169.log: /var/log/ceph/ceph-client.admin.56429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58538.log 2024-03-17T23:50:26.514 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45169.log.gz 2024-03-17T23:50:26.514 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7%/var/log/ceph/ceph-client.admin.62626.log: -- replaced with /var/log/ceph/ceph-client.admin.56429.log.gz 2024-03-17T23:50:26.514 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53391.log 2024-03-17T23:50:26.514 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62626.log.gz 2024-03-17T23:50:26.514 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47711.log 2024-03-17T23:50:26.514 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.58538.log: /var/log/ceph/ceph-client.admin.53391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54943.log 2024-03-17T23:50:26.514 INFO:teuthology.orchestra.run.smithi150.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58538.log.gz 2024-03-17T23:50:26.514 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47711.log: gzip -5 --verbose -- 89.0% /var/log/ceph/ceph-client.admin.53475.log 2024-03-17T23:50:26.515 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53391.log.gz 2024-03-17T23:50:26.515 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47711.log.gz 2024-03-17T23:50:26.515 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52851.log 2024-03-17T23:50:26.515 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54943.log.gz 2024-03-17T23:50:26.515 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58309.log 2024-03-17T23:50:26.515 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53475.log: /var/log/ceph/ceph-client.admin.52851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44808.log 2024-03-17T23:50:26.515 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53475.log.gz 2024-03-17T23:50:26.515 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52851.log.gz 2024-03-17T23:50:26.516 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.58309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57536.log 2024-03-17T23:50:26.516 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61867.log 2024-03-17T23:50:26.516 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44808.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58309.log.gz 2024-03-17T23:50:26.516 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55489.log 2024-03-17T23:50:26.516 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57536.log.gz 2024-03-17T23:50:26.516 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.61867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64223.log 2024-03-17T23:50:26.516 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58039.log 2024-03-17T23:50:26.516 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55489.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61867.log.gz 2024-03-17T23:50:26.517 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.64223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56513.log 2024-03-17T23:50:26.517 INFO:teuthology.orchestra.run.smithi150.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.55489.log.gz 2024-03-17T23:50:26.517 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64223.log.gz 2024-03-17T23:50:26.517 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.58039.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.44808.log.gz 2024-03-17T23:50:26.517 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51395.log 2024-03-17T23:50:26.517 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56513.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58039.log.gz 2024-03-17T23:50:26.517 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-17T23:50:26.517 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56513.log.gz 2024-03-17T23:50:26.518 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49230.log 2024-03-17T23:50:26.518 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51395.log: /var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64096.log 2024-03-17T23:50:26.518 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51395.log.gz 2024-03-17T23:50:26.518 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49104.log 2024-03-17T23:50:26.518 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.64096.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49230.log.gz 2024-03-17T23:50:26.518 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53731.log 2024-03-17T23:50:26.518 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49104.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.64096.log.gz -5 2024-03-17T23:50:26.518 INFO:teuthology.orchestra.run.smithi150.stderr: --verbose -- /var/log/ceph/ceph-client.admin.64181.log 2024-03-17T23:50:26.519 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53731.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49104.log.gz 2024-03-17T23:50:26.519 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63324.log 2024-03-17T23:50:26.519 INFO:teuthology.orchestra.run.smithi150.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53731.log.gz 2024-03-17T23:50:26.519 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.64181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45935.log 2024-03-17T23:50:26.519 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.63324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43459.log 2024-03-17T23:50:26.519 INFO:teuthology.orchestra.run.smithi150.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.64181.log.gz 2024-03-17T23:50:26.519 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63324.log.gz 2024-03-17T23:50:26.519 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44027.log 2024-03-17T23:50:26.520 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.54700.log 2024-03-17T23:50:26.520 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43459.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45935.log.gz 2024-03-17T23:50:26.520 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45641.log 2024-03-17T23:50:26.520 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43459.log.gz 2024-03-17T23:50:26.520 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.0.54700.log: gzip -5 --verbose -- 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44027.log.gz 2024-03-17T23:50:26.520 INFO:teuthology.orchestra.run.smithi150.stderr: /var/log/ceph/ceph-client.admin.55693.log 2024-03-17T23:50:26.520 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46492.log 2024-03-17T23:50:26.520 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45641.log.gz 2024-03-17T23:50:26.520 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59089.log 2024-03-17T23:50:26.521 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46492.log: gzip 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55693.log.gz 2024-03-17T23:50:26.521 INFO:teuthology.orchestra.run.smithi150.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.61515.log 2024-03-17T23:50:26.521 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.59089.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46492.log.gz 2024-03-17T23:50:26.521 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45353.log 2024-03-17T23:50:26.521 INFO:teuthology.orchestra.run.smithi150.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.59089.log.gz 2024-03-17T23:50:26.521 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.61515.log: 91.4% -- replaced with /var/log/ceph/ceph-client.0.54700.log.gz 2024-03-17T23:50:26.521 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63198.log 2024-03-17T23:50:26.521 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45353.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61515.log.gz 2024-03-17T23:50:26.522 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59597.log 2024-03-17T23:50:26.522 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45353.log.gz 2024-03-17T23:50:26.522 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.63198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51909.log 2024-03-17T23:50:26.522 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.59597.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63198.log.gz 2024-03-17T23:50:26.522 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52035.log 2024-03-17T23:50:26.522 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59597.log.gz 2024-03-17T23:50:26.522 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56759.log 2024-03-17T23:50:26.522 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52035.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51909.log.gz 2024-03-17T23:50:26.523 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60575.log 2024-03-17T23:50:26.523 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52035.log.gz 2024-03-17T23:50:26.523 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45977.log 2024-03-17T23:50:26.523 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.60575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48892.log 2024-03-17T23:50:26.523 INFO:teuthology.orchestra.run.smithi150.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.56759.log.gz 2024-03-17T23:50:26.523 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45977.log: gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60575.log.gz 2024-03-17T23:50:26.523 INFO:teuthology.orchestra.run.smithi150.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.60475.log 2024-03-17T23:50:26.523 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45977.log.gz 2024-03-17T23:50:26.523 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52599.log 2024-03-17T23:50:26.524 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.60475.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48892.log.gz 2024-03-17T23:50:26.524 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49524.log 2024-03-17T23:50:26.524 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60475.log.gz 2024-03-17T23:50:26.524 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51789.log 2024-03-17T23:50:26.524 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49524.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52599.log.gz 2024-03-17T23:50:26.524 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.49524.log.gz 2024-03-17T23:50:26.524 INFO:teuthology.orchestra.run.smithi150.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53782.log 2024-03-17T23:50:26.524 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45851.log 2024-03-17T23:50:26.524 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51789.log.gz 2024-03-17T23:50:26.525 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43625.log 2024-03-17T23:50:26.525 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53782.log.gz 2024-03-17T23:50:26.525 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52203.log 2024-03-17T23:50:26.525 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43625.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45851.log.gz 2024-03-17T23:50:26.525 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53061.log 2024-03-17T23:50:26.525 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43625.log.gz 2024-03-17T23:50:26.525 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55347.log 2024-03-17T23:50:26.525 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53061.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52203.log.gz 2024-03-17T23:50:26.526 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53061.log.gz 2024-03-17T23:50:26.526 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43965.log 2024-03-17T23:50:26.526 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51951.log 2024-03-17T23:50:26.526 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55347.log.gz 2024-03-17T23:50:26.526 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61457.log 2024-03-17T23:50:26.526 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61641.log 2024-03-17T23:50:26.526 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51951.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43965.log.gz 2024-03-17T23:50:26.526 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.61457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61825.log 2024-03-17T23:50:26.527 INFO:teuthology.orchestra.run.smithi150.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51951.log.gz 2024-03-17T23:50:26.527 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61457.log.gz 2024-03-17T23:50:26.527 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.61641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63562.log 2024-03-17T23:50:26.527 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60041.log 2024-03-17T23:50:26.527 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.61825.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61641.log.gz 2024-03-17T23:50:26.527 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.63562.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-17T23:50:26.527 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61825.log.gz 2024-03-17T23:50:26.527 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.60041.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.63562.log.gz 2024-03-17T23:50:26.527 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56203.log 2024-03-17T23:50:26.528 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60041.log.gz 2024-03-17T23:50:26.528 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47753.log 2024-03-17T23:50:26.528 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31447.log 2024-03-17T23:50:26.528 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56203.log: /var/log/ceph/ceph-client.admin.47753.log: gzip -5 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56203.log.gz 2024-03-17T23:50:26.528 INFO:teuthology.orchestra.run.smithi150.stderr: --verbose -- /var/log/ceph/ceph-client.admin.63688.log 2024-03-17T23:50:26.528 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph.tmp-client.admin.31447.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31447.log.gz 2024-03-17T23:50:26.528 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43540.log 2024-03-17T23:50:26.528 INFO:teuthology.orchestra.run.smithi150.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.47753.log.gz 2024-03-17T23:50:26.528 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.63688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58927.log 2024-03-17T23:50:26.529 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43540.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63688.log.gz 2024-03-17T23:50:26.529 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48272.log 2024-03-17T23:50:26.529 INFO:teuthology.orchestra.run.smithi150.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43540.log.gz 2024-03-17T23:50:26.529 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.58927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63072.log 2024-03-17T23:50:26.529 INFO:teuthology.orchestra.run.smithi150.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58927.log.gz 2024-03-17T23:50:26.529 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.0.48272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49062.log 2024-03-17T23:50:26.529 INFO:teuthology.orchestra.run.smithi150.stderr:gzip/var/log/ceph/ceph-client.admin.63072.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.46820.log 2024-03-17T23:50:26.530 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49062.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63072.log.gz 90.4% -- replaced with /var/log/ceph/ceph-client.0.48272.log.gz 2024-03-17T23:50:26.530 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-03-17T23:50:26.530 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63282.log 2024-03-17T23:50:26.530 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49062.log.gz 2024-03-17T23:50:26.530 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59891.log 2024-03-17T23:50:26.530 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46820.log: /var/log/ceph/ceph-client.admin.63282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46491.log 2024-03-17T23:50:26.530 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46820.log.gz 2024-03-17T23:50:26.530 INFO:teuthology.orchestra.run.smithi150.stderr: 89.2%gzip/var/log/ceph/ceph-client.admin.59891.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.55203.log 2024-03-17T23:50:26.531 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63282.log.gz 2024-03-17T23:50:26.531 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63366.log 2024-03-17T23:50:26.531 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59891.log.gz 2024-03-17T23:50:26.531 INFO:teuthology.orchestra.run.smithi150.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46491.log.gz 2024-03-17T23:50:26.531 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43793.log 2024-03-17T23:50:26.531 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55203.log: /var/log/ceph/ceph-client.admin.63366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52725.log 2024-03-17T23:50:26.531 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55203.log.gz/var/log/ceph/ceph-client.admin.43793.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63366.log.gz 2024-03-17T23:50:26.531 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 2024-03-17T23:50:26.531 INFO:teuthology.orchestra.run.smithi150.stderr: --verbose -- /var/log/ceph/ceph-client.admin.62542.log 2024-03-17T23:50:26.532 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52725.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43793.log.gz 2024-03-17T23:50:26.532 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49398.log 2024-03-17T23:50:26.532 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52725.log.gz 2024-03-17T23:50:26.532 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.62542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57913.log 2024-03-17T23:50:26.532 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62542.log.gz 2024-03-17T23:50:26.532 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56287.log 2024-03-17T23:50:26.532 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57913.log: 89.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43837.log 2024-03-17T23:50:26.532 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49398.log.gz 2024-03-17T23:50:26.533 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50847.log 2024-03-17T23:50:26.533 INFO:teuthology.orchestra.run.smithi150.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57913.log.gz 2024-03-17T23:50:26.533 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56287.log.gz 2024-03-17T23:50:26.533 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59257.log 2024-03-17T23:50:26.533 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43837.log: gzip -5 --verbose --/var/log/ceph/ceph-client.admin.50847.log: 87.7% /var/log/ceph/ceph-client.2.48770.log 2024-03-17T23:50:26.533 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43837.log.gz 2024-03-17T23:50:26.533 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.59257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60885.log 2024-03-17T23:50:26.533 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50847.log.gz 2024-03-17T23:50:26.534 INFO:teuthology.orchestra.run.smithi150.stderr: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-17T23:50:26.534 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59257.log.gz 2024-03-17T23:50:26.534 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.2.48770.log: gzip -5/var/log/ceph/ceph-client.admin.60885.log: --verbose -- /var/log/ceph/ceph-client.admin.59301.log 2024-03-17T23:50:26.534 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.0.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60885.log.gz 2024-03-17T23:50:26.534 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51111.log 2024-03-17T23:50:26.534 INFO:teuthology.orchestra.run.smithi150.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.2.48770.log.gz 2024-03-17T23:50:26.534 INFO:teuthology.orchestra.run.smithi150.stderr:gzip/var/log/ceph/ceph-client.admin.59301.log: -5 --verbose -- /var/log/ceph/ceph-client.0.55953.log 2024-03-17T23:50:26.534 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51111.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59301.log.gz 2024-03-17T23:50:26.535 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50291.log 2024-03-17T23:50:26.535 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51111.log.gz 2024-03-17T23:50:26.535 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44153.log 2024-03-17T23:50:26.535 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.0.55953.log: /var/log/ceph/ceph-client.admin.50291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51831.log 2024-03-17T23:50:26.535 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44153.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50291.log.gz 2024-03-17T23:50:26.535 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44153.log.gz 2024-03-17T23:50:26.535 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49608.log 2024-03-17T23:50:26.536 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51831.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51831.log.gz 2024-03-17T23:50:26.536 INFO:teuthology.orchestra.run.smithi150.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.0.55953.log.gz 2024-03-17T23:50:26.536 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61599.log 2024-03-17T23:50:26.536 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49440.log 2024-03-17T23:50:26.536 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.61599.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49608.log.gz 2024-03-17T23:50:26.536 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47585.log 2024-03-17T23:50:26.536 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61599.log.gz/var/log/ceph/ceph-client.admin.49440.log: 2024-03-17T23:50:26.536 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50889.log 88.2% 2024-03-17T23:50:26.537 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49440.log.gz 2024-03-17T23:50:26.537 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51353.log 2024-03-17T23:50:26.537 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47585.log.gz/var/log/ceph/ceph-client.admin.50889.log: 2024-03-17T23:50:26.537 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63802.log 2024-03-17T23:50:26.537 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50889.log.gz 2024-03-17T23:50:26.537 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49692.log 2024-03-17T23:50:26.537 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51353.log.gz 2024-03-17T23:50:26.537 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49734.log 2024-03-17T23:50:26.538 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.63802.log: /var/log/ceph/ceph-client.admin.49692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53996.log 2024-03-17T23:50:26.538 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63802.log.gz 2024-03-17T23:50:26.538 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9%/var/log/ceph/ceph-client.admin.49734.log: -- replaced with /var/log/ceph/ceph-client.admin.49692.log.gz 2024-03-17T23:50:26.538 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-17T23:50:26.538 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49734.log.gz 2024-03-17T23:50:26.538 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47627.log 2024-03-17T23:50:26.538 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53687.log 2024-03-17T23:50:26.539 INFO:teuthology.orchestra.run.smithi150.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53996.log.gz 2024-03-17T23:50:26.539 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46821.log 2024-03-17T23:50:26.539 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53687.log: 85.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-17T23:50:26.539 INFO:teuthology.orchestra.run.smithi150.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47627.log.gz 2024-03-17T23:50:26.539 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53687.log.gz 2024-03-17T23:50:26.539 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59681.log 2024-03-17T23:50:26.539 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43881.log 2024-03-17T23:50:26.539 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3%/var/log/ceph/ceph-client.admin.59681.log: -- replaced with /var/log/ceph/ceph-client.admin.46821.log.gz 2024-03-17T23:50:26.540 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57394.log 2024-03-17T23:50:26.540 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59681.log.gz 2024-03-17T23:50:26.540 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46155.log 2024-03-17T23:50:26.540 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62500.log 2024-03-17T23:50:26.540 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57394.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43881.log.gz 2024-03-17T23:50:26.540 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46155.log: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45001.log 2024-03-17T23:50:26.540 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57394.log.gz 2024-03-17T23:50:26.541 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.62500.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-17T23:50:26.541 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46155.log.gz -- replaced with /var/log/ceph/ceph-client.admin.62500.log.gz 2024-03-17T23:50:26.541 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-03-17T23:50:26.541 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45001.log: gzip -5 --verbose -- 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45001.log.gz 2024-03-17T23:50:26.541 INFO:teuthology.orchestra.run.smithi150.stderr: /var/log/ceph/ceph-client.admin.56639.log 2024-03-17T23:50:26.541 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51621.log 2024-03-17T23:50:26.541 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45437.log 2024-03-17T23:50:26.541 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56639.log.gz/var/log/ceph/ceph-client.admin.51621.log: 2024-03-17T23:50:26.542 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54122.log 2024-03-17T23:50:26.542 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51621.log.gz 2024-03-17T23:50:26.542 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60533.log 2024-03-17T23:50:26.542 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45437.log: /var/log/ceph/ceph-client.admin.54122.log: 87.6% 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45437.log.gz 2024-03-17T23:50:26.542 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54122.log.gz 2024-03-17T23:50:26.542 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50932.log 2024-03-17T23:50:26.542 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.60533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51068.log 2024-03-17T23:50:26.542 INFO:teuthology.orchestra.run.smithi150.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60533.log.gz 2024-03-17T23:50:26.543 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54206.log 2024-03-17T23:50:26.543 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7%/var/log/ceph/ceph-client.admin.51068.log: -- replaced with /var/log/ceph/ceph-client.admin.50932.log.gz 2024-03-17T23:50:26.543 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51068.log.gz 2024-03-17T23:50:26.543 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51437.log 2024-03-17T23:50:26.543 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54206.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-17T23:50:26.543 INFO:teuthology.orchestra.run.smithi150.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54206.log.gz 2024-03-17T23:50:26.543 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53264.log 2024-03-17T23:50:26.543 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3%/var/log/ceph/ceph-mds.c.log: -- replaced with /var/log/ceph/ceph-client.admin.51437.log.gz 2024-03-17T23:50:26.543 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49272.log 2024-03-17T23:50:26.544 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53264.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53264.log.gz 2024-03-17T23:50:26.544 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63970.log 2024-03-17T23:50:26.544 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49272.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49272.log.gz 2024-03-17T23:50:26.544 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53601.log 2024-03-17T23:50:26.544 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.63970.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63970.log.gz 2024-03-17T23:50:26.544 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62904.log 2024-03-17T23:50:26.544 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53601.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53601.log.gz 2024-03-17T23:50:26.544 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63156.log 2024-03-17T23:50:26.545 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.62904.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.62904.log.gz 2024-03-17T23:50:26.545 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46490.log 2024-03-17T23:50:26.545 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.63156.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63156.log.gz 2024-03-17T23:50:26.545 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64265.log 2024-03-17T23:50:26.545 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46490.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46490.log.gz 2024-03-17T23:50:26.545 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37465.log 2024-03-17T23:50:26.545 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.64265.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64265.log.gz 2024-03-17T23:50:26.545 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61909.log 2024-03-17T23:50:26.546 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.37465.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37465.log.gz 2024-03-17T23:50:26.546 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59471.log 2024-03-17T23:50:26.546 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.61909.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.61909.log.gz 2024-03-17T23:50:26.546 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64515.log 2024-03-17T23:50:26.546 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.59471.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59471.log.gz 2024-03-17T23:50:26.546 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43363.log 2024-03-17T23:50:26.546 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.64515.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.64515.log.gz 2024-03-17T23:50:26.546 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52809.log 2024-03-17T23:50:26.547 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43363.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43363.log.gz 2024-03-17T23:50:26.547 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60759.log 2024-03-17T23:50:26.547 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52809.log: 90.6% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-17T23:50:26.547 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52809.log.gz 2024-03-17T23:50:26.547 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45557.log 2024-03-17T23:50:26.547 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.60759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61023.log 2024-03-17T23:50:26.547 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60759.log.gz 2024-03-17T23:50:26.547 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49650.log 2024-03-17T23:50:26.547 INFO:teuthology.orchestra.run.smithi150.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.45557.log.gz 2024-03-17T23:50:26.548 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.61023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63886.log 2024-03-17T23:50:26.548 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61023.log.gz 2024-03-17T23:50:26.548 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47881.log 2024-03-17T23:50:26.548 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.63886.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49650.log.gz 2024-03-17T23:50:26.548 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47123.log 2024-03-17T23:50:26.548 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63886.log.gz 2024-03-17T23:50:26.548 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47375.log 2024-03-17T23:50:26.548 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47123.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47881.log.gz 2024-03-17T23:50:26.549 INFO:teuthology.orchestra.run.smithi150.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.47123.log.gz 2024-03-17T23:50:26.549 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50685.log 2024-03-17T23:50:26.549 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49902.log 2024-03-17T23:50:26.549 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50685.log: /var/log/ceph/ceph-client.admin.47375.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47375.log.gz 2024-03-17T23:50:26.549 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50685.log.gz 2024-03-17T23:50:26.549 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58496.log 2024-03-17T23:50:26.549 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59975.log 2024-03-17T23:50:26.549 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49902.log: /var/log/ceph/ceph-client.admin.58496.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49902.log.gz 2024-03-17T23:50:26.550 INFO:teuthology.orchestra.run.smithi150.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-17T23:50:26.550 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60701.log 2024-03-17T23:50:26.550 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58496.log.gz 2024-03-17T23:50:26.550 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53559.log 2024-03-17T23:50:26.550 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.59975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45269.log 2024-03-17T23:50:26.550 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.60701.log: /var/log/ceph/ceph-client.admin.53559.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60701.log.gz 2024-03-17T23:50:26.550 INFO:teuthology.orchestra.run.smithi150.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.59975.log.gz 2024-03-17T23:50:26.550 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53559.log.gz 2024-03-17T23:50:26.550 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64475.log 2024-03-17T23:50:26.551 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61557.log 2024-03-17T23:50:26.551 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45269.log: /var/log/ceph/ceph-client.admin.64475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54164.log 2024-03-17T23:50:26.551 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45269.log.gz 2024-03-17T23:50:26.551 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.61557.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64475.log.gz 2024-03-17T23:50:26.551 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61557.log.gz 2024-03-17T23:50:26.551 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52119.log 2024-03-17T23:50:26.551 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59639.log 2024-03-17T23:50:26.551 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52119.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54164.log.gz 2024-03-17T23:50:26.552 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55245.log 2024-03-17T23:50:26.552 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52119.log.gz 2024-03-17T23:50:26.552 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45395.log 2024-03-17T23:50:26.552 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.59639.log: /var/log/ceph/ceph-client.admin.55245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60927.log 2024-03-17T23:50:26.552 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45395.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59639.log.gz 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55245.log.gz 2024-03-17T23:50:26.552 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-03-17T23:50:26.552 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45395.log.gz 2024-03-17T23:50:26.552 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55405.log 2024-03-17T23:50:26.553 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.60927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55061.log 2024-03-17T23:50:26.553 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60927.log.gz 2024-03-17T23:50:26.553 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49482.log 2024-03-17T23:50:26.553 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55405.log.gz 2024-03-17T23:50:26.553 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43709.log 2024-03-17T23:50:26.553 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55061.log.gz 2024-03-17T23:50:26.553 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46159.log 2024-03-17T23:50:26.553 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49482.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-17T23:50:26.554 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43709.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49482.log.gz/var/log/ceph/ceph-client.admin.46159.log: 2024-03-17T23:50:26.554 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43709.log.gz 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51237.log 2024-03-17T23:50:26.554 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46159.log.gz 2024-03-17T23:50:26.554 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-03-17T23:50:26.554 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52455.log 2024-03-17T23:50:26.554 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53349.log 2024-03-17T23:50:26.554 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51237.log.gz 2024-03-17T23:50:26.554 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51993.log 2024-03-17T23:50:26.555 INFO:teuthology.orchestra.run.smithi150.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-17T23:50:26.555 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53349.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52455.log.gz 2024-03-17T23:50:26.555 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50805.log 2024-03-17T23:50:26.555 INFO:teuthology.orchestra.run.smithi150.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53349.log.gz 2024-03-17T23:50:26.555 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51195.log 2024-03-17T23:50:26.555 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51993.log.gz/var/log/ceph/ceph-client.admin.50805.log: 2024-03-17T23:50:26.555 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56245.log 2024-03-17T23:50:26.555 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50805.log.gz 2024-03-17T23:50:26.556 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61783.log 2024-03-17T23:50:26.556 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51195.log.gz 2024-03-17T23:50:26.556 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56245.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56245.log.gz 2024-03-17T23:50:26.556 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.61783.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61783.log.gz 2024-03-17T23:50:26.556 INFO:teuthology.orchestra.run.smithi157.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-17T23:50:26.583 INFO:teuthology.orchestra.run.smithi150.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-17T23:50:27.030 INFO:teuthology.orchestra.run.smithi150.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-17T23:50:27.033 INFO:teuthology.orchestra.run.smithi150.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-17T23:50:27.059 INFO:teuthology.orchestra.run.smithi157.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-17T23:50:27.068 INFO:teuthology.orchestra.run.smithi157.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-17T23:50:27.069 INFO:teuthology.orchestra.run.smithi157.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-17T23:50:27.097 INFO:teuthology.orchestra.run.smithi157.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-17T23:50:27.099 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-03-17T23:50:27.099 INFO:teuthology.orchestra.run.smithi157.stderr:real 0m0.807s 2024-03-17T23:50:27.099 INFO:teuthology.orchestra.run.smithi157.stderr:user 0m3.300s 2024-03-17T23:50:27.099 INFO:teuthology.orchestra.run.smithi157.stderr:sys 0m0.149s 2024-03-17T23:50:27.155 INFO:teuthology.orchestra.run.smithi150.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-17T23:50:27.173 INFO:teuthology.orchestra.run.smithi150.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-17T23:50:27.176 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-03-17T23:50:27.176 INFO:teuthology.orchestra.run.smithi150.stderr:real 0m0.911s 2024-03-17T23:50:27.176 INFO:teuthology.orchestra.run.smithi150.stderr:user 0m3.427s 2024-03-17T23:50:27.176 INFO:teuthology.orchestra.run.smithi150.stderr:sys 0m0.349s 2024-03-17T23:50:27.176 INFO:tasks.ceph:Archiving logs... 2024-03-17T23:50:27.177 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608488/remote/smithi150/log 2024-03-17T23:50:27.178 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-17T23:50:28.023 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608488/remote/smithi157/log 2024-03-17T23:50:28.024 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-17T23:50:28.554 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-17T23:50:28.565 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-17T23:50:28.565 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-17T23:50:28.568 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-17T23:50:28.644 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-17T23:50:28.654 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-17T23:50:28.654 DEBUG:teuthology.orchestra.run.smithi150:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-17T23:50:28.657 DEBUG:teuthology.orchestra.run.smithi157:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-17T23:50:28.670 INFO:teuthology.orchestra.run.smithi150.stderr:bash: line 1: ntpq: command not found 2024-03-17T23:50:28.671 INFO:teuthology.orchestra.run.smithi157.stderr:bash: line 1: ntpq: command not found 2024-03-17T23:50:28.673 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-17T23:50:28.674 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2024-03-17T23:50:28.674 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 51 +736us[ +738us] +/- 148ms 2024-03-17T23:50:28.674 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 51 +75us[ +75us] +/- 51ms 2024-03-17T23:50:28.674 INFO:teuthology.orchestra.run.smithi150.stdout:^* hv03.front.sepia.ceph.com 3 6 377 51 -511us[ -510us] +/- 70ms 2024-03-17T23:50:28.674 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-17T23:50:28.676 INFO:teuthology.orchestra.run.smithi157.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-17T23:50:28.676 INFO:teuthology.orchestra.run.smithi157.stdout:=============================================================================== 2024-03-17T23:50:28.676 INFO:teuthology.orchestra.run.smithi157.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 51 +752us[ +760us] +/- 148ms 2024-03-17T23:50:28.676 INFO:teuthology.orchestra.run.smithi157.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 50 +83us[ +83us] +/- 51ms 2024-03-17T23:50:28.677 INFO:teuthology.orchestra.run.smithi157.stdout:^* hv03.front.sepia.ceph.com 3 6 377 50 -502us[ -494us] +/- 70ms 2024-03-17T23:50:28.677 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-17T23:50:28.677 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-17T23:50:28.686 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-17T23:50:28.687 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-17T23:50:28.696 DEBUG:teuthology.orchestra.run.smithi150:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-17T23:50:28.797 DEBUG:teuthology.orchestra.run.smithi157:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-17T23:50:28.863 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-17T23:50:28.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-17T23:50:28.895 DEBUG:teuthology.orchestra.run.smithi157:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-17T23:50:28.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-17T23:50:28.925 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-17T23:50:28.935 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-17T23:50:28.945 INFO:teuthology.task.internal:Duration was 936.278160 seconds 2024-03-17T23:50:28.946 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-17T23:50:28.957 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-17T23:50:28.957 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-17T23:50:28.960 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-17T23:50:29.001 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-17T23:50:29.008 INFO:teuthology.orchestra.run.smithi157.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-17T23:50:29.276 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-17T23:50:29.276 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi150.front.sepia.ceph.com 2024-03-17T23:50:29.277 DEBUG:teuthology.orchestra.run.smithi150:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-17T23:50:29.296 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi157.front.sepia.ceph.com 2024-03-17T23:50:29.296 DEBUG:teuthology.orchestra.run.smithi157:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-17T23:50:29.326 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-17T23:50:29.326 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-17T23:50:29.339 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-17T23:50:29.395 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-17T23:50:29.395 DEBUG:teuthology.orchestra.run.smithi150:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-17T23:50:29.408 DEBUG:teuthology.orchestra.run.smithi157:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-17T23:50:29.535 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-17T23:50:29.546 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-17T23:50:29.546 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-17T23:50:29.573 DEBUG:teuthology.orchestra.run.smithi157:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-17T23:50:29.600 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-17T23:50:29.613 DEBUG:teuthology.orchestra.run.smithi150:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-17T23:50:29.615 DEBUG:teuthology.orchestra.run.smithi157:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-17T23:50:29.638 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = core 2024-03-17T23:50:29.666 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = core 2024-03-17T23:50:29.682 DEBUG:teuthology.orchestra.run.smithi150:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-17T23:50:29.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-17T23:50:29.708 DEBUG:teuthology.orchestra.run.smithi157:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-17T23:50:29.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-17T23:50:29.737 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-17T23:50:29.750 INFO:teuthology.task.internal:Transferring archived files... 2024-03-17T23:50:29.754 DEBUG:teuthology.misc:Transferring archived files from smithi150:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608488/remote/smithi150 2024-03-17T23:50:29.754 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-17T23:50:29.792 DEBUG:teuthology.misc:Transferring archived files from smithi157:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608488/remote/smithi157 2024-03-17T23:50:29.792 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-17T23:50:29.828 INFO:teuthology.task.internal:Removing archive directory... 2024-03-17T23:50:29.829 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-17T23:50:29.831 DEBUG:teuthology.orchestra.run.smithi157:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-17T23:50:29.878 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-17T23:50:29.896 INFO:teuthology.task.internal:Not uploading archives. 2024-03-17T23:50:29.896 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-17T23:50:29.911 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-17T23:50:29.911 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-17T23:50:29.915 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-17T23:50:29.928 INFO:teuthology.orchestra.run.smithi150.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 17 23:50 /home/ubuntu/cephtest 2024-03-17T23:50:29.933 INFO:teuthology.orchestra.run.smithi157.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 17 23:50 /home/ubuntu/cephtest 2024-03-17T23:50:29.934 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-17T23:50:29.945 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-17T23:50:29.978 INFO:teuthology.nuke:Checking targets against current locks 2024-03-17T23:50:30.004 DEBUG:teuthology.nuke:shortname: smithi150 2024-03-17T23:50:30.005 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-17T23:50:30.026 DEBUG:teuthology.nuke:shortname: smithi157 2024-03-17T23:50:30.026 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-17T23:50:30.030 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608488', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-17 23:24:02.780949', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEuznq3ZAT3syTm6Yet4MN3i/Y3sAFNkEz27pRQMqOK0NnuEKnOkPmWZl97ElRgzeZNsnze7EUL85fmO4o5EZ6M='} 2024-03-17T23:50:30.050 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608488', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-17 23:24:02.782200', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFQmXCS1lYraNILT38OgyDIOYfOlxAZXfrbtQYaTnpIiZzvkKmkBYkPjsJ1gnMMUKS7RHaMUrZ6fmo+5HkRvS2I='} 2024-03-17T23:50:30.080 INFO:teuthology.orchestra.console.smithi150:Power off 2024-03-17T23:50:30.081 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-17T23:50:30.106 DEBUG:teuthology.orchestra.console.smithi150:power off output: Chassis Power Control: Down/Off 2024-03-17T23:50:30.106 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-17T23:50:30.124 INFO:teuthology.orchestra.console.smithi157:Power off 2024-03-17T23:50:30.125 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-17T23:50:30.134 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-03-17T23:50:30.149 DEBUG:teuthology.orchestra.console.smithi157:power off output: Chassis Power Control: Down/Off 2024-03-17T23:50:30.149 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-17T23:50:30.173 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is on 2024-03-17T23:50:34.135 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-17T23:50:34.174 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-17T23:50:34.245 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is on 2024-03-17T23:50:34.284 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is on 2024-03-17T23:50:38.246 DEBUG:teuthology.orchestra.console.smithi150:pexpect command: ipmitool -H smithi150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-17T23:50:38.285 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-17T23:50:38.357 DEBUG:teuthology.orchestra.console.smithi150:check power output: Chassis Power is off 2024-03-17T23:50:38.395 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is off 2024-03-17T23:50:38.459 INFO:teuthology.orchestra.console.smithi150:Power off completed 2024-03-17T23:50:38.496 INFO:teuthology.orchestra.console.smithi157:Power off completed 2024-03-17T23:50:38.601 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/pool-perm} duration: 936.2781596183777 failure_reason: 'Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=810d613814e74b5196e016d9e432b81f status: fail success: false 2024-03-17T23:50:38.602 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-17T23:50:38.680 INFO:teuthology.run:FAIL